Estimating the reproducibility of psychological science

Aarts, Alexander A.

doi:10.1126/science.aac4716

Science

2015

DOI: 10.1126/science.aac4716

|Get access via publisher |Summarize |Cite

|

Sign up to set email alerts

Estimating the reproducibility of psychological science

Alexander A. Aarts¹

Abstract: 52% Yes, a signiicant crisis 3% No, there is no crisis 7% Don't know 38% Yes, a slight crisis 38% Yes, a slight crisis 1,576 RESEARCHERS SURVEYED M ore than 70% of researchers have tried and failed to reproduce another scientist's experiments, and more than half have failed to reproduce their own experiments. Those are some of the telling figures that emerged from Nature's survey of 1,576 researchers who took a brief online questionnaire on reproducibility in research. The data reveal sometimes-contradictory a… Show more

Help me understand this report

View preprint versions

Cited by 7,979 publications

(2,532 citation statements)

References 43 publications

Supporting

67

Mentioning

2,441

Contrasting

5

Unclassified

14

Order By: Relevance

“…However, a look at Figure 5 reveals that with studies published with pre-registration, hence potentially preventing publication bias, the correlation is indeed smaller but still far from zero. This result is in accordance with the result of the Reproducibility Project in Psychology (Open Science Collaboration, 2015), where for replication studies, the standard error was a significant predictor ( z = 3.47, p < 0.001) for the observed effect size. The authors concluded that “[b]ecause publication bias was absent, this positive effect of standard error was likely caused by using power analysis for replication studies, i.e., generally larger replication samples were used for smaller true effects” (Open Science Collaboration, 2015, supplemental information).…”

Section: Resultssupporting

confidence: 90%

“…This result is in accordance with the result of the Reproducibility Project in Psychology (Open Science Collaboration, 2015), where for replication studies, the standard error was a significant predictor ( z = 3.47, p < 0.001) for the observed effect size. The authors concluded that “[b]ecause publication bias was absent, this positive effect of standard error was likely caused by using power analysis for replication studies, i.e., generally larger replication samples were used for smaller true effects” (Open Science Collaboration, 2015, supplemental information). This general correlation between sample size and effect size due to statistical power might also have led to a learning effect: in research areas with larger effects, scientists may have learned that small samples are enough while in research areas with smaller effects, they know that larger samples are needed.…”

Section: Resultssupporting

confidence: 90%

See 1 more Smart Citation

The Meaningfulness of Effect Sizes in Psychological Research: Differences Between Sub-Disciplines and the Impact of Potential Biases

Schäfer

¹

,

Schwarz

²

2019

Get access via publisher Add to dashboard Cite

Exaggerated anticipatory anxiety is common in social anxiety disorder (SAD). Neuroimaging studies have revealed altered neural activity in response to social stimuli in SAD, but fewer studies have examined neural activity during anticipation of feared social stimuli in SAD. The current study examined the time course and magnitude of activity in threat processing brain regions during speech anticipation in socially anxious individuals and healthy controls (HC). Method Participants (SAD n = 58; HC n = 16) underwent functional magnetic resonance imaging (fMRI) during which they completed a 90s control anticipation task and 90s speech anticipation task.

show abstract

“…However, a look at Figure 5 reveals that with studies published with pre-registration, hence potentially preventing publication bias, the correlation is indeed smaller but still far from zero. This result is in accordance with the result of the Reproducibility Project in Psychology (Open Science Collaboration, 2015), where for replication studies, the standard error was a significant predictor ( z = 3.47, p < 0.001) for the observed effect size. The authors concluded that “[b]ecause publication bias was absent, this positive effect of standard error was likely caused by using power analysis for replication studies, i.e., generally larger replication samples were used for smaller true effects” (Open Science Collaboration, 2015, supplemental information).…”

Section: Resultssupporting

confidence: 90%

“…This result is in accordance with the result of the Reproducibility Project in Psychology (Open Science Collaboration, 2015), where for replication studies, the standard error was a significant predictor ( z = 3.47, p < 0.001) for the observed effect size. The authors concluded that “[b]ecause publication bias was absent, this positive effect of standard error was likely caused by using power analysis for replication studies, i.e., generally larger replication samples were used for smaller true effects” (Open Science Collaboration, 2015, supplemental information). This general correlation between sample size and effect size due to statistical power might also have led to a learning effect: in research areas with larger effects, scientists may have learned that small samples are enough while in research areas with smaller effects, they know that larger samples are needed.…”

Section: Resultssupporting

confidence: 90%

The Meaningfulness of Effect Sizes in Psychological Research: Differences Between Sub-Disciplines and the Impact of Potential Biases

Schäfer

¹

,

Schwarz

²

2019

Get access via publisher Add to dashboard Cite

Exaggerated anticipatory anxiety is common in social anxiety disorder (SAD). Neuroimaging studies have revealed altered neural activity in response to social stimuli in SAD, but fewer studies have examined neural activity during anticipation of feared social stimuli in SAD. The current study examined the time course and magnitude of activity in threat processing brain regions during speech anticipation in socially anxious individuals and healthy controls (HC). Method Participants (SAD n = 58; HC n = 16) underwent functional magnetic resonance imaging (fMRI) during which they completed a 90s control anticipation task and 90s speech anticipation task.

show abstract

“…An exploratory linear regression with the Big Five traits and social status predicting green behavior showed only one unique main effect, for Openness, β = 0.48, SE = 0.18, t(343) = 2.65, p = 0.009. Openness having the strongest relationship with pro-environmental behavior is consistent with previous research [29,31] and lends support to the idea that Study 3 measured a behavior that seemed pro-environmental to participants.…”

Section: Resultssupporting

confidence: 87%

When Does Being Watched Change Pro-Environmental Behaviors in the Laboratory?

Brick

¹

,

Sherman

²

2021

Get access via publisher Add to dashboard Cite

Exaggerated anticipatory anxiety is common in social anxiety disorder (SAD). Neuroimaging studies have revealed altered neural activity in response to social stimuli in SAD, but fewer studies have examined neural activity during anticipation of feared social stimuli in SAD. The current study examined the time course and magnitude of activity in threat processing brain regions during speech anticipation in socially anxious individuals and healthy controls (HC). Method Participants (SAD n = 58; HC n = 16) underwent functional magnetic resonance imaging (fMRI) during which they completed a 90s control anticipation task and 90s speech anticipation task.

show abstract

“…With these findings, this study documents challenges in achieving reproducibility of ecological studies on insect behavior. In comparison to other systematic replication studies [ 3 , 6 , 7 ], however, we observed higher reproducibility rates, suggesting that while reproducibility issues do exist in insect studies, they might be less pronounced than in other areas of science.…”

Section: Discussioncontrasting

confidence: 91%

Testing the reproducibility of ecological studies on insect behavior in a multi-laboratory setting identifies opportunities for improving experimental rigor

Mundinger

¹

,

Schulz

²

,

Singh

³

et al. 2025

Get access via publisher Add to dashboard Cite

Exaggerated anticipatory anxiety is common in social anxiety disorder (SAD). Neuroimaging studies have revealed altered neural activity in response to social stimuli in SAD, but fewer studies have examined neural activity during anticipation of feared social stimuli in SAD. The current study examined the time course and magnitude of activity in threat processing brain regions during speech anticipation in socially anxious individuals and healthy controls (HC). Method Participants (SAD n = 58; HC n = 16) underwent functional magnetic resonance imaging (fMRI) during which they completed a 90s control anticipation task and 90s speech anticipation task.

show abstract

Scite is an AI-powered platform that helps researchers discover and evaluate scientific literature through Smart Citations, showing whether studies support or contradict a claim. Now part of Research Solutions, Scite has indexed 1.6B+ citations, partners with 30+ publishers, and serves 2M users worldwide.

Contact Info

customersupport@researchsolutions.com

10624 S. Eastern Ave., Ste. A-614

Henderson, NV 89052, USA