Estimating the Reproducibility of Psychological Science
Preprint — published as 10.1126/science.aac4716 · Paper page
Abstract
Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Lack of replicability and replication may introduce error into bodies of published literature.
“Publication bias [20, 21], lack of replicability and replication [22], implication of a causal relationship even after stating that the evidence is insufficient to reach causal conclusions [23–25], and related phenomena may also introduce error into bodies of published literature.”
From Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): A systematic review (2018) · cites this paper as [22]
Cites
Lakens, D. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00863 · Cited by 10,303 · see citing works
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · see citing works
Self-citationRaise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · see citing works
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708 · see citing works
Self-citationBelieve it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · see citing works
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824 · see citing works
Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences
Schmidt, S. (2009) · Review of General Psychology · doi:10.1037/a0015108 · Cited by 1,125 · see citing works
Consequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022 · cited 1× · see citing works
“bias favoring positive results together produce a literature with upwardly biased effect sizes (14, 16, 33, 34).”
BackgroundSterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · cited 1× · see citing works
“bias favoring positive results together produce a literature with upwardly biased effect sizes (14, 16, 33, 34).”
BackgroundSmall Telescopes: Detectability and the Evaluation of Replication Results
Simonsohn, U. (2015) · Psychological Science · doi:10.1177/0956797614567341 · Cited by 742 · see citing works
The effect of experimenter bias on the performance of the albino rat
Rosenthal, R., & Fode, K. L. (2007) · Behavioral Science · doi:10.1002/bs.3830080302 · Cited by 384 · see citing works
Maximal Specificity and Lawlikeness in Probabilistic Explanation
Hempel, C. G. (1968) · Philosophy of Science · doi:10.1086/288197 · Cited by 121 · see citing works
Do Research Literatures Give Correct Answers?
Howard, G. S., Lau, M. Y., Maxwell, S. E., Venter, A., Lundy, R., & Sweeny, R. M. (2009) · Review of General Psychology · doi:10.1037/a0015468 · Cited by 21 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Haber, N., Smith, E. R., Moscoe, E., Andrews, K., Audy, R., Bell, W., Brennan, A. T., Breskin, A., Kane, J. C., Karra, M., et al. (2018) · PLoS ONE · doi:10.1371/journal.pone.0196346 · Cited by 97 · see citing works
Hatch, S. G., Knopp, K., Le, Y., Allen, M. O. T., Rothman, K., Rhoades, G. K., & Doss, B. D. (2022) · Family Process · doi:10.1111/famp.12710 · Cited by 28 · see citing works
- Authors
- Nosek, B. A., Cohoon, J., Kidwell, M., & Spies, J. R.
- Affiliations
- Center for Open Science (US)
- Publisher
- Center for Open Science
- Published
- 2016-10-05
- OpenAlex
- W4211125142
- Cited by
- 88
- References
- 13
- Citations by year
- 2025: 5 · 2024: 4 · 2023: 6 · 2022: 7 · 2021: 13 · 2020: 13 · 2019: 12 · 2018: 14 · 2017: 9 · 2016: 4 · 2015: 1
- Type
- posted content
- Language
- en
- Open access
- closed
- Locations
- RePEc: Research Papers in Economics (submittedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Keywords
- reproducibility, replication, statistical significance, confidence intervals, publication bias, quality of evidence, psychology research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
