Estimating the reproducibility of psychological science
Open accessPublished version of a preprint
The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index; 20 of the 57 sources its claims cite were checked for support.
Abstract
Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.
Overview
CoChat Summary
TL;DR
A large collaborative replication of 100 published psychology studies found that replication effects were about half the magnitude of original effects, and only 36% of replications had significant results versus 97% of originals. Replication success was more consistently related to the strength of the original evidence than to characteristics of the replication teams or implementation.
Key findings
- Original effects (M = .396, SD = .193) were reliably larger than replication effects (M = .198, SD = .255); 82 of 97 study pairs (82.8%) showed a stronger original effect.
- 97% of original studies had significant results (p < .05), but only 36% of replications did; 47% of original effect sizes fell within the 95% CI of the replication effect size, 38% were subjectively rated as replicated, and 70% of meta-analytic combinations were significant if no bias in original results was assumed.
- Reproducibility was stronger for cognitive than social psychology: 20 of 38 (53%) cognitive effects replicated by p < .05 versus 14 of 53 (26%) social effects; main/simple effects replicated more often (23/50, 46%) than interaction effects (8/37, 22%).
- Replication success correlated with original p-value, original effect size, and replication power, but not consistently with perceived importance, team expertise, or self-assessed replication quality.
Methods
- The Open Science Collaboration conducted 100 replications of studies published in 2008 in three psychology journals: Psychological Science, Journal of Personality and Social Psychology, and Journal of Experimental Psychology: Learning, Memory, and Cognition; 270 contributing authors completed the replications.
- Replication teams followed a structured, pre-registered protocol: selecting a key effect, contacting original authors for materials, writing and registering an analysis plan, obtaining review, conducting high-powered replications (average 92% power to detect the original effect size), and archiving data and materials.
- Analyses converted effects to correlation coefficients where possible and evaluated reproducibility with five indicators: significance testing, comparison of original vs. replication effect sizes, whether the original effect size fell within the replication 95% CI, meta-analytic combination, and subjective “Did it replicate?” ratings.
Limitations
- The paper states that it is unknown how far findings extend to other psychology subdisciplines or other disciplines; not all eligible studies were replicated, resource-intensive studies were less likely to be included, and the impact of selection bias is unknown.
- The paper notes no single indicator sufficiently describes replication success, and meta-analytic estimates assume no publication bias in original results; if original effects are inflated by publication, selection, or reporting biases, replication estimates may better reflect true effects.
- The text makes evident that the sample is limited to 100 completed replications from 2008 articles in three journals, and that unanticipated differences in sample, setting, or procedure could alter observed effect magnitudes despite efforts to maintain fidelity.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
8 Valid · 0 Not valid · 10 Not checked · 144 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In cell biology, two industrial laboratories reported success replicating the original results of landmark findings in only 11% and 25% of the attempted cases ( 10, 11 ).”
What the sources say
- Partly supportedcites [10] 10. C. G. Begley, L. M. Ellis, Raise standards for preclinical cancer research. Nature 483 , 531533 (2012). sourceExperimental — accuracy not yet measured
In CONFIRMING RESEARCH FINDINGS (p. 2), the paper reports that Amgen confirmed findings in 6 of 53 landmark papers (11%) and that a Bayer team validated about 25% of published preclinical studies sufficiently for projects to continue. The 25% figure is not specifically for landmark findings, and the Bayer studies were not all cell biology or cancer research; the paper says 70% were published cancer research. (LLM, conf=0.87)
- Not checked — no full text of the sourcecites [11] 11. F. Prinz, T. Schlange, K. Asadullah, Believe it or not: How much can we rely on published data on potential drug targets? Nat. Rev. Drug Disc. 10 , 712713 (2011 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“There is good evidence that low power of research designs combined with publication bias favoring positive results over negative results produces a literature with upwardly biased effect sizes ( 14, 16, 30, 31 ).”
What the sources say
- Supportedcites [14] 14. K. S. Button, et al ., Power failure: Why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14 , 112 (2013). source
The ‘Low power in the presence of other biases’ section says publication bias disproportionately affects underpowered studies and that small negative studies are more readily rejected. The ‘Results’ section says small studies are particularly susceptible to inflated effect-size estimates and publication bias. Page numbers were not supplied in our copy. (LLM, conf=0.92)
- Not checked — no full text of the sourcecites [16] 16. A. G. Greenwald, Consequences of prejudice against the null hypothesis. Psychol. Bull. 82 , 1-20 (1975). source
- Not checked — no full text of the sourcecites [30] 30. J. Cohen, The statistical power of abnormalsocial psychological research: A review. J. Abnorm. Soc. Psychol. 65 , 145153 (1962). source
- Not matchedcites [31] 31. T. D. Sterling, Publication decisions and their possible effects on inferences drawn from tests of significanceor vice versa. J. Am. Stat. Assoc. 54 , 3034 (1959).
More than one paper fits this citation.
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nonetheless, none of these efforts guarantee that unanticipated factors in the sample, setting, or procedure altered the observed effect magnitudes ( 32 ).”
What the sources say
- Supportedcites [32] 32. R. Klein, et al ., Investigating variation in replicability: A 'many labs' replication project. Soc. Psychol. 45 , 142152 (2014). source
The Overview of the Present Research (p. 5) says sample, setting, and procedural variations can influence effects, but that effect magnitudes may also vary no more than expected random error. The Discussion (p. 18) cautions against assuming differences between samples and limits its conclusions to the studies and settings examined. Together, these support the claim that the efforts do not guarantee that unanticipated factors altered the observed magnitudes. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Ninetyseven percent of original studies had significant results ( p < .05).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“47% of original effect sizes were in the 95% confidence interval of the replication effect size;”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
60 more claims
Create a free account to see all 162 claims and every source verdict.
- Probably not a substantial claim. 65
- It may state two separate claims as one. 49
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“First because, once again, such a low level of reproducibility was not ubiquitous but varied depending on subfield, methodology, and expertise of the authors conducting the replication (23 -25).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [23]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“a large-scale replication of experimental psychology studies found that while 97% of results in the initial studies were statistically significant, only 36% of results in the replication were significant, even when larger samples were used (Open Science Collaboration, 2015)”
From Acceptance and Commitment Therapy Processes and Mediation: Challenges and How to Address Them (2023) · cites this paper as Open Science Collaboration, 2015
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, sample sizes are often underpowered (4-6), at times leading to conclusions that do not hold up with larger-sized replications or metaanalyses (7).”
From Generic language in scientific communication (2019) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In the field of psychology, less than half of experimental and correlational studies are reportedly reproducible (11).”
From The Prevalence of Inappropriate Image Duplication in Biomedical Research Publications (2016) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“reanalyses of the data suggest that reproducibility in psychological science might be higher than originally claimed (23, 26, 27).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [23]
Experimental — accuracy not yet measured
25 more claims about this paper
Create a free account to see every claim other papers make about this one.
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Consistentχ²(128) = 155.83reported p = .048 · recomputed p = .048
“χ 2 (128) = 155.83, p = .048”
- Consistentχ²(1) = 59.1reported p < .001 · recomputed p < .001
“χ 2 (1) = 59.1, p < .001”
- Consistentt(98) = -8.22reported p < .001 · recomputed p < .001
“t (98) = 8.22, p < .001”
- Consistentt(96) = 9.33reported p < .001 · recomputed p < .001
“t (96) = 9.33, p < .001”
- ConsistentQ(73) = 310.45reported p < .001 · recomputed p < .001
“Q (73) = 310.45, p < .001”
17 more recomputed tests
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“Of the 97 studies for which an effect size in both the original and replication study could be calculated ( 28 ), 82 showed a stronger effect size in the original study (82.8%; p < .001, binomial test; see right panel of Figure 1).”
Reported 82.8 · recomputed 84.536082
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Sampling Frame and Study Selection · p. 23
“an initial pool of the first 30 articles from each journal in chronological order.”
§ Sampling Frame and Selection Biases · p. 32
“Initially, a total of 60 articles were made available from the sampling frame”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Evaluating Reproducibility · p. 8
“p 's = .0508, .0514, .0516, .0567”
§ Preliminary analyses · p. 39
“(.0503, .0509, .0514, and .0516)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Abstract · p. 2
“original effects ( M = .396, SD = .193)”
§ Discussion · p. 12
“original effects ( M = .396, SD = .196)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Evaluating Reproducibility · p. 8
“a significant reduction (McNemar test, ᵯ€ 2 (1) = 59.5, p < .001).”
§ Evaluating replication effect against null hypothesis of no effect . · p. 40
“a significant change (McNemar test, χ 2 (1) = 59.1, p < .001).”
1 more finding
Create a free account to see every finding of this check.
Citations
Cites
Conducting Meta-Analyses in R with the metafor Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464 · cited 1×
“We conducted fixed-effect meta-analyses using the R packagemetafor (27) on Fisher-transformed correlations for all study-pairs in subset MA and on study-pairs with the odds ratio as the dependent variable.”
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Lakens, D. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00863 · Cited by 10,303
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“Concernsaboutreproducibilityarewidespread acrossdisciplines( 9 – 21 ).”
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 1×
“Low-powerresearchdesignscombinedwithpublication bias favoring positive results together produce a literature with upwardly biased effect sizes ( 14 , 16 , 33 , 34 ).”
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“make it likely that more than half of research results are false and therefore irreproducible (9).”
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
Studies in the Logic of Explanation
Hempel, C. G., & Oppenheim, P. (1948) · Philosophy of Science · doi:10.1086/286983 · Cited by 3,174
Research works printed but not linked: 2
- 6. W. C. Salmon, in Introduction to the Philosophy of Science, M. H. Salmon Ed. (Hackett Publishing Company, Inc., Indianapolis, 1999) pp. 741.
- 25. Open Science Collaboration, in Implementing Reproducible Computational Research (A Volume in The R Series) , V. Stodden, F. Leisch, R. Peng, Eds. (Taylor & Francis, New York, 2014) pp. 299323.
Cited by
Items 1,701–1,800 of 8,626
Brederoo, S., Nieuwenstein, M. R., Cornelissen, F. W., & Lorist, M. M. (2018) · Cortex · doi:10.1016/j.cortex.2018.10.021 · Cited by 40
Munnich, E. L., & Ranney, M. A. (2018) · Topics in Cognitive Science · doi:10.1111/tops.12397 · Cited by 46
Chambers, C., Kong, G.-Q., Wei, K.-L., & Kording, K. P. (2019) · PLoS ONE · doi:10.1371/journal.pone.0217861 · Cited by 30
Testosterone administration increases social discounting in healthy males
Wu, Y., Liao, J., Zilioli, S., Wu, Y., Deng, H., Li, H., & Tobler, P. N. (2019) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2019.06.013 · Cited by 37
Hennig, M., & Hütter, M. (2020) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000173 · Cited by 45
Cauchoix, M., Chaine, A. S., & Barragan‐Jason, G. (2020) · Frontiers in Ecology and Evolution · doi:10.3389/fevo.2020.00106 · Cited by 45
Advice to New Human-Robot Interaction Researchers
Belpaeme, T. (2020) · Springer series on bio- and neurosystems · doi:10.1007/978-3-030-42307-0_14 · Cited by 45
Toward dynamic phenotypes and the scalable measurement of human behavior
Germine, L. T., Strong, R. W., Singh, S. M., & Sliwinski, M. J. (2020) · Neuropsychopharmacology · doi:10.1038/s41386-020-0757-1 · Cited by 37
Kong, X., ENIGMA Laterality Working Group, & Francks, C. (2020) · Human Brain Mapping · doi:10.1002/hbm.25154 · Cited by 35
Reproducibility and Replicability in the Context of the Contested Identities of Geography
Sui, D. Z., & Kedron, P. (2020) · Annals of the American Association of Geographers · doi:10.1080/24694452.2020.1806024 · Cited by 37
Improving norms in research culture to incentivize transparency and rigor
Mellor, D. T. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1902329 · Cited by 41
Scheffel, C., Graupner, S., Gärtner, A., Zerna, J., Strobel, A., & Dörfel, D. (2021) · Psychophysiology · doi:10.1111/psyp.13908 · Cited by 45
Cai, Y., Yang, Y., & Shi, W. (2021) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2021.1938393 · Cited by 41
Isolating the sources of pipeline‐variability in group‐level task‐fMRI results
Bowring, A., Nichols, T. E., & Maumet, C. (2021) · Human Brain Mapping · doi:10.1002/hbm.25713 · Cited by 44
The Effect of Replications on Citation Patterns: Evidence From a Large-Scale Reproducibility Project
Schafmeister, F. (2021) · Psychological Science · doi:10.1177/09567976211005767 · Cited by 45
“The “small-telescopes” approach of Simonsohn (2015) is based on the idea of detectability in that it tests whether the effect obtained in a replication study is smaller than an effect size that the original study had only 33% power to detect.”
BackgroundProposal of a Selection Protocol for Replication of Studies in Sports and Exercise Science
Murphy, J., Mesquida, C., Caldwell, A. R., Earp, B. D., & Warne, J. P. (2022) · Sports Medicine · doi:10.1007/s40279-022-01749-1 · Cited by 46
Embedding data provenance into the Learning Health System to facilitate reproducible research
Ćurčin, V. (2016) · Learning Health Systems · doi:10.1002/lrh2.10019 · Cited by 44
Reproducibility in Psychological Science: When Do Psychological Phenomena Exist?
Iso‐Ahola, S. E. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00879 · Cited by 42
Robust modeling in cognitive sciencePreprint
Lee, M., Criss, A. H., Devezer, B., et al. (2019) · doi:10.31234/osf.io/dmfhk · Cited by 36
Fallibility in Science: Responding to Errors in the Work of Oneself and Others
Bishop, D. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918776632 · Cited by 39
Mayer, M. M., Bell, R., & Buchner, A. (2021) · PLOS ONE · doi:10.1371/journal.pone.0261673 · Cited by 36
Vitriol, J. A., & Moskowitz, G. B. (2021) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2021.104165 · Cited by 35
Context Matters: How an Ecological-Belonging Intervention Can Reduce Inequities in STEM
Hammarlund, S. P., Scott, C., Binning, K. R., & Cotner, S. H. (2022) · BioScience · doi:10.1093/biosci/biab146 · Cited by 35
‘Neurorights’: A Human Rights–Based Approach for Governing Neurotechnologies
Kellmeyer, P. (2022) · Cambridge University Press eBooks · doi:10.1017/9781009207898.032 · Cited by 45
Borycz, J., Olendorf, R., Specht, A., Grant, B. W., Crowston, K., Tenopir, C., Allard, S. L., Rice, N. M., Hu, R., & Sandusky, R. J. (2023) · Humanities and Social Sciences Communications · doi:10.1057/s41599-023-01831-7 · Cited by 37
Perspective-shifting and free indirect discourse: Experimental investigations
Kaiser, E. M. (2015) · Proceedings from Semantics and Linguistic Theory · doi:10.3765/salt.v25i0.3436 · Cited by 37
Would Ecology Fail the Repeatability Test?
Schnitzer, S. A., & Carson, W. P. (2016) · BioScience · doi:10.1093/biosci/biv176 · Cited by 40
Internal conceptual replications do not increase independent replication success
Kunert, R. (2016) · Psychonomic Bulletin & Review · doi:10.3758/s13423-016-1030-9 · Cited by 36
Webster, G. D., Smith, C. V., Brunell, A. B., Paddock, E. L., & Nezlek, J. B. (2016) · Journal of Research in Personality · doi:10.1016/j.jrp.2016.06.005 · Cited by 39
Dearing, E., & Zachrisson, H. D. (2017) · Child Development Perspectives · doi:10.1111/cdep.12224 · Cited by 35
Branco, A., Cohen, K. B., Vossen, P. T. J. M., Ide, N., & Calzolari, N. (2017) · Language Resources and Evaluation · doi:10.1007/s10579-017-9380-0 · Cited by 35
The Heuristic Value of p in Inductive Statistical Inference
Krueger, J. I., & Heck, P. R. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00908 · Cited by 34
Heuristics and biases in mental arithmetic: revisiting and reversing operational momentum
Shaki, S., Pinhas, M., & Fischer, M. H. (2017) · Thinking & Reasoning · doi:10.1080/13546783.2017.1348987 · Cited by 35
Achterberg, M., Bakermans-Kranenburg, M. J., van Ijzendoorn, M. H., van der Meulen, M., Tottenham, N., & Crone, E. A. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2018.03.076 · Cited by 35
Lindner, C., Nagy, G., & Retelsdorf, J. (2018) · Social Psychology of Education · doi:10.1007/s11218-018-9455-9 · Cited by 43
A Call for Open Science in Giftedness Research
McBee, M. T., Makel, M. C., Peters, S. J., & Matthews, M. S. (2018) · Gifted Child Quarterly · doi:10.1177/0016986218784178 · Cited by 40
“The journal Psychological Science recently added a requirement for all manuscripts reporting new empirical data to include an open practices statement (https://www.psychologicalscience.org/publications/psychological_science/ ps-submissions#OPS).”
InfluentialNew Frontiers in Parkinson's Disease: From Genetics to the Clinic
Shihabuddin, L. S., Brundin, P., Greenamyre, J. T., Stephenson, D. T., & Sardi, S. P. (2018) · Journal of Neuroscience · doi:10.1523/jneurosci.1666-18.2018 · Cited by 40
Cho, H., Roh, S., & Park, B. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.07.001 · Cited by 46
“Given the recent controversies about the reproducibility of psychological findings (Open Science Collaboration, 2015), it is worthwhile to retest the robustness of the default effect found in study 1.”
BackgroundSagarra, O., Gutiérrez-Roig, M., Bonhoure, I., & Perelló, J. (2016) · Frontiers in Physics · doi:10.3389/fphy.2015.00093 · Cited by 59
Jekel, M., Fiedler, S., Torras, R. A., Mischkowski, D., Dorrough, A. R., & Glöckner, A. (2019) · Psychology Learning & Teaching · doi:10.1177/1475725719868149 · Cited by 39
Introduction to the Special Collection on the Fragile Families Challenge
Salganik, M. J., Lundberg, I., Kindel, A. T., & McLanahan, S. (2019) · Socius: Sociological Research for a Dynamic World · doi:10.1177/2378023119871580 · Cited by 33
The effects of psychosocial stress on intergroup resource allocation
Schweda, A., Faber, N. S., Crockett, M. J., & Kalenscher, T. (2019) · Scientific Reports · doi:10.1038/s41598-019-54954-w · Cited by 43
The temporal dynamics of insight problem solving – restructuring might not always be sudden
Bilalić, M., Graf, M., Vaci, N., & Danek, A. H. (2019) · Thinking & Reasoning · doi:10.1080/13546783.2019.1705912 · Cited by 48
How robust is our cumulative knowledge on turnover?
Field, J. G., Bosco, F. A., & Kepes, S. (2020) · Journal of Business and Psychology · doi:10.1007/s10869-020-09687-3 · Cited by 39
Cairo, A. H., Green, J. D., Forsyth, D. R., Behler, A. M. C., & Raldiris, T. L. (2020) · Personality and Social Psychology Bulletin · doi:10.1177/0146167220903896 · Cited by 34
Assessing heterogeneity and power in replications of psychological experiments.
Schauer, J. M., & Hedges, L. V. (2020) · Psychological Bulletin · doi:10.1037/bul0000232 · Cited by 41
Traditional and biomedical care pathways for mental well‐being in rural Nepal
Pham, T. V., Koirala, R., & Kohrt, B. A. (2021) · International Journal of Mental Health Systems · doi:10.1186/s13033-020-00433-z · Cited by 45
Implications of the open science era for educational psychology research syntheses
Patall, E. A. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1897009 · Cited by 38
Looking Back at the Next 40 Years of ASD Neuroscience Research
McPartland, J. C., Lerner, M. D., Bhat, A. N., Clarkson, T., Jack, A., Koohsari, S., Matuskey, D., McQuaid, G. A., Su, W.-C., & Trevisan, D. A. (2021) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-021-05095-5 · Cited by 39
“2010; Open Science Collaboration, 2015), the constraints of MRI as a method (particularly the cost of acquiring MRI data and the data’s susceptibility to noise, including noise from millimeter or sub-millimeter head motion (eg.”
MethodSabik, N. J., Matsick, J. L., McCormick‐Huhn, K., & Cole, E. R. (2021) · Psychology of Women Quarterly · doi:10.1177/03616843211035678 · Cited by 36
Transparent, Open, and Reproducible Prevention Science
Grant, S., Wendt, K. E., Leadbeater, B. J. R., Supplee, L. H., Mayo‐Wilson, E., Gardner, F. E. M., & Bradshaw, C. P. (2022) · Prevention Science · doi:10.1007/s11121-022-01336-w · Cited by 46
“Numerous large-scale collaborative efforts have found low reproducibility rates in psychology (Klein et al., 2014; Open Science, 2015), economics (Camerer et al., 2016; Chang & Li, 2015), the social sciences (Camerer et al., 2018), and medicine (Nosek & Errington, 2017).”
BackgroundExtrinsic rewards and crowding-out of prosocial behaviour
Wollbrant, C., Knutsson, M., & Martinsson, P. (2022) · Nature Human Behaviour · doi:10.1038/s41562-022-01293-y · Cited by 40
Hullman, J. R., Kapoor, S., Nanayakkara, P., Gelman, A., & Narayanan, A. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534196 · Cited by 59
“While psychologists discussed fundamental issues with conventional approaches to inference as early as the 1960s [50, 146], in the last decade critics brought concerns to the forefront, demonstrating how motivated researchers can obtain false positives under various conditions [74, 75, 188] and that many published con…”
BackgroundResonant Games: Design Principles for Learning Games that Connect Hearts, Minds, and the Everyday
Klopfer, E. D., Haas, J., Osterweil, S., & Rosenheck, L. (2018) · The MIT Press eBooks · doi:10.7551/mitpress/10887.001.0001 · Cited by 67
Psychology needs to get tired of winning
Haeffel, G. J. (2022) · Royal Society Open Science · doi:10.1098/rsos.220099 · Cited by 41
Beyond Random Effects: When Small-Study Findings Are More Heterogeneous
Stanley, T. D., Doucouliagos, H., & Ioannidis, J. P. A. (2022) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459221120427 · Cited by 43
Graßmann, C., & Schermuly, C. C. (2020) · International Coaching Psychology Review · doi:10.53841/bpsicpr.2020.15.2.99 · Cited by 39
A Replication Study of Operations Management Experiments in Management Science
Davis, A. M., Flicker, B., Hyndman, K., Katok, E., Keppler, S. M., Leider, S., Long, X., & Tong, J. D. (2022) · Management Science · doi:10.1287/mnsc.2023.4866 · Cited by 54
Nomological network of two-dimensional Machiavellianism
Monaghan, C., Bizumić, B., & Sellbom, M. (2018) · Personality and Individual Differences · doi:10.1016/j.paid.2018.03.047 · Cited by 44
Thresholds, Text Coverage, Vocabulary Size, and Reading Comprehension in Applied Linguistics
Larson, M. (2017) · Open Access Victoria University of Wellington | Te Herenga Waka (Figshare) · doi:10.26686/wgtn.17057906.v1 · Cited by 35
Estimating the Reproducibility of Experimental PhilosophyPreprint
Cova, F., Strickland, B., Abatista, A., Allard, A., Andow, J., Attie, M., Beebe, J. R., Berniûnas, R., Boudesseul, J., Colombo, M., et al. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/sxdah · Cited by 38
Bayesian Mindsponge FrameworkPreprint
Nguyen, M., & Le, T.-T. (2021) · doi:10.31219/osf.io/b4uny · Cited by 39
Revisiting the Stanford Prison Experiment: A Case Study in Organized Skepticism
Kulig, T. C., Pratt, T. C., & Cullen, F. T. (2016) · Journal of Criminal Justice Education · doi:10.1080/10511253.2016.1165855 · Cited by 35
Dunn, B. D., Burr, L. A., Smith, H. B., Hunt, A., Dadgostar, D., Dalglish, L., Smith, S., Attree, E., Jell, G., Martyn, J., et al. (2018) · Behaviour Research and Therapy · doi:10.1016/j.brat.2018.05.003 · Cited by 42
Osterhaus, C., & Koerber, S. (2020) · European Journal of Developmental Psychology · doi:10.1080/17405629.2020.1820861 · Cited by 38
“Confirmatory replication: Three-factor AToM model To replicate the three-factor AToM model from late elementary school, we conducted a confirmatory factor analysis of the 18 AToM items using MPlus 7 software (Muthén & Muthén, 2012) with a mean- and varianceadjusted weighted least-squares estimator.”
MethodRoyer, M., Houser, K. W., Durmus, D., Esposito, T., & Wei, M. (2021) · Lighting Research & Technology · doi:10.1177/14771535211019864 · Cited by 38
Range Restriction Affects Factor Analysis: Normality, Estimation, Fit, Loadings, and Reliability
Franco-Martínez, A., Alvarado, J. M., & Sorrel, M. Á. (2022) · Educational and Psychological Measurement · doi:10.1177/00131644221081867 · Cited by 48
Vuong, Q., Le, T.-T., Quang-Loc, N., Quang-Loc, N., & Nguyen, M. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph191912233 · Cited by 39
Impaired episodic memory in PTSD patients — A meta-analysis of 47 studies
Petzold, M., & Bunzeck, N. (2022) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2022.909442 · Cited by 48
Trübutschek, D., Yang, Y., Gianelli, C., Čėsnaitė, E., Fischer, N. L., Vinding, M. C., Marshall, T. R., Algermissen, J., Pascarella, A., Puoliväli, T., et al. (2023) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_02087 · Cited by 47
“Low replicability and reproducibility of research findings have eroded trust in even some of the most established results (e.g., Szucs & Ioannidis, 2017; Open Science Collaboration, 2015).”
ResultBackfisch, I., Sibley, L., Lachner, A., Kirchner, K. T., Hische, C., & Scheiter, K. (2024) · Teaching and Teacher Education · doi:10.1016/j.tate.2024.104532 · Cited by 68
Cui, Z., Li, N., & Zhou, H. (2025) · Nature Computational Science · doi:10.1038/s43588-025-00840-7 · Cited by 122
“For assessing replication success specifically, we analyzed a subsample of statistically significant original findings ( P 0.05), as replication success fundamentally involves reproducing previously demonstrated significant effects 28 .”
InfluentialSchaller, M. (2015) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.09.006 · Cited by 44
van IJzendoorn, M. H., & Bakermans‐Kranenburg, M. J. (2016) · Journal of Neuroendocrinology · doi:10.1111/jne.12355 · Cited by 34
Reinquiries in Advertising Research
Eisend, M., Franke, G. R., & Leigh, J. H. (2016) · Journal of Advertising · doi:10.1080/00913367.2015.1115381 · Cited by 35
Physiological Plausibility Can Increase Reproducibility in Cognitive Neuroscience
van Ede, F., & Maris, E. (2016) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2016.05.006 · Cited by 32
Design and Analysis of Replication Studies
Bonett, D. G. (2020) · Organizational Research Methods · doi:10.1177/1094428120911088 · Cited by 39
Oxytocin and vulnerable romantic relationships
Grebe, N. M., Kristoffersen, A. A., Grøntvedt, T. V., Emery Thompson, M., Kennair, L. E. O., & Gangestad, S. W. (2017) · Hormones and Behavior · doi:10.1016/j.yhbeh.2017.02.009 · Cited by 34
Replications in Development Economics
Sukhtankar, S. (2017) · American Economic Review · doi:10.1257/aer.p20171120 · Cited by 35
The dark personality and psychopathology: Toward a brighter future.
Thomaes, S. C. E., Brummelman, E., Miller, J. D., & Lilienfeld, S. O. (2017) · Journal of Abnormal Psychology · doi:10.1037/abn0000305 · Cited by 44
Pedersen, M. J., & Stritch, J. M. (2018) · Public Administration Review · doi:10.1111/puar.12910 · Cited by 38
Evidence-Based Policy in the Real World: A Cautionary View
Gamoran, A. (2018) · The Annals of the American Academy of Political and Social Science · doi:10.1177/0002716218770138 · Cited by 38
The sampling precision of research in five major areas of psychology
Trafimow, D., & Myüz, H. A. (2018) · Behavior Research Methods · doi:10.3758/s13428-018-1173-x · Cited by 37
“Third, we provide an Open Science Foundation link to the data file: https://osf.io/ms4zg/.”
BackgroundInfluentialThe reliability of estimating visual working memory capacity
Dai, M., Li, Y., Gan, S., & Du, F. (2019) · Scientific Reports · doi:10.1038/s41598-019-39044-1 · Cited by 40
Launes, G., Laukvik, I. L., Sunde, T., Klovning, I., Hagen, K., Solem, S., Öst, L., Hansen, B. K. A., & Kvale, G. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.01069 · Cited by 41
Empirical research on folk moral objectivism
Pölzler, T., & Wright, J. C. (2019) · Philosophy Compass · doi:10.1111/phc3.12589 · Cited by 48
Lateral Eye Movements Do Not Increase False-Memory Rates: A Failed Direct-Replication Study
van Schie, K., & Leer, A. (2019) · Clinical Psychological Science · doi:10.1177/2167702619859335 · Cited by 42
Kampa, M., Schick, A., Sebastian, A., Wessa, M., Tüscher, O., Kalisch, R., & Yuen, K. S. L. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.116223 · Cited by 34
2D:4D Suggests a Role of Prenatal Testosterone in Gender Dysphoria
Sadr-Bazzaz, M., Khorashad, B. S., Talaeı, A., Fazeli, N., & Hönekopp, J. (2020) · Archives of Sexual Behavior · doi:10.1007/s10508-020-01630-0 · Cited by 36
Wiik, E., Jones, J. P. G., Pynegar, E., Bottazzi, P., Asquith, N., Gibbons, J., & Kontoleon, A. V. (2020) · Conservation Biology · doi:10.1111/cobi.13508 · Cited by 37
Power, S. A., & Vélez, G. (2020) · Review of General Psychology · doi:10.1177/1089268020915841 · Cited by 39
ManyPrimates, Altschul, D., Beran, M. J., Bohn, M., Caspar, K., Fichtel, C., Försterling, M., Grebe, N. M., Hernandez-Aguilar, R. A., Kwok, S. C., et al. (2019) · PsyArXiv (OSF Preprints) · doi:10.24602/sjpr.62.3_205 · Cited by 42
Bakker, M., Veldkamp, C. L. S., van den Akker, O. R., van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H., & Wicherts, J. M. (2020) · PLoS ONE · doi:10.1371/journal.pone.0236079 · Cited by 45
Free to blame? Belief in free will is related to victim blaming
Genschow, O., & Vehlow, B. (2021) · Consciousness and Cognition · doi:10.1016/j.concog.2020.103074 · Cited by 46
The role of replication in psychological science
Fletcher, S. C. (2021) · European Journal for Philosophy of Science · doi:10.1007/s13194-020-00329-2 · Cited by 49
Toffalini, E., Giofrè, D., Pastore, M., Carretti, B., Fraccadori, F., & Szűcs, D. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01549-x · Cited by 53
“This is true not only because the real effect sizes in psychology are generally limited (Open Science Collaboration, 2015), but also because learning disorders are characterized by poor response to treatment by definition (DSM-5; APA, 2013).”
BackgroundAdvances in transparency and reproducibility in the social sciences
Freese, J., Rauf, T., & Voelkel, J. G. (2022) · Social Science Research · doi:10.1016/j.ssresearch.2022.102770 · Cited by 60
Ma, N., Zhang, Q., Murai, F., Braham, W. W., & Samuelson, H. W. (2023) · Building and Environment · doi:10.1016/j.buildenv.2023.110319 · Cited by 40
Jha, M. K., Minhajuddin, A., Greer, T. L., Carmody, T. J., Rush, A. J., & Trivedi, M. H. (2016) · PLoS ONE · doi:10.1371/journal.pone.0167901 · Cited by 35
Leppink, J. (2017) · Journal of Taibah University Medical Sciences · doi:10.1016/j.jtumed.2016.11.008 · Cited by 88
Metadata
- Authors
- Open Science Collaboration
- Corresponding author
- Open Science Collaboration
- Venue
- Science
- Published
- 2015-08-28
- Volume
- 349
- Issue
- 6251
- Pages
- aac4716-aac4716
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W1897139626
- Cited by
- 8,515
- References
- 39
- Citations by year
- 2026: 414 · 2025: 532 · 2024: 698 · 2023: 757 · 2022: 783 · 2021: 906 · 2020: 1001 · 2019: 989 · 2018: 938 · 2017: 800 · 2016: 607 · 2015: 79
- FWCI
- 736.90
- Type
- journal article
- Language
- en
- Open access
- Locations
- Science (publishedVersion)
- Research portal (Tilburg University) (publishedVersion, other-oa)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, other-oa)
- UvA-DARE (University of Amsterdam) (publishedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (publishedVersion)
- VU Research Portal (publishedVersion)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (acceptedVersion, other-oa)
- Repository of the Academy's Library (Library of the Hungarian Academy of Sciences) (acceptedVersion)
- London School of Economics and Political Science Theses Online (London School of Economics and Political Science) (acceptedVersion)
- Discovery Research Portal (University of Dundee) (submittedVersion, other-oa)
- Keele Research Repository (Keele University) (submittedVersion, cc-by)
- London School of Economics and Political Science Research Online (London School of Economics and Political Science) (submittedVersion)
- Pure (Coventry University) (submittedVersion, other-oa)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- Radboud Repository (Radboud University) (submittedVersion)
- Winchester Research Repository (submittedVersion)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion)
- ANU Open Research (Australian National University) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- TU/e Research Portal (submittedVersion)
- BOA (University of Milano-Bicocca) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- Max Planck Digital Library (submittedVersion)
- Coventry University Open Collections (Coventry university) (submittedVersion)
- The HKU Scholars Hub (University of Hong Kong) (submittedVersion)
- Research Portal (King's College London) (submittedVersion)
- publish.UP (University of Potsdam) (submittedVersion)
- Lirias (KU Leuven) (submittedVersion)
- Faculty of Philosophy (University of Belgrade) (submittedVersion)
- Biblos-e Archivo (Universidad Autónoma de Madrid) (submittedVersion, public-domain)
- Bristol Research (University of Bristol) (submittedVersion)
- UWE Research Repository (UWE Bristol) (submittedVersion, cc-by)
- Padua Research Archive (University of Padova) (submittedVersion, cc-by-nc-nd)
- Zurich Open Repository and Archive (University of Zurich) (submittedVersion)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, other-oa)
- The Open Collections (Coventry University) (submittedVersion)
- Research Publications (Maastricht University) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Funders
- Laura and John Arnold Foundation
- Center for Open Science
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Keywords
- reproducibility, replication, statistical significance, confidence intervals, publication bias, quality of evidence, psychology research
- MeSH terms
- Periodicals as Topic, Psychology, Publishing, Research, Research Design, Reproducibility of Results, Confidence Intervals, Publication Bias, Behavioral Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.