Estimating the reproducibility of psychological science
Open accessPublished version of a preprint
The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index; 20 of the 57 sources its claims cite were checked for support.
Abstract
Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.
Overview
CoChat Summary
TL;DR
A large collaborative replication of 100 published psychology studies found that replication effects were about half the magnitude of original effects, and only 36% of replications had significant results versus 97% of originals. Replication success was more consistently related to the strength of the original evidence than to characteristics of the replication teams or implementation.
Key findings
- Original effects (M = .396, SD = .193) were reliably larger than replication effects (M = .198, SD = .255); 82 of 97 study pairs (82.8%) showed a stronger original effect.
- 97% of original studies had significant results (p < .05), but only 36% of replications did; 47% of original effect sizes fell within the 95% CI of the replication effect size, 38% were subjectively rated as replicated, and 70% of meta-analytic combinations were significant if no bias in original results was assumed.
- Reproducibility was stronger for cognitive than social psychology: 20 of 38 (53%) cognitive effects replicated by p < .05 versus 14 of 53 (26%) social effects; main/simple effects replicated more often (23/50, 46%) than interaction effects (8/37, 22%).
- Replication success correlated with original p-value, original effect size, and replication power, but not consistently with perceived importance, team expertise, or self-assessed replication quality.
Methods
- The Open Science Collaboration conducted 100 replications of studies published in 2008 in three psychology journals: Psychological Science, Journal of Personality and Social Psychology, and Journal of Experimental Psychology: Learning, Memory, and Cognition; 270 contributing authors completed the replications.
- Replication teams followed a structured, pre-registered protocol: selecting a key effect, contacting original authors for materials, writing and registering an analysis plan, obtaining review, conducting high-powered replications (average 92% power to detect the original effect size), and archiving data and materials.
- Analyses converted effects to correlation coefficients where possible and evaluated reproducibility with five indicators: significance testing, comparison of original vs. replication effect sizes, whether the original effect size fell within the replication 95% CI, meta-analytic combination, and subjective “Did it replicate?” ratings.
Limitations
- The paper states that it is unknown how far findings extend to other psychology subdisciplines or other disciplines; not all eligible studies were replicated, resource-intensive studies were less likely to be included, and the impact of selection bias is unknown.
- The paper notes no single indicator sufficiently describes replication success, and meta-analytic estimates assume no publication bias in original results; if original effects are inflated by publication, selection, or reporting biases, replication estimates may better reflect true effects.
- The text makes evident that the sample is limited to 100 completed replications from 2008 articles in three journals, and that unanticipated differences in sample, setting, or procedure could alter observed effect magnitudes despite efforts to maintain fidelity.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
8 Valid · 0 Not valid · 10 Not checked · 144 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In cell biology, two industrial laboratories reported success replicating the original results of landmark findings in only 11% and 25% of the attempted cases ( 10, 11 ).”
What the sources say
- Partly supportedcites [10] 10. C. G. Begley, L. M. Ellis, Raise standards for preclinical cancer research. Nature 483 , 531533 (2012). sourceExperimental — accuracy not yet measured
In CONFIRMING RESEARCH FINDINGS (p. 2), the paper reports that Amgen confirmed findings in 6 of 53 landmark papers (11%) and that a Bayer team validated about 25% of published preclinical studies sufficiently for projects to continue. The 25% figure is not specifically for landmark findings, and the Bayer studies were not all cell biology or cancer research; the paper says 70% were published cancer research. (LLM, conf=0.87)
- Not checked — no full text of the sourcecites [11] 11. F. Prinz, T. Schlange, K. Asadullah, Believe it or not: How much can we rely on published data on potential drug targets? Nat. Rev. Drug Disc. 10 , 712713 (2011 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“There is good evidence that low power of research designs combined with publication bias favoring positive results over negative results produces a literature with upwardly biased effect sizes ( 14, 16, 30, 31 ).”
What the sources say
- Supportedcites [14] 14. K. S. Button, et al ., Power failure: Why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14 , 112 (2013). source
The ‘Low power in the presence of other biases’ section says publication bias disproportionately affects underpowered studies and that small negative studies are more readily rejected. The ‘Results’ section says small studies are particularly susceptible to inflated effect-size estimates and publication bias. Page numbers were not supplied in our copy. (LLM, conf=0.92)
- Not checked — no full text of the sourcecites [16] 16. A. G. Greenwald, Consequences of prejudice against the null hypothesis. Psychol. Bull. 82 , 1-20 (1975). source
- Not checked — no full text of the sourcecites [30] 30. J. Cohen, The statistical power of abnormalsocial psychological research: A review. J. Abnorm. Soc. Psychol. 65 , 145153 (1962). source
- Not matchedcites [31] 31. T. D. Sterling, Publication decisions and their possible effects on inferences drawn from tests of significanceor vice versa. J. Am. Stat. Assoc. 54 , 3034 (1959).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nonetheless, none of these efforts guarantee that unanticipated factors in the sample, setting, or procedure altered the observed effect magnitudes ( 32 ).”
What the sources say
- Supportedcites [32] 32. R. Klein, et al ., Investigating variation in replicability: A 'many labs' replication project. Soc. Psychol. 45 , 142152 (2014). source
The Overview of the Present Research (p. 5) says sample, setting, and procedural variations can influence effects, but that effect magnitudes may also vary no more than expected random error. The Discussion (p. 18) cautions against assuming differences between samples and limits its conclusions to the studies and settings examined. Together, these support the claim that the efforts do not guarantee that unanticipated factors altered the observed magnitudes. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Ninetyseven percent of original studies had significant results ( p < .05).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“47% of original effect sizes were in the 95% confidence interval of the replication effect size;”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
60 more claims
Create a free account to see all 162 claims and every source verdict.
- Probably not a substantial claim. 65
- It may state two separate claims as one. 49
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“First because, once again, such a low level of reproducibility was not ubiquitous but varied depending on subfield, methodology, and expertise of the authors conducting the replication (23 -25).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [23]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“a large-scale replication of experimental psychology studies found that while 97% of results in the initial studies were statistically significant, only 36% of results in the replication were significant, even when larger samples were used (Open Science Collaboration, 2015)”
From Acceptance and Commitment Therapy Processes and Mediation: Challenges and How to Address Them (2023) · cites this paper as Open Science Collaboration, 2015
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, sample sizes are often underpowered (4-6), at times leading to conclusions that do not hold up with larger-sized replications or metaanalyses (7).”
From Generic language in scientific communication (2019) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In the field of psychology, less than half of experimental and correlational studies are reportedly reproducible (11).”
From The Prevalence of Inappropriate Image Duplication in Biomedical Research Publications (2016) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“reanalyses of the data suggest that reproducibility in psychological science might be higher than originally claimed (23, 26, 27).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [23]
Experimental — accuracy not yet measured
25 more claims about this paper
Create a free account to see every claim other papers make about this one.
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Consistentχ²(128) = 155.83reported p = .048 · recomputed p = .048
“χ 2 (128) = 155.83, p = .048”
- Consistentχ²(1) = 59.1reported p < .001 · recomputed p < .001
“χ 2 (1) = 59.1, p < .001”
- Consistentt(98) = -8.22reported p < .001 · recomputed p < .001
“t (98) = 8.22, p < .001”
- Consistentt(96) = 9.33reported p < .001 · recomputed p < .001
“t (96) = 9.33, p < .001”
- ConsistentQ(73) = 310.45reported p < .001 · recomputed p < .001
“Q (73) = 310.45, p < .001”
17 more recomputed tests
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“Of the 97 studies for which an effect size in both the original and replication study could be calculated ( 28 ), 82 showed a stronger effect size in the original study (82.8%; p < .001, binomial test; see right panel of Figure 1).”
Reported 82.8 · recomputed 84.536082
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Sampling Frame and Study Selection · p. 23
“an initial pool of the first 30 articles from each journal in chronological order.”
§ Sampling Frame and Selection Biases · p. 32
“Initially, a total of 60 articles were made available from the sampling frame”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Evaluating Reproducibility · p. 8
“p 's = .0508, .0514, .0516, .0567”
§ Preliminary analyses · p. 39
“(.0503, .0509, .0514, and .0516)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Abstract · p. 2
“original effects ( M = .396, SD = .193)”
§ Discussion · p. 12
“original effects ( M = .396, SD = .196)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Evaluating Reproducibility · p. 8
“a significant reduction (McNemar test, ᵯ€ 2 (1) = 59.5, p < .001).”
§ Evaluating replication effect against null hypothesis of no effect . · p. 40
“a significant change (McNemar test, χ 2 (1) = 59.1, p < .001).”
1 more finding
Create a free account to see every finding of this check.
Citations
Cites
Conducting Meta-Analyses in R with the metafor Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464 · cited 1×
“We conducted fixed-effect meta-analyses using the R packagemetafor (27) on Fisher-transformed correlations for all study-pairs in subset MA and on study-pairs with the odds ratio as the dependent variable.”
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Lakens, D. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00863 · Cited by 10,303
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“Concernsaboutreproducibilityarewidespread acrossdisciplines( 9 – 21 ).”
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 1×
“Low-powerresearchdesignscombinedwithpublication bias favoring positive results together produce a literature with upwardly biased effect sizes ( 14 , 16 , 33 , 34 ).”
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“make it likely that more than half of research results are false and therefore irreproducible (9).”
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
Studies in the Logic of Explanation
Hempel, C. G., & Oppenheim, P. (1948) · Philosophy of Science · doi:10.1086/286983 · Cited by 3,174
Research works printed but not linked: 2
- W. C. Salmon in Introduction to the Philosophy of Science M. H. Salmon Ed. (Hackett Publishing Company Indianapolis 1999) pp. 7–41.
- Open Science Collaboration in Implementing Reproducible Computational Research (A Volume in The R Series) V. Stodden F. Leisch R. Peng Eds. (Taylor & Francis New York 2014) pp. 299–323.
Cited by
Items 5,701–5,800 of 8,625
Transparent Science: A More Credible, Reproducible, and Publishable Way to Do Science
Mellor, D. T., Vazire, S., & Lindsay, D. S. (2018) · Cambridge University Press eBooks · doi:10.1017/9781108304443.020 · Cited by 2
Johnson, B. A., Addolorato, G., Lesch, O., Liu, L., & Rodd, Z. A. (2023) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2023.1271229 · Cited by 3
“The Psychology Reproductivity Project reported that only 36% of ‘highly influential’ published research in highly ranked journals were reproduced (2).”
BackgroundCzy kryzys wiarygodności w psychologii?
Brzeziński, J. M. (2023) · Przegląd Psychologiczny · doi:10.31648/przegldpsychologiczny.9456 · Cited by 5
Trail making test B in postoperative delirium: a replication study
Fislage, M., Feinkohl, I., Borchers, F., Heinrich, M., Pischon, T., Veldhuijzen, D. S., Slooter, A. J. C., Spies, C., Winterer, G., & Zacharias, N. (2023) · BJA Open · doi:10.1016/j.bjao.2023.100239 · Cited by 2
Our vision is your opportunity
Brügger, A., & Richter, I. (2023) · Global Environmental Psychology · doi:10.5964/gep.11841 · Cited by 2
“…decades, standard research practices have increasingly come under scru tiny because they undermine key principles of good science such as transparency and reproducibility (Open Science Collaboration, 2015); while some practices are “merely” questionable, others are simply fraud (Spellman, 2015).”
BackgroundThériault, R., & Dandeneau, S. (2023) · Mindfulness · doi:10.1007/s12671-023-02246-w · Cited by 9
“In the current era of the replication crisis (e.g., Camerer et al., 2018; Ioannidis, 2005; Open Science Collaboration, 2015), preregistered replication studies seem more necessary than ever (Cesario, 2014; Nosek et al., 2018; Nosek & Lakens, 2014).”
Vermessen? Zur Möglichkeit philosophischer Beiträge für den Diskurs der quantitativen Psychologie
Ramminger, J. J. (2023) · cultura & psyché · doi:10.1007/s43638-023-00081-3 · Cited by 3
“Die Replikationskrise beschreibt den Umstand, dass ein großer Anteil an psycho-logischen Studien bei Wiederholung nicht bestätigt werden kann (Open Science Collaboration 2015).”
Do reconciliation events serve as a conciliatory signal?
Ohtsubo, Y., Himichi, T., Inamasu, K., Kohama, S., Mifune, N., & Tago, A. (2023) · European Journal of Social Psychology · doi:10.1002/ejsp.3028 · Cited by 3
Hutchinson, J., Murphy, J., Lucia, B. D., O’Neill, E., Curtis, D., Mellano, K. T., Pelton, L. M., & Coker, N. A. (2024) · Communications in Kinesiology · doi:10.51224/cik.2024.60 · Cited by 4
“In psychology, for example, a widespread replication attempt by the Open Science Collaboration (2015) demonstrated that more than half of the empirical findings under scrutiny did not replicate.”
BackgroundIntegrating exploration and prediction in computational psychotherapy science: proof of concept
Fisher, H., Stone, S. J., Zilcha‐Mano, S., Goldstein, P., & Anderson, T. (2024) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2023.1274764 · Cited by 2
“…evidence for the potential utility of implementations of data-driven and machine learning approaches in psychotherapy research has emerged in recent years, answering questions of treatment personalization (16–18) while providing critical solutions to p -hacking (19) and replicability issues (10).”
BackgroundPsychotherapy Expectations and Preferences Among Black and White Undergraduate Women
Rodriguez, T. R., Woodling, C. M., O’Dell, C. C., Charles, N. E., Finn, J. A., & Anestis, J. C. (2024) · Journal of College Student Mental Health · doi:10.1080/28367138.2024.2303076 · Cited by 2
“Given the need for increased replication efforts in psychological science (e.g., Open Science Collaboration, 2015), we first aimed to determine whether the findings from Charles et al. (2021) are replicable in a sample of undergraduate women.”
Li, T., Frank, K. A., & Chen, M. (2024) · Mathematics · doi:10.3390/math12030388 · Cited by 5
“The probabilities of reproducibility are driven by the reproducibility of a scholarly finding rather than its statistical significance, thus becoming tools for tackling the reproducibility crisis [60,61].”
BackgroundThe best medicine: Lessons from health for policy randomistas
Leigh, A. (2024) · Evaluation Journal of Australasia · doi:10.1177/1035719x231226437 · Cited by 2
“A 2015 study led by psychologist Brian Nosek looked at a hundred studies published in top journals and found that they could replicate only one in three (Open Science Collaboration, 2015).”
BackgroundTesting the efficacy of the cognitive interview to road traffic accident investigations
Maulina, D., Irwanda, D. Y., Guritnaningsih, Otgaar, H., Nurfajriah, D. S., Armas, L. O. M. A., & Dewi, A. I. (2024) · Applied Cognitive Psychology · doi:10.1002/acp.4177 · Cited by 3
The Alternative Factors Leading to Replication Crisis: Prediction and Evaluation
Chernov, G. (2024) · Evaluation Review · doi:10.1177/0193841x241229106 · Cited by 3
Reporting Standards, Certification/Accreditation, and Reproducibility
Simon, G., & Aliferis, C. (2024) · Health informatics · doi:10.1007/978-3-031-39355-6_17 · Cited by 2
The State of Open Science Practices in Psychometric Studies of Suicide: A Systematic Review
Flores-Kanter, P. E., & Alvarado, J. M. (2024) · Assessment · doi:10.1177/10731911241236315 · Cited by 4
Costigan, S., Ruscio, J., & Crawford, J. T. (2024) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459241227865 · Cited by 4
Exploring the multiverse of analysis options for the alcohol Stroop
Jones, A., Stafford, T., & Petrovskaya, E. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02377-5 · Cited by 6
“For example, in 2015 the Open Science Collaboration attempted to replicate 100 psychology studies published in high-impact journals (Open Science Collaboration, 2015), with increased statistical power.”
BackgroundAssociations between cognitive test scores and pain tolerance: The Tromsø study
Melum, T. A., Steingrímsdóttir, Ó. A., Jacobsen, H. B., Johnsen, B., Stubhaug, A., Schirmer, H., Mathiesen, E. B., & Nielsen, C. S. (2024) · Scandinavian Journal of Pain · doi:10.1515/sjpain-2023-0082 · Cited by 2
McDaniel, J. (2024) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-024-06279-5 · Cited by 3
“Therefore, LRR effects can support meta-analytic work, which is important for addressing replication concerns and as a means for influencing research and policy decisions (Berchialla et al., 2021; Chow et al., 2021; Nosek et al., 2015).”
Hyper-ambition and the Replication Crisis: Why Measures to Promote Research Integrity can Falter
Erden, Y. J. (2024) · Journal of Academic Ethics · doi:10.1007/s10805-024-09528-5 · Cited by 5
“For instance with increased funding for, say, large-scale collaborations that in turn produce landmark replications of the kind generated by the Open Science Collaboration ( 2015).”
Time-dependent proteomics and drug response in expanding cancer cells
Pan, Y., Xuan, Y., Hao, P., Chen, X., Yan, R., Zhang, C., Ke, X., Qu, Y., & Zhang, X. (2024) · Pharmacological Research · doi:10.1016/j.phrs.2024.107208 · Cited by 3
Screening p -hackers: Dissemination noise as bait
Echenique, F., & He, K. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2400787121 · Cited by 5
Ni, H., Price, P. C., & Jones, C. J. (2024) · School Psychology Review · doi:10.1080/2372966x.2024.2365618 · Cited by 3
“Systematic attempts to replicate published studies in psychology have suggested that at least half of published results are not reproducible (Open Science Collaboration, 2015).”
What drives preventative health behaviors one year into a pandemic? A replication and extension
Disabato, D. J., Foust, J. L., Taber, J. M., Thompson, C. A., Sidney, P. G., & Coifman, K. G. (2024) · Psychology and Health · doi:10.1080/08870446.2024.2372651 · Cited by 3
“However, only 1/3 of original studies in the behavioral sciences replicate their primary effects (Open Science Collaboration, 2015).”
The Language of (Non)Replicable Social Science
Herzenstein, M., Rosario, S., Oblander, S., & Netzer, O. (2024) · Psychological Science · doi:10.1177/09567976241254037 · Cited by 6
“…replication efforts of original studies in psychology and behavioral economics, we compiled a data set of 299 studies: 96 studies were replicated by Open Science Collaboration (2015; RPP), 49 by Many Labs (Ebersole et al., 2016; Klein et al., 2014, 2018, 2022; ML), 18 by Camerer et al. (2016; EE),…”
Mirabella, G. (2024) · Journal of Neurophysiology · doi:10.1152/jn.00359.2024 · Cited by 2
Kravitz, D. J., Mitroff, S. R., Flintoft, C. C., & Oie, K. S. (2024) · Policy Insights from the Behavioral and Brain Sciences · doi:10.1177/23727322241274745 · Cited by 2
“Reproducibility — that a study will produce similar results if run again — is a problem in the basic sciences (Open Science Collaboration, 2015) but can be even worse in applied contexts where there can be fi rmer deadlines to produce results, potential con fl icts of interest for internal…”
Rengasamy, M. L., & Price, R. B. (2024) · Psychiatry Research · doi:10.1016/j.psychres.2024.116190 · Cited by 4
“Through prior replication projects (Camerer et al., 2018; Collaboration, 2015; Klein et al., 2018), have explored replication of experimental studies, our analyses here identify the meaningfulness of both different replicability metrics and analytic decisions, as our findings indicate that two…”
ResultToward Better Reporting in Oral Health Research
Jakubovics, N. S., & Schwendicke, F. (2024) · Journal of Dental Research · doi:10.1177/00220345241275459 · Cited by 4
“Large-scale replication studies in psychology and economics have found that replications often provide weaker evidence than the original studies or fail to reproduce the original findings and conclusions at all (Open Science Collaboration 2015; Camerer et al. 2016).”
Schmitt, H., Sindermann, C., & Montag, C. (2024) · Deviant Behavior · doi:10.1080/01639625.2024.2398529 · Cited by 4
“Following up from this consideration, a more general issue is the deficient reproducibility of findings obtained in psychological research, emphasizing the need for tests on the robustness of study findings (Open Science Collaboration 2015).”
Fischhoff, B. (2024) · Journal of Risk and Uncertainty · doi:10.1007/s11166-024-09437-3 · Cited by 3
“Psychology’s slowly growing reproducibility crisis was crystalized by a landmark study finding that results failed to replicate for roughly 2/3 of 50 published results in social psychology and 1/3 of 50 in cognitive psychology (Open Science Collaboration, 2015).”
Langfus, J. A., Youngstrom, E. A., Daniel, D., Busner, J., & Findling, R. L. (2024) · Journal of Child and Adolescent Psychopharmacology · doi:10.1089/cap.2024.0078 · Cited by 3
Recovering the Anchoring of Economic Valuations
Guo, L. (2021) · American Economic Journal Microeconomics · doi:10.1257/mic.20200320 · Cited by 5
Foreign funding of U.S. higher education relates to sanctioning of scholars and antisemitism
Bass, M., Ross, A. R., Wolfson, B., Finkelstein, J., Yanovsky, S., Finkelstein, D., Stevens, S. T., Honeycutt, N., Paresky, P., Feinberg, A. K., et al. (2024) · Frontiers in Social Psychology · doi:10.3389/frsps.2024.1408913 · Cited by 2
Acting for Reasons: In defence of Common-sense Psychology
Borg, E. (2024) · doi:10.1093/9780198929031.001.0001 · Cited by 4
Cimino, S. R., Crozier, O., Lizotte, D. J., Shabbir, A., Stoikos, J., & Wolfe, D. L. (2024) · Frontiers in Rehabilitation Sciences · doi:10.3389/fresc.2024.1336549 · Cited by 2
“Involving students from various disciplines not only provided them with an opportunity to put their theoretical knowledge into practice, but provided our team invaluable perspectives on the development of the different aspects of the program.”
Verberne, S. (2024) · ACM eBooks · doi:10.1145/3674127.3674141 · Cited by 5
Ferro, N., & Maistro, M. (2024) · ACM eBooks · doi:10.1145/3674127.3674132 · Cited by 5
Privacy in Information Retrieval
Soboroff, I. M. (2024) · ACM eBooks · doi:10.1145/3674127.3674139 · Cited by 2
Preregistration and predictivism
Choi, H. H. (2024) · Synthese · doi:10.1007/s11229-024-04827-1 · Cited by 6
Gilbert, E. A., Guinn, A. D., & Reppucci, N. D. (2024) · Frontiers in Social Psychology · doi:10.3389/frsps.2024.1456591 · Cited by 2
“The data for these experiments was collected at a time (2014– 2015) when the U.S. media was covering a series of cases about police officers harming Black men and juveniles (e.g., Michael Brown in Ferguson, Missouri; Eric Garner in New York City).”
InfluentialVahey, N. A., Nicholson, E., & Barnes‐Holmes, Y. (2025) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/j.jbtep.2024.102016 · Cited by 2
Introduction to “autonomous human–machine teams”
Lawless, W. F., Mittu, R., Sofge, D., & Fouad, H. Y. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-29246-0.00003-1 · Cited by 2
McNamee, B., Forscher, P. S., & Lenczner, M. (2024) · New Directions for Evaluation · doi:10.1002/ev.20624 · Cited by 4
Ferr, H. (2025) · PLOS mental health. · doi:10.1371/journal.pmen.0000242 · Cited by 4
“Studies show that only a fraction of psychological findings replicate successfully, raising questions about the reliability of the field [12].”
That’s replicable: building cumulative knowledge in the face of fads, obsessions and malpractices
Gupta, V. K., Mortal, S. C., & Skiadopoulou, A. (2025) · Organization Management Journal · doi:10.1108/omj-11-2024-2382 · Cited by 3
Bate, S. T., Stanford, S. C., & Page, L. (2025) · Journal of Psychopharmacology · doi:10.1177/02698811251319453 · Cited by 2
“Within laboratory-based scientific research there has been a growing awareness of poor reproducibility of experimental findings (Errington et al., 2021; Macleod and Mohan, 2019; Science Collaboration, 2015).”
The Overlooking of Subtractive Changes: Replication and Extension to Stronger Cues and Social Norms
Fillon, A. A., Girandola, F., Bonnardel, N., & Souchet, L. (2025) · The Journal of Creative Behavior · doi:10.1002/jocb.1535 · Cited by 5
“Given the growing recognition of the importance of reproducibility and replicability in psychological science (e.g., Isager et al., 2020; Open Science Collaboration, 2015; van ‘t Veer & Giner-Sorolla, 2016; Zwaan et al., 2018), we revisited the overlooking of subtractive changes by embarking on a…”
Lindsay, R. M. (2025) · Advances in management accounting · doi:10.1108/s1474-787120250000037001 · Cited by 9
The Reproducibility Crisis and Kinesiology: Confronting the Human & Statistical Limitations
Twietmeyer, G. (2025) · International Journal of Kinesiology in Higher Education · doi:10.1080/24711616.2025.2468469 · Cited by 2
“When scholars from disciplines outside of kinesiology have looked carefully at their own research literature, what they have seen has indicated a pressing need for reform (Begely & Ellis, 2012; Chang & Li, 2015; Open Science Collaboration, 2015).”
Resch, M., Nagelmann, E., & Bellhäuser, H. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1465512 · Cited by 3
“The presumptions of the SOS theory as a foundation of EA-SI were not taken for granted but were statistically investigated to counter the replication crisis (Open Science Collaboration, 2015).”
The missing link: Theory in social psychology as key contribution to transdisciplinary research
Ellemers, N. (2025) · European Review of Social Psychology · doi:10.1080/10463283.2025.2489867 · Cited by 3
Dombrowski, S. C. (2025) · Journal of School Psychology · doi:10.1016/j.jsp.2025.101462 · Cited by 2
Duong, T. M. P., Li, D., Nguyen, M., & Vuong, Q. (2025) · Sustainable Water Resources Management · doi:10.1007/s40899-025-01227-2 · Cited by 5
“The short-ages of frequentist paradigm has been suggested to contribute to the ongoing reproducibility crisis in socio-psychological research (Open Science Collaboration 2015; Camerer et al. 2018), largely due to the instability of p -values across different samples (Halsey et al. 2015 ).”
The Role of Faculty in Cultivating Thriving Campus STEM Ecosystems
Killpack, T. L., Sandoval, C. L., & Dewsbury, B. M. (2025) · CBE—Life Sciences Education · doi:10.1187/cbe.24-02-0041 · Cited by 3
“…by funding sources and funding scarcity (Als-Nielsen et al. , 2003; Harding, 2004; Alberts et al., 2014; Belluz et al. , 2016), in issues with error or reproducibility in studies (Ioannidis, 2005; Open Science Collaboration, 2015; Nichols et al. , 2021), and even in scientific fraud (Byrne, 2019).”
Quantifying the Alignment of a Data Analysis Between Analyst and Audience
McGowan, L. D., Peng, R. D., & Hicks, S. C. (2025) · Journal of Data Science · doi:10.6339/25-jds1189 · Cited by 3
Salet, X., Gelissen, J., Moors, G., & Wicherts, J. M. (2025) · Royal Society Open Science · doi:10.1098/rsos.242001 · Cited by 4
Richardson, G. B., Bates, D., McLaughlin, L. L., Mcgee, N. D., Tse, W. W.-Y., & Lai, M. H. C. (2025) · Human Nature · doi:10.1007/s12110-025-09497-7 · Cited by 4
“Our findings suggest omitted cross-loading bias joins the likes of p -hacking and HARKing (Kerr, 1998; Open Science Collaboration, 2015; Smaldino & McElreath, 2016); lack of formal theorizing (Muthukrishna & Henrich, 2019; Smaldino, 2020); overrepresentation of samples from undergraduate and WEIRD populations (Pollet…”
Hut, R., & Hall, C. (2025) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2024.0296 · Cited by 2
“More concerning is that conclusions may not be reproducible—whether due to false positives or inadvertently building on research with subtle flaws not caught during peer review (e.g. [8]).”
Robust inference and errors in studies of wildlife control
Treves, A., & Khorozyan, I. (2025) · Scientific Reports · doi:10.1038/s41598-025-18497-7 · Cited by 3
“With normally distributed response variables like L and change in L, Pearson’s r is unbiased, normal (Anderson-Darling test A = 0.78, p = 0.05 and A = 0.37, p =0.38, respectively).”
InfluentialHow to Improve Theory Specification in Psychology: A Review and Five Policy Suggestions
Glöckner, A., & Fiedler, S. (2025) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000599 · Cited by 3
The epistemic responsibilities in medicine of beneficence and respect for patient autonomy
Schwab, A. P. (2017) · Ethics Medicine and Public Health · doi:10.1016/j.jemep.2017.04.015 · Cited by 2
Replicability, Reproducibility, and Agent-based Simulation of Interventions.
Hum, R. S., & Kleinberg, S. (2017) · PubMed · Cited by 5
Teaching qualitative psychology at undergraduate level: Challenges and responses
Etheridge, L., McDermott, H. J., Sullivan, C., et al. (2017) · QMiP Bulletin · doi:10.53841/bpsqmip.2017.1.24.56 · Cited by 2
An argument for how (and why) to incentivise replication
Howe, P. D. L., & Perfors, A. (2018) · Behavioral and Brain Sciences · doi:10.1017/s0140525x18000705 · Cited by 3
Malgaroli, M. (2018) · Columbia Academic Commons (Columbia University) · doi:10.7916/d87m1rfq · Cited by 1
Development of Number Understanding: Different Theoretical Perspectives
Ansari, D. (2019) · Springer eBooks · doi:10.1007/978-3-319-97148-3_7 · Cited by 2
The Structure of Scientific Fraud: The Relationship Between Paradigms and Misconduct
Trubody, B. (2019) · Synthese Library/Synthese library · doi:10.1007/978-3-030-23769-1_5 · Cited by 2
StatBreak : Identifying “Lucky” Data Points Through Genetic Algorithms
Rosenbusch, H., Hilbert, L. P., Evans, A. M., & Zeelenberg, M. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920917950 · Cited by 2
Safra, L., Baumard, N., Wyart, V., & Chevallier, C. (2020) · PLoS ONE · doi:10.1371/journal.pone.0230011 · Cited by 2
The Application of Comparative Psychology to Parapsychology: A Recommendation
Somers, A., Byrd, M. D., Curliss, B., Lay, V., Jones, I. T., & Abramson, C. I. (2020) · Natural Science · doi:10.4236/ns.2020.126029 · Cited by 2
A platform for reproducibility
Casarotto, P. C., & Brembs, B. (2020) · Journal for Reproducibility in Neuroscience · doi:10.31885/jrn.1.2020.303 · Cited by 2
Sakstrup, C., & Hansen, K. J. (2020) · International Journal of Public Opinion Research · doi:10.1093/ijpor/edaa032 · Cited by 2
Philosophical or empirical incommensurability of frequentist versus Bayesian thinking
Trafimow, D. (2021) · Econometrics · doi:10.15611/eada.2021.1.02 · Cited by 2
Popova, K. (2021) · Science Technology & Human Values · doi:10.1177/01622439211032392 · Cited by 3
Wdrożenia wiedzy psychologicznej – pięć tez i przykład
Cieciuch, J. (2021) · Przegląd Psychologiczny · doi:10.31648/pp.7318 · Cited by 2
Faye, C. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108290876.013 · Cited by 2
Replicability and Meta-Analysis
Schauer, J. M. (2022) · Springer eBooks · doi:10.1007/978-3-031-04968-2_14 · Cited by 3
When should researchers cite study differences in response to a failure to replicate?
Colaço, D., Bickle, J., & Walters, B. J. (2022) · Biology & Philosophy · doi:10.1007/s10539-022-09873-y · Cited by 2
Counterfactuals with Experimental and Quasi-Experimental Variation
Battistin, E., & Bertoni, M. (2023) · Texts in quantitative political analysis · doi:10.1007/978-3-031-12982-7_3 · Cited by 3
Perceived timing of postural instability onset
McIlroy, R. E., & Barnett‐Cowan, M. (2023) · Gait & Posture · doi:10.1016/j.gaitpost.2023.07.003 · Cited by 2
Bridging consensus and controversy in fear conditioning research via meta-analysis
Garofalo, S. (2023) · Peer community in neuroscience. · doi:10.24072/pci.neuro.100144 · Cited by 1
Statistical Learning as a Regression Problem
Berk, R. A. (2016) · Springer texts in statistics · doi:10.1007/978-3-319-44048-4_1 · Cited by 2
Lats, T. V., Kim, P. Y., & Diekema, D. A. (2016) · Psi Chi Journal of Psychological Research · doi:10.24839/2164-8204.jn21.4.252 · Cited by 3
A Functional MRI Index of Spatial Context Effects in Vision
Feigenson, K. A., Hanson, C., Papathomas, T. V., & Silverstein, S. M. (2015) · Psychology · doi:10.4236/psych.2015.616211 · Cited by 2
Picking Up the Gauntlet: Richard Thaler's Defence of Behavioural Economics
Ormerod, P. (2016) · Economic Affairs · doi:10.1111/ecaf.12159 · Cited by 3
Applying Heuristics and Biases More Broadly and Cautiously
Schwab, A. P. (2016) · The American Journal of Bioethics · doi:10.1080/15265161.2016.1159764 · Cited by 2
Agent-based simulation of herding in financial markets
Liu, X., & Brooks, R. J. (2016) · Lancaster EPrints (Lancaster University) · Cited by 2
The Brief Pain Inventory (BPI) – Revisited and rejuvenated?
Werner, M. U. (2016) · Scandinavian Journal of Pain · doi:10.1016/j.sjpain.2016.04.003 · Cited by 2
On the road from WEIRD to STEM, psychology hits a bump.
Suedfeld, P. (2016) · Canadian Psychology/Psychologie canadienne · doi:10.1037/cap0000044 · Cited by 2
Davidsson, P. (2016) · International studies in entrepreneurship · doi:10.1007/978-3-319-26692-3_3 · Cited by 2
Are you coming on to me? Accuracy and bias in couples' perceptions of sexual advances
Dobson, K., Campbell, L., & Stanton, S. C. E. (2016) · Scholarship@Western (Western University) · Cited by 2
Empirische basis van conclusies
Eerland, A., & van den Bergh, H. (2016) · Tijdschrift voor taalbeheersing · doi:10.5117/tvt2016.2.eerl · Cited by 2
Police studies on PTSD in Spanish-speaking nations: A systematic review.
Verbeek, I. C., & van der Velden, P. G. (2016) · Traumatology An International Journal · doi:10.1037/trm0000084 · Cited by 4
Ayer, V., Wiljes, C., Cimiano, P., Pietsch, C., Vompras, J., Schirrwagen, J., & Jahn, N. (2017) · D-Lib Magazine · doi:10.1045/january2017-ayer · Cited by 6
“In particular, we describe the goals and objectives of the project as well as the underlying system architecture which relies on a DCVS system for storing data, and on continuous integration principles to foster data quality.”
MethodWahres und Wissenswertes in der Sportpsychologie
Seiler, R. (2016) · Zeitschrift für Sportpsychologie · doi:10.1026/1612-5010/a000166 · Cited by 2
Consume, Reproduce, Extend and Connect: Sustaining Our Research Lifecycle
Johnson, R. P. (2017) · Bulletin of the Association for Information Science and Technology · doi:10.1002/bul2.2017.1720430407 · Cited by 2
Magid, K. S., Sarkol, V., & Mesoudi, A. (2017) · Royal Society Open Science · doi:10.1098/rsos.161025 · Cited by 4
Metadata
- Authors
- Open Science Collaboration
- Corresponding author
- Open Science Collaboration
- Venue
- Science
- Published
- 2015-08-28
- Volume
- 349
- Issue
- 6251
- Pages
- aac4716-aac4716
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W1897139626
- Cited by
- 8,515
- References
- 39
- Citations by year
- 2026: 414 · 2025: 532 · 2024: 698 · 2023: 757 · 2022: 783 · 2021: 906 · 2020: 1001 · 2019: 989 · 2018: 938 · 2017: 800 · 2016: 607 · 2015: 79
- FWCI
- 736.90
- Type
- journal article
- Language
- en
- Open access
- Locations
- Science (publishedVersion)
- Research portal (Tilburg University) (publishedVersion, other-oa)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, other-oa)
- UvA-DARE (University of Amsterdam) (publishedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (publishedVersion)
- VU Research Portal (publishedVersion)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (acceptedVersion, other-oa)
- Repository of the Academy's Library (Library of the Hungarian Academy of Sciences) (acceptedVersion)
- London School of Economics and Political Science Theses Online (London School of Economics and Political Science) (acceptedVersion)
- Discovery Research Portal (University of Dundee) (submittedVersion, other-oa)
- Keele Research Repository (Keele University) (submittedVersion, cc-by)
- London School of Economics and Political Science Research Online (London School of Economics and Political Science) (submittedVersion)
- Pure (Coventry University) (submittedVersion, other-oa)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- Radboud Repository (Radboud University) (submittedVersion)
- Winchester Research Repository (submittedVersion)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion)
- ANU Open Research (Australian National University) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- TU/e Research Portal (submittedVersion)
- BOA (University of Milano-Bicocca) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- Max Planck Digital Library (submittedVersion)
- Coventry University Open Collections (Coventry university) (submittedVersion)
- The HKU Scholars Hub (University of Hong Kong) (submittedVersion)
- Research Portal (King's College London) (submittedVersion)
- publish.UP (University of Potsdam) (submittedVersion)
- Lirias (KU Leuven) (submittedVersion)
- Faculty of Philosophy (University of Belgrade) (submittedVersion)
- Biblos-e Archivo (Universidad Autónoma de Madrid) (submittedVersion, public-domain)
- Bristol Research (University of Bristol) (submittedVersion)
- UWE Research Repository (UWE Bristol) (submittedVersion, cc-by)
- Padua Research Archive (University of Padova) (submittedVersion, cc-by-nc-nd)
- Zurich Open Repository and Archive (University of Zurich) (submittedVersion)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, other-oa)
- The Open Collections (Coventry University) (submittedVersion)
- Research Publications (Maastricht University) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Funders
- Laura and John Arnold Foundation
- Center for Open Science
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Keywords
- reproducibility, replication, statistical significance, confidence intervals, publication bias, quality of evidence, psychology research
- MeSH terms
- Periodicals as Topic, Psychology, Publishing, Research, Research Design, Reproducibility of Results, Confidence Intervals, Publication Bias, Behavioral Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.