Estimating the reproducibility of psychological science
Open accessPublished version of a preprint
The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index; 20 of the 57 sources its claims cite were checked for support.
Abstract
Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.
Overview
CoChat Summary
TL;DR
A large collaborative replication of 100 published psychology studies found that replication effects were about half the magnitude of original effects, and only 36% of replications had significant results versus 97% of originals. Replication success was more consistently related to the strength of the original evidence than to characteristics of the replication teams or implementation.
Key findings
- Original effects (M = .396, SD = .193) were reliably larger than replication effects (M = .198, SD = .255); 82 of 97 study pairs (82.8%) showed a stronger original effect.
- 97% of original studies had significant results (p < .05), but only 36% of replications did; 47% of original effect sizes fell within the 95% CI of the replication effect size, 38% were subjectively rated as replicated, and 70% of meta-analytic combinations were significant if no bias in original results was assumed.
- Reproducibility was stronger for cognitive than social psychology: 20 of 38 (53%) cognitive effects replicated by p < .05 versus 14 of 53 (26%) social effects; main/simple effects replicated more often (23/50, 46%) than interaction effects (8/37, 22%).
- Replication success correlated with original p-value, original effect size, and replication power, but not consistently with perceived importance, team expertise, or self-assessed replication quality.
Methods
- The Open Science Collaboration conducted 100 replications of studies published in 2008 in three psychology journals: Psychological Science, Journal of Personality and Social Psychology, and Journal of Experimental Psychology: Learning, Memory, and Cognition; 270 contributing authors completed the replications.
- Replication teams followed a structured, pre-registered protocol: selecting a key effect, contacting original authors for materials, writing and registering an analysis plan, obtaining review, conducting high-powered replications (average 92% power to detect the original effect size), and archiving data and materials.
- Analyses converted effects to correlation coefficients where possible and evaluated reproducibility with five indicators: significance testing, comparison of original vs. replication effect sizes, whether the original effect size fell within the replication 95% CI, meta-analytic combination, and subjective “Did it replicate?” ratings.
Limitations
- The paper states that it is unknown how far findings extend to other psychology subdisciplines or other disciplines; not all eligible studies were replicated, resource-intensive studies were less likely to be included, and the impact of selection bias is unknown.
- The paper notes no single indicator sufficiently describes replication success, and meta-analytic estimates assume no publication bias in original results; if original effects are inflated by publication, selection, or reporting biases, replication estimates may better reflect true effects.
- The text makes evident that the sample is limited to 100 completed replications from 2008 articles in three journals, and that unanticipated differences in sample, setting, or procedure could alter observed effect magnitudes despite efforts to maintain fidelity.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
8 Valid · 0 Not valid · 10 Not checked · 144 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In cell biology, two industrial laboratories reported success replicating the original results of landmark findings in only 11% and 25% of the attempted cases ( 10, 11 ).”
What the sources say
- Partly supportedcites [10] 10. C. G. Begley, L. M. Ellis, Raise standards for preclinical cancer research. Nature 483 , 531533 (2012). sourceExperimental — accuracy not yet measured
In CONFIRMING RESEARCH FINDINGS (p. 2), the paper reports that Amgen confirmed findings in 6 of 53 landmark papers (11%) and that a Bayer team validated about 25% of published preclinical studies sufficiently for projects to continue. The 25% figure is not specifically for landmark findings, and the Bayer studies were not all cell biology or cancer research; the paper says 70% were published cancer research. (LLM, conf=0.87)
- Not checked — no full text of the sourcecites [11] 11. F. Prinz, T. Schlange, K. Asadullah, Believe it or not: How much can we rely on published data on potential drug targets? Nat. Rev. Drug Disc. 10 , 712713 (2011 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“There is good evidence that low power of research designs combined with publication bias favoring positive results over negative results produces a literature with upwardly biased effect sizes ( 14, 16, 30, 31 ).”
What the sources say
- Supportedcites [14] 14. K. S. Button, et al ., Power failure: Why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14 , 112 (2013). source
The ‘Low power in the presence of other biases’ section says publication bias disproportionately affects underpowered studies and that small negative studies are more readily rejected. The ‘Results’ section says small studies are particularly susceptible to inflated effect-size estimates and publication bias. Page numbers were not supplied in our copy. (LLM, conf=0.92)
- Not checked — no full text of the sourcecites [16] 16. A. G. Greenwald, Consequences of prejudice against the null hypothesis. Psychol. Bull. 82 , 1-20 (1975). source
- Not checked — no full text of the sourcecites [30] 30. J. Cohen, The statistical power of abnormalsocial psychological research: A review. J. Abnorm. Soc. Psychol. 65 , 145153 (1962). source
- Not matchedcites [31] 31. T. D. Sterling, Publication decisions and their possible effects on inferences drawn from tests of significanceor vice versa. J. Am. Stat. Assoc. 54 , 3034 (1959).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nonetheless, none of these efforts guarantee that unanticipated factors in the sample, setting, or procedure altered the observed effect magnitudes ( 32 ).”
What the sources say
- Supportedcites [32] 32. R. Klein, et al ., Investigating variation in replicability: A 'many labs' replication project. Soc. Psychol. 45 , 142152 (2014). source
The Overview of the Present Research (p. 5) says sample, setting, and procedural variations can influence effects, but that effect magnitudes may also vary no more than expected random error. The Discussion (p. 18) cautions against assuming differences between samples and limits its conclusions to the studies and settings examined. Together, these support the claim that the efforts do not guarantee that unanticipated factors altered the observed magnitudes. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Ninetyseven percent of original studies had significant results ( p < .05).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“47% of original effect sizes were in the 95% confidence interval of the replication effect size;”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
60 more claims
Create a free account to see all 162 claims and every source verdict.
- Probably not a substantial claim. 65
- It may state two separate claims as one. 49
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“First because, once again, such a low level of reproducibility was not ubiquitous but varied depending on subfield, methodology, and expertise of the authors conducting the replication (23 -25).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [23]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“a large-scale replication of experimental psychology studies found that while 97% of results in the initial studies were statistically significant, only 36% of results in the replication were significant, even when larger samples were used (Open Science Collaboration, 2015)”
From Acceptance and Commitment Therapy Processes and Mediation: Challenges and How to Address Them (2023) · cites this paper as Open Science Collaboration, 2015
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, sample sizes are often underpowered (4-6), at times leading to conclusions that do not hold up with larger-sized replications or metaanalyses (7).”
From Generic language in scientific communication (2019) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In the field of psychology, less than half of experimental and correlational studies are reportedly reproducible (11).”
From The Prevalence of Inappropriate Image Duplication in Biomedical Research Publications (2016) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“reanalyses of the data suggest that reproducibility in psychological science might be higher than originally claimed (23, 26, 27).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [23]
Experimental — accuracy not yet measured
25 more claims about this paper
Create a free account to see every claim other papers make about this one.
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Consistentχ²(128) = 155.83reported p = .048 · recomputed p = .048
“χ 2 (128) = 155.83, p = .048”
- Consistentχ²(1) = 59.1reported p < .001 · recomputed p < .001
“χ 2 (1) = 59.1, p < .001”
- Consistentt(98) = -8.22reported p < .001 · recomputed p < .001
“t (98) = 8.22, p < .001”
- Consistentt(96) = 9.33reported p < .001 · recomputed p < .001
“t (96) = 9.33, p < .001”
- ConsistentQ(73) = 310.45reported p < .001 · recomputed p < .001
“Q (73) = 310.45, p < .001”
17 more recomputed tests
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“Of the 97 studies for which an effect size in both the original and replication study could be calculated ( 28 ), 82 showed a stronger effect size in the original study (82.8%; p < .001, binomial test; see right panel of Figure 1).”
Reported 82.8 · recomputed 84.536082
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Sampling Frame and Study Selection · p. 23
“an initial pool of the first 30 articles from each journal in chronological order.”
§ Sampling Frame and Selection Biases · p. 32
“Initially, a total of 60 articles were made available from the sampling frame”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Evaluating Reproducibility · p. 8
“p 's = .0508, .0514, .0516, .0567”
§ Preliminary analyses · p. 39
“(.0503, .0509, .0514, and .0516)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Abstract · p. 2
“original effects ( M = .396, SD = .193)”
§ Discussion · p. 12
“original effects ( M = .396, SD = .196)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Evaluating Reproducibility · p. 8
“a significant reduction (McNemar test, ᵯ€ 2 (1) = 59.5, p < .001).”
§ Evaluating replication effect against null hypothesis of no effect . · p. 40
“a significant change (McNemar test, χ 2 (1) = 59.1, p < .001).”
1 more finding
Create a free account to see every finding of this check.
Citations
Cites
Conducting Meta-Analyses in R with the metafor Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464 · cited 1×
“We conducted fixed-effect meta-analyses using the R packagemetafor (27) on Fisher-transformed correlations for all study-pairs in subset MA and on study-pairs with the odds ratio as the dependent variable.”
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Lakens, D. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00863 · Cited by 10,303
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“Concernsaboutreproducibilityarewidespread acrossdisciplines( 9 – 21 ).”
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 1×
“Low-powerresearchdesignscombinedwithpublication bias favoring positive results together produce a literature with upwardly biased effect sizes ( 14 , 16 , 33 , 34 ).”
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“make it likely that more than half of research results are false and therefore irreproducible (9).”
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
Studies in the Logic of Explanation
Hempel, C. G., & Oppenheim, P. (1948) · Philosophy of Science · doi:10.1086/286983 · Cited by 3,174
Research works printed but not linked: 2
- W. C. Salmon in Introduction to the Philosophy of Science M. H. Salmon Ed. (Hackett Publishing Company Indianapolis 1999) pp. 7–41.
- Open Science Collaboration in Implementing Reproducible Computational Research (A Volume in The R Series) V. Stodden F. Leisch R. Peng Eds. (Taylor & Francis New York 2014) pp. 299–323.
Cited by
Items 8,401–8,500 of 8,625
Wolf, B., & Klager, C. (2026) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2026.2638432
“A major concern is whether interventions shown to be effective once can produce similar results in other contexts, with different populations, when replicated, or when scaling up (Chhin et al., 2018; Open Science Collaboration, 2015; Stuart et al., 2017; Tipton & Olsen, 2018).”
Data Availability Statement: A Tool for Enhancing Research Trust
Тихонова, Е. В., & Mekeko, N. (2024) · Journal of Employment and Career · doi:10.56414/jeac.2024.2.72
The Cost of Prediction: How Computational Methods Compromise Reproducibility
Lenhard, J., Stephan, S., & Hasse, H. (2026) · Boston studies in the philosophy of science · doi:10.1007/978-3-032-11706-9_12 · Cited by 1
Peng, Y., Wang, X., Zhou, Z. E., & Zhang, W. (2026) · Occupational Health Science · doi:10.1007/s41542-026-00261-6
“The convergence of results across these two samples with two heterogeneous samples can help improve the generalizability of results across different cultural backgrounds and alleviates the concern about the lack of reproducibility (Makel et al., 2012; Open Science Collaboration, 2015).”
Weinstein, N., Ghai, S., Moin, T., Legate, N., Matos, L., & Przybylski, A. K. (2026) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459261423052 · Cited by 1
“At present, open-science practices are largely predicated on the values of having accurate a priori prediction and careful control, both of which drive a value toward transparency (Open Science Collaboration, 2015).”
aggreCAT: An R Package for Mathematically Aggregating Expert JudgmentsPreprint
Gray, C. T., Wilkinson, D. P., Groenewegen, R., Willcox, A., O’Dea, R. E., & Gould, E. (2026) · MetArXiv (OSF Preprints)
Räsänen, O. J. (2026) · Advances in child development and behavior · doi:10.1016/bs.acdb.2026.03.001 · Cited by 2
“For lexical evaluation, the model is asked to provide acceptability scores for isolated words that occur in CHILDES transcripts (MacWhinney & Snow, 1985), thereby approximating the vocabulary actually heard by real infants.”
Brown, M., Sacco, D. F., Raible, C., Wallace, K. J., & Lemacks, J. L. (2026) · Accountability in Research · doi:10.1080/08989621.2026.2659673
“For example, in one large-scale replication effort involving 100 classic psychology studies, researchers were only able to replicate the originally reported findings 33% of the time (Open Science Collaboration 2015).”
Açık Bilim ve Yeniden Üretilebilirlik: Çok Kriterli Karar Alma Vaka Çalışmasından Perspektifler
Yayla, K., & Soyuer, H. (2025) · Türk Kütüphaneciliği · doi:10.24146/tk.1694565
Neural activation by vocal threat requires attention
Witteman, J., Axioti, S., & van Tol, A. (2026) · Cortex · doi:10.1016/j.cortex.2026.04.009
Balkin, R. S. (2026) · Measurement and Evaluation in Counseling and Development · doi:10.1080/07481756.2026.2658009
Gomziakova, N. Y., Kovalev, P. A., Kovalkova, L. A., et al. (2026) · Psychological Science and Education · doi:10.17759/pse.2026310218
Anthropological and Psychological Approaches to Culture, Identity, and Essentialism
Moya, C., & Wagner, W. (2026) · Theory and history in the human and social sciences · doi:10.1007/978-3-032-11918-6_1
How to become a Bayesian in eight easy steps: An annotated reading listPreprint
Baribault, B., Vandekerckhove, J., Dablander, F., Etz, A., Gronau, Q. F., & Edelsbrunner, P. A. (2018) · OSF Preprints (OSF Preprints)
Psychology’s RenaissancePreprint
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2022) · OSF Preprints (OSF Preprints)
Nieuwenhuys, A., Furley, P., & Bijlstra, G. (2023) · OSF Preprints (OSF Preprints)
Predicting citation in financial accounting and auditing research: The role of replication
Salterio, S. E., Luo, Y., & Adamson, C. (2026) · Accounting Open · doi:10.1016/j.accop.2026.100007
Doing Psycholinguistics in Applied Linguistics: Foundations, Methods, and Future Directions
Godfroid, A. (2026) · TESOL Quarterly · doi:10.1002/tesq.70137
Geukes), K. U. (., Leckelt, M., Nestler, S., Humberg, S., Schmukle, S., Schröder–Abé, M., Mota, S., & Back, M. D. (2024) · OSF Preprints (OSF Preprints)
Verschuère, B., Kleinberg, B., & Warmelink, L. (2023) · OSF Preprints (OSF Preprints)
Flexible Yet Fair: Blinding Analyses in Experimental PsychologyPreprint
Wagenmakers, E., Sarafoglou, A., & Dutilh, G. (2019) · OSF Preprints (OSF Preprints)
Estimating the Evidential Value of Significant Results in Psychological SciencePreprint
Szaszi, B., Pálfi, B., & Aczel, B. (2022) · OSF Preprints (OSF Preprints)
Open Science is Robust SciencePreprint
Zickar, M. J., Grubbs, J. B., & McAbee, S. T. (2017) · OSF Preprints (OSF Preprints)
Preregistered replication of the auditory deviant effect: A robust benchmark findingPreprint
Buchner, A., Bell, R., Mieth, L., & Röer, J. P. (2019) · OSF Preprints (OSF Preprints)
Using the CNI Model to Investigate Individual Differences in Moral Dilemma JudgmentsPreprint
Gawronski, B., Deutsch, R., & Körner, A. (2020) · OSF Preprints (OSF Preprints)
Nguyen, M.-H., Tran, T. M. A., Sari, N. P. W. P., Trần, T. B., & Vuong, Q. (2026) · Urban forestry & urban greening · doi:10.1016/j.ufug.2026.129485
das Neves, L. O., Morales, M. L. P., Soroldoni, S., Nagata, R. M., & Pinho, G. L. L. (2026) · Marine Environmental Research · doi:10.1016/j.marenvres.2026.108099
Tuned Out, Not Drawn In: Blunted Neural Sensitivity to Social Reward in Narcissism
Zhou, Z., Angus, D. J., Sedikides, C., & Kelley, N. J. (2026) · European Journal of Personality · doi:10.1177/08902070261449063
“Only 25% of fi ndings in social psychology were found to be reproducible in a large-scale effort (Open Science Collaboration, 2015), with accompanying updated effect size expectations and sample size recommendations (Lovakov & Agadullina, 2021; Sch¨onbrodt & Perugini, 2013).”
Creative Cognition: A Collaborative Research Agenda
Ross, W., Agnoli, S., Baas, M., Danek, A. H., Lebuda, I., Ackerman, R., Ammalainen, A. V., Benedek, M., Bai, H., Ball, L. J., et al. (2026) · The Journal of Creative Behavior · doi:10.1002/jocb.70097 · Cited by 1
The Effect of Animacy on Structural Priming: A Replication of Bock, Loebell and Morey (1992)Preprint
Zhang, C., Hartsuiker, R. J., Slim, M. S., Chen, X., & Muylle, M. (2023) · OSF Preprints (OSF Preprints)
Ten easy steps to conducting transparent, reproducible meta-analyses for infant researchersPreprint
Tsuji, S., Gasparini, L., & Bergmann, C. (2024) · OSF Preprints (OSF Preprints)
The benefits of preregistration for hypothesis-driven bilingualism researchPreprint
Vasishth, S., Mertzen, D., & Lago, S. (2022) · OSF Preprints (OSF Preprints)
Pre-registration for confirmatory and exploratory research
Dunleavy, P., & Monteath, T. (2026) · LSE Press eBooks · doi:10.31389/lsepress.dos.d
Silva, S. F. D. S., & Bonini, S. (2026) · Humans · doi:10.3390/humans6020017
“…what is contested is the size, reliability, and generalizability of purported direct cue to behavior effects (Bargh et al., 1996), given prominent challenges, replication failures, and broader reproducibility concerns in psychology (Doyen et al., 2012; Open Science Collaboration, 2015).”
Constraint Generative Theory: Typed Constraint Effects and Scientific AvailabilityPreprint
Takahashi, K. (2026) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.20199440
Jin, Y., Guo, K., & Rothenhaüsler, D. (2026) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.1093/jrsssc/qlag019 · Cited by 10
Wolff, F., Schumacher, J., Etzel, J. M., Junge, F., Hörsch, H., & Møller, J. (2026) · Social Psychology of Education · doi:10.1007/s11218-026-10209-x · Cited by 1
Tragoudas, G., Ünal, Z. E., Geary, D. C., & Menon, V. (2026) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.70168
“The Open Science Collaboration’s attempt to replicate 100 psychology studies found that only 39% achieved significant results, with effect sizes approximately half of those originally reported (Collaboration, 2015).”
Exploring COVID-19 Research Credibility Among Spanish Scientists.Preprint
Angulo-Brunet, A., Corradi, G., García‐Garzón, E., Barrada, J. R., & Lecuona, O. Z. (2022) · OSF Preprints (OSF Preprints)
Multiverse robustness tests in macroeconomic calibration models
Kvarven, A., Lilletvedt, J. S., Strømland, E., & Johannesson, M. (2026) · Research in Economics · doi:10.1016/j.rie.2026.101135
Kozula, M., DeVito, N., Onghena, P., Colombo, C., Dudda, L. A., Teno, P. M., & den Eynden, V. V. (2026) · PLoS ONE · doi:10.1371/journal.pone.0348512
Reproducibility, Transparency, and Open Science in AI-Driven Statistical Education
Cruz, V. P. J. D., & Musni-tagaytay, F. M. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-6259-5.ch011
Forty years of changes in scientific publishing: from conflict of interest to generative AI
Bauchner, H. (2026) · Health Affairs Scholar · doi:10.1093/haschl/qxag125
“13 These observations crystallized concerns about both reproducibility, defined by the NIH as “the ability of independent researchers to test a hypothesis through multiple methods and consistently achieve results that confirm or refute it, ensuring findings are generalizable and robust across…”
Following the money: an econometric investigation into health valuations
Ásgeirsdóttir, T. L., & Kárason, V. A. (2026) · Scandinavian Journal of Economics · doi:10.1111/sjoe.70026
Costa, T. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1799078
“The replication crisis has made this tension explicit: when study power is low, biases are present, and the base rate of true effects is modest, many published findings will overstate what the data can rationally support (Ioannidis, 2005; Open Science Collaboration, 2015).”
Procedural Rigor and Reproducibility in NMR Metabolomics: Community Practices and Challenges
Cheng, L. L., Powers, R., Brua, R. B., et al. (2026) · Critical Reviews in Analytical Chemistry · doi:10.1080/10408347.2026.2675511
Yang, Y., Liu, L., Gao, L., & Zhang, Y. (2026) · Current Psychology · doi:10.1007/s12144-026-09524-5
A response to Pek et al.’s commentary on Z-curve: clarifying the assumptions of selection models
Schimmack, U., & Soto, M. (2026) · Cognition & Emotion · doi:10.1080/02699931.2026.2678998 · Cited by 2
The Social Sciences Reconsidered
Söilen, K. S. (2026) · Springer eBooks · doi:10.1007/978-3-032-23069-0_6
GAZARIAN, P. K., Crawford, S. L., Quill, C., & Rossetti, V. (2026) · Worldviews on Evidence-Based Nursing · doi:10.1111/wvn.70149
Cognition, Emotion, and Sociality
Söilen, K. S. (2026) · Springer eBooks · doi:10.1007/978-3-032-23069-0_3
Humanism and Wisdom in the AI Era
Söilen, K. S. (2026) · Springer eBooks · doi:10.1007/978-3-032-23069-0_10
Study preregistration: an early example and analysis
Wiseman, R. J., & Watt, C. A. (2026) · PeerJ · doi:10.7717/peerj.21360 · Cited by 1
“Several strands of evidence supported this argument, including: surveys showing that experimenters frequently engage in QRPs ( Agnoli et al., 2017 ; John, Loewenstein & Prelec, 2012 ; Makel et al., 2021 ); papers containing an implausibly large number of significant results ( Fanelli, 2010 ); researchers struggling to…”
Modeling human social vision with cinematic stimuli: An integrative approach
Santavirta, S., Paranko, B., Seppälä, K., Hyönä, J., & Nummenmaa, L. (2026) · Journal of Vision · doi:10.1167/jov.26.5.9 · Cited by 3
“Due to serious concerns about the replicability of results in psychology (Open Science Collaboration, 2015), it is critical to conduct large-scale studies using cross-validated models and report replicable findings across independent samples.”
Scientific workflow in experimental economics
Dreber, A., & Toussaert, S. (2026) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2024.0606
Veltri, G. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-40622-5.00046-3
The Hypothesis Race Model for evaluation of research findingsPreprint
Kelly, R. E. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.05.28.728385
Harun, M. F., & Taib, M. M. (2026) · Europe’s Journal of Psychology · doi:10.5964/ejop.16341
“Replication studies conducted across diverse organizational and cultural contexts will be especially important for validating observed relationships and refining effect size estimates (Open Science Collaboration, 2015).”
Towards a Genre Repertoire Approach to Research Transparency
Koljonen, T., & Spencer, L. (2026) · Organization Studies · doi:10.1177/01708406261459081
Vuong, Q., Nguyen, P.-T., & Nguyen, M.-H. (2026) · Research Square · doi:10.21203/rs.3.rs-9456872/v1
Oo, C. Z., Eryılmaz, N., & Strietholt, R. (2026) · Studies In Educational Evaluation · doi:10.1016/j.stueduc.2026.101618
Parfenov, E. A., Baranova, E., & Duncan, N. W. (2026) · Royal Society Open Science · doi:10.1098/rsos.251532
Can an army of babies and dogs rescue psychology from its reproducibility crisis?
Fleming, N. (2026) · Nature · doi:10.1038/d41586-026-01727-x
Staller, M. S., & Körner, S. (2026) · Frontiers in Sociology · doi:10.3389/fsoc.2026.1758071 · Cited by 2
“…temporal vector can be observed particularly clearly: replication projects, preregistration initiatives and registered reports organize research processes along explicit time axes—before and after registration, before and after replication (Nosek et al., 2018; Open Science Collaboration, 2015).”
Lovett, J. M., Galloway, C., Bhanushali, P. D., Sarda, S., Bledsoe, A., Bourke, K. T., & Schonert-Reichl, K. A. (2026) · Frontiers in Social Psychology · doi:10.3389/frsps.2026.1804116 · Cited by 1
Howard, M. C., Henderson, J., & Simonson, P. D. (2026) · Supply Chain Analytics · doi:10.1016/j.sca.2026.100214 · Cited by 1
Sharma, R. K. (2026) · Research Square · doi:10.21203/rs.3.rs-10332758/v1
Huang, X. (2026) · Transactions in GIS · doi:10.1111/tgis.70285
Creating Business Value in Unpredictable Times Aiming for Antifragility
Botjes, E., Huygh, T., Bollen, L. H. H., et al. (2026) · doi:10.18690/um.fov.4.2026.73
Lee, C. J. (2026) · Social Epistemology · doi:10.1080/02691728.2026.2669578
“…that a study successfully replicated in cases where a replication had statistically significant results in the same direction as the original study (Open Science Collaboration 2015b), which suggests that the latter measure captures the epistemic basis for replication team member judgements.”
Class, B., Bourguet, D., & Guillemaud, T. (2026) · Insights the UKSG journal · doi:10.1629/uksg.744 · Cited by 2
1236 Research ethics in communication science: practical and aspirational landscapes
Lahman, M. K. E., Gittner, K. B., & Brown, C. (2026) · De Gruyter eBooks · doi:10.1515/9783111087139-006
1999 Establishing scale unidimensionality
Shen, L. (2026) · De Gruyter eBooks · doi:10.1515/9783111087139-009
Tillit og turbulens: Endringer i tillit til forskning i Norge 2019–2024
Wollebæk, D., Fladmoe, A., Saglie, J., Segaard, S. B., & Winsvold, M. (2026) · Norsk statsvitenskapelig tidsskrift · doi:10.18261/nst.42.2.1
Peter, B. (2026) · International Journal of Clinical and Experimental Hypnosis · doi:10.1080/00207144.2026.2669241
Better for self or better than others, the secret of relative position in behavior
Celse, J., & Chang, K. (2026) · Review of Behavioral Economics · doi:10.1108/rbe-12-2025-0126
Meier, M., Sep, M. S. C., Bastiaanssen, T. F. S., Clephas, E., Quaedflieg, C. W. E. M., Rajčáni, J., Richer, R., Szaszkó, B., Vogel, S., Voulgaropoulou, S. D., et al. (2026) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2026.108004
Statistical Methods for Assessing Non-Replicable, Outlying, and Influential Studies
Yang, Y., & Nakagawa, S. (2026) · Mathematics · doi:10.3390/math14193436
“…in the same direction as the original effect (e.g., a two-sided p-value < 0.05) in the same direction as the original effect (e.g., Jaljuli et al. (2023)), a criterion employed in most large-scale replication efforts (Collaboration (2015), Camerer et al. (2016), Errington, Mathur, et al. (2021)).”
InfluentialCheung, K., Gangaram, S., Ge, M. P. X., Sung, J., Wong, K., Howatson, L., Lai, A., Li, S., Tsang, B., & Gerlai, R. T. (2026) · Current Research in Behavioral Sciences · doi:10.1016/j.crbeha.2026.100216 · Cited by 1
Mascarenhas, M. A., & Zakzanis, K. K. (2026) · Archives of Clinical Neuropsychology · doi:10.1093/arclin/acag044
Fernández-García, M. P., Vallejo, G., & Livacic-Rojas, P. E. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1835500
Thorne, T., Milyavskaya, M., & Leduc‐Cummings, I. (2026) · European Journal of Personality · doi:10.1177/08902070261463054
Leadership in Your Pocket: Enhancing Leadership Self-Schemas via Mobile App Attribute Conditioning
Ashlock, L., Leung, A., Sy, T., & Panda, K. (2026) · Journal of Leadership & Organizational Studies · doi:10.1177/15480518261461824
“…work indicates that changes in attitudes and self-ef fi cacy often translate to modest behavioral effects (e.g., Conner & Norman, 2022), and that simple psychological interventions may not reliably replicate or generalize across contexts (Camerer et al., 2018; Open Science Collaboration, 2015).”
Limited evidence that reputation-based partner choice facilitates information sharing in humans
Jiménez, Á. V., Chang, L., Jensen, K., Lu, H. J., & Mesoudi, A. (2026) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2026.0876
Beck, E. D., Willroth, E. C., Delius, J. A. M., Bennett, D. A., Barnes, L. L., James, B. D., Lipton, R. B., Katz, M., Hassing, L. B., Huisman, M. A., et al. (2026) · Personality and Social Psychology Bulletin · doi:10.1177/01461672261451901 · Cited by 2
Creating common virtual ground: Protocols to democratize open VR research
Zelderen, A. P. A. V., Masters-Waage, T. C., Affinito, S. J., Bekbergenova, A., Cowan, K. L., Hartmann, T., Hausfeld, M. M., Jolink, A., Korbel, J. J., Lynn, T., et al. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2524991123
Post-landslide vegetation recovery: research gaps and future opportunities
Williams, E., Dowling, T., Burns, B. R., Brock, J. M. R., & Perry, G. L. W. (2026) · Landscape Ecology · doi:10.1007/s10980-026-02402-2
“The lack of documentation in the workflow processes and software used also mirrors the issues in reproducibility noted across scientific research (Baker 2016; Open Science Collaboration, 2015; K. K. Singh et al. 2024).”
Nudge Without Evidence: The Hidden Risks of Behavioural Policy
Pawinska, M. (2026) · Archiwum Filozofii Prawa i Filozofii Społecznej · doi:10.36280/afpifs.2026.2.92
Reliability and the Hidden Translational Gaps in Multidisciplinary Clinical Psychological Science
Germine, L. T. (2026) · Clinical Psychological Science · doi:10.1177/21677026251407591 · Cited by 1
“These culminated in an article from the Open Science Collaboration (2015) estimating that only about a third of results from psychological science might be reproducible.”
The sociologist and the sirens: iconography and inspiration of a metaphor thirty years later
Atkinson, P., & Cataldi, S. (2026) · SOCIOLOGIA E RICERCA SOCIALE · doi:10.3280/sr2025-138002
The Adoption of Open Science Practices in Communication Research: Taking Stock of the Field in 2025
Kruijt, J., Gao, C., Ke, X., Pribylova, S., Yang, W., & Bakker, B. N. (2026) · Media and Communication · doi:10.17645/mac.12178
“…that similar studies with new data do not always yield comparable findings (Tyner et al., 2026)—have been documented in multiple scientific fields, including psychology (Open Science Collaboration, 2015) and other areas of the social sciences (Camerer et al., 2018; Klein et al., 2014).”
Fairbairn, C. E., & Bosch, N. (2026) · Clinical Psychological Science · doi:10.1177/21677026251379441 · Cited by 2
“…the past several decades have witnessed a sharp rise in concern surrounding the prevalence of false-positive findings in psychological science (Open Science Collaboration, 2015) paired with awareness of the prevalence of research practices in the discipline that invite such outcomes (John et…”
Do emotional faces modulate pupillary pseudoneglect?
Xu, J., Ni, K., Burns, N. E., & McIntosh, R. D. (2026) · Laterality Asymmetries of Body Brain and Cognition · doi:10.1080/1357650x.2026.2694335 · Cited by 1
“…ECFT effect size from Voyer et al. (2012) may be inflated due to being derived from behavioural measures that use binary responding (where a small bias can lead to a categorical change in response), and possible publication bias within the literature (Open Science Collaboration, 2015; Smart, 1964).”
Sultanaamin, K., Balneaves, L. G., Kumar, S., Leach, M., Su, K., Veziari, Y., Wardle, J., Cramer, H., & Ng, J. Y. (2026) · Integrative Medicine Research · doi:10.1016/j.imr.2026.101385 · Cited by 1
Ioannidis, J. P. A. (2026) · European Journal of Clinical Investigation · doi:10.1111/eci.70240 · Cited by 1
Tostlebe, J. J. (2026) · Crime & Delinquency · doi:10.1177/00111287261460006
Dörner, D., Güss, C. D., & Lutsevich, A. (2026) · Springer eBooks · doi:10.1007/978-3-662-72921-2_1
Open Science for Missions and Observatories: Practical Choices to Shape Your Observing Adventure
Ringuette, R., Barnum, J., Bishop, R., et al. (2026) · F1000Research · doi:10.12688/f1000research.183314.1
Metadata
- Authors
- Open Science Collaboration
- Corresponding author
- Open Science Collaboration
- Venue
- Science
- Published
- 2015-08-28
- Volume
- 349
- Issue
- 6251
- Pages
- aac4716-aac4716
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W1897139626
- Cited by
- 8,515
- References
- 39
- Citations by year
- 2026: 414 · 2025: 532 · 2024: 698 · 2023: 757 · 2022: 783 · 2021: 906 · 2020: 1001 · 2019: 989 · 2018: 938 · 2017: 800 · 2016: 607 · 2015: 79
- FWCI
- 736.90
- Type
- journal article
- Language
- en
- Open access
- Locations
- Science (publishedVersion)
- Research portal (Tilburg University) (publishedVersion, other-oa)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, other-oa)
- UvA-DARE (University of Amsterdam) (publishedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (publishedVersion)
- VU Research Portal (publishedVersion)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (acceptedVersion, other-oa)
- Repository of the Academy's Library (Library of the Hungarian Academy of Sciences) (acceptedVersion)
- London School of Economics and Political Science Theses Online (London School of Economics and Political Science) (acceptedVersion)
- Discovery Research Portal (University of Dundee) (submittedVersion, other-oa)
- Keele Research Repository (Keele University) (submittedVersion, cc-by)
- London School of Economics and Political Science Research Online (London School of Economics and Political Science) (submittedVersion)
- Pure (Coventry University) (submittedVersion, other-oa)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- Radboud Repository (Radboud University) (submittedVersion)
- Winchester Research Repository (submittedVersion)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion)
- ANU Open Research (Australian National University) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- TU/e Research Portal (submittedVersion)
- BOA (University of Milano-Bicocca) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- Max Planck Digital Library (submittedVersion)
- Coventry University Open Collections (Coventry university) (submittedVersion)
- The HKU Scholars Hub (University of Hong Kong) (submittedVersion)
- Research Portal (King's College London) (submittedVersion)
- publish.UP (University of Potsdam) (submittedVersion)
- Lirias (KU Leuven) (submittedVersion)
- Faculty of Philosophy (University of Belgrade) (submittedVersion)
- Biblos-e Archivo (Universidad Autónoma de Madrid) (submittedVersion, public-domain)
- Bristol Research (University of Bristol) (submittedVersion)
- UWE Research Repository (UWE Bristol) (submittedVersion, cc-by)
- Padua Research Archive (University of Padova) (submittedVersion, cc-by-nc-nd)
- Zurich Open Repository and Archive (University of Zurich) (submittedVersion)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, other-oa)
- The Open Collections (Coventry University) (submittedVersion)
- Research Publications (Maastricht University) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Funders
- Laura and John Arnold Foundation
- Center for Open Science
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Keywords
- reproducibility, replication, statistical significance, confidence intervals, publication bias, quality of evidence, psychology research
- MeSH terms
- Periodicals as Topic, Psychology, Publishing, Research, Research Design, Reproducibility of Results, Confidence Intervals, Publication Bias, Behavioral Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.