A Bayesian Perspective on the Reproducibility Project: Psychology
Open accessThe DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Abstract
We revisit the results of the recent Reproducibility Project: Psychology by the Open Science Collaboration. We compute Bayes factors-a quantity that can be used to express comparative evidence for an hypothesis but also for the null hypothesis-for a large subset (N = 72) of the original papers and their corresponding replication attempts. In our computation, we take into account the likely scenario that publication bias had distorted the originally published results. Overall, 75% of studies gave qualitatively similar results in terms of the amount of evidence provided. However, the evidence was often weak (i.e., Bayes factor < 10). The majority of the studies (64%) did not provide strong evidence for either the null or the alternative hypothesis in either the original or the replication, and no replication attempts provided strong evidence in favor of the null. In all cases where the original paper provided strong evidence but the replication did not (15%), the sample size in the replication was smaller than the original. Where the replication provided strong evidence but the original did not (10%), the replication sample size was larger. We conclude that the apparent failure of the Reproducibility Project to replicate many target effects can be adequately explained by overestimation of effect sizes (or overestimation of evidence against the null hypothesis) due to small sample sizes and publication bias in the psychological literature. We further conclude that traditional sample sizes are insufficient and that a more widespread adoption of Bayesian methods is desirable.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“reanalyses of the data suggest that reproducibility in psychological science might be higher than originally claimed (23, 26, 27).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [27]
In the citing paper
“Second, and more importantly, because how reproducibility ought to be measured is the subject of a growing methodological and philosophical debate”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [27]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1972) · Cited by 17,806
Kass, R. E., & Raftery, A. E. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476572 · Cited by 12,558
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 2×
“Studies that do not meet the criteria 57 are relegated to lab file drawers [9].”
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Jeffreys, H., Moran, P. A. P., & Chatfield, C. (1922) · Nature · doi:10.1093/oso/9780198503682.001.0001 · Cited by 7,878
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 1×
“…research designs undermine the credibility of statistically significant results in addition to increasing the probability of nonsignificant ones (see (Button et al., 2013) and the references therein for a detailed argument); furthermore, low-powered studies generally provide only small amounts of…”
The American Psychological Association
Langfeld, H. S. (1919) · Science · doi:10.1126/science.49.1254.51 · Cited by 7,475
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 2×
“These two weaknesses of our field have conspired 285 to create a literature that is rife with false alarms [27].”
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688 · cited 1×
“Our prior expectation regarding the 82 effect size under HA is represented by a normal distribution centered on zero with 83 variance equal to 1 (this is a unit information prior, which carries a weight equivalent to 84 approximately one observation [17]).”
Bayesian statistical inference for psychological research.
Edwards, W., Lindman, H., & Savage, L. J. (1963) · Psychological Review · doi:10.1037/h0044139 · Cited by 1,734 · cited 1×
“306 Decades of reliance on orthodox statistical inference—which is known to overstate 307 the evidence against a null hypothesis [29–32]—have obfuscated the widespread problem 308 of small samples in psychological studies in general and in replication studies specifically.”
Why Most Discovered True Associations Are Inflated
Ioannidis, J. P. A. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818131e7 · Cited by 1,495 · cited 2×
“Studies with smaller, non-significant effects are rarely published, driving up the average effect size (Ioannidis, 2008).”
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321 · cited 2×
“A selective preference for publishing studies that 58 reject H0 is now known as publication bias, and is recognized as one cause of the current 59 crisis of confidence in psychology [10].”
JEFFREYS, H. (1922) · Nature · doi:10.1038/109132a0 · Cited by 1,217
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042 · cited 1×
“[34] for more detail and see Matzke et al.”
Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?
Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015) · American Psychologist · doi:10.1037/a0039400 · Cited by 1,005 · cited 2×
“The small degrees of evidence provided by these published reports, taken at face value, are consistent with observations by Wetzels et al. (2011) as well as the cautionary messages by Johnson (2013) and Maxwell, Lau, and Howard (2015).”
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · cited 2×
“Nearly six decades ago, Sterling [8] noted this 48 anomaly in the public record: In four prominent psychology journals, 95 to 99% of 49 studies that performed a significance test rejected the null hypothesis (i.”
Statistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 t Tests
Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406923 · Cited by 945
Stanley, T. D., & Doucouliagos, H. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1095 · Cited by 916 · cited 3×
“A unique feature of this method (compared to other 191 bias mitigation methods such as PET-PEESE [22]) is that it allows us to quantify 192 mitigated evidence for or against the null hypothesis on a continuous scale—a feature 193 that will become useful when we compare original and replicated studies, below.”
InfluentialThe test of significance in psychological research.
Bakan, D. (1966) · Psychological Bulletin · doi:10.1037/h0020412 · Cited by 911 · cited 2×
“Readers who wish to 65 evaluate original findings and replications alike must take into account the fact that our 66 “very publication practices themselves are part and parcel of the probabilistic processes 67 on which we base our conclusions concerning the nature of psychological 68 phenomena” [12] (p.”
Revised standards for statistical evidence
Johnson, V. E. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1313476110 · Cited by 808 · cited 1×
“The small degrees of evidence provided by these published reports, taken at face value, are consistent with observations by Wetzels et al. (2011) as well as the cautionary messages by Johnson (2013) and Maxwell, Lau, and Howard (2015).”
Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence
Berger, J. O., & Sellke, T. (1987) · Journal of the American Statistical Association · doi:10.1080/01621459.1987.10478397 · Cited by 775 · cited 1×
“Decades of reliance on orthodox statistical inference—which is known to overstate the evidence against a null hypothesis (Berger & Delampady, 1987; Berger & Sellke, 1987; Edwards, Lindman, & Savage, 1963; Goodman, 2008)—have obfuscated the widespread problem of small samples in psychological…”
A Dirty Dozen: Twelve P-Value Misconceptions
Goodman, S. (2008) · Seminars in Hematology · doi:10.1053/j.seminhematol.2008.04.003 · Cited by 680 · cited 2×
“306 Decades of reliance on orthodox statistical inference—which is known to overstate 307 the evidence against a null hypothesis [29–32]—have obfuscated the widespread problem 308 of small samples in psychological studies in general and in replication studies specifically.”
Berger, J. O., & Delampady, M. (1987) · Statistical Science · doi:10.1214/ss/1177013238 · Cited by 664 · cited 1×
“Decades of reliance on orthodox statistical inference—which is known to overstate the evidence against a null hypothesis (Berger & Delampady, 1987; Berger & Sellke, 1987; Edwards, Lindman, & Savage, 1963; Goodman, 2008)—have obfuscated the widespread problem of small samples in psychological…”
Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences.
Schönbrodt, F. D., Wagenmakers, E.-J., Zehetleitner, M., & Perugini, M. (2017) · Psychological Methods · doi:10.1037/met0000061 · Cited by 567
Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence
Berger, J. O., & Sellke, T. (1987) · Journal of the American Statistical Association · doi:10.2307/2289131 · Cited by 548
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.2307/2282137 · Cited by 524
Why Current Publication Practices May Distort Science
Young, N. S., Ioannidis, J. P. A., & Al-Ubaydli, O. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050201 · Cited by 502 · cited 1×
“Since studies in which the null hypothesis is confidently rejected are published at a higher rate than those in which it is not, the literature is “unrepresentative of scientists’ repeated samplings of the real world” [21].”
Optional stopping: No problem for Bayesians
Rouder, J. N. (2014) · Psychonomic Bulletin & Review · doi:10.3758/s13423-014-0595-4 · Cited by 492 · cited 2×
“A major selling point of Bayesian statistical methods is that sample sizes need not be determined in advance (Rouder, 2014), which allows analysts to monitor the incoming data and stop data collection when the results are deemed adequately informative; see (Wagenmakers, Wetzels, Borsboom, van der…”
Fraley, R. C., & Vazire, S. (2014) · PLoS ONE · doi:10.1371/journal.pone.0109019 · Cited by 409 · cited 1×
“Our analysis in that sense echoes that of Fraley and Vazire (2014).”
Bayesian tests to quantify the result of a replication attempt.
Verhagen, J., & Wagenmakers, E.-J. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0036731 · Cited by 291
Dirty Dozen: Twelve P-Value Misconceptions
Goodman, T. (2008) · Cited by 238
A Proposal for a New Editorial Policy in the Social Sciences
Walster, G. W., & Cleary, T. A. (1970) · The American Statistician · doi:10.1080/00031305.1970.10478884 · Cited by 144 · cited 1×
“A Differently put, the publication criteria should be considered part of the experimental design (Walster & Cleary, 1970).”
The effect of horizontal eye movements on free recall: A preregistered adversarial collaboration.
Matzke, D., Nieuwenhuis, S., van Rijn, H., Slagter, H. A., van der Molen, M. W., & Wagenmakers, E. (2015) · Journal of Experimental Psychology General · doi:10.1037/xge0000038 · Cited by 118 · cited 1×
“…when the results are deemed adequately informative; see (Wagenmakers, Wetzels, Borsboom, van der Maas, & Kievit, 2012) for more detail and see (Matzke et al., 2015) for an implementation of this kind of sampling plan, and also see (Schönbrodt, Wagenmakers, Zehetleitner, & Perugini, in press)…”
A Bayesian approach to mitigation of publication bias
Guan, M., & Vandekerckhove, J. (2015) · Psychonomic Bulletin & Review · doi:10.3758/s13423-015-0868-6 · Cited by 68 · cited 8×
“For the current project, we choose to account for publication bias by modeling the publication process as a part of the data collection procedure, using a Bayesian model averaging method proposed by Guan and Vandekerckhove (2016) and detailed in Section 2.2 below.”
InfluentialSelf-citationA Proposal for a New Editorial Policy in the Social Sciences
Walster, G. W., & Cleary, T. A. (1970) · The American Statistician · doi:10.2307/2681924 · Cited by 16
[Testing Precise Hypotheses]: Rejoinder
Berger, J. O., & Delampady, M. (1987) · Statistical Science · doi:10.1214/ss/1177013245 · Cited by 13 · cited 1×
“Decades of reliance on orthodox statistical inference—which is known to overstate the evidence against a null hypothesis (Berger & Delampady, 1987; Berger & Sellke, 1987; Edwards, Lindman, & Savage, 1963; Goodman, 2008)—have obfuscated the widespread problem of small samples in psychological…”
Origin of the Bayes FactorPreprint
Etz, A. J., & Wagenmakers, E. (2015) · arXiv (Cornell University) · Cited by 1
Self-citation
Cited by
Siddaway, A. P., Wood, A. M., & Hedges, L. V. (2019) · Annual Review of Psychology · doi:10.1146/annurev-psych-010418-102803 · Cited by 2,550
“…(although the contribution 3Bayesian statistics might even be used to compare the weight of evidence for competing (null and alternative) hypotheses (Braver et al. 2014, Etz & Vandekerckhove 2016). www.annualreviews.org • Systematic Reviews 9.9 Review in Advance first posted on August 8, 2018.”
BackgroundInfluentialCamerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521
Makowski, D., Ben-Shachar, M., & Lüdecke, D. (2019) · Journal of Open Source Software · doi:10.21105/joss.01541 · Cited by 1,483
“…accuracy (particularly in noisy data and small samples), the possibility of incorporating prior knowledge into the analysis, and the intuitive interpretation of results (Andrews & Baguley, 2013; Etz & Vandekerckhove, 2016; Kruschke, 2010; Kruschke, Aguinis, & Joo, 2012; Wagenmakers et al., 2017).”
BackgroundPower Analysis and Effect Size in Mixed Effects Models: A Tutorial
Brysbaert, M., & Stevens, M. (2018) · Journal of Cognition · doi:10.5334/joc.10 · Cited by 1,420
“At the same time, it is becoming clear that the definition of replication success may be less straightforward than assumed at first (Amrhein, Korner-Nievergelt, & Roth, 2017; Etz & Vandekerckhove, 2016).”
BackgroundIndices of Effect Existence and Significance in the Bayesian Framework
Makowski, D., Ben-Shachar, M. S., Chen, S. H. A., & Lüdecke, D. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02767 · Cited by 983
“The Bayesian framework is quickly gaining popularity among psychologists and neuroscientists (Andrews and Baguley, 2013), for reasons such as flexibility, better accuracy in noisy data and small samples, less proneness to type I errors, the possibility of introducing prior knowledge into the analysis and the intuitive…”
BackgroundMethod
Metadata
- Authors
- Etz, A., & Vandekerckhove, J.
- Affiliations
- University of Amsterdam (NL)
- University of California, Irvine (US)
- Corresponding author
- Etz, Alexander
- Vandekerckhove, Joachim
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-02-26
- Volume
- 11
- Issue
- 2
- Pages
- e0149794
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4769355
- OpenAlex
- W2282205924
- Cited by
- 295
- References
- 37
- Citations by year
- 2026: 11 · 2025: 14 · 2024: 8 · 2023: 15 · 2022: 25 · 2021: 28 · 2020: 50 · 2019: 34 · 2018: 48 · 2017: 45 · 2016: 17
- FWCI
- 44.56
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- eScholarship (California Digital Library) (submittedVersion)
- Funders
- National Science Foundation (1230118, 48192, 1534472)
- John Templeton Foundation (1534472, 1230118, 48192)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- Reproducibility Project: Psychology, replication study, Bayes factor, publication bias, small sample size, effect size overestimation, Bayesian hypothesis testing, Open Science Collaboration
- MeSH terms
- Bayes Theorem, Humans, Psychology, Research Design, Reproducibility of Results, Publication Bias, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.