The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
A Bayesian Perspective on the Reproducibility Project: Psychology
Open accessAbstract
We revisit the results of the recent Reproducibility Project: Psychology by the Open Science Collaboration. We compute Bayes factors-a quantity that can be used to express comparative evidence for an hypothesis but also for the null hypothesis-for a large subset (N = 72) of the original papers and their corresponding replication attempts. In our computation, we take into account the likely scenario that publication bias had distorted the originally published results. Overall, 75% of studies gave qualitatively similar results in terms of the amount of evidence provided. However, the evidence was often weak (i.e., Bayes factor < 10). The majority of the studies (64%) did not provide strong evidence for either the null or the alternative hypothesis in either the original or the replication, and no replication attempts provided strong evidence in favor of the null. In all cases where the original paper provided strong evidence but the replication did not (15%), the sample size in the replication was smaller than the original. Where the replication provided strong evidence but the original did not (10%), the replication sample size was larger. We conclude that the apparent failure of the Reproducibility Project to replicate many target effects can be adequately explained by overestimation of effect sizes (or overestimation of evidence against the null hypothesis) due to small sample sizes and publication bias in the psychological literature. We further conclude that traditional sample sizes are insufficient and that a more widespread adoption of Bayesian methods is desirable.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“reanalyses of the data suggest that reproducibility in psychological science might be higher than originally claimed (23, 26, 27).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [27]
In the citing paper
“Second, and more importantly, because how reproducibility ought to be measured is the subject of a growing methodological and philosophical debate”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [27]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1972) · Cited by 17,806
Kass, R. E., & Raftery, A. E. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476572 · Cited by 12,558
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 2×
“Studies that do not meet the criteria 57 are relegated to lab file drawers [9].”
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Jeffreys, H., Moran, P. A. P., & Chatfield, C. (1922) · Nature · doi:10.1093/oso/9780198503682.001.0001 · Cited by 7,878
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 1×
“…research designs undermine the credibility of statistically significant results in addition to increasing the probability of nonsignificant ones (see (Button et al., 2013) and the references therein for a detailed argument); furthermore, low-powered studies generally provide only small amounts of…”
The American Psychological Association
Langfeld, H. S. (1919) · Science · doi:10.1126/science.49.1254.51 · Cited by 7,475
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 2×
“These two weaknesses of our field have conspired 285 to create a literature that is rife with false alarms [27].”
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688 · cited 1×
“Our prior expectation regarding the 82 effect size under HA is represented by a normal distribution centered on zero with 83 variance equal to 1 (this is a unit information prior, which carries a weight equivalent to 84 approximately one observation [17]).”
Cited by
Siddaway, A. P., Wood, A. M., & Hedges, L. V. (2019) · Annual Review of Psychology · doi:10.1146/annurev-psych-010418-102803 · Cited by 2,550
“…(although the contribution 3Bayesian statistics might even be used to compare the weight of evidence for competing (null and alternative) hypotheses (Braver et al. 2014, Etz & Vandekerckhove 2016). www.annualreviews.org • Systematic Reviews 9.9 Review in Advance first posted on August 8, 2018.”
BackgroundInfluentialCamerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521
Makowski, D., Ben-Shachar, M., & Lüdecke, D. (2019) · Journal of Open Source Software · doi:10.21105/joss.01541 · Cited by 1,483
“…accuracy (particularly in noisy data and small samples), the possibility of incorporating prior knowledge into the analysis, and the intuitive interpretation of results (Andrews & Baguley, 2013; Etz & Vandekerckhove, 2016; Kruschke, 2010; Kruschke, Aguinis, & Joo, 2012; Wagenmakers et al., 2017).”
BackgroundPower Analysis and Effect Size in Mixed Effects Models: A Tutorial
Brysbaert, M., & Stevens, M. (2018) · Journal of Cognition · doi:10.5334/joc.10 · Cited by 1,420
“At the same time, it is becoming clear that the definition of replication success may be less straightforward than assumed at first (Amrhein, Korner-Nievergelt, & Roth, 2017; Etz & Vandekerckhove, 2016).”
BackgroundIndices of Effect Existence and Significance in the Bayesian Framework
Makowski, D., Ben-Shachar, M. S., Chen, S. H. A., & Lüdecke, D. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02767 · Cited by 983
“The Bayesian framework is quickly gaining popularity among psychologists and neuroscientists (Andrews and Baguley, 2013), for reasons such as flexibility, better accuracy in noisy data and small samples, less proneness to type I errors, the possibility of introducing prior knowledge into the analysis and the intuitive…”
BackgroundMethod
Metadata
- Authors
- Etz, A., & Vandekerckhove, J.
- Affiliations
- University of Amsterdam (NL)
- University of California, Irvine (US)
- Corresponding author
- Etz, Alexander
- Vandekerckhove, Joachim
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-02-26
- Volume
- 11
- Issue
- 2
- Pages
- e0149794
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4769355
- OpenAlex
- W2282205924
- Cited by
- 295
- References
- 37
- Citations by year
- 2026: 11 · 2025: 14 · 2024: 8 · 2023: 15 · 2022: 25 · 2021: 28 · 2020: 50 · 2019: 34 · 2018: 48 · 2017: 45 · 2016: 17
- FWCI
- 44.56
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- eScholarship (California Digital Library) (submittedVersion)
- Funders
- National Science Foundation (1230118, 48192, 1534472)
- John Templeton Foundation (1534472, 1230118, 48192)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- Reproducibility Project: Psychology, replication study, Bayes factor, publication bias, small sample size, effect size overestimation, Bayesian hypothesis testing, Open Science Collaboration
- MeSH terms
- Bayes Theorem, Humans, Psychology, Research Design, Reproducibility of Results, Publication Bias, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Bayesian Perspective on the Reproducibility Project: Psychology. https://app.cochat.ai/papers/10.1371/journal.pone.0149794
- The paper
- Etz, A., & Vandekerckhove, J. (2016). A Bayesian Perspective on the Reproducibility Project: Psychology. PLOS ONE. https://doi.org/10.1371/journal.pone.0149794
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.