Reproducing Statistical Results
Open accessThe DOI matches its Crossref record; 99 of the 100 references it cites were found in a registry or an index.
Abstract
The reproducibility of statistical findings has become a concern not only for statisticians, but for all researchers engaged in empirical discovery. Section 2 of this article identifies key reasons statistical findings may not replicate, including power and sampling issues; misapplication of statistical tests; the instability of findings under reasonable perturbations of data or models; lack of access to methods, data, or equipment; and cultural barriers such as researcher incentives and rewards. Section 3 discusses five proposed remedies for these replication failures: improved prepublication and postpublication validation of findings; the complete disclosure of research steps; assessment of the stability of statistical findings; providing access to digital research objects, in particular data and software; and ensuring these objects are legally reusable.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, the utility of any subsequent data reuse is undermined because it is unclear what the results of any novel analyses should be compared with, or whether the outcomes are sufficiently credible to be included in meta-analyses [37].”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition (2018) · cites this paper as [37]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Being able to verify the analytic reproducibility of reported outcomes is necessary to establish a baseline standard for all subsequent data reuse, and is a minimal threshold of scientific credibility [37].”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition (2018) · cites this paper as [37]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
A Simple Sequentially Rejective Multiple Test Procedure
(1979) · Scandinavian Journal of Statistics · doi:10.2307/4615733 · Cited by 21,526
POLANYI, M. (1997) · Knowledge in Organisations · doi:10.1016/b978-0-7506-9718-7.50010-x · Cited by 12,408
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“Steps taken by the National Science Foundation (NSF) and the National Institutes of Health (NIH) to recognize software and data set contributions in the biographical sketch of a grant applicant encourage greater data and code production and disclosure (NSF 2013, Rockey 2014).”
BackgroundPersonal Knowledge: Towards a Post-Critical Philosophy
Reid, L. A., & Polanyi, M. (1959) · British Journal of Educational Studies · doi:10.2307/3119338 · Cited by 7,538
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 2×
“Ioannidis (2005) lists several additional reasons why empirical findings may not replicate (some of which I have already discussed): small sample size, small effect size, the number of statistical tests carried out relative www.annualreviews.org • Reproducing Statistical Results 7 A nn ua l R ev…”
BackgroundResultThe Consensus Coding Sequences of Human Breast and Colorectal Cancers
Sjöblom, T., Jones, S., Wood, L. D., Parsons, D. W., Lin, J., Barber, T. D., Mandelker, D., Leary, R. J., Ptak, J., Silliman, N., et al. (2006) · Science · doi:10.1126/science.1133427 · Cited by 3,169
Research works printed but not linked: 17
- Abdi H, 2007, Encycl. Meas. Stat., 3, 103
- Am. Libr. Assoc, 2014, The Fair Access to Science and Technology Research Act (FASTR).
- Boyle R, 2007, The Sceptical Chymistor Chymico-Physical Doubts & Paradoxes, Touching the Spagyrist's Principles Commonly Call'd Hypostatical; As They Are Wont to Be Propos'd and Defended by the Generality of Alchymists. Whereunto Is Præmis'd Part of Another Discourse Relating to the Same Subject.
- Creative Commons, 2013, CC0 1.0 Universal (CC0 1.0) Public Domain Dedication
- Creative Commons, 2013, Creative Commons—Attribution 4.0 International (CC BY 4.0)
- Donohue JJ, 2006, Stanford Law Rev., 58, 791
- Holm S, 1979, Scand. J. Stat., 6, 65
- NSF (Nat. Sci. Found.), 2011, NSF Data Management Plan Requirements
- NSF (Nat. Sci. Found.), 2013, GPG Summary of Significant Changes
- Open Source Initiative, 2013, The BSD 2-Clause License
- Open Source Initiative, 2013, The MIT License (MIT)
- Panel Sci. Responsib. Conduct Res., Comm. Sci. Eng. Public Policy, Nat. Acad. Sci., Nat. Acad. Eng., Inst. Med, 1992, Responsible Science
- 10.1056/nejm200504073521422 · DOI
- Stock JH, 2011, Introduction to Econometrics
- Stodden V, 2009, Intl. J. Comm. Pol., 13, 1
- 10.1056/nejm200504073521422 · DOI
- WSSSPE (Work. Sustain. Softw. Sci. Pract. Exp.). 2013. Contributions.WSSSPE Workshop 1, Denver, CO, Nov. 17.http://wssspe.researchcomputing.org.uk/wssspe1/contributions/
Cited by
Hardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C., Hofelich Mohr, A., Clayton, E., Yoon, E. J., Henry Tessler, M., et al. (2018) · Royal Society Open Science · doi:10.1098/rsos.180448 · Cited by 336
Reproducible Research and GIScience: An Evaluation Using GIScience Conference Papers
Ostermann, F. O., Nüst, D., Granell, C., Hofer, B., & Konkol, M. (2018) · arXiv (Cornell University) · doi:10.4230/lipics.giscience.2021.ii.2 · Cited by 135
A Practical Guide for Transparency in Psychological Science
Klein, O., Hardwicke, T. E., Aust, F., Breuer, J., Danielsson, H., Mohr, A. H., IJzerman, H., Nilsonne, G., Vanpaemel, W., & Frank, M. C. (2018) · Collabra: Psychology · doi:10.1525/collabra.158 · Cited by 233
REFORMS: Consensus-based Recommendations for Machine-learning-based Science
Kapoor, S., Cantrell, E. M., Peng, K., Pham, T. H., Bail, C. A., Gundersen, O. E., Hofman, J. M., Hullman, J., Lones, M. A., Malik, M. M., et al. (2024) · Science Advances · doi:10.1126/sciadv.adk3452 · Cited by 129
Bon-EV: an improved multiple testing procedure for controlling false discovery rates
Li, D., Xie, Z., Zand, M., Fogg, T., & Dye, T. (2017) · BMC Bioinformatics · doi:10.1186/s12859-016-1414-x · Cited by 230
Metadata
- Authors
- Stodden, V.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Corresponding author
- Stodden, Victoria
- ORCID
- Stodden, Victoria
- Venue
- Annual Review of Statistics and Its Application
- Publisher
- Annual Reviews
- Published
- 2015-04-10
- Volume
- 2
- Issue
- 1
- Pages
- 1-19
- ISSN
- 2326-8298 (ISSN-L)
- 2326-831X
- OpenAlex
- W2126645252
- Cited by
- 77
- References
- 100
- Citations by year
- 2026: 3 · 2025: 7 · 2024: 6 · 2023: 4 · 2022: 7 · 2021: 5 · 2020: 7 · 2019: 8 · 2018: 10 · 2017: 10 · 2016: 8 · 2015: 2
- FWCI
- 3.29
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Annual Review of Statistics and Its Application (publishedVersion, cc-by)
- Funders
- Alfred P. Sloan Foundation (PG004545)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- reproducibility, replication failure, statistical power, sampling issues, researcher incentives
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Reproducing Statistical Results. https://app.cochat.ai/papers/10.1146/annurev-statistics-010814-020127
- The paper
- Stodden, V. (2015). Reproducing Statistical Results. Annual Review of Statistics and Its Application. https://doi.org/10.1146/annurev-statistics-010814-020127
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.

