CoChat

Reproducing Statistical Results

Open access
Stodden, Victoria · 2015 · Annual Review of Statistics and Its Application

The DOI matches its Crossref record; 99 of the 100 references it cites were found in a registry or an index.

Abstract

The reproducibility of statistical findings has become a concern not only for statisticians, but for all researchers engaged in empirical discovery. Section 2 of this article identifies key reasons statistical findings may not replicate, including power and sampling issues; misapplication of statistical tests; the instability of findings under reasonable perturbations of data or models; lack of access to methods, data, or equipment; and cultural barriers such as researcher incentives and rewards. Section 3 discusses five proposed remedies for these replication failures: improved prepublication and postpublication validation of findings; the complete disclosure of research steps; assessment of the stability of statistical findings; providing access to digital research objects, in particular data and software; and ensuring these objects are legally reusable.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1146/annurev-statistics-010814-020127

Cites

99 of 99 references verified0 citation errors · 0 not checkable

Statistics

Figures

Image 1
CC-BY · From the PDF
Image 2
CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 99 references

Research works printed but not linked: 17

  1. Abdi H, 2007, Encycl. Meas. Stat., 3, 103
  2. Am. Libr. Assoc, 2014, The Fair Access to Science and Technology Research Act (FASTR).
  3. Boyle R, 2007, The Sceptical Chymistor Chymico-Physical Doubts & Paradoxes, Touching the Spagyrist's Principles Commonly Call'd Hypostatical; As They Are Wont to Be Propos'd and Defended by the Generality of Alchymists. Whereunto Is Præmis'd Part of Another Discourse Relating to the Same Subject.
  4. Creative Commons, 2013, CC0 1.0 Universal (CC0 1.0) Public Domain Dedication
  5. Creative Commons, 2013, Creative Commons—Attribution 4.0 International (CC BY 4.0)
  6. Donohue JJ, 2006, Stanford Law Rev., 58, 791
  7. Holm S, 1979, Scand. J. Stat., 6, 65
  8. NSF (Nat. Sci. Found.), 2011, NSF Data Management Plan Requirements
  9. NSF (Nat. Sci. Found.), 2013, GPG Summary of Significant Changes
  10. Open Source Initiative, 2013, The BSD 2-Clause License
  11. Open Source Initiative, 2013, The MIT License (MIT)
  12. Panel Sci. Responsib. Conduct Res., Comm. Sci. Eng. Public Policy, Nat. Acad. Sci., Nat. Acad. Eng., Inst. Med, 1992, Responsible Science
  13. 10.1056/nejm200504073521422 · DOI
  14. Stock JH, 2011, Introduction to Econometrics
  15. Stodden V, 2009, Intl. J. Comm. Pol., 13, 1
  16. 10.1056/nejm200504073521422 · DOI
  17. WSSSPE (Work. Sustain. Softw. Sci. Pract. Exp.). 2013. Contributions.WSSSPE Workshop 1, Denver, CO, Nov. 17.http://wssspe.researchcomputing.org.uk/wssspe1/contributions/

Cited by

See all 86

Metadata

Authors
Stodden, V.
Affiliations
University of Illinois Urbana-Champaign (US)
Corresponding author
Stodden, Victoria
Venue
Annual Review of Statistics and Its Application
Published
2015-04-10
Volume
2
Issue
1
Pages
1-19
ISSN
  • 2326-8298 (ISSN-L)
  • 2326-831X
OpenAlex
W2126645252
Cited by
77
References
100
Citations by year
2026: 3 · 2025: 7 · 2024: 6 · 2023: 4 · 2022: 7 · 2021: 5 · 2020: 7 · 2019: 8 · 2018: 10 · 2017: 10 · 2016: 8 · 2015: 2
FWCI
3.29
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
Annual Review of Statistics and Its Application (publishedVersion, cc-by)
Funders
Alfred P. Sloan Foundation (PG004545)
Topics
  • Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
  • Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
Keywords
reproducibility, replication failure, statistical power, sampling issues, researcher incentives

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Reproducing Statistical Results. https://app.cochat.ai/papers/10.1146/annurev-statistics-010814-020127
The paper
Stodden, V. (2015). Reproducing Statistical Results. Annual Review of Statistics and Its Application. https://doi.org/10.1146/annurev-statistics-010814-020127

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.