The DOI matches its Crossref record; 38 of the 41 references it cites were found in a registry or an index.
What does research reproducibility mean?
Abstract
The language and conceptual framework of “research reproducibility” are nonstandard and unsettled across the sciences.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“High quality reporting is an important step when considering issues related to reproducibility [50].”
From CONSORT 2025 statement: Updated guideline for reporting randomised trials (2025) · cites this paper as [50]
In the citing paper
“These factors are inherent to modern computational pipelines in deep learning (Desai et al., 2025; Goodman et al., 2016; Semmelrock et al., 2025)”
From CiteFusion: an ensemble framework for citation intent classification harnessing dual-model binary couples and SHAP analyses (2025) · cites this paper as Goodman et al., 2016
In the citing paper
“for reasons that have nothing to do with questionable research and publication practices (28).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
In the citing paper
“but the reproducibility of results and inferences is likely to be lower and to vary across subfields and methodologies”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
In the citing paper
“the reproducibility of research methods can in principle be expected to be 100%”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“The classic file-drawer publication bias problem (wherein nonsignificant or “uninteresting” results are not published) (29) results in bias under the assumption that multiple studies are being produced independently but a biased sample is published.”
BackgroundEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Pseudoreplication and the Design of Ecological Field Experiments
Hurlbert, S. H. (1984) · Ecological Monographs · doi:10.2307/1942661 · Cited by 8,377
The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2016 update
Afgan, E., Baker, D., van den Beek, M., Blankenberg, D., Bouvier, D., Čech, M., Chilton, J., Clements, D., Coraor, N., Eberhard, C., et al. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkw343 · Cited by 2,148
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
A Reality Check for Data Snooping
White, H. (2000) · Econometrica · doi:10.1111/1468-0262.00152 · Cited by 1,851
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500
The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant
Gelman, A., & Stern, H. (2006) · The American Statistician · doi:10.1198/000313006x152649 · Cited by 1,407
Policy: NIH plans to enhance reproducibility
Collins, F. S., & Tabak, L. A. (2014) · Nature · doi:10.1038/505612a · Cited by 1,247
Toward Evidence-Based Medical Statistics. 1: The P Value Fallacy
Goodman, S. N. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-12-199906150-00008 · Cited by 1,185
Self-citationSpecification Searches: Ad Hoc Inference with Non-Experimental Data.
Richard, J.-F., & Leamer, E. E. (1981) · Economica · doi:10.2307/2553491 · Cited by 1,114
Reproducibility in Science: Improving the Standard for Basic and Preclinical Research
Begley, C. G., & Ioannidis, J. P. A. (2015) · Circulation Research · doi:10.1161/circresaha.114.303819 · Cited by 1,035
Self-citationEffects of Adjuvant Tamoxifen and of Cytotoxic Therapy on Mortality in Early Breast Cancer
Early Breast Cancer Trialists' Collaborative Group (1988) · New England Journal of Medicine · doi:10.1056/nejm198812293192601 · Cited by 933
Toward Evidence-Based Medical Statistics. 2: The Bayes Factor
Goodman, S. N. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-12-199906150-00019 · Cited by 912
Self-citationComment on “Estimating the reproducibility of psychological science.”
Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D. (2016) · Science · doi:10.1126/science.aad7243 · Cited by 474 · cited 2×
“The lack of a single accepted definition opened the door to controversy about their methodological approach and conclusions (19).”
BackgroundA comment on replication,P‐values and evidence
Goodman, S. N. (1992) · Statistics in Medicine · doi:10.1002/sim.4780110705 · Cited by 309
Self-citationReproducible Epidemiologic Research
Peng, R. D., Dominici, F., & Zeger, S. L. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj093 · Cited by 292
Sharing Clinical Trial Data: Maximizing Benefits, Minimizing Risk
Lo, B. (2015) · JAMA · doi:10.1001/jama.2015.292 · Cited by 233
THE MEANING OF FAILED REPLICATIONS: A REVIEW AND PROPOSAL
Clemens, M. A. (2017) · Journal of Economic Surveys · doi:10.1111/joes.12139 · Cited by 213
Electronic documents give reproducible research a new meaning
Claerbout, J. F., & Karrenbach, M. (1992) · SEG Technical Program Expanded Abstracts 1992 · doi:10.1190/1.1822162 · Cited by 205
Is everything we eat associated with cancer? A systematic cookbook review
Schoenfeld, J. D., & Ioannidis, J. P. (2013) · The American Journal of Clinical Nutrition · doi:10.3945/ajcn.112.047142 · Cited by 198
Self-citationBibliometric Evidence for a Hierarchy of the Sciences
Fanelli, D., & Glänzel, W. (2013) · PLoS ONE · doi:10.1371/journal.pone.0066938 · Cited by 197
Self-citationA Bayes factor meta-analysis of Bem’s ESP claim
Rouder, J. N., & Morey, R. D. (2011) · Psychonomic Bulletin & Review · doi:10.3758/s13423-011-0088-7 · Cited by 194
Mills, J. L. (1993) · New England Journal of Medicine · doi:10.1056/nejm199310143291613 · Cited by 182
P-Value Precision and Reproducibility
Boos, D. D., & Stefanski, L. A. (2011) · The American Statistician · doi:10.1198/tas.2011.10129 · Cited by 166
Rothman, K. J. (1986) · Annals of Internal Medicine · doi:10.7326/0003-4819-105-3-445 · Cited by 145
WHAT'S TO KNOW ABOUT THE CREDIBILITY OF EMPIRICAL ECONOMICS?
Ioannidis, J. P. A., & Doucouliagos, H. (2013) · Journal of Economic Surveys · doi:10.1111/joes.12032 · Cited by 142
Self-citationCancer on Trial: Oncology as a New Style of Practice
Keating, P., & Cambrosio, A. (2011) · Cited by 128
Keating, P., & Cambrosio, A. (2011) · University of Chicago Press eBooks · doi:10.7208/chicago/9780226428932.001.0001 · Cited by 62
Multiplicities in Cancer Research: Ubiquitous and Necessary Evils
Berry, D. A. (2012) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djs301 · Cited by 50
Failure to replicate the STAP cell phenomenon
Angeles, A. D. L., Ferrari, F., Fujiwara, Y., Mathieu, R., Lee, S., Lee, S., Tu, H.-C., Ross, S. J., Chou, S. S., Nguyen, M., et al. (2015) · Nature · doi:10.1038/nature15513 · Cited by 49
Irons, E. E., Flatt, J. W., Doronin, K., Fox, T. L., Acchione, M., Stewart, P. L., & Shayakhmetov, D. M. (2013) · Journal of Virology · doi:10.1128/jvi.01070-13 · Cited by 34
Solutions for quantifying P-value uncertainty and replication power
Lazzeroni, L. C., Lü, Y., & Belitskaya‐Lévy, I. (2016) · Nature Methods · doi:10.1038/nmeth.3741 · Cited by 34
An Unappreciated Challenge to Oncology Drug Discovery: Pitfalls in Preclinical Research
Begley, C. G. (2013) · American Society of Clinical Oncology Educational Book · doi:10.14694/edbook_am.2013.33.466 · Cited by 24
Lewenstein, B. V., & Baur, W. (1991) · Journal of Radioanalytical and Nuclear Chemistry · doi:10.1007/bf02042161 · Cited by 11
Iorns, E. J., Gunn, W., Erath, J., Rodríguez, A., Zhou, J., Benzinou, M., & Initiative, T. R. (2014) · PLoS ONE · doi:10.1371/journal.pone.0114614 · Cited by 7
Detecting and describing heterogeneity in meta-analysis
Hardy, R. J., & Thompson, S. G. (1998) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19980430)17:8<841::aid-sim781>3.0.co;2-d
Research works printed but not linked: 2
- S. N. Goodman J. Gerson Mechanistic Evidence in Evidence-Based Medicine: A Conceptual Framework (Agency for Healthcare Research and Quality Rockville MD 2013); http://www.ncbi.nlm.nih.gov/books/NBK154584/.
- E. E. Leamer Specification Searches: Ad Hoc Inference with Nonexperimental Data (Wiley Hoboken NJ 1978).
Cited by
The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000410 · Cited by 5,939
The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · Experimental Physiology · doi:10.1113/ep088870 · Cited by 3,497
The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research*
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1177/0271678x20943823 · Cited by 1,443
“Why good reporting is important In recent years, concerns about the reproducibility of research findings have been raised by scientists, funders, research users, and policy makers [1, 2].”
CONSORT 2025 statement: updated guideline for reporting randomised trials
Hopewell, S., Chan, A.-W., Collins, G. S., Hróbjartsson, A., Moher, D., Schulz, K. F., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081123 · Cited by 1,195
“50 We encourage trial authors to detail what was done and to acknowledge if something was not done or was modified, ensuring alignment of information with that reported in the trial protocol, statistical analysis plan, and trial registry.”
Self-citationThe ARRIVE guidelines 2.0: Updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · British Journal of Pharmacology · doi:10.1111/bph.15193 · Cited by 1,081
Metadata
- Authors
- Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A.
- Affiliations
- Stanford University (US)
- Corresponding author
- Goodman, Steven N.
- Venue
- Science Translational Medicine
- Published
- 2016-06
- Volume
- 8
- Issue
- 341
- Pages
- 341ps12-341ps12
- ISSN
- 1946-6234 (ISSN-L)
- 1946-6242
- OpenAlex
- W2414283705
- Cited by
- 1,293
- References
- 41
- Citations by year
- 2026: 85 · 2025: 93 · 2024: 146 · 2023: 110 · 2022: 174 · 2021: 151 · 2020: 168 · 2019: 140 · 2018: 112 · 2017: 83 · 2016: 30
- FWCI
- 18.01
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science Translational Medicine (publishedVersion)
- PubMed (publishedVersion)
- Funders
- John and Laura Arnold Foundation
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- research reproducibility, terminology, conceptual framework, truth in science
- MeSH terms
- Reproducibility of Results, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.