The DOI matches its Crossref record; 38 of the 41 references it cites were found in a registry or an index.
What does research reproducibility mean?
Abstract
The language and conceptual framework of “research reproducibility” are nonstandard and unsettled across the sciences.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“High quality reporting is an important step when considering issues related to reproducibility [50].”
From CONSORT 2025 statement: Updated guideline for reporting randomised trials (2025) · cites this paper as [50]
In the citing paper
“These factors are inherent to modern computational pipelines in deep learning (Desai et al., 2025; Goodman et al., 2016; Semmelrock et al., 2025)”
From CiteFusion: an ensemble framework for citation intent classification harnessing dual-model binary couples and SHAP analyses (2025) · cites this paper as Goodman et al., 2016
In the citing paper
“for reasons that have nothing to do with questionable research and publication practices (28).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
In the citing paper
“but the reproducibility of results and inferences is likely to be lower and to vary across subfields and methodologies”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
In the citing paper
“the reproducibility of research methods can in principle be expected to be 100%”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“The classic file-drawer publication bias problem (wherein nonsignificant or “uninteresting” results are not published) (29) results in bias under the assumption that multiple studies are being produced independently but a biased sample is published.”
BackgroundEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Pseudoreplication and the Design of Ecological Field Experiments
Hurlbert, S. H. (1984) · Ecological Monographs · doi:10.2307/1942661 · Cited by 8,377
The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2016 update
Afgan, E., Baker, D., van den Beek, M., Blankenberg, D., Bouvier, D., Čech, M., Chilton, J., Clements, D., Coraor, N., Eberhard, C., et al. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkw343 · Cited by 2,148
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
A Reality Check for Data Snooping
White, H. (2000) · Econometrica · doi:10.1111/1468-0262.00152 · Cited by 1,851
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500
The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant
Gelman, A., & Stern, H. (2006) · The American Statistician · doi:10.1198/000313006x152649 · Cited by 1,407
Policy: NIH plans to enhance reproducibility
Collins, F. S., & Tabak, L. A. (2014) · Nature · doi:10.1038/505612a · Cited by 1,247
Research works printed but not linked: 2
- S. N. Goodman J. Gerson Mechanistic Evidence in Evidence-Based Medicine: A Conceptual Framework (Agency for Healthcare Research and Quality Rockville MD 2013); http://www.ncbi.nlm.nih.gov/books/NBK154584/.
- E. E. Leamer Specification Searches: Ad Hoc Inference with Nonexperimental Data (Wiley Hoboken NJ 1978).
Cited by
The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000410 · Cited by 5,939
The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · Experimental Physiology · doi:10.1113/ep088870 · Cited by 3,497
The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research*
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1177/0271678x20943823 · Cited by 1,443
“Why good reporting is important In recent years, concerns about the reproducibility of research findings have been raised by scientists, funders, research users, and policy makers [1, 2].”
CONSORT 2025 statement: updated guideline for reporting randomised trials
Hopewell, S., Chan, A.-W., Collins, G. S., Hróbjartsson, A., Moher, D., Schulz, K. F., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081123 · Cited by 1,195
“50 We encourage trial authors to detail what was done and to acknowledge if something was not done or was modified, ensuring alignment of information with that reported in the trial protocol, statistical analysis plan, and trial registry.”
Self-citationThe ARRIVE guidelines 2.0: Updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · British Journal of Pharmacology · doi:10.1111/bph.15193 · Cited by 1,081
Metadata
- Authors
- Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A.
- Affiliations
- Stanford University (US)
- Corresponding author
- Goodman, Steven N.
- Venue
- Science Translational Medicine
- Published
- 2016-06
- Volume
- 8
- Issue
- 341
- Pages
- 341ps12-341ps12
- ISSN
- 1946-6234 (ISSN-L)
- 1946-6242
- OpenAlex
- W2414283705
- Cited by
- 1,293
- References
- 41
- Citations by year
- 2026: 85 · 2025: 93 · 2024: 146 · 2023: 110 · 2022: 174 · 2021: 151 · 2020: 168 · 2019: 140 · 2018: 112 · 2017: 83 · 2016: 30
- FWCI
- 18.01
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science Translational Medicine (publishedVersion)
- PubMed (publishedVersion)
- Funders
- John and Laura Arnold Foundation
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- research reproducibility, terminology, conceptual framework, truth in science
- MeSH terms
- Reproducibility of Results, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: What does research reproducibility mean? https://app.cochat.ai/papers/10.1126/scitranslmed.aaf5027
- The paper
- Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016). What does research reproducibility mean? Science Translational Medicine. https://doi.org/10.1126/scitranslmed.aaf5027
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.