The DOI matches its Crossref record; 38 of the 41 references it cites were found in a registry or an index.
What does research reproducibility mean?
Abstract
The language and conceptual framework of “research reproducibility” are nonstandard and unsettled across the sciences.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“High quality reporting is an important step when considering issues related to reproducibility [50].”
From CONSORT 2025 statement: Updated guideline for reporting randomised trials (2025) · cites this paper as [50]
In the citing paper
“These factors are inherent to modern computational pipelines in deep learning (Desai et al., 2025; Goodman et al., 2016; Semmelrock et al., 2025)”
From CiteFusion: an ensemble framework for citation intent classification harnessing dual-model binary couples and SHAP analyses (2025) · cites this paper as Goodman et al., 2016
In the citing paper
“for reasons that have nothing to do with questionable research and publication practices (28).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
In the citing paper
“but the reproducibility of results and inferences is likely to be lower and to vary across subfields and methodologies”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
In the citing paper
“the reproducibility of research methods can in principle be expected to be 100%”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“The classic file-drawer publication bias problem (wherein nonsignificant or “uninteresting” results are not published) (29) results in bias under the assumption that multiple studies are being produced independently but a biased sample is published.”
BackgroundEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Pseudoreplication and the Design of Ecological Field Experiments
Hurlbert, S. H. (1984) · Ecological Monographs · doi:10.2307/1942661 · Cited by 8,377
The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2016 update
Afgan, E., Baker, D., van den Beek, M., Blankenberg, D., Bouvier, D., Čech, M., Chilton, J., Clements, D., Coraor, N., Eberhard, C., et al. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkw343 · Cited by 2,148
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
A Reality Check for Data Snooping
White, H. (2000) · Econometrica · doi:10.1111/1468-0262.00152 · Cited by 1,851
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500
The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant
Gelman, A., & Stern, H. (2006) · The American Statistician · doi:10.1198/000313006x152649 · Cited by 1,407
Policy: NIH plans to enhance reproducibility
Collins, F. S., & Tabak, L. A. (2014) · Nature · doi:10.1038/505612a · Cited by 1,247
Research works printed but not linked: 2
- S. N. Goodman J. Gerson Mechanistic Evidence in Evidence-Based Medicine: A Conceptual Framework (Agency for Healthcare Research and Quality Rockville MD 2013); http://www.ncbi.nlm.nih.gov/books/NBK154584/.
- E. E. Leamer Specification Searches: Ad Hoc Inference with Nonexperimental Data (Wiley Hoboken NJ 1978).
Cited by
The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000410 · Cited by 5,939
The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · Experimental Physiology · doi:10.1113/ep088870 · Cited by 3,497
The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research*
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1177/0271678x20943823 · Cited by 1,443
“Why good reporting is important In recent years, concerns about the reproducibility of research findings have been raised by scientists, funders, research users, and policy makers [1, 2].”
CONSORT 2025 statement: updated guideline for reporting randomised trials
Hopewell, S., Chan, A.-W., Collins, G. S., Hróbjartsson, A., Moher, D., Schulz, K. F., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081123 · Cited by 1,195
“50 We encourage trial authors to detail what was done and to acknowledge if something was not done or was modified, ensuring alignment of information with that reported in the trial protocol, statistical analysis plan, and trial registry.”
Self-citationThe ARRIVE guidelines 2.0: Updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · British Journal of Pharmacology · doi:10.1111/bph.15193 · Cited by 1,081
The ARRIVE guidelines 2019: updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/703181 · Cited by 1,036
Machine Learning Approaches for Clinical Psychology and Psychiatry
Dwyer, D. B., Falkai, P., & Koutsouleris, N. (2018) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-032816-045037 · Cited by 966
A standard protocol for reporting species distribution models
Zurell, D., Franklin, J., König, C., Bouchet, P. J., Dormann, C. F., Elith, J., Fandos, G., Feng, X., Guillera‐Arroita, G., Guisan, A., et al. (2020) · Ecography · doi:10.1111/ecog.04960 · Cited by 945
“In the the lexicon of research reproducibility (Goodman et al. 2016), methods reproducibility means that sufficient details are provided on data and methods in order to independently repeat the study, while results reproducibility means that the same results can be obtained from an independent…”
MethodA solution to minimum sample size for regressions
Jenkins, D. G., & Quintana-Ascencio, P. F. (2020) · PLOS ONE · doi:10.1371/journal.pone.0229345 · Cited by 665
“Small N has been discussed as one part of a larger reproducibility problem, where recommended solutions include registered studies, conflict of interest statements, and publication of data [11,13,14,15,16,17].”
BackgroundThe ARRIVE guidelines 2.0: updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · The Journal of Physiology · doi:10.1113/jp280389 · Cited by 593
“In recent years, concerns about the reproducibility of research findings have been raised by scientists, funders, research users, and policy makers (Begley & Ioannidis, 2015, Goodman et al. 2016).”
BackgroundCONSORT 2025 statement: updated guideline for reporting randomised trials
Hopewell, S., Chan, A.-W., Collins, G. S., Hróbjartsson, A., Moher, D., Schulz, K. F., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., et al. (2025) · The Lancet · doi:10.1016/s0140-6736(25)00672-5 · Cited by 510
Self-citationVollmer, S., Mateen, B. A., Bohner, G., Király, F. J., Ghani, R., Jonsson, P., Cumbers, S., Jonas, A., McAllister, K. S. L., Myles, P., et al. (2020) · BMJ · doi:10.1136/bmj.l6927 · Cited by 497
“…is associated with results reproducibility (that is, the production of corroborating results in a new study, having followed the same experimental methods 65 ), whether it is a consequence of the inadequate reporting observed in the modelling literature, 59 or other related issues is unclear.”
ResultSelf-citationThe ARRIVE guidelines 2.0: Updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · BMC Veterinary Research · doi:10.1186/s12917-020-02451-y · Cited by 496
“Why good reporting is important In recent years, concerns about the reproducibility of research findings have been raised by scientists, funders, research users, and policy makers [1, 2].”
On doing better science: From thrill of discovery to policy implications
Antonakis, J. (2017) · The Leadership Quarterly · doi:10.1016/j.leaqua.2017.01.006 · Cited by 475
What You See Is What You Get? Enhancing Methodological Transparency in Management Research
Aguinis, H., Ramani, R. S., & Alabduljader, N. (2018) · Academy of Management Annals · doi:10.5465/annals.2016.0011 · Cited by 463
Elhassan, Y. S., Kluckova, K., Fletcher, R. S., Schmidt, M. S., Garten, A., Doig, C. L., Cartwright, D. M., Oakey, L., Burley, C. V., Jenkinson, N., et al. (2019) · Cell Reports · doi:10.1016/j.celrep.2019.07.043 · Cited by 448
Pratt, M. G., Kaplan, S., & Whittington, R. (2020) · Administrative Science Quarterly · doi:10.1177/0001839219887663 · Cited by 458
“…the claims made are in fact true by replicating the study (i.e., conducting the same study but with a new sample/dataset) or by attempting to reproduce identical findings using the same data and procedures as the initial study (Ketokivi and Mantere, 2010; Goodman, Fanelli, and Ioannidis, 2016).”
ResultIs science really facing a reproducibility crisis, and do we need it to?
Fanelli, D. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708272114 · Cited by 416
Self-citationReproducibility vs. Replicability: A Brief History of a Confused Terminology
Plesser, H. E. (2018) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2017.00076 · Cited by 424
“To solve the terminology confusion, Goodman et al. (2016) propose a new lexicon for research reproducibility with the following definitions: • Methods reproducibility: provide sufficient detail about procedures and data so that the same procedures could be exactly repeated.”
BackgroundA systematic approach to quantitative Western blot analysis
Pillai-Kastoori, L., Schutz-Geschwender, A. R., & Harford, J. A. (2020) · Analytical Biochemistry · doi:10.1016/j.ab.2020.113608 · Cited by 407
CONSORT 2025 Statement: Updated Guideline for Reporting Randomized Trials
Hopewell, S., Chan, A.-W., Collins, G. S., Hróbjartsson, A., Moher, D., Schulz, K. F., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., et al. (2025) · JAMA · doi:10.1001/jama.2025.4347 · Cited by 369
Self-citationSwedberg, R. (2020) · The Production of Knowledge · doi:10.1017/9781108762519.002 · Cited by 293
Reproducibility of animal research in light of biological variation
Voelkl, B., Altman, N. S., Forsman, A., Forstmeier, W., Gurevitch, J., Jaric, I., Karp, N. A., Kas, M. J., Schielzeth, H., Van de Casteele, T., et al. (2020) · Nature Reviews Neuroscience · doi:10.1038/s41583-020-0313-3 · Cited by 390
Rubin, M. (2021) · Synthese · doi:10.1007/s11229-021-03276-4 · Cited by 371
Effects of tDCS on motor learning and memory formation: A consensus and critical position paper
Buch, E. R., Santarnecchi, E., Antal, A., Born, J., Celnik, P. A., Classen, J., Gerloff, C., Hallett, M., Hummel, F. C., Nitsche, M. A., et al. (2017) · Clinical Neurophysiology · doi:10.1016/j.clinph.2017.01.004 · Cited by 367
A translational framework to DELIVER nanomedicines to the clinic
Joyce, P., Allen, C. J., Alonso, M. J., Ashford, M., Bradbury, M. S., Germain, M., Kavallaris, M., Langer, R., Lammers, T., Peracchia, M. T., et al. (2024) · Nature Nanotechnology · doi:10.1038/s41565-024-01754-7 · Cited by 333
O. B. O. T. C.-S. G., Grant, S., Mayo-Wilson, E., Montgomery, P., Macdonald, G., Michie, S., Hopewell, S., & Moher, D. (2018) · Trials · doi:10.1186/s13063-018-2735-z · Cited by 348
“Item 3a: description of trial design (such as parallel, factorial) including allocation ratio Unambiguous details about trial design help readers assess the suitability of trial methods for addressing trial objectives [46, 57], and clear and transparent reporting of all design features of a trial facilitates reproduci…”
BackgroundHardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C., Hofelich Mohr, A., Clayton, E., Yoon, E. J., Henry Tessler, M., et al. (2018) · Royal Society Open Science · doi:10.1098/rsos.180448 · Cited by 336
“In total, we attempted to reproduce 1324 individual values and, after author assistance was provided, 64 major numerical errors (5%, CI [4, 6]) and two insufficient information errors remained.”
MethodThe importance of open data and software: Is energy research lagging behind?
Pfenninger, S., DeCarolis, J., Hirth, L., Quoilin, S., & Staffell, I. (2017) · Energy Policy · doi:10.1016/j.enpol.2016.11.046 · Cited by 335
“A balanced societal and political debate requires transparent arguments based on scientific justifications, but escalating concern about reproducibility in some fields is shaking public confidence in scientific research (Goodman et al., 2016).”
BackgroundCheng, C.-W., Wu, T.-X., Shang, H.-C., Li, Y.-P., Altman, D. G., Moher, D., Bian, Z.-X., & for the CONSORT-CHM Formulas 2017 Group* (2017) · Annals of Internal Medicine · doi:10.7326/m16-2977 · Cited by 323
The earth is flat ( p > 0.05): significance thresholds and the crisis of unreplicable research
Amrhein, V., Korner-Nievergelt, F., & Roth, T. (2017) · PeerJ · doi:10.7717/peerj.3544 · Cited by 308
Critical Issues for the Translation of Cardioprotection
Heusch, G. (2017) · Circulation Research · doi:10.1161/circresaha.117.310820 · Cited by 293
State of the Art: Reproducibility in Artificial Intelligence
Gundersen, O. E., & Kjensmo, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11503 · Cited by 293
“Replication, as used by (Drummond 2009) and (Stodden 2011), is in line with methods reproducibility as proposed by (Goodman et al. 2016) while reproducibility seems to entail both results reproducibility and inferential reproducibility.”
BackgroundMethodInfluentialCONSORT 2025 statement: updated guideline for reporting randomized trials
Hopewell, S., Chan, A.-W., Collins, G. S., Hróbjartsson, A., Moher, D., Schulz, K. F., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03635-5 · Cited by 283
Self-citationEvaluating Recommender Systems: Survey and Framework
Zangerle, E., & Bauer, C. (2023) · ACM Computing Surveys · doi:10.1145/3556536 · Cited by 283
“Reproducibility is a minimum necessary condition for a inding to be believable and informative” [91].”
BackgroundResultSchloss, P. D. (2018) · mBio · doi:10.1128/mbio.00525-18 · Cited by 280
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · BMJ Open Science · doi:10.1136/bmjos-2020-100115 · Cited by 273
Basic & Clinical Pharmacology & Toxicology Policy for Experimental and Clinical studies
Tveden‐Nyborg, P., Bergmann, T. K., & Lykkesfeldt, J. (2018) · Basic & Clinical Pharmacology & Toxicology · doi:10.1111/bcpt.13059 · Cited by 271
Wallach, J. D., Boyack, K. W., & Ioannidis, J. P. A. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2006930 · Cited by 269
Self-citationArtificial intelligence in cancer research, diagnosis and therapy
Elemento, O., Leslie, C., Lundin, J., & Tourassi, G. (2021) · Nature Reviews Cancer · doi:10.1038/s41568-021-00399-1 · Cited by 264
“Such a framework will help us monitor the collected data for potential biases and for measuring reproducibility and repeatability based on statistically and clinically appropriate standards 33,34 .”
MethodReproducibility of preclinical animal research improves with heterogeneity of study samples
Voelkl, B., Vogt, L., Sena, E. S., & Würbel, H. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2003693 · Cited by 262
CONSORT 2010 statement: extension checklist for reporting within person randomised trials
Pandis, N., Chung, B., Scherer, R. W., Elbourne, D., & Altman, D. G. (2017) · BMJ · doi:10.1136/bmj.j2835 · Cited by 257
Inherited Retinal Degenerations: Current Landscape and Knowledge Gaps
Duncan, J. L., Pierce, E. A., Laster, A. M., Daiger, S. P., Birch, D. G., Ash, J. D., Iannaccone, A., Flannery, J. G., Sahel, J. A., Zack, D. J., et al. (2018) · Translational Vision Science & Technology · doi:10.1167/tvst.7.4.6 · Cited by 257
A Practical Guide for Transparency in Psychological Science
Klein, O., Hardwicke, T. E., Aust, F., Breuer, J., Danielsson, H., Mohr, A. H., IJzerman, H., Nilsonne, G., Vanpaemel, W., & Frank, M. C. (2018) · Collabra: Psychology · doi:10.1525/collabra.158 · Cited by 233
“Across statistical frameworks, conducting multiple tests and then selectively reporting only a subset may lead to improper and ungeneralisable conclusions (Goodman et al., 2016; Wasserstein & Lazar, 2016).”
MethodNaudet, F., Sakarovitch, C., Janiaud, P., Cristea, I., Fanelli, D., Moher, D., & Ioannidis, J. P. A. (2018) · BMJ · doi:10.1136/bmj.k400 · Cited by 227
Self-citationUK phenomics platform for developing and validating electronic health record phenotypes: CALIBER
Denaxas, S., Gonzalez-Izquierdo, A., Direk, K., Fitzpatrick, N. K., Fatemifar, G., Banerjee, A., Dobson, R. J. B., Howe, L. J., Kuan, V., Lumbers, R. T., et al. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz105 · Cited by 225
Standardized multi-omics of Earth’s microbiomes reveals microbial and metabolite diversity
Shaffer, J. P., Nothias, L.-F., Thompson, L. R., Sanders, J. G., Salido, R. A., Couvillion, S. P., Brejnrod, A. D., Lejzerowicz, F., Haiminen, N., Huang, S., et al. (2022) · Nature Microbiology · doi:10.1038/s41564-022-01266-x · Cited by 220
Reproducibility and replicability of rodent phenotyping in preclinical studies
Kafkafi, N., Agassi, J., Chesler, E. J., Crabbe, J. C., Crusio, W. E., Eilam, D., Gerlai, R., Golani, I., Gomez-Marin, A., Heller, R., et al. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.01.003 · Cited by 215
“Ironically, there is currently no scientific consensus even over the name of the problem and the meaning of basic terms, confusing the discussion even further (Goodman et al., 2016).”
Klemm, C., & Vennemann, P. (2021) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2020.110206 · Cited by 208
Automation in the Life Science Research Laboratory
Holland, I., & Davies, J. A. (2020) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2020.571777 · Cited by 200
The case for formal methodology in scientific reform
Devezer, B., Navarro, D. J., Vandekerckhove, J., & Ozge Buzbas, E. (2021) · Royal Society Open Science · doi:10.1098/rsos.200805 · Cited by 188
“We limit our discussion to statistical reproducibility of results only (similar to results reproducibility in [26]), and exclude other types such as computational or methods reproducibility—whether the materials, methods, procedures, algorithms, analyses used in an original study are reported in a sufficiently detaile…”
ResultThe case for formal methodology in scientific reform
Devezer, B., Navarro, D. J., Vandekerckhove, J., & Buzbas, E. O. (2020) · doi:10.1101/2020.04.26.048306 · Cited by 195
Standardized and reproducible measurement of decision-making in mice
Laboratory, T. I. B., Aguillon-Rodriguez, V., Angelaki, D., Bayer, H., Bonacchi, N., Carandini, M., Cazettes, F., Chapuis, G., Churchland, A. K., Dan, Y., et al. (2021) · eLife · doi:10.7554/elife.63711 · Cited by 189
Children's Reading Comprehension Difficulties
Cain, K. (2022) · The Science of Reading · doi:10.1002/9781119705116.ch14 · Cited by 187
Comprehensive multi-center assessment of small RNA-seq methods for quantitative miRNA profiling
Giraldez, M. D., Spengler, R. M., Etheridge, A., Godoy, P. M., Barczak, A. J., Srinivasan, S., De Hoff, P. L., Tanriverdi, K., Courtright, A., Lu, S., et al. (2018) · Nature Biotechnology · doi:10.1038/nbt.4183 · Cited by 177
Cohen-Boulakia, S., Belhajjame, K., Collin, O., Chopard, J., Froidevaux, C., Gaignard, A., Hinsen, K., Larmande, P., Bras, Y. L., Lemoine, F., et al. (2017) · Future Generation Computer Systems · doi:10.1016/j.future.2017.01.012 · Cited by 181
“We introduce the terminology we will follow in this paper based a review of the various definitions provided in the literature [2, 30, 17, 31, 16, 32].”
MethodAssessing data availability and research reproducibility in hydrology and water resources
Stagge, J. H., Rosenberg, D. E., Abdallah, A. M., Akbar, H., Attallah, N. A., & James, R. (2019) · Scientific Data · doi:10.1038/sdata.2019.30 · Cited by 165
Gerstner, K., Moreno‐Mateos, D., Gurevitch, J., Beckmann, M., Kambach, S., Jones, H. P., & Seppelt, R. (2017) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12758 · Cited by 159
A systematic survey on multi-step attack detection
Navarro, J., Deruyver, A., & Parrend, P. (2018) · Computers & Security · doi:10.1016/j.cose.2018.03.001 · Cited by 158
A Systematic Review of Gesture Elicitation Studies: What Can We Learn from 216 Studies?
Villarreal-Narvaez, S., Vanderdonckt, J. M., Vatavu, R.-D., & Wobbrock, J. O. (2020) · Designing Interactive Systems Conference · doi:10.1145/3357236.3395511 · Cited by 171
“An explicit definition of the GES context of use is the cornerstone for reproducible research [41], defined by ACM via three badges: repeatability (same team, same experimental setup), replicability (different team with the same experimental setup), and reproducibility (different team and a different…”
MethodApplications of Google Earth Engine in fluvial geomorphology for detecting river channel change
Boothroyd, R. J., Williams, R. D., Hoey, T. B., Barrett, B., & Prasojo, O. A. (2021) · WIREs Water · doi:10.1002/wat2.1496 · Cited by 140
“Reporting this important methodological information would improve the future transparency, methods reproducibility, and completeness of analytic reporting (Goodman, Fanelli, & Ioannidis, 2016).”
BackgroundEvaluating the Use of Nonrandomized Real‐World Data Analyses for Regulatory Decision Making
Franklin, J. M., Glynn, R. J., Martin, D., & Schneeweiß, S. (2019) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.1351 · Cited by 159
“Direct replicability the ability to exactly reproduce an analysis in a given dataset based on reporting of the methods without accompanying programming code(47), is distinct from reproducibility, which assumes programming code is available, as well as conceptual replicability, which is focused on whether findings repl…”
BackgroundGuiding questions to avoid data leakage in biological machine learning applications
Bernett, J., Blumenthal, D. B., Grimm, D. G., Haselbeck, F., Joeres, R., Kalinina, O. V., & List, M. (2024) · Nature Methods · doi:10.1038/s41592-024-02362-y · Cited by 223
Methods to increase reproducibility in differential gene expression via meta-analysis
Sweeney, T. E., Haynes, W. A., Vallania, F., Ioannidis, J. P. A., & Khatri, P. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkw797 · Cited by 146
“As previously defined (2,8), our silver standard measures results reproducibility as defined by replication validity.”
BackgroundSelf-citationMayo-Wilson, E., Fusco, N., Li, T., Hong, H., Canner, J. K., & Dickersin, K. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.05.007 · Cited by 131
Why imaging data alone is not enough: AI-based integration of imaging, omics, and clinical data
Holzinger, A., Haibe‐Kains, B., & Jurišica, I. (2019) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-019-04382-9 · Cited by 143
Assessment of transparency indicators across the biomedical literature: How open is open?
Serghiou, S., Contopoulos-Ioannidis, D. G., Boyack, K. W., Riedel, N., Wallach, J. D., & Ioannidis, J. P. A. (2021) · PLOS Biology · doi:10.1371/journal.pbio.3001107 · Cited by 139
“Research reproducibility [1] is a fundamental tenet of science, yet recent reports suggest that reproducibility of published findings should not be taken for granted [2].”
BackgroundSelf-citationReproducibility in systems biology modelling
Tiwari, K., Kananathan, S., Roberts, M. G., Meyer, J. P., Sharif Shohan, M. U., Xavier, A., Maire, M., Zyoud, A., Men, J., Ng, S., et al. (2021) · Molecular Systems Biology · doi:10.15252/msb.20209982 · Cited by 131
“Although the lack of reproducibility has been discussed within the community (Mendes, 2018; Papin et al , 2020), it was not expected to be this adverse.”
The reproducibility crisis in the age of digital medicine
Stupple, A., Singerman, D., & Celi, L. A. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0079-z · Cited by 137
“” For our purposes here, we take reproducibility to mean “obtaining the same results from the conduct of an independent study whose procedures are as closely matched to the original experiment as possible” (also known as “research reproducibility” or simply “replicability”).(3) A sense of crisis itself began with the…”
MethodHan, S., Olonisakin, T. F., Pribis, J. P., Zupetic, J., Yoon, J. H., Holleran, K. M., Jeong, K., Shaikh, N., Rubio, D. M., & Lee, J. S. (2017) · PLOS ONE · doi:10.1371/journal.pone.0183591 · Cited by 131
An update to SPIRIT and CONSORT reporting guidelines to enhance transparency in randomized trials
Hopewell, S., Boutron, I., Chan, A.-W., Collins, G. S., de Beyer, J. A., Hróbjartsson, A., Nejstgaard, C. H., Østengaard, L., Schulz, K. F., Tunn, R., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01989-8 · Cited by 127
Promises and pitfalls of data sharing in qualitative research
Tsai, A. C., Kohrt, B. A., Matthews, L. T., Betancourt, T. S., Lee, J. K., Papachristos, A. V., Weiser, S. D., & Dworkin, S. L. (2016) · Social Science & Medicine · doi:10.1016/j.socscimed.2016.08.004 · Cited by 196
“This definition corresponds to the concept of “results reproducibility” suggested by Goodman et al. (2016).”
BackgroundThe Production of Knowledge: Enhancing Progress in Social Science
(2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519 · Cited by 240
CONSORT 2025 statement: Updated guideline for reporting randomised trials
Hopewell, S., Chan, A.-W., Collins, G. S., Hróbjartsson, A., Moher, D., Schulz, K. F., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., et al. (2025) · PLOS Medicine · doi:10.1371/journal.pmed.1004587 · Cited by 139
Self-citationPromoting Open Science to Increase the Trustworthiness of Evidence in Special Education
Adelson, J. L., Barton, E., Bradshaw, C., Nosek, B. A., Bryant, D., Cook, B. G., Coyne, M., DeBettencourt, L., DeHaven, A. C., Dymond, S., et al. (2018) · Exceptional Children · doi:10.1177/0014402918793138 · Cited by 143
Marek, S. A., Tervo‐Clemmens, B., Nielsen, A. N., Wheelock, M. D., Miller, R. L., Laumann, T. O., Earl, E., Foran, W., Cordova, M., Doyle, O. J., et al. (2019) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2019.100706 · Cited by 138
“Leveraging data collected from the large 21-site developmental ABCD study, the current project provides a framework of analytic approaches to quantify results reproducibility (hereafter referred to as reproducibility for brevity; see (Goodman et al., 2016)) in RSFC, which is critical for studies examining neurodevelop…”
MethodSofia, G. (2020) · Geomorphology · doi:10.1016/j.geomorph.2020.107055 · Cited by 142
The Experimental Design Assistant
Percie du Sert, N., Bamsey, I., Bate, S. T., Berdoy, M., Clark, R. A., Cuthill, I., Fry, D., Karp, N. A., Macleod, M., Moon, L., et al. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2003779 · Cited by 118
Reproducibility in machine‐learning‐based research: Overview, barriers, and drivers
Semmelrock, H., Ross‐Hellauer, T., Kopeinik, S., Theiler, D., Haberl, A., Thalmann, S., & Kowald, D. (2025) · AI Magazine · doi:10.1002/aaai.70002 · Cited by 118
Never waste a good crisis: confronting reproducibility in translational research
Drucker, D. J. (2016) · Cell Metabolism · doi:10.1016/j.cmet.2016.08.006 · Cited by 117
Adolescent Brain Cognitive Development (ABCD) Community MRI Collection and UtilitiesPreprint
Feczko, E., Conan, G., Marek, S. A., Tervo‐Clemmens, B., Cordova, M., Doyle, O. J., Earl, E., Perrone, A. J., Sturgeon, D., Klein, R., et al. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.07.09.451638 · Cited by 137
“The Brain Imaging Data Structure (BIDS) provides neuroimaging researchers with a standard for organizing and describing MRI datasets (Gorgolewski et al., 2016).”
BackgroundMethodReproducible research practices are underused in systematic reviews of biomedical interventions
Page, M. J., Altman, D. G., Shamseer, L., McKenzie, J. E., Ahmadzai, N., Wolfe, D., Yazdi, F., Catalá-López, F., Tricco, A. C., & Moher, D. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.10.017 · Cited by 115
“Such practices include providing a detailed description of the data collected and used for analysis, along with descriptive metadata, clearly reporting the analysis methods and results, and sharing the data set and statistical code used to perform analyses [1,4].”
MethodHow to Crack Pre-registration: Toward Transparent and Open Science
Yamada, Y. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.01831 · Cited by 122
“One such method is selective reporting, which is a type of data fabrication in which data that do not support the hypothesis are not reported (Goodman et al., 2016).”
MethodChen, X., Dallmeier-Tiessen, S., Dasler, R., Feger, S. S., Fokianos, P., Gonzalez, J. B., Hirvonsalo, H., Kousidis, D., Lavasa, A., Mele, S., et al. (2018) · Nature Physics · doi:10.1038/s41567-018-0342-2 · Cited by 114
Everything Matters: The ReproNim Perspective on Reproducible Neuroimaging
Kennedy, D. N., Abraham, S. A., Bates, J. F., Crowley, A., Ghosh, S., Gillespie, T., Goncalves, M., Grethe, J. S., Halchenko, Y. O., Hanke, M., et al. (2019) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2019.00001 · Cited by 113
Reporting guidelines: doing better for readers
Moher, D. (2018) · BMC Medicine · doi:10.1186/s12916-018-1226-0 · Cited by 107
Measuring behavior in the home cage: study design, applications, challenges, and perspectives
Grieco, F., Bernstein, B. J., Biemans, B., Bikovski, L., Burnett, C. J., Cushman, J. D., van Dam, E. A., Fry, S. A., Richmond-Hacham, B., Homberg, J. R., et al. (2021) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2021.735387 · Cited by 109
“, the ability of researchers to duplicate the results of a prior study (Goodman et al., 2016) is a growing concern in preclinical behavioral sciences (Steckler et al.”
BackgroundPackaging Data Analytical Work Reproducibly Using R (and Friends)
Marwick, B., Boettiger, C., & Mullen, L. A. (2017) · The American Statistician · doi:10.1080/00031305.2017.1375986 · Cited by 123
“This has resulted in extensive discussion of how to define reproducibility, and how to enhance it across diverse research contexts (Stodden et al. 2016; Goodman, Fanelli, and Ioannidis 2016; Marwick 2017).”
BackgroundImproving natural product research translation: From source to clinical trial
Sorkin, B. C., Kuszak, A. J., Bloss, G., Fukagawa, N. K., Hoffman, F. A., Jafari, M., Barrett, B. P., Brown, P. N., Bushman, F. D., Casper, S. J., et al. (2019) · The FASEB Journal · doi:10.1096/fj.201902143r · Cited by 121
Reproducibility and Rigor in Animal-Based Research
Macleod, M., & Mohan, S. (2019) · ILAR Journal · doi:10.1093/ilar/ilz015 · Cited by 104
“Articulating this broader definition, and borrowing from Goodman [1], one might consider a hierarchy of characteristics that might give confidence in the ‘truth’ of a research finding: first, ‘reproducibility’ as originally described, based on reanalysis of an existing dataset (‘reproducibility of analysis’); secondly…”
BackgroundInfluentialDeep learning in pulmonary nodule detection and segmentation: a systematic review
Gao, C., Wu, L., Wu, W., Huang, Y., Wang, X., Sun, Z., Xu, M., & Gao, C. (2024) · European Radiology · doi:10.1007/s00330-024-10907-0 · Cited by 129
“Public datasets offer abundant annotated data for training deep-learning models and enable researchers to validate and compare algorithms, enhancing research reproducibility [56, 57].”
Calibrating the Scientific Ecosystem Through Meta-Research
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98
“Goodman et al. (2016) proposed one framework that delineates methods reproducibility (obtaining similar results given the same data and analytical tools; often called analytic reproducibility or computational reproducibility), results reproducibility (obtaining similar results given the same…”
Self-citationReproducibility2020: Progress and priorities
Freedman, L. P., Venugopalan, G., & Wisman, R. (2017) · F1000Research · doi:10.12688/f1000research.11334.1 · Cited by 102
“The Report draws upon the definitions promulgated by the framework proposed by Goodman et al.[7]: “methods reproducibility” refers to the complete and transparent reporting of information required for another researcher to repeat protocols and analytical methods; “results reproducibility” refers to independent attempt…”
MethodNaldi, A., Hernandez, C., Levy, N., Stoll, G., Monteiro, P. T., Chaouiya, C., Helikar, T., Zinovyev, A. Y., Calzone, L., Cohen‐Boulakia, S., et al. (2018) · Frontiers in Physiology · doi:10.3389/fphys.2018.00680 · Cited by 113
Neural stimulation and recording performance in human sensorimotor cortex over 1500 days
Hughes, C. L., Flesher, S. N., Weiss, J. M., Downey, J. E., Boninger, M. L., Collinger, J. L., & Gaunt, R. A. (2021) · Journal of Neural Engineering · doi:10.1088/1741-2552/ac18ad · Cited by 106
Hadjiiski, L. M., Cha, K., Chan, H., Drukker, K., Morra, L., Näppi, J. J., Sahiner, B., Yoshida, H., Chen, Q., Deserno, T. M., et al. (2022) · Medical Physics · doi:10.1002/mp.16188 · Cited by 122
Progress toward openness, transparency, and reproducibility in cognitive neuroscience
Gilmore, R. O., Diaz, M. T., Wyble, B. A., & Yarkoni, T. (2017) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.13325 · Cited by 95
A survey of gene expression meta-analysis: methods and applications
Toro‐Domínguez, D., Villatoro-García, J. A., Martorell‐Marugán, J., Montoya, Y. R., Alarcón-Riquelme, M. E., & Carmona‐Sáez, P. (2020) · Briefings in Bioinformatics · doi:10.1093/bib/bbaa019 · Cited by 114
Perception of microstimulation frequency in human somatosensory cortex
Hughes, C. L., Flesher, S. N., Weiss, J. M., Boninger, M. L., Collinger, J. L., & Gaunt, R. A. (2021) · eLife · doi:10.7554/elife.65128 · Cited by 99
Marwick, B., Barton, C. M., Bates, L., Bollwerk, E., Bocinsky, R. K., Carter, A. K., Conrad, C., Costa, S., Crema, E. R., Davies, B., et al. (2017) · doi:10.31235/osf.io/72n8g · Cited by 17
Metadata
- Authors
- Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A.
- Affiliations
- Stanford University (US)
- Corresponding author
- Goodman, Steven N.
- Venue
- Science Translational Medicine
- Published
- 2016-06
- Volume
- 8
- Issue
- 341
- Pages
- 341ps12-341ps12
- ISSN
- 1946-6234 (ISSN-L)
- 1946-6242
- OpenAlex
- W2414283705
- Cited by
- 1,293
- References
- 41
- Citations by year
- 2026: 85 · 2025: 93 · 2024: 146 · 2023: 110 · 2022: 174 · 2021: 151 · 2020: 168 · 2019: 140 · 2018: 112 · 2017: 83 · 2016: 30
- FWCI
- 18.01
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science Translational Medicine (publishedVersion)
- PubMed (publishedVersion)
- Funders
- John and Laura Arnold Foundation
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- research reproducibility, terminology, conceptual framework, truth in science
- MeSH terms
- Reproducibility of Results, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.