The DOI matches its Crossref record; 38 of the 41 references it cites were found in a registry or an index.
What does research reproducibility mean?
Abstract
The language and conceptual framework of “research reproducibility” are nonstandard and unsettled across the sciences.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“High quality reporting is an important step when considering issues related to reproducibility [50].”
From CONSORT 2025 statement: Updated guideline for reporting randomised trials (2025) · cites this paper as [50]
In the citing paper
“These factors are inherent to modern computational pipelines in deep learning (Desai et al., 2025; Goodman et al., 2016; Semmelrock et al., 2025)”
From CiteFusion: an ensemble framework for citation intent classification harnessing dual-model binary couples and SHAP analyses (2025) · cites this paper as Goodman et al., 2016
In the citing paper
“for reasons that have nothing to do with questionable research and publication practices (28).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
In the citing paper
“but the reproducibility of results and inferences is likely to be lower and to vary across subfields and methodologies”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
In the citing paper
“the reproducibility of research methods can in principle be expected to be 100%”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“The classic file-drawer publication bias problem (wherein nonsignificant or “uninteresting” results are not published) (29) results in bias under the assumption that multiple studies are being produced independently but a biased sample is published.”
BackgroundEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Pseudoreplication and the Design of Ecological Field Experiments
Hurlbert, S. H. (1984) · Ecological Monographs · doi:10.2307/1942661 · Cited by 8,377
The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2016 update
Afgan, E., Baker, D., van den Beek, M., Blankenberg, D., Bouvier, D., Čech, M., Chilton, J., Clements, D., Coraor, N., Eberhard, C., et al. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkw343 · Cited by 2,148
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
A Reality Check for Data Snooping
White, H. (2000) · Econometrica · doi:10.1111/1468-0262.00152 · Cited by 1,851
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500
The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant
Gelman, A., & Stern, H. (2006) · The American Statistician · doi:10.1198/000313006x152649 · Cited by 1,407
Policy: NIH plans to enhance reproducibility
Collins, F. S., & Tabak, L. A. (2014) · Nature · doi:10.1038/505612a · Cited by 1,247
Research works printed but not linked: 2
- S. N. Goodman J. Gerson Mechanistic Evidence in Evidence-Based Medicine: A Conceptual Framework (Agency for Healthcare Research and Quality Rockville MD 2013); http://www.ncbi.nlm.nih.gov/books/NBK154584/.
- E. E. Leamer Specification Searches: Ad Hoc Inference with Nonexperimental Data (Wiley Hoboken NJ 1978).
Cited by
Items 701–800 of 1,003
Howell, E. L. (2020) · Harvard Data Science Review · doi:10.1162/99608f92.f2823096 · Cited by 9
Hanin, L. G. (2021) · Mathematics · doi:10.3390/math9060603 · Cited by 6
“Reproducibility of research findings is also adversely affected by insufficient standardi‐ zation of research protocols and inadequate reporting of experimental procedures [10].”
BackgroundPinto, W. G., Alguacil, A., & Bauerheim, M. (2021) · Data-Centric Engineering · doi:10.1017/dce.2022.18 · Cited by 6
Reproducible manuscript preparation with RMarkdown application to JMSACL and other Elsevier Journals
Holmes, D. T., Mobini, M., & McCudden, C. R. (2021) · Journal of Mass Spectrometry and Advances in the Clinical Lab · doi:10.1016/j.jmsacl.2021.09.002 · Cited by 6
Gerring, J., Mahoney, J., & Elman, C. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.001 · Cited by 6
OncoPubMiner: a platform for mining oncology publications
Xu, Q., Liu, Y., Hu, J., Duan, X., Song, N., Zhou, J., Zhai, J.-C., Su, J., Liu, S., Chen, F., et al. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac383 · Cited by 5
Stopsack, K. H., Mucci, L. A., Tworoger, S. S., Kang, J. H., Eliassen, A. H., Willett, W. C., & Stampfer, M. J. (2023) · Epidemiology · doi:10.1097/ede.0000000000001599 · Cited by 6
Creating Interactive Scientific Publications using Bindings
Konkol, M., Kray, C., & Suleiman, J. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3331158 · Cited by 8
Ethical implementation of precision psychiatry
Ball, T. M., Kalinowski, A., & Williams, L. M. (2019) · Personalized Medicine in Psychiatry · doi:10.1016/j.pmip.2019.05.003 · Cited by 5
Role of Replication Research in Biostatistics Graduate Education
Smith, L. M., Yu, F., & Schmid, K. K. (2020) · Journal of Statistics and Data Science Education · doi:10.1080/10691898.2020.1844105 · Cited by 9
UiLab, a Workbench for Conducting and Reproducing Experiments in GUI Visual Design
Burny, N., & Vanderdonckt, J. M. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3457143 · Cited by 5
“Following RepliCHI [68, 69], the term “replicability” has been widely adopted in HCI where “reproducibility” has been used for the same purpose in general science [24].”
BackgroundMethodResultApplication of Pre-Trained Deep Learning Models for Clinical ECGs
Bender, T., Seidler, T., Bengel, P., Sax, U., & Krefting, D. (2021) · Studies in health technology and informatics · doi:10.3233/shti210539 · Cited by 4
Self-Scaling Clusters and Reproducible Containers to Enable Scientific Computing
Vaillancourt, P. Z., Coulter, J. E., Knepper, R., & Barker, B. (2020) · IEEE High Performance Extreme Computing Conference · doi:10.1109/hpec43674.2020.9286208 · Cited by 11
“as ”methods reproducibility”, ”results reproducibility”, and ”inferential reproducibility” [24].”
Foldes, S. T., Chandrasekaran, S., Camerone, J., Lowe, J., Ramdeo, R., Ebersole, J. S., & Bouton, C. E. (2021) · Frontiers in Neurology · doi:10.3389/fneur.2021.739693 · Cited by 5
Can Orthopaedics become the Gold Standard for Reproducibility? A Roadmap to SuccessPreprint
Fladie, I. A., Evans, S., Checketts, J. X., Tritz, D., Norris, B. L., & Vassar, M. B. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/715144 · Cited by 4
Lynnerup, N. A., Nolling, L., Andersen, R. H., & Hallam, J. C. T. (2019) · University of Southern Denmark Research Portal (University of Southern Denmark) · doi:10.48550/arxiv.1909.03772 · Cited by 7
“[27] argues that the definitions should be clear in principle but operationally elusive, why they provide many operational examples specific to certain scientific fields.”
BackgroundInfluentialThe Development, Function, and Plasticity of the Immune Macroenvironment in CancerPreprint
Allen, B. M., Hiam-Galvez, K. J., Burnett, C. E., Venida, A. C., DeBarge, R., Carmi, Y., & Spitzer, M. H. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/805473 · Cited by 5
Giraldez, M., Spengler, R., Etheridge, A., Godoy, P., Barczak, A., Srinivasan, S., De Hoff, P., Tanriverdi, K., Courtright, A., Lu, S., et al. (2017) · doi:10.1101/113050 · Cited by 5
Nanivadekar, A. C., Bose, R., Petersen, B. A., Okorokova, E. V., Sarma, D., Farooqui, J., Dalrymple, A. N., Levy, I., Helm, E. R., Mièle, V., et al. (2022) · medRxiv · doi:10.1101/2022.09.15.22279956 · Cited by 6
Implementing Code Review in the Scientific Workflow: Insights from Ecology and Evolutionary BiologyPreprint
Ivimey‐Cook, E. R., Pick, J. L., Bairos‐Novak, K. R., Čulina, A., Gould, E., Grainger, M., Marshall, B. M., Moreau, D., Paquet, M., Royauté, R., et al. (2023) · doi:10.32942/x2cg64 · Cited by 5
playOmics: A multi-omics pipeline for interpretable predictions and biomarker discoveryPreprint
Głowacka‐Walas, J., Sijko, K., Wojdan, K., & Gambin, T. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.03.12.584088 · Cited by 5
“Ensuring such consistency not only builds trust in the findings, but also facilitates scientific collaboration, allowing other researchers to validate and build upon existing work [8, 9].”
BackgroundLemaitre, J., Popelka, D., Ribotta, B., Westlake, H., Chakrabarti, S., Li, X., Hanson, M., Jiang, H.-B., Cara, F. D., Kurant, E., et al. (2026) · bioRxiv · doi:10.1101/2025.07.07.663460 · Cited by 4
“Thus, our conclusions do not directly challenge the initial results leading to a claim, but rather the general applicability (i.e. generalizability) of the claim itself (Goodman et al., 2016; Meng, 2020).”
Reproducibility in systems biology modelling
Tiwari, K., Kananathan, S., Roberts, M. G., Meyer, J. P., Sharif Shohan, M. U., Xavier, A., Maire, M., Zyoud, A., Men, J., Ng, S., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.08.07.239855 · Cited by 131
Improving research reproducibility in surgery: A narrative review on caveats in research methodology
Liukkonen, R. J., Vaajala, M., Kuitunen, I., Mattila, V. M., Ponkilainen, V., Uimonen, M., & Reito, A. R. (2025) · Scandinavian Journal of Surgery · doi:10.1177/14574969251371880 · Cited by 4
Challenges in applying the EU AI act research exemptions to contemporary AI research
Mészáros, J., Huys, I., & Ioannidis, J. P. A. (2026) · npj Digital Medicine · doi:10.1038/s41746-025-02263-0 · Cited by 4
Self-citationResponsible assessment of what research? Beware of epistemic diversity!
Ulpts, S. (2024) · Meta-Psychology · doi:10.15626/mp.2023.3797 · Cited by 6
“…Nosek et al. (2022), who the authors of the target articles cite, use the concept of process reproducibility, which is quite similar to what Goodman et al. (2016) call methods reproducibility, but there is also amorass of meanings of reproducibility that contradict each other (Barba, 2018).”
BackgroundLi, L., Liu, J., Zheng, Y., Shi, C., & Bai, W. (2025) · Depression and Anxiety · doi:10.1155/da/7645625 · Cited by 4
Chen, Y., Jarecka, D., Abraham, S. A., Gau, R., Ng, E., Low, D. M., Bevers, I., Johnson, A. E. W., Keshavan, A., Klein, A., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/63343 · Cited by 9
Williams, R., Bolton, T., Jenkins, D. A., Mizani, M. A., Sperrin, M., Sudlow, C. L. M., Wood, A., Heald, A. H., & Peek, N. B. (2025) · PLoS ONE · doi:10.1371/journal.pone.0326335 · Cited by 4
“Goodman et al. define three terms for discussing research reproducibility: methods reproducibility, results reproducibility and inferential reproducibility [5].”
ter Riet, G., Storosum, B. W. C., & Zwinderman, A. H. (2019) · F1000Research · doi:10.12688/f1000research.17615.1 · Cited by 4
Replication for Quantitative Research
Freese, J., & Peterson, D. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.010 · Cited by 4
Highland, H. M., Gamazon, E. R., & Below, J. E. (2021) · Circulation Research · doi:10.1161/circresaha.120.317360 · Cited by 4
A systematic review on reporting of refinement measures in mouse ECG telemetry implantation surgery
Gkrouzoudi, A., Tsingotjidou, A., & Jirkof, P. (2022) · Laboratory Animals · doi:10.1177/00236772221115492 · Cited by 4
Mericskay, B., & Demoraes, F. (2022) · Cybergeo · doi:10.4000/cybergeo.39583 · Cited by 4
Matarese, V. (2023) · Inquiry · doi:10.1080/0020174x.2023.2278032 · Cited by 1
Schenk, P. O., & Kern, C. A. (2024) · AStA Wirtschafts- und Sozialstatistisches Archiv · doi:10.1007/s11943-024-00344-2 · Cited by 16
“They all describe under which conditions the results from a new data analysis must be the same or qualitatively similar to those from a first data analysis (Goodman et al., 2016): Methods reproducibility is defined as obtaining identical results, using the same data and the same methods .”
ResultTrippas, J. R. (2024) · ACM eBooks · doi:10.1145/3674127.3674135 · Cited by 6
Cenci, J., Franco, M. C., Pereira‐Cenci, T., Corrêa, M. B., Helal, L. C. A. S., Moher, D., Bouter, L. M., Huysmans, M. C. D. N. J. M., & Cenci, M. S. (2024) · Brazilian Oral Research · doi:10.1590/1807-3107bor-2024.vol38.0135 · Cited by 6
The prevalence of gas exchange data processing methods: a semi-automated scoping review
Hesse, A. S., White, M., & Lundstrom, C. J. (2025) · International Journal of Sports Medicine · doi:10.1055/a-2495-5364 · Cited by 5
Millet, B., Harika‐Germaneau, G., Maatoug, R., Naudet, F., Reymann, J., Turmel, V., Batail, J.-M. V., Soulabaille, J., Jaafari, N., & Drapier, D. (2025) · PLoS ONE · doi:10.1371/journal.pone.0317597 · Cited by 6
“Thus, our definition of reproducibility includes both the concept of result reproducibility and generalizability, as outlined by Goodman et al. [27].”
Usefulness of Analytical Research: Rethinking Analytical R&D&T Strategies
Valcárcel, M. (2017) · Analytical Chemistry · doi:10.1021/acs.analchem.7b03935 · Cited by 4
Transparency and Reproducibility: Conceptualizing the Problem
Christensen, G., & Miguel, E. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.006 · Cited by 4
Measurement Replication in Qualitative and Quantitative Studies
Reiter, D. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.011 · Cited by 4
Evaluating Reproducibility and Transparency in Emergency Medicine Publications
Johnson, B. S., Rauh, S. L., Tritz, D., Schiesel, M., & Vassar, M. B. (2021) · Western Journal of Emergency Medicine · doi:10.5811/westjem.2021.3.50078 · Cited by 3
Situating Linguistics in the Social Science Data Movement
Gawne, L., & Styles, S. J. (2022) · The MIT Press eBooks · doi:10.7551/mitpress/12200.003.0006 · Cited by 11
Open Science: Recommendations for Research on School Bullying
Noret, N. F., Hunter, S. C., Pimenta, S. M., Taylor, R., & Johnson, R. (2022) · International Journal of Bullying Prevention · doi:10.1007/s42380-022-00130-0 · Cited by 7
“Replication reflects the ability of a researcher to duplicate the results of a prior study with new data (Goodman et al., 2018).”
The Role of Statistics in Promoting Data Reusability and Research Transparency
Nusser, S. M. (2022) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-033121-105114 · Cited by 4
Ultrahigh evaporative heat transfer measured locally in submicron water films
Wang, X., Ghaffarizadeh, S. A., He, X., McGaughey, A. J. H., & Malen, J. A. (2022) · Scientific Reports · doi:10.1038/s41598-022-26182-2 · Cited by 4
Forghani, R., Goodnight, B., Latchoumane, C.-F. V., & Karumbaiah, L. (2023) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2023.109798 · Cited by 5
“The AutoRG presents an attractive, scalable, and affordable solution to the problem of reproducibility in science (Collins and Tabak, 2014; Goodman et al., 2016;).”
Reproducibility and FAIR principles: the case of a segment polarity network model
Mendes, P. (2023) · Frontiers in Cell and Developmental Biology · doi:10.3389/fcell.2023.1201673 · Cited by 5
“As previously (Mendes, 2018), I will follow the definitions of Goodman et al. (2016), which specifies three distinct types of reproducibility: • reproducibility of methods requires one to be able to exactly reproduce the results using the same methods on the same data; • reproducibility of results…”
ASA 2022 Data-Driven Decision Making: Book of short papers
Edited by: Enrico di Bella, L. F., Fabbris, L., di Bella, E., & Lagazio, C. (2023) · Proceedings e report · doi:10.36253/979-12-215-0106-3 · Cited by 7
Effects of home‐cage elevation on behavioral tests in mice
Ueno, H., Takahashi, Y., Murakami, S., Wani, K., Matsumoto, Y., Okamoto, M., & Ishihara, T. (2023) · Brain and Behavior · doi:10.1002/brb3.3269 · Cited by 5
Savva, K., Zachariou, M., Bourdakou, M. M., Dietis, N., & Spyrou, G. M. (2024) · International Journal of Molecular Sciences · doi:10.3390/ijms25105319 · Cited by 5
Aspbury, M., Mansfield, R. C., Baxter, L., Bhatt, A., Cobo, M. M., Fitzgibbon, S. P., Hartley, C., Hauck, A., Marchant, S., Monk, V., et al. (2024) · Cortex · doi:10.1016/j.cortex.2024.05.023 · Cited by 5
“Reproducibility studies can establish whether clinical research findings are generalisable (Goodman, Fanelli, & Ioannidis, 2016; Nosek & Errington, 2020).”
BackgroundVsevolozhskaya, O. A., Kuo, C., Ruiz, G., Diatchenko, L. B., & Zaykin, D. V. (2017) · Genetic Epidemiology · doi:10.1002/gepi.22064 · Cited by 4
From academia to industry: a road more travelled
Pao, W. J. (2017) · Annals of Oncology · doi:10.1093/annonc/mdx319 · Cited by 4
Considerations on Preclinical Neuromuscular Disease Gene Therapy Studies
Duan, D. (2019) · Springer eBooks · doi:10.1007/978-3-030-03095-7_17 · Cited by 6
Enhancing Collaboration between Clinician-Researchers and Methodologists in Clinical Research
Amuah, J. E., Mutasingwa, D. R., & Amankwah, E. K. (2020) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2020.02.035 · Cited by 4
Lama-Larenas, P. A., Mora-Chaparro, J. C., Gomez-Romero, J., Canet, C., Cruz-Pérez, M. Á., García-Alonso, E. J., & Salgado-Martínez, E. (2021) · Geoheritage · doi:10.1007/s12371-021-00546-y · Cited by 4
Janero, D. R. (2016) · Expert Opinion on Drug Discovery · doi:10.1080/17460441.2016.1212014 · Cited by 6
“However, replicability implies stringently identical technical conditions among the original study and all repeats, whereas reproducibility does not and may be conducted under experimental conditions somewhat different from those in the original investigation [66].”
ResultDoes Only Size Matter or Is There Still a Place for Single-Center Studies in the Era of Big Data?
Einav, S., & O’Connor, M. (2016) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000001614 · Cited by 4
Mapping landscape units in Galicia (Spain): A first step for assessment and management?
Rico, E. C., & Sánchez, J. M. T. (2017) · Change and Adaptation in Socio-Ecological Systems · doi:10.1515/cass-2017-0008 · Cited by 4
Farrell, B., & Bengtson, J. (2019) · PLoS ONE · doi:10.1371/journal.pone.0223984 · Cited by 5
Analyzing the heterogeneity of rule-based EHR phenotyping algorithms in CALIBER and the UK BiobankPreprint
Denaxas, S., Parkinson, H., Fitzpatrick, N. K., Sudlow, C. L. M., & Hemingway, H. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/685156 · Cited by 5
Evans, S., Fladie, I. A., Anderson, J. M., Tritz, D., & Vassar, M. B. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/773473 · Cited by 3
Wu, J., Dong, Q., Gui, J., Zhang, J., Su, Y., Chen, K., Thompson, P. M., Caselli, R. J., Reiman, E. M., Ye, J., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.10.16.343137 · Cited by 5
High admixture between forest an cultivated chestnut ( Castanea sativa Mill.) in FrancePreprint
Bouffartigue, C., Debille, S., Fabreguette, O., Ramos‐Cabrer, A. M., Pereira‐Lorenzo, S., Flutre, T., & Harvengt, L. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/792259 · Cited by 4
“Reproducibility To facilitate method reproducibility (Goodman et al. 2016), most of our analyses were performed in R (R Core Team 2019); the scripts are available at https://data.inra.fr/privateurl.xhtml?token=8c03a83c-be4d-4984-972f-7808558b4539.”
MethodResearch data management for open science
Lefebvre, A. (2021) · Utrecht University Repository (Utrecht University) · doi:10.33540/447 · Cited by 6
A fully automated, transparent, reproducible, and blind protocol for sequential analysesPreprint
Beffara, B., Bret, A., & Nalborczyk, L. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/v7xpg · Cited by 5
Sbierski‐Kind, J., Cautivo, K. M., Wagner, J. C., Dahlgren, M. W., Nilsson, J., Krasilnikov, M., Mroz, N. M., Lizama, C. O., Gan, A. L., Matatia, P. R., et al. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.04.26.537913 · Cited by 5
Is everything wrong in analytical chemistry? A study on reproducibility
Ferreira, B. D., Olivares, I. R. B., Carrilho, E., & Pacces, V. H. P. (2025) · Accreditation and Quality Assurance · doi:10.1007/s00769-025-01649-7 · Cited by 3
An international consensus on core reproducibility items in research
Banzi, R., Varga, M., Gelsleichter, Y. A., Vinatier, C., Moher, D., Naudet, F., & on behalf of the OSIRIS-Delphi Study Group (2026) · PLoS Biology · doi:10.1371/journal.pbio.3003726 · Cited by 7
“…results from the conduct of an independent study whose procedures are as closely as possible matched to the original experiment); and inferential reproducibility (drawing qualitatively similar conclusions from either an independent replication of a study or a reanalysis of the original study) [8].”
Sirago, G., Zotti, F., Mele, F., Misceo, F., Solarino, B., Dell'Erba, A., & Ferorelli, D. (2026) · Journal of Hospital Infection · doi:10.1016/j.jhin.2026.04.019 · Cited by 6
Anderson, J. M., Niemann, A., Johnson, A., Cook, C., Tritz, D., & Vassar, M. B. (2019) · JMIR Dermatology · doi:10.2196/16078 · Cited by 3
“Precisely outlining methodology is essential for study reproducibility [31], whether this information is provided within the publication or in supplementary materials [32].”
Aviyente, S., & Karaaslanli, A. (2022) · IEEE Signal Processing Magazine · doi:10.1109/msp.2022.3149471 · Cited by 3
Defining Replicability of Prediction Rules
Parmigiani, G. (2023) · Statistical Science · doi:10.1214/23-sts891 · Cited by 4
Blixhavn, C. H., Reiten, I., Kleven, H., Øvsthus, M., Yates, S. C., Schlegel, U., Puchades, M. A., Schmid, O., Bjaalie, J. G., Bjerke, I. E., et al. (2024) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2024.1284107 · Cited by 6
“…is well established that the amount and consistency of metadata provided with research data varies considerably (see Bjerke et al., 2018a, 2020a), which in turn also contributes to the known problems with low replicability and reproducibility of studies (Goodman et al., 2016; Stupple et al., 2019).”
BackgroundIkäheimonen, A., Li, J., Yao, K., Zuo, S., Aledavood, T., & Hölttä‐Otto, K. (2024) · Journal of Engineering Design · doi:10.1080/09544828.2024.2396194 · Cited by 3
“Sharing well-documented and organised code used in the analysis promotes the methodological reproducibility of the research (Goodman, Fanelli, and Ioannidis 2016).”
Hartmann, S. (2024) · Studies in corpus linguistics · doi:10.1075/scl.118.06har · Cited by 12
Dunn, J., Mishra, V., Shandhi, M. M. H., Jeong, H., Yamane, N., Watanabe, Y., Chen, B. B., & Goodwin, M. S. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2024.1467424 · Cited by 5
“Scienti fi c endeavors are experiencing both a crisis of method reproducibility (the ability to achieve congruent results from a given dataset and analysis) and results reproducibility (the ability to recreate results independently (13, 14)).”
Researching public health datasets in the era of deep learning: a systematic literature review
Obeidat, R., Alsmadi, I. M., Baker, Q. B., Al-Njadat, A., Srinivasan, S., Ashong, G., & Osigwe, I. (2025) · Health Informatics Journal · doi:10.1177/14604582241307839 · Cited by 5
“145,146 Reproducibility is also critical for deep learning research, whose goal is to develop algorithms to reliably solve complex tasks at scale.”
Eryılmaz, B., Koras, O., Schlötterer, J., & Seifert, C. (2024) · International Conference on Cognitive Machine Intelligence · doi:10.1109/cogmi62246.2024.00042 · Cited by 7
“Goodman et al. [8] provide a foundational definition of reproducibility, framing the standards we adopted in our study.”
When to adjust for multiplicity in cancer clinical trials
Unger, J. M., Mazza, G. L., Elsaid, M. I., Duan, F., Dressler, E. V. M., Snavely, A. C., Enserro, D. M., & Pugh, S. L. (2025) · JNCI Monographs · doi:10.1093/jncimonographs/lgae051 · Cited by 4
Replication: A Critique and Overview for Kinesiology
Martin, J. J., & Yukhymenko–Lescroart, M. A. (2025) · Kinesiology Review · doi:10.1123/kr.2024-0057 · Cited by 5
Eleftheriou, C., Giachetti, S., Hickson, R. D. L., Kaminioti-Dumont, L., Templaar, R., Aaltonen, A., Tsoukala, E., Kim, N., Fryer-Petridis, L., Henley, C., et al. (2025) · Molecular Autism · doi:10.1186/s13229-025-00663-3 · Cited by 3
Lu, R., Zhang, R., & Liu, Y. (2025) · Frontiers in Microbiology · doi:10.3389/fmicb.2025.1609142 · Cited by 3
Context is Key for Reproducibility of Empirical Studies in Human-Computer Interaction
Vanderdonckt, J. M., & Vatavu, R.-D. (2025) · ACM Conference on Reproducibility and Replicability (ACM REP) · doi:10.1145/3736731.3746142 · Cited by 6
Reproducibility literature analysis - a federal information professional perspective
Antognoli, E., Avila, R. L., Sears, J., Christiansen, L. L., Tieman, J., & Hart, J. (2020) · IASSIST Quarterly · doi:10.29173/iq967 · Cited by 2
Reproducibility in speech rate convergence experiments
Fuscone, S., Favre, B., & Prévot, L. (2021) · Language Resources and Evaluation · doi:10.1007/s10579-021-09528-6 · Cited by 5
Rethinking the Academic Artefacts
Almevik, G., & Westin, J. (2022) · Acta Universitatis Gothoburgensis eBooks · doi:10.21524/kriterium.40.b · Cited by 3
Deney Hayvanları Araştırmalarında Standardizasyonun Yeri ve Önemi: Geleneksel Derleme
Çelik, A., Baksi, N., & Güneli, M. E. (2023) · Uludağ Üniversitesi Tıp Fakültesi Dergisi · doi:10.32708/uutfd.1216412 · Cited by 3
Kazemi, A., & Törnblom, K. (2023) · Social Justice Research · doi:10.1007/s11211-023-00423-9 · Cited by 3
Hussein, H., Farfar, K. E., Oelen, A., Karras, O., & Auer, S. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8088-8_19 · Cited by 6
Firmino, V. C., Brasil, L. S., Juen, L., Hamada, N., & Martins, R. T. (2024) · Neotropical Entomology · doi:10.1007/s13744-024-01150-9 · Cited by 3
“By reducing methodological gaps, microcosm experiments under laboratory conditions can become more standardized and reproducible, which can increase the accuracy of the results and reduce uncertainties and bias (Goodman et al. 2016; Cooke et al. 2017).”
Stagge, J. H., Rosenberg, D. E., Castronova, A. M., Ostfeld, A., & Jones, A. S. (2024) · Journal of Water Resources Planning and Management · doi:10.1061/jwrmd5.wreng-6559 · Cited by 4
Grant, S., & Khatua, S. (2024) · Journal of Development Effectiveness · doi:10.1080/19439342.2024.2388102 · Cited by 6
“…the ability to obtain the same results as a previous study by implementing the same methodological and computational procedures on the same data, or corroborate a previous study by implementing the same methodological procedures with a new independent dataset (Goodman, Fanelli, and Ioannidis 2016).”
Recommendations for a Complete Reporting of Statistical Methods in Veterinary Pharmacology
Villarino, N. F. (2025) · Journal of Veterinary Pharmacology and Therapeutics · doi:10.1111/jvp.70001 · Cited by 6
Enabling reproducible obia with open-source software in docker containers
Knoth, C., & Nüst, D. (2016) · International Conference on Geographic Object-Based Image Analysis (GEOBIA) · doi:10.3990/2.456 · Cited by 3
“, 2016) focuses on traceability, others treat “reproducibility” and “replicability” either as interchangeable (Gentleman and Lang, 2007) or completely different terms (Goodman et al., 2016).”
BackgroundReplicating Relevance-Ranked Synonym Discovery in a New Language and Domain
Yates, A., & Unterkalmsteiner, M. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-15712-8_28 · Cited by 3
Expanding the Scope of Reproducibility Research Through Data Analysis Replications
Hofman, J. M., Goldstein, D. G., Sen, S., & Poursabzi-Sandegh, F. (2020) · International World Wide Web Conference (WWW) · doi:10.1145/3366424.3383417 · Cited by 7
“…is not to debate the semantics of different terms used to categorize replication attempts, among which there is a good deal of confusion and disagreement (Christensen, Freese, & Miguel, 2019; Goodman, Fanelli, & Ioannidis, 2016; Nosek & Errington, 2019; Patil, Peng, & Leek, 2019; Plesser, 2018).”
Metadata
- Authors
- Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A.
- Affiliations
- Stanford University (US)
- Corresponding author
- Goodman, Steven N.
- Venue
- Science Translational Medicine
- Published
- 2016-06
- Volume
- 8
- Issue
- 341
- Pages
- 341ps12-341ps12
- ISSN
- 1946-6234 (ISSN-L)
- 1946-6242
- OpenAlex
- W2414283705
- Cited by
- 1,293
- References
- 41
- Citations by year
- 2026: 85 · 2025: 93 · 2024: 146 · 2023: 110 · 2022: 174 · 2021: 151 · 2020: 168 · 2019: 140 · 2018: 112 · 2017: 83 · 2016: 30
- FWCI
- 18.01
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science Translational Medicine (publishedVersion)
- PubMed (publishedVersion)
- Funders
- John and Laura Arnold Foundation
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- research reproducibility, terminology, conceptual framework, truth in science
- MeSH terms
- Reproducibility of Results, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.