Reproducibility in machine learning for health research: Still a ways to go
Open accessAbstract
Machine learning for health must be reproducible to ensure reliable clinical use. We evaluated 511 scientific papers across several machine learning subfields and found that machine learning for health compared poorly to other areas regarding reproducibility metrics, such as dataset and code accessibility. We propose recommendations to address this problem.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Especially when sample sizes are small, significance testing and uncertainty quantification are important steps toward reproducibility.
“Especially when sample sizes are small, significance testing and uncertainty quantification are important steps toward reproducibility.48,85”
From Leakage and the reproducibility crisis in machine-learning-based science (2023) · cites this paper as [48]
In “Statistical Reproducibility” (page unavailable), the paper calls for reporting variance around results and warns that relatively small ML4H datasets may heighten reproducibility problems. “Adopt Rigorous Statistical Methods” recommends proper procedures for model comparisons, but our copy does not specifically identify significance testing as a necessary step. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
1 retracted
Retraction rates are on the rise
Cokol, M., Ozbay, F., & Rodriguez‐Esteban, R. (2008) · The EMBO Reports · doi:10.1038/sj.embor.7401143 · Cited by 149 · see citing works
RetractedSudlow, C., Gallacher, J., Allen, N., Beral, V., Burton, P., Danesh, J., Downey, P., Elliott, P., Green, J., Landray, M., et al. (2015) · PLOS Medicine · doi:10.1371/journal.pmed.1001779 · Cited by 12,728 · see citing works
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · see citing works
Self-citation1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209 · cited 1× · see citing works
“INTRODUCTION Reproducibility is required for scientific research, but many subfields of science have recently experienced a reproducibility crisis, eroding trust in processes and results and potentially influencing the rising rates of scientific paper retractions (1, 2).”
BackgroundGebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469 · see citing works
The “All of Us” Research Program
The All of Us Research Program Investigators (2019) · New England Journal of Medicine · doi:10.1056/nejmsr1809937 · Cited by 2,440 · see citing works
Pollard, T. J., Johnson, A. E. W., Raffa, J. D., Celi, L. A., Mark, R. G., & Badawi, O. (2018) · Scientific Data · doi:10.1038/sdata.2018.178 · Cited by 1,993 · see citing works
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685 · see citing works
Nagendran, M., Chen, Y., Lovejoy, C. A., Gordon, A. C., Komorowski, M., Harvey, H., Topol, E. J., Ioannidis, J. P. A., Collins, G. S., & Maruthappu, M. (2020) · BMJ · doi:10.1136/bmj.m689 · Cited by 1,152 · see citing works
George, H., Duke Jon, D., Shah Nigam, H., Reich Christian, G., Vojtech, H., Schuemie Martijn, J., Suchard Marc, A., Woong, P. R., Kei, W. I. C., Rijnbeek Peter, R., et al. (2015) · Studies in Health Technology and Informatics · doi:10.3233/978-1-61499-564-7-574 · Cited by 1,107 · cited 1× · see citing works
“org/) (19) provides publicly available code and guidance on best practices to run observational health studies across multiple institutions and countries.”
BackgroundThe “All of Us” Research Program
Sullivan, F., McKinstry, B., & Vasishta, S. (2019) · New England Journal of Medicine · doi:10.1056/nejmc1912496 · Cited by 855 · see citing works
Overview of the BioBank Japan Project: Study design and profile
Nagai, A., Hirata, M., Kamatani, Y., Muto, K., Matsuda, K., Kiyohara, Y., Ninomiya, T., Tamakoshi, A., Yamagata, Z., Mushiroda, T., et al. (2017) · Journal of Epidemiology · doi:10.1016/j.je.2016.12.005 · Cited by 739 · see citing works
Validation of a common data model for active safety surveillance research
Overhage, J. M., Ryan, P. B., Reich, C. G., Hartzema, A. G., & Stang, P. E. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000376 · Cited by 548 · see citing works
Split learning for health: Distributed deep learning without sharing raw patient data
Vepakomma, P., Gupta, O., Swedish, T., & Raskar, R. (2018) · arXiv · doi:10.48550/arxiv.1812.00564 · Cited by 513 · see citing works
Walonoski, J. A., Kramer, M. A., Nichols, J., Quina, A., Moesel, C., Hall, D., Duffett, C., Dube, K., Gallagher, T., & McLachlan, S. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx079 · Cited by 440 · see citing works
Reproducibility vs. Replicability: A Brief History of a Confused Terminology
Pleßer, H. E. (2018) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2017.00076 · Cited by 426 · see citing works
Challenges to the Reproducibility of Machine Learning Models in Health Care
Beam, A. L., Manrai, A. K., & Ghassemi, M. (2020) · JAMA · doi:10.1001/jama.2019.20866 · Cited by 366 · see citing works
Self-citationState of the Art: Reproducibility in Artificial Intelligence
Gundersen, O. E., & Kjensmo, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11503 · Cited by 293 · see citing works
Relabeling internal and external validity for applied social scientists
Campbell, D. T. (1986) · New Directions for Program Evaluation · doi:10.1002/ev.1434 · Cited by 237 · see citing works
Replicability is not Reproducibility:Nor is it Good Science
Drummond, C. (2009) · Cited by 227 · see citing works
Registration of observational studies
Loder, E., Groves, T., & Macauley, D. (2010) · BMJ · doi:10.1136/bmj.c950 · Cited by 131 · see citing works
Reproducibility in critical care: a mortality prediction case study
Johnson, A. E. W., Pollard, T., & Mark, R. G. (2017) · Machine Learning for Healthcare Conference · Cited by 99 · see citing works
Hu, Q., & Greene, C. S. (2018) · Pacific Symposium on Biocomputing (PSB) · doi:10.1142/9789813279827_0033 · Cited by 68 · see citing works
Hu, Q., & Greene, C. S. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/385534 · Cited by 44 · see citing works
Lash, T. L., & Vandenbroucke, J. P. (2012) · Epidemiology · doi:10.1097/ede.0b013e318245c05b · Cited by 42 · see citing works
The Blue Button Project: Engaging Patients in Healthcare by a Click of a Button.
Mohsen, M. O., & Aziz, H. A. (2015) · PubMed · Cited by 17 · see citing works
Generalizability of predictive models for intensive care unit patients
Johnson, A. E. W., Pollard, T. J., & Naumann, T. (2018) · arXiv · doi:10.48550/arxiv.1812.02275 · Cited by 12 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [10]Recht B, Roelofs R, Schmidt L, Shankar V, Do ImageNet Classifiers Generalize to ImageNet?, arXiv:1806.00451 [cs, stat] (2018). ArXiv: 1806.00451. · arXiv
- [12]Nestor B,, Feature robustness in non-stationary health records: Caveats to deployable model performance in common clinical machine learning task, Proceedings of MAchine Learning Research 106: Machine Learning for Healthcare (Ann Arbor, MI, USA, 2019).
Cited by
Sorted by citations
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Collins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-078378 · Cited by 4,718 · see citing works
“…2015, there have been numerous methodological advances in prediction modelling, including sample size guidance for developing models 23-27 and evaluating their performance, 28-32 and greater recognition of operationalising fairness, 33 reproducibility, 34 and adopting open science principles.”
BackgroundSelf-citationLeakage and the reproducibility crisis in machine-learning-based science
Kapoor, S., & Narayanan, A. (2023) · Patterns · doi:10.1016/j.patter.2023.100804 · Cited by 1,387 · see citing works
“Especially when sample sizes are small, significance testing and uncertainty quantification are important steps toward reproducibility.(48,85) As an illustration, we examine this issue in detail in the case of Blair and Sambanis(86) because their test dataset has a particularly small number of instances of civil war o…”
BackgroundMethodMoons, K. G. M., Damen, J. A. A., Kaul, T., Hooft, L., Andaur Navarro, C., Dhiman, P., Beam, A. L., Van Calster, B., Celi, L. A., Denaxas, S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-082505 · Cited by 1,069 · see citing works
Self-citation(2024) · BMJ · doi:10.1136/bmj.q902 · Cited by 416 · see citing works
“…2015, there have been numerous methodological advances in prediction modelling, including sample size guidance for developing models 23-27 and evaluating their performance, 28-32 and greater recognition of operationalising fairness, 33 reproducibility, 34 and adopting open science principles.”
Large language models in health care: Development, applications, and challenges
Yang, R., Tan, T. F., Lu, W., Thirunavukarasu, A. J., Ting, D. S. W., & Liu, N. (2023) · Health Care Science · doi:10.1002/hcs2.61 · Cited by 366 · see citing works
Cutillo, C. M., Sharma, K. R., Foschini, L., Kundu, S., Mackintosh, M., Mandl, K. D., MI in Healthcare Workshop Working Group, Beck, T., Collier, E., Colvis, C., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0254-2 · Cited by 337 · see citing works
Self-citationJoshi, G., Jain, A., Araveeti, S. R., Adhikari, S., Garg, H., & Bhandari, M. (2024) · Electronics · doi:10.3390/electronics13030498 · Cited by 294 · see citing works
Big data in basic and translational cancer research
Jiang, P., Sinha, S., Aldape, K., Hannenhalli, S., Sahinalp, C., & Ruppin, E. (2022) · Nature Reviews Cancer · doi:10.1038/s41568-022-00502-0 · Cited by 254 · see citing works
Accelerating antibiotic discovery through artificial intelligence
Melo, M. C. R., Maasch, J. R. M. A., & de la Fuente-Nunez, C. (2021) · Communications Biology · doi:10.1038/s42003-021-02586-0 · Cited by 234 · see citing works
Evaluation and Mitigation of Racial Bias in Clinical Machine Learning Models: Scoping Review
Huang, J., Galal, G., Etemadi, M., & Vaidyanathan, M. (2022) · JMIR Medical Informatics · doi:10.2196/36388 · Cited by 186 · see citing works
Yang, J., Soltan, A. A. S., & Clifton, D. A. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00614-9 · Cited by 181 · see citing works
The Diagnostic and Triage Accuracy of the GPT-3 Artificial Intelligence Model
Levine, D. M., Tuwani, R., Kompa, B., Varma, A., Finlayson, S. G., Mehrotra, A., & Beam, A. (2023) · doi:10.1101/2023.01.30.23285067 · Cited by 106 · see citing works
London, A. J. (2022) · Cell Reports Medicine · doi:10.1016/j.xcrm.2022.100622 · Cited by 76 · see citing works
Niazi, S. K. (2025) · Pharmaceuticals · doi:10.3390/ph18091271 · Cited by 61 · see citing works
Solving medicine's data bottleneck: Nightingale Open Science
Mullainathan, S., & Obermeyer, Z. (2022) · Nature Medicine · doi:10.1038/s41591-022-01804-4 · Cited by 43 · see citing works
Cunningham, J. W., Singh, P., Reeder, C., Claggett, B., Marti-Castellote, P. M., Lau, E. S., Khurshid, S., Batra, P., Lubitz, S. A., Maddah, M., et al. (2024) · JAMA Cardiology · doi:10.1001/jamacardio.2023.4859 · Cited by 38 · see citing works
Venkatesh, K., Santomartino, S. M., Sulam, J., & Yi, P. H. (2022) · Radiology: Artificial Intelligence · doi:10.1148/ryai.220081 · Cited by 31 · see citing works
- Authors
- McDermott, M. B. A., Wang, S., Marinsek, N., Ranganath, R., Foschini, L., & Ghassemi, M.
- Affiliations
- Massachusetts Institute of Technology (US)
- University of Toronto (CA)
- Evidation Health (United States) (US)
- Courant Institute of Mathematical Sciences (US)
- Corresponding author
- McDermott, Matthew B. A.
- Venue
- Science Translational Medicine
- Published
- 2021-03-24
- Volume
- 13
- Issue
- 586
- ISSN
- 1946-6234 (ISSN-L)
- 1946-6242
- PMC
- PMC12862614
- OpenAlex
- W3138114698
- Cited by
- 273
- References
- 28
- Citations by year
- 2026: 46 · 2025: 53 · 2024: 53 · 2023: 58 · 2022: 42 · 2021: 17 · 2020: 2
- FWCI
- 8.03
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Science Translational Medicine (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- Funders
- National Institutes of Health (P50-MH106933)
- Canadian Institute for Advanced Research (AI Chair)
- Natural Sciences and Engineering Research Council of Canada (Discovery Grant)
- Mitacs (Globalink Research Award)
- National Institutes of Health National Institute of Mental Health (P50-MH106933)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Keywords
- reproducibility, machine learning, health research, machine learning in healthcare, reproducibility metrics, research recommendations
- MeSH terms
- Machine Learning, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
