The reproducibility crisis in the age of digital medicine
Open accessAbstract
If anyone doubts the explosive growth of interest in digital medicine, consider a recent conference and workshop in Beijing, jointly organized by the People’s Liberation Army General Hospital and MIT Critical Data to showcase the opportunities and challenges of applying machine learning to the kind of data routinely collected during the provision of care. 1 In person, 500 attendees heard a keynote and panels and participated in a health data hackathon. Online, however, the event was streamed to more than one million unique viewers. 2
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
AI's transparency and interpretability are challenged by factors such as patient privacy, algorithm interpretability, publication bias, etc.
“In addition, AI's transparency and interpretability are challenged by factors such as patient privacy, algorithm interpretability, publication bias, etc. 39,40.”
From Artificial intelligence in pancreatic cancer (2022) · cites this paper as [39]
The “Peer review would not protect us” section describes a publication-bias problem: negative or null findings are rarely published. The conclusion also calls for data and code sharing so others can reproduce studies. Our copy does not identify patient privacy or algorithm interpretability as challenges to AI transparency or interpretability. (LLM, conf=0.84)
Experimental — accuracy not yet measuredThe reproducibility and replicability crisis is important for digital health and mHealth research, including wrist-worn heart rate accuracy and studies utilizing wearable heart rate data, because this new field allows for large data samples with various sensors that increase the ease for p-hacking.
“Recent information on the reproducibility and replicability crisis has swept the sciences and as such these issues are very important for digital health and mHealth research, which includes wrist-worn HR accuracy and any studies that utilize wearable HR data, as this new field will allow for large data samples31 with various sensors that will increase the ease for p-hacking.”
From Guidelines for wrist-worn consumer wearable assessment of heart rate in biobehavioral research (2020) · cites this paper as [31]
Many researchers have begun to push for increased focus on reproducible and open practices in science.
“Therefore, many researchers have begun to push for increased focus on reproducible and open practices in science31,53–55.”
From Guidelines for wrist-worn consumer wearable assessment of heart rate in biobehavioral research (2020) · cites this paper as [31]
mHealth research will be particularly susceptible to research practices that degrade quality, consistency, generalizability, and therefore replicability and reproducibility of findings, as data sensors allow vast collection of big data samples vulnerable to publication bias, P-hacking, HARKing, and low power.
“Lastly, from a study design perspective, mHealth research will be particularly susceptible to a number of research practices, which have been shown to degrade the quality, consistency, generalizability, and therefore replicability and reproducibility of findings31 as data sensors in this research field allow for the vast collection of big data samples, which will be vulnerable to some of the “four”
From Guidelines for wrist-worn consumer wearable assessment of heart rate in biobehavioral research (2020) · cites this paper as [31]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · see citing works
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209 · see citing works
Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life
Traynham, J. G., Shapin, S., & Schaffer, S. (1987) · The Journal of Interdisciplinary History · doi:10.2307/204299 · Cited by 2,054 · see citing works
Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition
Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01125.x · Cited by 1,475 · see citing works
What does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293 · cited 1× · see citing works
“” For our purposes here, we take reproducibility to mean “obtaining the same results from the conduct of an independent study whose procedures are as closely matched to the original experiment as possible” (also known as “research reproducibility” or simply “replicability”).(3) A sense of crisis itself began with the…”
MethodLederer, D. J., Bell, S. C., Branson, R. D., Chalmers, J. D., Marshall, R., Maslove, D. M., Ost, D. E., Punjabi, N. M., Schatz, M., Smyth, A. R., et al. (2019) · Annals of the American Thoracic Society · doi:10.1513/annalsats.201808-564ps · Cited by 715 · see citing works
Guidelines for reinforcement learning in healthcare
Gottesman, O., Johansson, F., Komorowski, M., Faisal, A., Sontag, D., Doshi-Velez, F., & Celi, L. A. (2019) · Nature Medicine · doi:10.1038/s41591-018-0310-5 · Cited by 395 · see citing works
Self-citationThe Private Science of Louis Pasteur
Geison, G. L. (1996) · doi:10.1515/9781400864089 · Cited by 284 · see citing works
Reproducibility: A tragedy of errors
Allison, D. B., Brown, A. W., George, B. J., & Kaiser, K. A. (2016) · Nature · doi:10.1038/530027a · Cited by 255 · see citing works
Peer review: Troubled from the start
Csiszar, A. (2016) · Nature · doi:10.1038/532306a · Cited by 172 · see citing works
The Use of Superlatives in Cancer Research
Abola, M. V., & Prasad, V. K. (2015) · JAMA Oncology · doi:10.1001/jamaoncol.2015.3931 · Cited by 91 · see citing works
The Air We Breathe: A Critical Look at Practices and Alternatives in the Peer-Review Process
Suls, J. M., & Martin, R. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01105.x · Cited by 85 · see citing works
Celi, L. A., Citi, L., Ghassemi, M., & Pollard, T. (2019) · PLoS ONE · doi:10.1371/journal.pone.0210232 · Cited by 45 · see citing works
Self-citationBaldwin, M. (2017) · Physics Today · doi:10.1063/pt.3.3463 · Cited by 43 · see citing works
Ferreira, C., Bastille‐Rousseau, G., Bennett, A. M., Ellington, E. H., Terwissen, C. V., Austin, C., Borlestean, A., Boudreau, M. R., Chan, K., Forsythe, A., et al. (2015) · Biological reviews/Biological reviews of the Cambridge Philosophical Society · doi:10.1111/brv.12185 · Cited by 34 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [1]Jia, G. Chinese Peoples Liberation Army General Hospital—MIT Critical Data Conference and Workshop. http://www.plagh-mit-datathon.com (2018) (Accessed 16 Dec 2018).
- [14]Fyfe, A. Peer review: not as old as you might think. Times Higher Education, 25 (2015).
Cited by
Sorted by citations
Cutillo, C. M., Sharma, K. R., Foschini, L., Kundu, S., Mackintosh, M., Mandl, K. D., MI in Healthcare Workshop Working Group, Beck, T., Collier, E., Colvis, C., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0254-2 · Cited by 337 · see citing works
Thoral, P. J., Peppink, J. M., Driessen, R. H., Sijbrands, E. J. G., Kompanje, E. J. O., Kaplan, L., Bailey, H., Kesecioglu, J., Cecconi, M., Churpek, M., et al. (2021) · Critical Care Medicine · doi:10.1097/ccm.0000000000004916 · Cited by 257 · see citing works
Guidelines for wrist-worn consumer wearable assessment of heart rate in biobehavioral research
Nelson, B. W., Low, C. A., Jacobson, N., Areán, P., Torous, J., & Allen, N. B. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0297-4 · Cited by 241 · see citing works
Looking beyond the hype: applied AI and machine learning in translational medicine
Toh, T. S., Dondelinger, F., & Wang, D. (2019) · EBioMedicine · doi:10.1016/j.ebiom.2019.08.027 · Cited by 159 · see citing works
Artificial intelligence in pancreatic cancer
Huang, B., Huang, H., Zhang, S., Zhang, D., Shi, Q., Liu, J., & Guo, J. (2022) · Theranostics · doi:10.7150/thno.77949 · Cited by 142 · see citing works
Gallifant, J., Zhang, J., del Pilar Arias Lopez, M., Zhu, T., Camporota, L., Celi, L. A., & Formenti, F. (2022) · British Journal of Anaesthesia · doi:10.1016/j.bja.2021.09.025 · Cited by 72 · see citing works
Self-citationSharing Data Is Essential for the Future of AI in Medical Imaging
Bell, L. C., & Shimron, E. (2024) · Radiology: Artificial Intelligence · doi:10.1148/ryai.230337 · Cited by 30 · see citing works
SEQUIN is an R/Shiny framework for rapid and reproducible analysis of RNA-seq data.
Weber, C., Hirst, M. B., Ernest, B., Schaub, N. J., Wilson, K. M., Wang, K., Baskir, H. M., Chu, P.-H., Tristan, C. A., & Singeç, I. (2023) · Cell Reports Methods · doi:10.1016/j.crmeth.2023.100420 · Cited by 11 · see citing works
- Authors
- Stupple, A., Singerman, D., & Celi, L. A.
- Affiliations
- Beth Israel Deaconess Medical Center (US)
- University of Virginia (US)
- Corresponding author
- Celi, Leo Anthony
- ORCID
- Celi, Leo Anthony
- Venue
- npj Digital Medicine
- Published
- 2019-01-29
- Volume
- 2
- Issue
- 1
- Pages
- 2-2
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC6550262
- OpenAlex
- W2912652107
- Cited by
- 137
- References
- 16
- Citations by year
- 2026: 5 · 2025: 9 · 2024: 18 · 2023: 25 · 2022: 28 · 2021: 24 · 2020: 19 · 2019: 9
- FWCI
- 6.36
- Type
- journal article
- Language
- en
- Update notices
- correction (2019-03-19)
- Open access
- gold
- doi.org
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
- Funders
- National Institute of Health
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- replication crisis, digital medicine, machine learning, health data, research reproducibility, clinical data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
