Bridging the implementation gap of machine learning in healthcare
Open accessThe DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Applications of machine learning on clinical data are now attaining levels of performance that match or exceed human clinicians.1–3 Fields involving image interpretation—radiology, pathology and dermatology—have led the charge due to the power of convolutional neural networks, the existence of standard data formats and large data repositories. We have also seen powerful diagnostic and predictive algorithms built using a range of other data, including electronic health records (EHR), -omics, monitoring signals, insurance claims and patient-generated data.4 The looming extinction of doctors has captured the public imagination, with editorials such as ‘The AI Doctor Will See You Now’.5 The prevailing view among experts is more balanced: that doctors who use artificial intelligence (AI) will replace those who do not.6 Amid such inflated expectations, the elephant in the room is the implementation gap of machine learning in healthcare.7 8 Very few of these algorithms ever make it to the bedside; and even the most technology-literate academic medical centres are not routinely using AI in clinical workflows. A recent systematic review of deep learning applications using EHR data highlighted the need to focus on the last mile of implementation: ‘for direct clinical impact, deployment and automation of deep learning models must be considered’.9 The typical life-cycle of an algorithm remains: train on historical data, publish a good receiver-operator curve and then collect dust in the ‘model graveyard’. This begs the question: if model performance is so …
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Despite the limited number of ICU datasets, the flurry of excellent modeling work afforded by these freely available intensive care datasets has exposed a severe translational gap, with implementation at the bedside and demonstration of improved patient outcomes using those models proving very challenging [7, 8].”
From Use of artificial intelligence in critical care: opportunities and obstacles (2024) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Indeed, the previously described “implementation gap” for medical AI extends to predicting clinical deterioration and sepsis CDSS [7].”
From Use of artificial intelligence in critical care: opportunities and obstacles (2024) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The rapid advances in AI, and in particular the release of publicly available generative AI applications leveraging advanced large language models, have greatly accelerated discussions considering the promises and pitfalls related to AI deployment in society and healthcare [7–17].”
From Use of artificial intelligence in critical care: opportunities and obstacles (2024) · cites this paper as [7]
Experimental — accuracy not yet measuredIn the citing paper
“Real-time operational models also need to cover the ‘last mile’ of AI deployment; this means that the applications that generate predictions can run end-to-end without human intervention.”
From Machine learning for real-time aggregated prediction of hospital admission for emergency patients (2022) · cites this paper as [29]
In the citing paper
“This last mile is the most neglected29, leading to calls for a delivery science for AI, in which AI is viewed as an enabling component within an operational workflow, rather than an end in itself30.”
From Machine learning for real-time aggregated prediction of hospital admission for emergency patients (2022) · cites this paper as [29]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
Artificial intelligence in healthcare
Yu, K.-H., Beam, A. L., & Kohane, I. S. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0305-z · Cited by 2,817
Clinically applicable deep learning for diagnosis and referral in retinal disease
De Fauw, J., Ledsam, J. R., Romera-Paredes, B., Nikolov, S., Tomasev, N., Blackwell, S., Askham, H., Glorot, X., O’Donoghue, B., Visentin, D., et al. (2018) · Nature Medicine · doi:10.1038/s41591-018-0107-6 · Cited by 2,639
Potential Biases in Machine Learning Algorithms Using Electronic Health Record Data
Gianfrancesco, M. A., Tamang, S., Yazdany, J., & Schmajuk, G. (2018) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.3763 · Cited by 1,395 · cited 1×
“…patient safety, such as protecting against algorithmic bias (will certain ethnic or socioeconomic groups be systematically disadvantaged by an algorithm trained on historical prejudices 15 ) and model brittleness (given the recent evidence showing adversarial attacks on deep neural networks 16 ).”
BackgroundKomorowski, M., Celi, L. A., Badawi, O., Gordon, A. C., & Faisal, A. A. (2018) · Nature Medicine · doi:10.1038/s41591-018-0213-5 · Cited by 1,290
Machine Learning and Prediction in Medicine — Beyond the Peak of Inflated Expectations
Chen, J. H., & Asch, S. M. (2017) · New England Journal of Medicine · doi:10.1056/nejmp1702071 · Cited by 1,171
Sittig, D. F., & Singh, H. (2010) · Quality and Safety in Health Care · doi:10.1136/qshc.2010.042085 · Cited by 897
Xiao, C., Choi, E., & Sun, J. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy068 · Cited by 792
What This Computer Needs Is a Physician: Humanism and Artificial Intelligence
Verghese, A., Shah, N. H., & Harrington, R. A. (2018) · JAMA · doi:10.1001/jama.2017.19198 · Cited by 468
Self-citationArbabshirani, M. R., Fornwalt, B. K., Mongelluzzo, G. J., Suever, J. D., Geise, B. D., Patel, A. A., & Moore, G. J. (2018) · npj Digital Medicine · doi:10.1038/s41746-017-0015-z · Cited by 443
Research works printed but not linked: 3
- Rajpurkar P , Hannun AY , Haghpanahi M , et al . Cardiologist-Level Arrhythmia Detection with Convolutional Neural Networks [Internet]. arXiv [cs.CV], 2017. Available: http://arxiv.org/abs/1707.01836 · arXiv
- Finlayson SG , Chung HW , Kohane IS , et al . Adversarial Attacks Against Medical Deep Learning Systems [Internet]. arXiv [cs.CR], 2018. Available: http://arxiv.org/abs/1804.05296 · arXiv
- Garg R , Dong S , Shah S , et al . A Bootstrap Machine Learning Approach to Identify Rare Disease Patients from Electronic Health Records [Internet]. arXiv [cs.LG] 2016.
Cited by
Significance of machine learning in healthcare: Features, pillars and applications
Javaid, M., Haleem, A., Pratap Singh, R., Suman, R., & Rab, S. (2022) · International Journal of Intelligent Networks · doi:10.1016/j.ijin.2022.05.002 · Cited by 641
Higgins, O., Short, B., Chalup, S. K., & Wilson, R. L. (2023) · International Journal of Mental Health Nursing · doi:10.1111/inm.13114 · Cited by 219
Kumar, D., Haque, M. A., Mishra, K., Islam, F., Mishra, B. K., & Ahmad, S. (2023) · Metaverse Basic and Applied Research · doi:10.56294/mr202355 · Cited by 213
Application of Artificial Intelligence for the Diagnosis and Treatment of Liver Diseases
Ahn, J. C., Connell, A., Simonetto, D. A., Hughes, C., & Shah, V. H. (2021) · Hepatology · doi:10.1002/hep.31603 · Cited by 196
Peer review of GPT-4 technical report and systems card
Gallifant, J., Fiske, A., Strekalova, Y. A., Osorio-Valencia, J. S., Parke, R. L., Mwavu, R., Martinez‐Martin, N., Gichoya, J. W., Ghassemi, M., Demner‐Fushman, D., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000417 · Cited by 181
Metadata
- Authors
- Seneviratne, M. G., Shah, N. H., & Chu, L.
- Affiliations
- Stanford Medicine (US)
- Stanford University (US)
- Corresponding author
- Shah, Nigam H
- Venue
- BMJ Innovations
- Publisher
- BMJ
- Published
- 2019-12-20
- Volume
- 6
- Issue
- 2
- Pages
- 45-47
- ISSN
- 2055-642X (ISSN-L)
- 2055-8074
- OpenAlex
- W2994958466
- Cited by
- 173
- References
- 23
- Citations by year
- 2026: 37 · 2025: 28 · 2024: 39 · 2023: 33 · 2022: 26 · 2021: 4 · 2020: 6
- FWCI
- 2.85
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- BMJ Innovations (publishedVersion, cc-by-nc)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- deep learning deployment, electronic health records, clinical workflow integration, AI adoption in healthcare, clinical decision support
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Bridging the implementation gap of machine learning in healthcare. https://app.cochat.ai/papers/10.1136/bmjinnov-2019-000359
- The paper
- Seneviratne, M. G., Shah, N. H., & Chu, L. (2020). Bridging the implementation gap of machine learning in healthcare. BMJ Innovations. https://doi.org/10.1136/bmjinnov-2019-000359
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.