Crossing the chasm from model performance to clinical impact: the need to improve implementation and evaluation of AI
Open accessAbstract
Artificial intelligence (AI) has been the subject of considerable interest for many years for its potential to improve clinical care—yet its actual impact on patient outcomes when deployed in clinical settings remains largely unknown. In a recent systematic review by Zhou et al. 1 , the authors surprisingly show that its impact so far has been quite limited. They reviewed 65 randomized controlled trials (RCTs) evaluating AI-based clinical interventions and found that there was no clinical benefit of using AI prediction tools compared to the standard of care in nearly 40% of studies. Among a subset of trials that the authors identified as having a low risk of bias, the clinical benefit of using deep learning (DL) predictive models over traditional statistical (TS) risk calculators was only minimal, and there was no benefit in using machine learning (ML) models over TS tools. Somewhat counterintuitively, most of the AI tools in these trials exhibited an excellent area under the receiver operating characteristic (AUROC; a common performance metric for predictive models) during development (median AUROC 0.81, IQR 0.75–0.90) and validation (median AUROC 0.83, IQR 0.79–0.97): a humbling reminder that robust predictive utility does not guarantee clinical impact at the bedside. As the science of building accurate predictive models progresses, our ability to translate these advancements into real-world clinical utility remains comparatively limited. How can we bridge this gap between AUROCs and clinical benefit?
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
This requires the user to figure out how to combine the results of individual evaluations for decision-making-an unfilled gap commonly described as the 'AI Chasm'.
“This requires the user to figure out how to combine the results of individual evaluations for decision-making-an unfilled gap commonly described as the 'AI Chasm' 8 .”
From Standing on FURM Ground: A Framework for Evaluating Fair, Useful, and Reliable AI Models in Health Care Systems (2024) · cites this paper as [8]
The paper’s title and its 'Building out the implementation science of AI' section describe a gap between model performance and clinical impact, including the unresolved challenge of incorporating predictions into care decisions (page not available). It does not describe that gap as users combining the results of individual evaluations, or establish 'AI Chasm' as a commonly used term. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The false hope of current approaches to explainable artificial intelligence in health care
Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00208-9 · Cited by 1,724 · see citing works
Trust and medical AI: the challenges we face and the expertise needed to overcome them
Quinn, T. P., Senadeera, M., Jacobs, S., Coghlan, S., & Le, V. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa268 · Cited by 287 · see citing works
Zhou, Q., Chen, Z.-H., Cao, Y.-H., & Peng, S. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00524-2 · Cited by 126 · see citing works
Levy, J., Álvarez, D., Rosenberg, A. A., Alexandrovich, A., del Campo, F., & Behar, J. A. (2021) · npj Digital Medicine · doi:10.1038/s41746-020-00373-5 · Cited by 125 · see citing works
A framework for making predictive models useful in practice
Jung, K., Kashyap, S., Avati, A., Harman, S. M., Shaw, H., Li, R., Smith, M. A., Shum, K., Javitz, J., Vetteth, Y., et al. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa318 · Cited by 82 · see citing works
Administrative Expenses in the US Health Care System
Chernew, M. E., & Mintz, H. (2021) · JAMA · doi:10.1001/jama.2021.17318 · Cited by 59 · see citing works
Golas, S. B., Nikolova-Simons, M., Palacholla, R. S., den Buijs, J. O., Garberg, G., Orenstein, A., & Kvedar, J. C. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00463-y · Cited by 23 · see citing works
Self-citationPrecision population analytics: population management at the point-of-care
Tang, P. C., Miller, S., Stavropoulos, H., Kartoun, U., Zambrano, J. A., & Ng, K. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa247 · Cited by 19 · see citing works
Barak‐Corren, Y., Agarwal, I., Michelson, K. A., Lyons, T. W., Neuman, M. I., Lipsett, S. C., Kimia, A. A., Eisenberg, M. A., Capraro, A., Levy, J. A., et al. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab076 · Cited by 18 · see citing works
Comment on: Truth and truthiness: evidence, experience and clinical judgement in surgery
Marwaha, J. S., Beaulieu-Jones, B. R., Yuan, W., & Brat, G. A. (2021) · British journal of surgery · doi:10.1093/bjs/znab319 · Cited by 3 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Derraz, B., Breda, G., Kaempf, C., Baenke, F., Cotte, F., Reiche, K., Köhl, U., Kather, J. N., Eskenazy, D., & Gilbert, S. (2024) · npj Precision Oncology · doi:10.1038/s41698-024-00517-w · Cited by 94 · see citing works
Artificial intelligence in hematological diagnostics: Game changer or gadget?
Walter, W., Pohlkamp, C., Meggendorfer, M., Nadarajah, N., Kern, W., Haferlach, C., & Haferlach, T. (2022) · Blood Reviews · doi:10.1016/j.blre.2022.101019 · Cited by 86 · see citing works
To warrant clinical adoption AI models require a multi-faceted implementation evaluation
van de Sande, D., Chung, E. F. F., Oosterhoff, J. H. F., van Bommel, J., Gommers, D. A. M. P. J., & van Genderen, M. E. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01064-1 · Cited by 73 · see citing works
Callahan, A., McElfresh, D., Banda, J. M., Bunney, G., Char, D., Chen, J., Corbin, C. K., Dash, D., Downing, N. L., Jain, S. S., et al. (2024) · NEJM Catalyst · doi:10.1056/cat.24.0131 · Cited by 61 · see citing works
The digital transformation of surgery
Marwaha, J. S., Raza, M. M., & Kvedar, J. C. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00846-3 · Cited by 44 · see citing works
Self-citationArtificial intelligence in risk prediction and diagnosis of vertebral fractures
Namireddy, S. R., Gill, S. S., Peerbhai, A., Kamath, A. G., Ramsay, D. S. C., Ponniah, H. S., Salih, A. M., Janković, D., Kalasauskas, D., Neuhoff, J., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-75628-2 · Cited by 35 · see citing works
A clinically motivated self-supervised approach for content-based image retrieval of CT liver images
Wickstrøm, K., Østmo, E. A., Radiya, K., Mikalsen, K. Ø., Kampffmeyer, M., & Jenssen, R. (2023) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2023.102239 · Cited by 25 · see citing works
An operational guide to translational clinical machine learning in academic medical centers
Poddar, M., Marwaha, J. S., Yuan, W., Brufau, S. R., & Brat, G. A. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01094-9 · Cited by 24 · see citing works
Self-citationThe Environmental Costs of Artificial Intelligence for Healthcare
Katirai, A. (2024) · Asian Bioethics Review · doi:10.1007/s41649-024-00295-4 · Cited by 27 · see citing works
McCoy, L. G., Bihorac, A., Celi, L. A., Elmore, M., Kewalramani, D., Kwaga, T., Martinez‐Martin, N., Prôa, R., Schamroth, J., Shaffer, J. D., et al. (2025) · BMC Global and Public Health · doi:10.1186/s44263-025-00158-6 · Cited by 24 · see citing works
Youssef, A., Abràmoff, M. D., & Char, D. S. (2023) · The American Journal of Bioethics · doi:10.1080/15265161.2023.2191052 · Cited by 20 · see citing works
Appraising the Quality of Development and Reporting in Surgical Prediction Models
Marwaha, J. S., Chen, H. W., Habashy, K. J., Choi, J., Spain, D. A., & Brat, G. A. (2022) · JAMA Surgery · doi:10.1001/jamasurg.2022.4488 · Cited by 19 · see citing works
Self-citationWarman, A., Kalluri, A. L., & Azad, T. D. (2023) · Neurosurgical FOCUS · doi:10.3171/2023.3.focus2386 · Cited by 18 · see citing works
Predictive Models for Health Deterioration: Understanding Disease Pathways for Personalized Medicine
Eskofier, B. M., & Klucken, J. (2023) · Annual Review of Biomedical Engineering · doi:10.1146/annurev-bioeng-110220-030247 · Cited by 14 · see citing works
DASS Good: Explainable Data Mining of Spatial Cohort Data
Wentzel, A., Floricel, C., Canahuate, G. M., Naser, M. A., Mohamed, A. S., Fuller, C. D., van Dijk, L. V., & Marai, G. E. (2023) · Computer Graphics Forum · doi:10.1111/cgf.14830 · Cited by 13 · see citing works
Oakley, W., Tandle, S., Perkins, Z., & Marsden, M. E. R. (2024) · The Journal of Trauma: Injury, Infection, and Critical Care · doi:10.1097/ta.0000000000004385 · Cited by 14 · see citing works
Gill, S. S., Ponniah, H. S., Giersztein, S., Anantharaj, R. M., Namireddy, S. R., Killilea, J., Ramsay, D. S. C., Salih, A. M., Thavarajasingam, A., Scurtu, D., et al. (2025) · Brain and Spine · doi:10.1016/j.bas.2025.104208 · Cited by 15 · see citing works
The role of digital technology in surgical home hospital programs
Pathak, K., Marwaha, J. S., & Tsai, T. C. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00750-w · Cited by 13 · see citing works
Self-citationCrossing the AI Chasm in Neurocritical Care
Cascella, M., Montomoli, J., Bellini, V., Vittori, A., Biancuzzi, H., Mas, F. D., & Bignami, E. G. (2023) · Computers · doi:10.3390/computers12040083 · Cited by 12 · see citing works
Machine learning in the evaluation of voice and swallowing in the head and neck cancer patient
Srinivasan, Y., Liu, A. Y., & Rameau, A. (2023) · Current Opinion in Otolaryngology & Head & Neck Surgery · doi:10.1097/moo.0000000000000948 · Cited by 9 · see citing works
Developing a Chatbot–Clinician Model for Hypertension Management
Lee, N. S., Luong, T., Rosin, R. M., Asch, D. A., Sevinc, C., Balachandran, M., Josephs, M. D., Rusk, M. H., McHenry, N. L., Asch, E. L., et al. (2022) · NEJM Catalyst · doi:10.1056/cat.22.0228 · Cited by 9 · see citing works
Wentzel, A., Mohamed, A., Naser, M. A., van Dijk, L. V., Hutcheson, K. A., Moreno, A. M., Fuller, C. D., Canahuate, G. M., & Marai, G. E. (2023) · Frontiers in Oncology · doi:10.3389/fonc.2023.1210087 · Cited by 7 · see citing works
Rösler, W., Roiss, M., & Widmer, C. (2024) · healthbook TIMES Oncology Hematology · doi:10.36000/hbt.oh.2024.20.146 · Cited by 6 · see citing works
Important Design Questions for Algorithmic Ethics Consultation
Char, D. S. (2022) · The American Journal of Bioethics · doi:10.1080/15265161.2022.2075054 · Cited by 6 · see citing works
Augmenting care in hepatocellular carcinoma with artificial intelligence
Xu, F. W. X., Tang, S. S. K., Soh, H. N., Pang, N. Q., & Bonney, G. K. (2023) · Artificial Intelligence Surgery · doi:10.20517/ais.2022.33 · Cited by 5 · see citing works
Predicting emergency department visits among children with asthma in two academic medical systems
Gorham, T. J., Tumin, D., Groner, J. A., Allen, E. D., Retzke, J., Hersey, S., Liu, S. B., Macias, C., Alachraf, K., Smith, A. W., et al. (2023) · Journal of Asthma · doi:10.1080/02770903.2023.2225603 · Cited by 4 · see citing works
European regulation of digital respiratory healthcare
Gilbert, S. (2023) · European Respiratory Society eBooks · doi:10.1183/2312508x.10000923 · Cited by 5 · see citing works
Wilson, B., Natali, C., Roach, M., Scott, D., Rahat, A. A.-A. M., Rawlinson, D. R., & Cabitza, F. (2025) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-025-09514-4 · Cited by 3 · see citing works
Thomas, N. R., Anitha, J., Popîrlan, C., Popirlan, C.-I., & Hemanth, D. J. (2025) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2025.070689 · Cited by 3 · see citing works
Defining operational safety in clinical artificial intelligence systems
Kim, Y.-T., Kim, H., Bahl, M., Lev, M., González, R. G., Gee, M. S., & Do, S. (2026) · npj Digit. Medicine · doi:10.1038/s41746-026-02450-7 · Cited by 4 · see citing works
Strengthening Machine Learning Reproducibility for Image Classification
Shao, G., Zhang, H., Shao, J., Woeste, K., & Tang, L. (2022) · Advances in Artificial Intelligence and Machine Learning · doi:10.54364/aaiml.2022.1132 · Cited by 3 · see citing works
Modeling Access Differences to Reduce Disparity in Resource Allocation
Andrews, K., Ohannessian, M. I., & Berger-Wolf, T. Y. (2022) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3551624.3555302 · Cited by 2 · see citing works
Shaw, K. M., Shao, Y., Ghanta, M., Junior, V. F. M., Kimchi, E. Y., Houle, T. T., Akeju, O., & Westover, M. B. (2024) · JMIR Medical Informatics · doi:10.2196/60442 · Cited by 2 · see citing works
Impact of an AI prognostic tool on clinician performance in colorectal liver metastases
Chen, Q., Tong, J., Deng, Y., Bi, X., Li, Y., Li, K., & Zhao, H. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02606-5 · Cited by 1 · see citing works
Artificial Intelligence and Pain Medicine: an Introduction [Letter]
Cascella, M., Monaco, F., & Piazza, O. (2024) · Journal of Pain Research · doi:10.2147/jpr.s476359 · Cited by 1 · see citing works
Pedersen, M., Pardoe, H., de Weger, A., Hutchison, D. M., Abbott, D. F., Verspoor, K. M., & Jackson, G. D. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103192 · Cited by 5 · see citing works
Application of artificial intelligence to overcome clinical information overload in cancer
Stenzl, A., Ghith, J., & Schijvenaars, B. J. A. (2025) · Oxford University Press eBooks · doi:10.1093/med/9780192867636.003.0022
Multimorbidity measurement strategies for predicting hospital visits
Neves, B. D., Moreira, J. M., Gonçalves, S., Cerejo, J., Mota, I., Silva, N. A., Leite, F., & Silva, M. J. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-13817-z
Huo, T., Chen, X., Liu, P., Liu, J., Yan, Z., Yang, J., Liu, S., Lu, L., Zhang, J., Shao, J., et al. (2026) · International Journal of Surgery · doi:10.1097/js9.0000000000005138
Trust by design: Crossing the chasm between clinical AI/ML innovation and practice
Idris, M. Y., & Chalk, M. B. (2026) · Journal of the National Medical Association · doi:10.1016/j.jnma.2026.03.008
Emovon, A., Driggers-Jones, L., Engelhard, M., Maslow, G., Dawson, G., Goldstein, B. A., & Franz, L. (2026) · JAMIA Open · doi:10.1093/jamiaopen/ooag145
Masood, H., Bajwa, M. N., Shafait, S., Fraz, M. M., Faisal Shafait, B., & Khan, H. A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-35390-0_6
Artificial Intelligence in Physiotherapy and the Limits of Clinical Reasoning: A Narrative Review
Çırak, N. (2026) · İstanbul Esenyurt Üniversitesi Sağlık Bilimleri Dergisi · doi:10.67801/iesujhs.1971525
From algorithm performance to clinical value in AI-assisted skin cancer diagnostics
Ocker, L., Livingstone, E., Lodde, G. C., Stockfleth, E., Rached, N. A., Schadendorf, D., & Nüesch, S. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1896521
Özel, P., & Olamat, A. (2026) · The International Journal of Frontier Sciences · doi:10.37978/tijfs.v08i01.005
Ringel, C., Terhardt, A., Sure, M., Kaftan, P., Schlüter, C., Dakkak, S. A., Staritzbichler, R., & Kaiser, T. (2026) · Journal of Personalized Medicine · doi:10.3390/jpm16110555
- Authors
- Marwaha, J. S., & Kvedar, J. C.
- Affiliations
- Beth Israel Deaconess Medical Center (US)
- Harvard University (US)
- Corresponding author
- Marwaha, Jayson S.
- Venue
- npj Digital Medicine
- Published
- 2022-03-03
- Volume
- 5
- Issue
- 1
- Pages
- 25-25
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC8894388
- OpenAlex
- W4214874725
- Cited by
- 46
- References
- 9
- Citations by year
- 2026: 10 · 2025: 7 · 2024: 10 · 2023: 13 · 2022: 6
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- U.S. Department of Health & Human Services | NIH | U.S. National Library of Medicine (T15LM007092)
- Harvard University (Biomedical Informatics and Data Science Research Training (BIRT) Program)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- clinical implementation of AI, AI implementation, prediction model performance, machine learning prediction tool
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
