The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index; 2 of the 7 sources its claims cite were checked for support.
The Clinician and Dataset Shift in Artificial Intelligence
Open accessAbstract
Dataset Shift in Clinical Trials This letter outlines how to identify, and potentially mitigate, common sources of “dataset shift” in machine-learning systems. This occurs when the model “training ...
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 of 7 cited sources checked · 1 supported · 0 flagged · accuracy not yet measured
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“A major driver of AI system malfunction is known as “dataset shift.”4,5”
§ The Clinician and Dataset Shift in Artificial Intelligence > TO THE EDITOR:
What the sources say
- Supportedcites [4] Subbaswamy A, Saria S. From development to deployment: dataset shift, causality, and shift-stable models in health AI. Biostatistics 2020; 21: 345– 52. 31742354 source
The paper’s opening discussion says shifts between training and deployment are common and can cause poor performance or dangerous decisions. It explicitly calls this problem of shifting conditions that prevent generalization “dataset shift,” supporting the citing sentence’s paraphrase. (LLM, conf=0.92)
- Not checked yetcites [5] Saria S, Subbaswamy A. Tutorial: safe and reliable machine learning. In: Proceedings of the conference on fairness, accountability, and transparency, January 29–31, 2019. New York: Association for Computing Machinery, 2019. DOI: 10.1056/NEJMc2104626
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Rests on an assumption: Most clinical AI systems today use machine learning.
- Supported
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This was a case in which dataset shift fundamentally altered the relationship between fevers and bacterial sepsis, leading the hospital’s clinical AI governing committee (which one of the authors of this letter chairs) to decommission its use.”
§ The Clinician and Dataset Shift in Artificial Intelligence > TO THE EDITOR:
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes- Rests on an assumption: In April 2020, the University of Michigan Hospital deactivated the Epic sepsis-alerting model because of spurious alerting owing to changes in patients’ demographic characteristics associated with the COVID-19 pandemic.
In the paper
“Artificial intelligence (AI) systems are now regularly being used in medical settings,1”
§ The Clinician and Dataset Shift in Artificial Intelligence > TO THE EDITOR:
What the sources say
- Not checked — no full text of the sourcecites [1] Rajkomar A, Dean J, Kohane I. Machine learning in medicine. N Engl J Med 2019; 380: 1347– 58. 30943338 source
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Probably not a substantial claim. 3
- It may state two separate claims as one. 2
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Successful recognition and mitigation of dataset shift require both vigilant clinicians and sound technical oversight through AI governance teams.4,5”
§ The Clinician and Dataset Shift in Artificial Intelligence > TO THE EDITOR:
What the sources say
- Partly supportedcites [4] Subbaswamy A, Saria S. From development to deployment: dataset shift, causality, and shift-stable models in health AI. Biostatistics 2020; 21: 345– 52. 31742354 sourceExperimental — accuracy not yet measured
Section 4, “Open Challenges and Looking Forward” (page not supplied), calls for monitoring to detect shifts, maintenance to address them, and collaboration with domain experts when constructing models. It does not say that vigilant clinicians and AI governance teams are both required for this work. (LLM, conf=0.86)
- Not checked yetcites [5] Saria S, Subbaswamy A. Tutorial: safe and reliable machine learning. In: Proceedings of the conference on fairness, accountability, and transparency, January 29–31, 2019. New York: Association for Computing Machinery, 2019. DOI: 10.1056/NEJMc2104626
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
- Partly supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“we group into changes in technology (e.g., software vendors), changes in population and setting (e.g., new demographics), and changes in behavior (e.g., new reimbursement incentives)”
§ The Clinician and Dataset Shift in Artificial Intelligence > TO THE EDITOR:
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“although regulatory oversight is inconsistent and undeveloped.2,3”
§ The Clinician and Dataset Shift in Artificial Intelligence > TO THE EDITOR:
What the sources say
- Not checked yetcites [2] Ross C Hospitals are using AI to predict the decline of Covid-19 patients — before knowing it works. Stat 4 24, 2020 (https://www.statnews.com/2020/04/24/coronavirus-hospitals-use-ai-to-predict-patient-decline-before-knowing-it-works/).
- Not checked yetcites [3] Artificial intelligence/machine learning (AI/ML)-based software as a medical device (SaMD) action plan Silver Spring, MD: Food and Drug Administration, January 2021 (https://www.fdagov/media/145022/download).
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Safe deployment of clinical AI requires informed clinician-users”
§ The Clinician and Dataset Shift in Artificial Intelligence > TO THE EDITOR:
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
- Rests on an assumption: Clinicians are generally responsible for identifying and reporting emerging problems with clinical AI.
The paper relies on this without support.
Clinicians are generally responsible for identifying and reporting emerging problems with clinical AI.
“who are generally responsible for identifying and reporting emerging problems.”
The paper relies on this without support.
Most clinical AI systems today use machine learning.
“Most clinical AI systems today use machine learning, algorithms that leverage statistical methods to learn key patterns from clinical data.”
The paper relies on this without support.
In April 2020, the University of Michigan Hospital deactivated the Epic sepsis-alerting model because of spurious alerting owing to changes in patients’ demographic characteristics associated with the COVID-19 pandemic.
“the University of Michigan Hospital implemented the widely used sepsis-alerting model developed by Epic Systems; in April 2020, the model had to be deactivated because of spurious alerting owing to changes in patients’ demographic characteristics associated with the coronavirus disease 2019 pandemic.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This quality becomes crucial when predictive models are deployed across different hospitals or healthcare systems, where variations in patient demographics, clinical practices, and data collection procedures can lead to inaccurate predictions [17, 18].”
From Toward generalizable and interpretable machine learning models in healthcare: Insights from ICU outcome predictions (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Small changes in the distribution of the underlying data between the algorithm training and clinical evaluation populations (i.e., dataset shift) can lead to substantial variation in clinical performance and expose patients to potential unexpected harm [21, 22].”
From Use of artificial intelligence in critical care: opportunities and obstacles (2024) · cites this paper as [21]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Finlayson et al. argue that dataset shift in medicine is “the rule rather than the exception,” driven by changes in care delivery, clinical incentives, and documentation rather than by random variation alone [8].”
From Validation is not enough: Longitudinal evidence of post-deployment fragility in clinical AI systems (2026) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Between 3 and 4 weeks after its first COVID-19 hospitalization, the University of Michigan paused ESM-generated alerts in April 2020 after nursing reports of overalerting.5”
From Quantification of Sepsis Model Alerts in 24 US Hospitals Before and During the COVID-19 Pandemic (2021) · cites this paper as [5]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Finlayson et al 55 have argued that clinicians who are well-prepared to recognize when AI tools are misbehaving are essential partners in safe deployment, and that institutions need governance structures, monitoring protocols, and clear pathways for frontline staff to escalate concerns.”
From Artificial Intelligence in Oncology: Practical Applications Across Clinical Care, Scholarship, and Translation. (2026) · cites this paper as [55]
The cited source says what the claim attributes to it.
In the citing paper
“Understanding persistence and degradation is perhaps particularly critical for ML models, where dataset shift should be expected to cause declines in performance when the data used to train a model begins to look different from the data in a new context [8].”
From Managing a 'responsibility vacuum' in AI monitoring and governance in healthcare: a qualitative study (2025) · cites this paper as [8]
The cited source says what the claim attributes to it.
In the citing paper
“This supports a workflow-aware interpretation of post-deployment fragility: degradation may be linked to how data are produced within evolving workflows and information systems, not only to who the patients are [8,18,33,35].”
From Validation is not enough: Longitudinal evidence of post-deployment fragility in clinical AI systems (2026) · cites this paper as [8]
The cited source says what the claim attributes to it.
In the citing paper
“As AI solutions have emerged, a wide variety of governance best practices in the form of frameworks, governance structures, checklists, and guidelines have been proposed7–23.”
From Advancing healthcare AI governance through a comprehensive maturity model based on systematic review (2026) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“It is well-known that such models may not generalize across settings or over time due to changes in patient populations, practice patterns, and data processes — a phenomenon known as dataset shift.1”
From Challenges in the postmarket surveillance of clinical prediction models. (2025) · cites this paper as [1]
In the citing paper
“The healthcare landscape is constantly changing; therefore, there is no guarantee that an AI algorithm with high performance will maintain its high performance in the future [89].”
From Fairness of artificial intelligence in healthcare: review and recommendations (2024) · cites this paper as [89]
In the citing paper
“The models should be monitored regularly to address biases that may emerge over time [89].”
From Fairness of artificial intelligence in healthcare: review and recommendations (2024) · cites this paper as [89]
In the citing paper
“Consequently, these changes necessitate the periodic retraining and re-evaluation of AI systems to maintain their relevance and efficacy [69, 70].”
From Integrating machine learning and artificial intelligence in life-course epidemiology: pathways to innovative public health solutions (2024) · cites this paper as [70]
In the citing paper
“The dynamic nature of clinical practice, fluctuations in the prevalence of comorbidities, and various socio-environmental factors can precipitate shifts in the distribution of patient characteristics.”
From Integrating machine learning and artificial intelligence in life-course epidemiology: pathways to innovative public health solutions (2024) · cites this paper as [70]
In the citing paper
“Laboratory performance does not always translate directly to clinical practice [142,143,144,145,146].”
From Artificial Intelligence in Clinical Oncology: From Productivity Enhancement to Creative Discovery (2025) · cites this paper as [144]
Citations
Cites
Rajkomar, A., Dean, J., & Kohane, I. (2019) · New England Journal of Medicine · doi:10.1056/nejmra1814259 · Cited by 4,579
Self-citationFrom development to deployment: dataset shift, causality, and shift-stable models in health AI
Subbaswamy, A., & Saria, S. (2019) · Biostatistics · doi:10.1093/biostatistics/kxz041 · Cited by 276
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- Saria S Subbaswamy A. Tutorial: safe and reliable machine learning. In: Proceedings of the conference on fairness accountability and transparency January 29–31 2019. New York: Association for Computing Machinery 2019.
Cited by
Sorted by citations
Items 101–200 of 848
Developing medical imaging AI for emerging infectious diseases
Huang, S.-C., Chaudhari, A. S., Langlotz, C. P., Shah, N. H., Yeung, S. Y., & Lungren, M. P. (2022) · Nature Communications · doi:10.1038/s41467-022-34234-4 · Cited by 42
Digitising the prediction and management of sepsis
Kheterpal, S., Singh, K., & Topol, E. J. (2022) · The Lancet · doi:10.1016/s0140-6736(22)00658-4 · Cited by 41
Self-citationBending the patient safety curve: how much can AI help?
Classen, D. C., Longhurst, C., & Thomas, E. J. (2023) · npj Digital Medicine · doi:10.1038/s41746-022-00731-5 · Cited by 44
Strumpf, Z. B., Gu, W., Tsai, C.-W., Chen, P., Yeh, E., Leung, L., Cheung, C., Wu, I.-C., Strohl, K. P., Tsai, T., et al. (2023) · Sleep Health · doi:10.1016/j.sleh.2023.05.001 · Cited by 47
Automatic correction of performance drift under acquisition shift in medical image classification
Roschewitz, M., Khara, G. S., Yearsley, J., Sharma, N., James, J. J., Ambrózay, É., Heroux, A., Kecskemethy, P. D., Rijken, T. M., & Glocker, B. (2023) · Nature Communications · doi:10.1038/s41467-023-42396-y · Cited by 49
Ethics and artificial intelligence
Galiana, L. I., Corral-Gudino, L., & Miramontes‐González, J. P. (2024) · Revista Clínica Española (English Edition) · doi:10.1016/j.rceng.2024.02.003 · Cited by 53
McCoy, L. G., Ng, F. Y. C., Sauer, C. M., Legaspi, K. E. Y., Jain, B., Gallifant, J., McClurkin, M., Hammond, A., Goode, D., Gichoya, J. W., et al. (2024) · BMC Medical Education · doi:10.1186/s12909-024-06048-z · Cited by 52
“Note Bloat” impacts deep learning-based NLP models for clinical prediction tasks
Liu, J., Daniel Capurro, N., Nguyen, A. N., & Verspoor, K. M. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104149 · Cited by 42
Deep learning to detect left ventricular structural abnormalities in chest X-rays
Bhave, S., Rodriguez, V., Poterucha, T., Mutasa, S., Aberle, D., Capaccione, K. M., Chen, Y., Dsouza, B., Dumeer, S., Goldstein, J., et al. (2024) · European Heart Journal · doi:10.1093/eurheartj/ehad782 · Cited by 38
Artificial intelligence in the neonatal intensive care unit: the time is now
Beam, K., Sharma, P. K., Levy, P. T., & Beam, A. L. (2023) · Journal of Perinatology · doi:10.1038/s41372-023-01719-z · Cited by 44
Artificial Intelligence and Pain Medicine: An Introduction
Hagedorn, J. M., George, T. K., Aiyer, R., Schmidt, K., Halamka, J. D., & D'Souza, R. S. (2024) · Journal of Pain Research · doi:10.2147/jpr.s429594 · Cited by 42
Thériault-Lauzier, P., Corbin, D., Tastet, O., Langlais, É. L., Taji, B., Kang, G., Chong, A., So, D. Y. F., Tang, A., Gichoya, J. W., et al. (2024) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2024.05.025 · Cited by 42
Machine Learning for the Orthopaedic Surgeon
Alsoof, D., McDonald, C. L., Kuris, E. O., & Daniels, A. H. (2022) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.21.01305 · Cited by 40
The diagnostic, therapeutic, and ethical impact of artificial intelligence in modern medicine
Katwaroo, A. R., Adesh, V. S., Lowtan, A., & Umakanthan, S. (2023) · Postgraduate Medical Journal · doi:10.1093/postmj/qgad135 · Cited by 44
Evaluating transparency in AI/ML model characteristics for FDA-reviewed medical devices
Mehta, V., Komanduri, A., Bhadouriya, R. S., Mehta, V., Johnson, M. D., Shrestha, P., Nikolov, M. C., Jain, B., Shah, N. H., & Schulman, K. A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02052-9 · Cited by 47
Artificial Intelligence in Infection Management in the ICU
Corte, T. D., Hoecke, S. V., & Waele, J. J. D. (2022) · Critical Care · doi:10.1186/s13054-022-03916-2 · Cited by 38
Tsitou, V.-M., Rallis, D., Cekova, M., & Yanev, N. (2024) · Biotechnology & Biotechnological Equipment · doi:10.1080/13102818.2024.2349587 · Cited by 49
Chen, S., & Bai, W. (2025) · Frontiers in Cell and Developmental Biology · doi:10.3389/fcell.2025.1576465 · Cited by 40
Yang, H. S., Rhoads, D. D., Sepulveda, J. L., Zang, C., Chadburn, A., & Wang, F. (2022) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2021-0635-ra · Cited by 40
Cresswell, K. M., Rigby, M., Magrabi, F., Scott, P., Brender, J., Craven, C. K., Wong, Z. S.-Y., Kukhareva, P., Ammenwerth, E., Georgiou, A., et al. (2023) · Health Policy · doi:10.1016/j.healthpol.2023.104889 · Cited by 48
A Framework for Interpretability in Machine Learning for Medical Imaging
Wang, A. Q., Karaman, B. K., Kim, H., Rosenthal, J., Saluja, R., Young, S. I., & Sabuncu, M. R. (2024) · IEEE Access · doi:10.1109/access.2024.3387702 · Cited by 41
Algorithmovigilance, lessons from pharmacovigilance
Balendran, A., Benchoufi, M., Evgeniou, T., & Ravaud, P. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01237-y · Cited by 34
Mccradden, M., Odusi, O., Joshi, S., Akrout, I., Ndlovu, K., Glocker, B., Maicas, G., Liu, X., Mazwi, M., Garnett, T., et al. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3594096 · Cited by 33
Advances in Machine Learning Approaches to Heart Failure with Preserved Ejection Fraction
Ahmad, F. S., Luo, Y., Wehbe, R. M., Thomas, J. D., & Shah, S. J. (2022) · Heart Failure Clinics · doi:10.1016/j.hfc.2021.12.002 · Cited by 34
Open questions and research gaps for monitoring and updating AI-enabled tools in clinical settings
Davis, S. E., Walsh, C. G., & Matheny, M. E. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.958284 · Cited by 33
Fogelberg, K., Chamarthi, S., Maron, R. C., Niebling, J., & Brinker, T. J. (2023) · New Biotechnology · doi:10.1016/j.nbt.2023.04.006 · Cited by 42
Heitmann, J., Glangetas, A., Doenz, J., Dervaux, J., Shama, D. M., Hinjos, D., Benissa, M.-R., Cantais, A., Perez, A., Müller, D., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00838-3 · Cited by 33
Reliability in Machine Learning
Grote, T., Genin, K., & Sullivan, E. (2024) · Philosophy Compass · doi:10.1111/phc3.12974 · Cited by 45
AI-guided precision parenteral nutrition for neonatal intensive care units
Phongpreecha, T., Ghanem, M., Reiss, J. D., Oskotsky, T., Mataraso, S. J., Francesco, D. D., Reincke, S. M., Espinosa, C. A., Chung, P., Ng, T., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03601-1 · Cited by 37
Silva, G. D. S., Barcellos, F., Wichmann, R. M., da Silva, F. C., & Chiavegatto, A. D. P. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104902 · Cited by 37
Sanchez-Pinto, L. N., López, M. D. P. A., Scott, H. F., GIBBONS, K. S., Moor, M., Watson, R. S., Wiens, M. O., Schlapbach, L. J., & Bennett, T. D. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00141-9 · Cited by 34
Machine Learning for Acute Kidney Injury Prediction in the Intensive Care Unit
Gottlieb, E. R., Samuel, M. C., Bonventre, J. V., Celi, L. A., & Mattie, H. (2022) · Advances in Chronic Kidney Disease · doi:10.1053/j.ackd.2022.06.005 · Cited by 36
Evans, H., & Snead, D. R. J. (2023) · Histopathology · doi:10.1111/his.15071 · Cited by 37
Malerba, S., Vladimirov, M., Goyal, A., Dulskas, A., Baušys, A., Cwalinski, T., Girnyi, S., Skokowski, J., Duka, R., Молчанов, Р., et al. (2026) · Journal of Clinical Medicine · doi:10.3390/jcm15062208 · Cited by 33
The Causal Cookbook: Recipes for Propensity Scores, G-Computation, and Doubly Robust Standardization
Chatton, A., & Rohrer, J. M. (2024) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459241236149 · Cited by 39
Foroushani, H. M., Hamzehloo, A., Kumar, A., Chen, Y., Heitsch, L., Słowik, A., Strbian, D., Lee, J., Marcus, D. S., & Dhar, R. (2021) · Neurocritical Care · doi:10.1007/s12028-021-01325-x · Cited by 33
Chen, Y., Taib, A. G., Darker, I. T., & James, J. J. (2023) · Radiology · doi:10.1148/radiol.223299 · Cited by 33
Algorithm Fairness in AI for Medicine and HealthcarePreprint
Chen, R. J., Chen, T., Lipková, J., Wang, J. J., Williamson, D. F. K., Lu, M. Y., Sahai, S., & Mahmood, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.00603 · Cited by 31
Machine learning for healthcare that matters: Reorienting from technical novelty to equitable impact
Balagopalan, A., Baldini, I., Celi, L. A., Gichoya, J. W., McCoy, L. G., Naumann, T., Shalit, U., van der Schaar, M., & Wagstaff, K. L. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000474 · Cited by 34
Implementation and Updating of Clinical Prediction Models: A Systematic Review
Saelmans, A., Seinen, T. M., Pera, V., Markus, A. F., Fridgeirsson, E. A., John, L. H., Schiphof‐Godart, L., Rijnbeek, P. R., Reps, J., & Williams, R. D. (2025) · Mayo Clinic Proceedings Digital Health · doi:10.1016/j.mcpdig.2025.100228 · Cited by 34
Wagner, S. K., Liefers, B. J., Radia, M., Zhang, G., Struyven, R., Faes, L., Than, J., Balal, S., Hennings, C., Kilduff, C. L. S., et al. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00050-x · Cited by 29
Generative artificial intelligence: implications for biomedical and health professions education
Hersh, W. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-094756 · Cited by 28
Attia, A. M., Rowe, I. A. C., Harrison, E. M., Gordon‐Walker, T., & Stutchfield, B. M. (2023) · The Lancet · doi:10.1016/s0140-6736(23)00114-9 · Cited by 28
Mathis, M. R., Steffner, K. R., Subramanian, H., Gill, G., Girardi, N. I., Bansal, S., Bartels, K., Khanna, A. K., & Huang, J. (2024) · Journal of Cardiothoracic and Vascular Anesthesia · doi:10.1053/j.jvca.2024.02.004 · Cited by 30
Afrifa‐Yamoah, E., Adua, E., Peprah‐Yamoah, E., Anto, E. O., Opoku‐Yamoah, V., Acheampong, E., Macartney, M. J., & Hashmi, R. (2024) · Chronic Diseases and Translational Medicine · doi:10.1002/cdt3.137 · Cited by 34
The Use of Artificial Intelligence for Cancer Therapeutic Decision-Making
Elemento, O., Khozin, S., & Sternberg, C. N. (2025) · NEJM AI · doi:10.1056/aira2401164 · Cited by 28
Armato, S. G., Drukker, K., & Hadjiiski, L. M. (2023) · British Journal of Radiology · doi:10.1259/bjr.20221152 · Cited by 31
Promise and Perils of Large Language Models for Cancer Survivorship and Supportive Care
Bitterman, D. S., Downing, A., Maués, J., & Lustberg, M. B. (2024) · Journal of Clinical Oncology · doi:10.1200/jco.23.02439 · Cited by 28
Enabling collaborative governance of medical AI
Price, W. N., Sendak, M., Balu, S., & Singh, K. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00699-1 · Cited by 27
Self-citationNot All Clinical AI Monitoring Systems Are Created Equal: Review and Recommendations
Feng, J., Xia, F., Singh, K., & Pirracchio, R. (2025) · NEJM AI · doi:10.1056/aira2400657 · Cited by 28
Self-citationA Review of the Clinical Applications of Artificial Intelligence in Abdominal Imaging
Mervak, B. M., Fried, J. G., & Wasnik, A. P. (2023) · Diagnostics · doi:10.3390/diagnostics13182889 · Cited by 26
Large language models in neurosurgery: a systematic review and meta-analysis
Patil, A., Serrato, P., Chisvo, N. S., Arnaout, O. M., See, P. A., & Huang, K. T. (2024) · Acta Neurochirurgica · doi:10.1007/s00701-024-06372-9 · Cited by 26
Association of Postoperative Undertriage to Hospital Wards With Mortality and Morbidity
Loftus, T. J., Ruppert, M. M., Ozrazgat‐Baslanti, T., Balch, J. A., Efron, P. A., Tighe, P. J., Hogan, W. R., Rashidi, P., Upchurch, G. R., & Bihorac, A. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.31669 · Cited by 26
Liu, L., Wu, X., Lin, D., Zhao, L., Li, M., Yun, D., Lin, Z., Pang, J., Li, L., Wu, Y., et al. (2023) · Cell Reports Medicine · doi:10.1016/j.xcrm.2022.100912 · Cited by 26
Duan, J., Vargas, C., Yu, N. Y., Laughlin, B. S., Toesca, D. A. S., Keole, S. R., Rwigema, J. M., Wong, W. W., Schild, S. E., Feng, X., et al. (2023) · Medical Physics · doi:10.1002/mp.16537 · Cited by 26
Jain, S. S., Goto, S., Hall, J. L., Khan, S. S., MacRae, C. A., Ofori, C., Pegus, C., Pencina, M., Peterson, E. D., Schwamm, L. H., et al. (2025) · Circulation · doi:10.1161/cir.0000000000001400 · Cited by 23
Power and reproducibility in the external validation of brain-phenotype predictions
Rosenblatt, M., Tejavibulya, L., Sun, H., Camp, C., Khaitova, M., Adkinson, B. D., Jiang, R., Westwater, M. L., Noble, S., & Scheinost, D. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01931-7 · Cited by 26
Out-of-Distribution Detection and Radiological Data Monitoring Using Statistical Process Control
Zamzmi, G., Venkatesh, K., Nelson, B., Prathapan, S., Yi, P. H., Sahiner, B., & Delfino, J. G. (2024) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-024-01212-9 · Cited by 35
Understanding Model Drift and Its Impact on Health Care Policy
Wong, A. H., & Sussman, J. B. (2025) · JAMA Health Forum · doi:10.1001/jamahealthforum.2025.2724 · Cited by 25
Clinical Decision Support for Laboratory Testing
Hughes, A. E. O., & Jackups, R. (2021) · Clinical Chemistry · doi:10.1093/clinchem/hvab201 · Cited by 26
Mapping the Landscape of Care Providers’ Quality Assurance Approaches for AI in Diagnostic Imaging
Lundström, C. F., & Lindvall, M. (2022) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-022-00731-7 · Cited by 28
From FDG and beyond: the evolving potential of nuclear medicine
Hirata, K., Kamagata, K., Ueda, D., Yanagawa, M., Kawamura, M., Nakaura, T., Ito, R., Tatsugami, F., Matsui, Y., Yamada, A., et al. (2023) · Annals of Nuclear Medicine · doi:10.1007/s12149-023-01865-6 · Cited by 25
Mind the Gap — Machine Learning, Dataset Shift, and History in the Age of Clinical Algorithms
Lea, A. S., & Jones, D. S. (2024) · New England Journal of Medicine · doi:10.1056/nejmp2311015 · Cited by 25
Proactive vs Reactive Machine Learning in Health Care: Lessons From the COVID-19 Pandemic
Luo, Y., Wunderink, R. G., & Lloyd-Jones, D. (2022) · JAMA · doi:10.1001/jama.2021.24935 · Cited by 24
An operational guide to translational clinical machine learning in academic medical centers
Poddar, M., Marwaha, J. S., Yuan, W., Brufau, S. R., & Brat, G. A. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01094-9 · Cited by 24
Revolutionising Acute Cardiac Care With Artificial Intelligence: Opportunities and Challenges
Doolub, G., Khurshid, S. S., Thériault-Lauzier, P., Lapalme, A. N., Tastet, O., So, D. Y. F., Langlais, É. L., Corbin, D., & Avram, R. (2024) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2024.06.011 · Cited by 57
Zang, C., Hou, Y., Lyu, D., Jin, J., Sacco, S. J., Chen, K., Aseltine, R. H., & Wang, F. (2024) · Translational Psychiatry · doi:10.1038/s41398-024-03034-3 · Cited by 25
Feng, G., Weng, F., Lu, W. P., Xu, L., Zhu, W., Tan, M., & Weng, P. (2025) · International Journal of General Medicine · doi:10.2147/ijgm.s516247 · Cited by 31
Black Swan Events and Intelligent Automation for Routine Safety Surveillance
Kjoersvik, O., & Bate, A. (2022) · Drug Safety · doi:10.1007/s40264-022-01169-0 · Cited by 24
Chiang, D.-H., Tian, C., Zeyu, J., Yu-Shan, O.-Y., & Yung-Yan, L. (2022) · Critical Care · doi:10.1186/s13054-022-04088-9 · Cited by 24
Mathis, M. R., Engoren, M. C., Williams, A. M., Biesterveld, B. E., Croteau, A. J., Cai, L., Kim, R. B., Liu, G., Ward, K. R., Najarian, K., et al. (2022) · Anesthesiology · doi:10.1097/aln.0000000000004345 · Cited by 24
Machine Learning and Statistics in Clinical Research Articles—Moving Past the False Dichotomy
Finlayson, S. G., Beam, A. L., & van Smeden, M. (2023) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/jamapediatrics.2023.0034 · Cited by 24
Self-citationPham, N., Hill, V. B., Rauschecker, A. M., Lui, Y. W., Niogi, S. N., Filippi, C. G., Chang, P., Zaharchuk, G., & Wintermark, M. (2023) · American Journal of Neuroradiology · doi:10.3174/ajnr.a7850 · Cited by 26
Early Breast Cancer Risk Assessment: Integrating Histopathology with Artificial Intelligence
Ivanova, M. D., Pescia, C., Trapani, D., Venetis, K., Frascarelli, C., Mane, E., Cursano, G., Sajjadi, E., Scatena, C., Cerbelli, B., et al. (2024) · Cancers · doi:10.3390/cancers16111981 · Cited by 25
Performance of a Deep Learning Diabetic Retinopathy Algorithm in India
Brant, A. R., Singh, P., Yin, X., Yang, L., Nayar, J., Jeji, D., Matias, Y., Corrado, G. S., Webster, D. R., Virmani, S., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.0984 · Cited by 23
van der Werff, S. D., van Rooden, S. M., Henriksson, A., Behnke, M., Aghdassi, S. J. S., van Mourik, M. S. M., & Nauclér, P. (2025) · Journal of Internal Medicine · doi:10.1111/joim.20100 · Cited by 28
Mantena, S., Arévalo, A. R., Maley, J. H., Vieira, S. M., Mateo-Collado, R., Sousa, J. M. C., & Celi, L. A. (2021) · Journal of Clinical Monitoring and Computing · doi:10.1007/s10877-021-00760-7 · Cited by 23
Recalibration Methods for Improved Clinical Utility of Risk Scores
Mishra, A., McClelland, R. L., Inoue, L. Y. T., & Kerr, K. F. (2021) · Medical Decision Making · doi:10.1177/0272989x211044697 · Cited by 23
Baumgart, D. C. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01005-y · Cited by 28
Leveraging reimbursement strategies to guide value-based adoption and utilization of medical AI
Venkatesh, K. P., Raza, M. M., Diao, J. A., & Kvedar, J. C. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00662-1 · Cited by 23
Artificial Intelligence in Cardiovascular Disease Prevention: Is it Ready for Prime Time?
Parsa, S., Somani, S. S., Dudum, R., Jain, S. S., & Rodríguez, F. (2024) · Current Atherosclerosis Reports · doi:10.1007/s11883-024-01210-w · Cited by 22
Performance of Machine Learning Suicide Risk Models in an American Indian Population
Haroz, E. E., Rebman, P., Goklish, N. A., Garcia, M., Suttle, R., Maggio, D., Clattenburg, E. J., Mega, J., & Adams, R. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.39269 · Cited by 23
Challenges in the postmarket surveillance of clinical prediction models.
Ansari, S., Baur, B., Singh, K., & Admon, A. J. (2025) · NEJM AI · doi:10.1056/aip2401116 · Cited by 22
Self-citationGastroNet-5M: A Multicenter Dataset for Developing Foundation Models in Gastrointestinal Endoscopy
Jong, M. R., Boers, T. G. W., Fockens, K. N., Jukema, J. B., Kusters, C. H. J., Jaspers, T. J. M., van Heslinga, R. A. H. V. E., Slooter, F. C., Struyvenberg, M. R., Bisschops, R., et al. (2025) · Gastroenterology · doi:10.1053/j.gastro.2025.07.030 · Cited by 22
A unifying causal framework for analyzing dataset shift-stable learning algorithms
Subbaswamy, A., Chen, B., & Saria, S. (2022) · Journal of Causal Inference · doi:10.1515/jci-2021-0042 · Cited by 22
Self-citationOtokiti, A. U., Ozoude, M. M., Williams, K. S., Sadiq-Onilenla, R. A., Ojo, S. A., Warsame, L., Walsh, S., & Edomwande, M. (2022) · JMIR Research Protocols · doi:10.2196/37685 · Cited by 26
Ivanics, T., So, D., Claasen, M. P. A. W., Wallace, D., Patel, M. S., Gravely, A., Choi, W. J., Shwaartz, C. G., Walker, K., Erdman, L., et al. (2023) · American Journal of Transplantation · doi:10.1016/j.ajt.2022.12.002 · Cited by 22
Foy, B. H., Stefely, J. A., Bendapudi, P. K., Hasserjian, R. P., Al-Samkari, H. T., Louissaint, A., Fitzpatrick, M. J., Hutchison, B. M., Mow, C., Collins, J., et al. (2023) · Blood Advances · doi:10.1182/bloodadvances.2022008967 · Cited by 22
APPFLx: Providing Privacy-Preserving Cross-Silo Federated Learning as a Service
Li, Z., He, S., Chaturvedi, P., Hoàng, T.-H., Ryu, M., Huerta, E. A., Kindratenko, V., Fuhrman, J. D., Giger, M. L., Chard, R., et al. (2023) · IEEE International Conference on eScience · doi:10.1109/e-science58273.2023.10254842 · Cited by 21
Utilizing large language models for gastroenterology research: a conceptual framework
Berry, P., Dhanakshirur, R. R., & Khanna, S. (2025) · Therapeutic Advances in Gastroenterology · doi:10.1177/17562848251328577 · Cited by 26
Loftus, T. J., Ruppert, M. M., Shickel, B., Ozrazgat‐Baslanti, T., Balch, J. A., Hu, D., Javed, A., Madbak, F., Skarupa, D. J., Guirgis, F. W., et al. (2022) · Journal of the American College of Surgeons · doi:10.1097/xcs.0000000000000471 · Cited by 21
Brandão‐de‐Resende, C., Melo, M., Lee, E., Jindal, A., Neo, Y. N., Sanghi, P., Freitas, J. R., Castro, P. V. I. P., Rosa, V. O. M., Valentim, G., et al. (2023) · EClinicalMedicine · doi:10.1016/j.eclinm.2023.102331 · Cited by 20
Implementing AI models in clinical workflows: a roadmap
Wang, F., & Beecy, A. (2024) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2023-112727 · Cited by 22
Subbaswamy, A., Sahiner, B., Petrick, N., Pai, V. M., Adams, R., Diamond, M. C., & Saria, S. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01275-6 · Cited by 27
Self-citationCabanillas Silva, P., Sun, H., Rezk, M., Roccaro-Waldmeyer, D. M., Fliegenschmidt, J., Hulde, N., von Dossow, V., Meesseman, L., Depraetere, K., Stieg, J., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/51409 · Cited by 20
Out-of-Distribution Detection for Medical Applications: Guidelines for Practical Evaluation
Zadorozhny, K., Thoral, P. J., Elbers, P., & Cinà, G. (2022) · Studies in computational intelligence · doi:10.1007/978-3-031-14771-5_10 · Cited by 22
Predicting counterfactual risks under hypothetical treatment strategies: an application to HIV
Dickerman, B. A., Dahabreh, I. J., Cantos, K., Logan, R. W., Lodi, S., Rentsch, C. T., Justice, A. C., & Hernán, M. Á. (2022) · European Journal of Epidemiology · doi:10.1007/s10654-022-00855-8 · Cited by 20
Bias in artificial intelligence in vascular surgery
Tran, Z. K., Byun, J., Lee, H. Y., Boggs, H. K., Tomihama, E. Y., & Kiang, S. C. (2023) · Seminars in Vascular Surgery · doi:10.1053/j.semvascsurg.2023.07.003 · Cited by 20
Participant flow diagrams for health equity in AI
Ellen, J., de Matos, J. C. R. G., Viola, M., Gallifant, J., Quion, J. M., Celi, L. A., & Hussein, N. S. A. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104631 · Cited by 22
Cummings, B. C., Blackmer, J. M., Motyka, J. R., Farzaneh, N., Cao, L., Bisco, E. L., Glassbrook, J. D., Roebuck, M. D., Gillies, C. E., Admon, A. J., et al. (2023) · Critical Care Medicine · doi:10.1097/ccm.0000000000005837 · Cited by 19
Self-citation
Metadata
- Authors
- Finlayson, S. G., Subbaswamy, A., Singh, K., Bowers, J., Kupke, A., Zittrain, J., Kohane, I. S., & Saria, S.
- Affiliations
- Harvard University (US)
- Johns Hopkins University (US)
- University of Michigan (US)
- Yale University (US)
- Boston University (US)
- Bayesian Health, New York, NY
- ORCID
- Venue
- New England Journal of Medicine
- Publisher
- Massachusetts Medical Society
- Published
- 2021-07-15
- Volume
- 385
- Issue
- 3
- Pages
- 283-286
- ISSN
- 0028-4793 (ISSN-L)
- 1533-4406
- PMC
- PMC8665481
- OpenAlex
- W3177617320
- Cited by
- 855
- References
- 2
- Citations by year
- 2026: 336 · 2025: 141 · 2024: 141 · 2023: 135 · 2022: 83 · 2021: 18
- FWCI
- 27.15
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- New England Journal of Medicine (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Science Foundation (1840088)
- Alfred P. Sloan Foundation (FG-2018-10877)
- National Institute of General Medical Sciences (T32GM007753)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- distribution shift, clinical trials, machine learning, clinician bias, model generalization, bias mitigation, clinical decision support
- MeSH terms
- Machine Learning, Information Technology, Artificial Intelligence, Humans, Medical Informatics, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-02, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-02 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
