Detecting and Remediating Harmful Data Shifts for the Responsible Deployment of Clinical AI Models
Open accessAbstract
Importance: Clinical artificial intelligence (AI) systems are susceptible to performance degradation due to data shifts, which can lead to erroneous predictions and potential patient harm. Proactively detecting and mitigating these shifts is crucial for maintaining AI effectiveness and safety in clinical practice. Objectives: To develop and evaluate a proactive, label-agnostic monitoring pipeline to detect and mitigate harmful data shifts in clinical AI systems and to assess the use of transfer learning and continual learning strategies in maintaining model performance. Design, Setting, and Participants: This prognostic study was conducted using electronic health record data for admissions to general internal medicine wards of 7 large hospitals (5 academic and 2 community) in Toronto, Canada, between January 1, 2010, to August 31, 2020. Inpatients (aged ≥18 years) with a hospital stay of at least 24 hours were included. Data analysis was performed from January to August 2022. Exposures: Data shifts due to changes in hospital type, critical laboratory assays, patient demographics, admission type, and the COVID-19 pandemic. Main Outcomes and Measures: The primary outcome was predictive performance for all-cause in-hospital mortality within the next 2 weeks, evaluated using the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC). Data shifts were detected using a label-agnostic monitoring pipeline employing a black box shift estimator with maximum mean discrepancy testing. Results: Data were available for 143 049 adult inpatients (mean [SD] age, 67.8 [19.6] years; 50.7% female). Significant data shifts were detected as a result of changes in younger age groups and admissions from nursing homes and acute care centers, transferring from community to academic hospitals, and changes in brain natriuretic peptide and D-dimer. Transfer learning improved model performance of community hospitals in a hospital type-dependent manner (Delta AUROC [SD], 0.05 [0.03]; Delta AUPRC [SD], 0.06 [0.04]). During the COVID-19 pandemic, drift-triggered continual learning improved overall model performance (Delta AUROC [SD], 0.44 [0.02]; P = .007, Mann-Whitney U test). Conclusions and Relevance: In this prognostic study, a proactive, label-agnostic monitoring pipeline detected harmful data shifts for a clinical AI system predicting in-hospital mortality. Transfer learning and drift-triggered continual learning strategies mitigated performance degradation, maintaining model performance across health care settings. These findings suggest that the approach used here may ensure the robust and equitable deployment of clinical AI models. Future research should explore the generalizability of this framework across diverse clinical domains, data modalities, and longer deployment periods to further validate its effectiveness.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
These frameworks primarily focus on evidence generation at the point of evaluation and provide limited guidance on how performance degradation should be detected and managed during routine clinical operation, particularly when outcome labels are delayed, incomplete, or influenced by the model's own use.
“However, these frameworks primarily focus on evidence generation at the point of evaluation and provide limited guidance on how performance degradation should be detected and managed during routine clinical operation, particularly when outcome labels are delayed, incomplete, or influenced by the model’s own use [25–27].”
From Validation is not enough: Longitudinal evidence of post-deployment fragility in clinical AI systems (2026) · cites this paper as [27]
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredLaboratory performance does not always translate directly to clinical practice.
“Laboratory performance does not always translate directly to clinical practice [142,143,144,145,146].”
From Artificial Intelligence in Clinical Oncology: From Productivity Enhancement to Creative Discovery (2025) · cites this paper as [142]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(2010) · Cited by 8,261 · see citing works
Scalable and accurate deep learning with electronic health records
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0029-1 · Cited by 2,648 · see citing works
Advances in Neural Information Processing Systems 32 (NeurIPS 2019)
Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché–Buc, F., Fox, E. B., & Garnett, R. (2020) · HAL · Cited by 2,450 · see citing works
New ICD-10 version of the Charlson comorbidity index predicted in-hospital mortality
Sundararajan, V., Henderson, T., Perry, C., Muggivan, A., Quan, H., & Ghali, W. A. (2004) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.03.012 · Cited by 2,439 · see citing works
Recurrent Neural Networks for Multivariate Time Series with Missing Values
Che, Z., Purushotham, S., Cho, K., Sontag, D., & Liu, Y. (2018) · Scientific Reports · doi:10.1038/s41598-018-24271-9 · Cited by 2,370 · see citing works
Scalable and accurate deep learning with electronic health records.
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · eScholarship (California Digital Library) · Cited by 2,242 · see citing works
Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., & Oermann, E. K. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002683 · Cited by 1,719 · see citing works
Potential Biases in Machine Learning Algorithms Using Electronic Health Record Data
Gianfrancesco, M. A., Tamang, S., Yazdany, J., & Schmajuk, G. (2018) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.3763 · Cited by 1,395 · see citing works
An interpretable mortality prediction model for COVID-19 patients
Li, Y., Zhang, H., Gonçalves, J. M., Xiao, Y., Wang, M., Guo, Y., Sun, C., Tang, X., Jing, L., Zhang, M., et al. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-0180-7 · Cited by 1,304 · see citing works
Wong, A., Otles, E., Donnelly, J. P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., et al. (2021) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2021.2626 · Cited by 995 · see citing works
Multitask learning and benchmarking with clinical time series data
Harutyunyan, H., Khachatrian, H., Kale, D. C., Ver Steeg, G., & Galstyan, A. (2019) · Scientific Data · doi:10.1038/s41597-019-0103-9 · Cited by 958 · see citing works
Seyyed-Kalantari, L., Zhang, H., McDermott, M. B. A., Chen, I. Y., & Ghassemi, M. (2021) · Nature Medicine · doi:10.1038/s41591-021-01595-0 · Cited by 928 · see citing works
Learning to Diagnose with LSTM Recurrent Neural Networks
Lipton, Z. C., Kale, D. C., Elkan, C., & Wetzel, R. (2015) · arXiv · doi:10.48550/arxiv.1511.03677 · Cited by 879 · see citing works
The Clinician and Dataset Shift in Artificial Intelligence
Finlayson, S. G., Subbaswamy, A., Singh, K., Bowers, J., Kupke, A., Zittrain, J., Kohane, I. S., & Saria, S. (2021) · New England Journal of Medicine · doi:10.1056/nejmc2104626 · Cited by 855 · see citing works
Collins, G. S., Dhiman, P., Andaur Navarro, C. L., Ma, J., Hooft, L., Reitsma, J. B., Logullo, P., Beam, A. L., Peng, L., Van Calster, B., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-048008 · Cited by 852 · see citing works
Addressing bias in artificial intelligence in health care
Parikh, R. B., Teeple, S., & Navathe, A. S. (2019) · JAMA · doi:10.1001/jama.2019.18058 · Cited by 764 · see citing works
Dataset Shift in Machine Learning
Adams, N. M. (2009) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2009.00624_10.x · Cited by 716 · see citing works
Ethical Machine Learning in Healthcare
Chen, I. Y., Pierson, E., Rose, S., Joshi, S., Ferryman, K., & Ghassemi, M. (2021) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-092820-114757 · Cited by 645 · see citing works
Minimum information about clinical artificial intelligence modeling: the MI-CLAIM checklist.
Norgeot, B., Quer, G., Beaulieu-Jones, B. K., Torkamani, A., Dias, R., Gianfrancesco, M., Arnaout, R., Kohane, I. S., Saria, S., Topol, E., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-1041-y · Cited by 632 · see citing works
Early prediction of circulatory failure in the intensive care unit using machine learning
Hyland, S. L., Faltys, M., Hüser, M., Lyu, X., Gumbsch, T., Esteban, C., Bock, C., Horn, M., Moor, M., Rieck, B., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0789-4 · Cited by 477 · see citing works
The myth of generalisability in clinical research and machine learning in health care
Futoma, J., Simons, M., Panch, T., Doshi-Velez, F., & Celi, L. A. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30186-2 · Cited by 467 · see citing works
In Search of Lost Domain Generalization
Gulrajani, I., & Lopez-Paz, D. (2020) · arXiv · doi:10.48550/arxiv.2007.01434 · Cited by 440 · see citing works
Feng, J., Phillips, R. V., Malenica, I., Bishara, A., Hubbard, A. E., Celi, L. A., & Pirracchio, R. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00611-y · Cited by 423 · see citing works
Shifting machine learning for healthcare from development to deployment and from models to data
Zhang, A., Xing, L., Zou, J., & Wu, J. C. (2022) · Nature Biomedical Engineering · doi:10.1038/s41551-022-00898-y · Cited by 420 · see citing works
Adams, R., Henry, K. E., Sridharan, A., Soleimani, H., Zhan, A., Rawat, N., Johnson, L., Hager, D. N., Cosgrove, S. E., Markowski, A., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01894-0 · Cited by 365 · see citing works
Machine learning based early warning system enables accurate mortality risk prediction for COVID-19
Gao, Y., Cai, G., Fang, W., Li, H., Wang, S., Chen, L., Yu, Y., Liu, D., Xu, S., Cui, P., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-18684-2 · Cited by 349 · see citing works
A Review of Challenges and Opportunities in Machine Learning for Health
Ghassemi, M., Naumann, T., Schulam, P., Beam, A. L., Chen, I. Y., & Ranganath, R. (2018) · PubMed Central · doi:10.48550/arxiv.1806.00388 · Cited by 323 · see citing works
Calibration drift in regression and machine learning models for acute kidney injury
Davis, S. E., Lasko, T. A., Chen, G., Siew, E. D., & Matheny, M. E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx030 · Cited by 312 · see citing works
Learning from streaming data with concept drift and imbalance: an overview
Hoens, T. R., Robi Polikar, A., & Chawla, N. V. (2012) · Progress in Artificial Intelligence · doi:10.1007/s13748-011-0008-0 · Cited by 301 · see citing works
The limits of fair medical imaging AI in real-world generalization
Yang, Y., Zhang, H., Gichoya, J. W., Katabi, D., & Ghassemi, M. (2024) · Nature Medicine · doi:10.1038/s41591-024-03113-4 · Cited by 285 · see citing works
Mitigating bias in machine learning for medicine
Vokinger, K. N., Feuerriegel, S., & Kesselheim, A. S. (2021) · Communications Medicine · doi:10.1038/s43856-021-00028-w · Cited by 269 · see citing works
Clinical applications of continual learning machine learning
Lee, C. S., & Lee, A. Y. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30102-3 · Cited by 254 · see citing works
Norgeot, B., Glicksberg, B. S., Trupin, L. S., Lituiev, D., Gianfrancesco, M. A., Oskotsky, B., Schmajuk, G., Yazdany, J., & Butte, A. J. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.0606 · Cited by 224 · see citing works
Detecting and Correcting for Label Shift with Black Box PredictorsPreprint
Lipton, Z. C., Wang, Y.-X., & Smola, A. J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.03916 · Cited by 216 · see citing works
The DECIDE-AI Steering Group (2021) · Nature Medicine · doi:10.1038/s41591-021-01229-5 · Cited by 184 · see citing works
Machine Learning–Based Early Warning Systems for Clinical Deterioration: Systematic Scoping Review
Muralitharan, S., Nelson, W., Di, S., McGillion, M. H., Devereaux, P. J., Barr, N. G., & Petch, J. (2020) · Journal of Medical Internet Research · doi:10.2196/25187 · Cited by 179 · see citing works
Inconsistent Performance of Deep Learning Models on Mammogram Classification
Wang, X., Liang, G., Zhang, Y., Blanton, H., Bessinger, Z., & Jacobs, N. (2020) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2020.01.006 · Cited by 176 · see citing works
Leveraging electronic health records for data science: common pitfalls and how to avoid them
Sauer, C. M., Chen, L.-C., Hyland, S. L., Girbes, A. R. J., Elbers, P., & Celi, L. A. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00154-6 · Cited by 173 · see citing works
Verma, A. A., Guo, Y., Kwan, J. L., Lapointe-Shaw, L., Rawal, S., Tang, T., Weinerman, A., Cram, P., Dhalla, I. A., Hwang, S. W., et al. (2017) · CMAJ Open · doi:10.9778/cmajo.20170097 · Cited by 145 · see citing works
Self-citationImputation of missing values for electronic health record laboratory data
Li, J., Yan, X. S., Chaudhary, D., Avula, V., Mudiganti, S., Husby, H., Shahjouei, S., Afshar, A., Stewart, W. F., Yeasin, M., et al. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00518-0 · Cited by 133 · see citing works
Henry, K. E., Adams, R., Parent, C., Soleimani, H., Sridharan, A., Johnson, L., Hager, D. N., Cosgrove, S. E., Markowski, A., Klein, E. Y., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01895-z · Cited by 127 · see citing works
Variation of Arterial and Central Venous Catheter Use in United States Intensive Care Units
Gershengorn, H. B., Garland, A., Kramer, A. A., Scales, D. C., Rubenfeld, G. D., & Wunsch, H. (2013) · Anesthesiology · doi:10.1097/aln.0000000000000008 · Cited by 124 · see citing works
Wiens, J., Guttag, J., & Horvitz, E. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-002162 · Cited by 123 · see citing works
Kraljevic, Z., Bean, D., Shek, A., Bendayan, R., Hemingway, H., Yeung, J. A., Deng, A., Balston, A., Ross, J., Idowu, E., et al. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00025-6 · Cited by 118 · see citing works
Duckworth, C., Chmiel, F. P., Burns, D. K., Zlatev, Z. D., White, N. M., Daniels, T. W. V., Kiuber, M., & Boniface, M. J. (2021) · Scientific Reports · doi:10.1038/s41598-021-02481-y · Cited by 109 · see citing works
Dynamic Memory to Alleviate Catastrophic Forgetting in Continual Learning with Medical Imaging
Perkonigg, M., Hofmanninger, J., Herold, C. J., Brink, J. A., Pianykh, O., Prosch, H., & Langs, G. (2021) · Nature Communications · doi:10.1038/s41467-021-25858-z · Cited by 106 · see citing works
Verma, A. A., Pasricha, S. V., Jung, H. Y., Kushnir, V., Mak, D. Y. F., Koppula, R., Guo, Y., Kwan, J. L., Lapointe‐Shaw, L., Rawal, S., et al. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa225 · Cited by 100 · see citing works
Rasmy, L., Nigo, M., Kannadath, B. S., Xie, Z., Mao, B., Patel, K., Zhou, Y., Zhang, W., Ross, A., Xu, H., et al. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00049-8 · Cited by 90 · see citing works
Nestor, B. A., McDermott, M. B. A., Boag, W., Berner, G., Naumann, T., Hughes, M. C., Goldenberg, A., & Ghassemi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.00690 · Cited by 78 · see citing works
Problems in the deployment of machine-learned models in health care
Cohen, J., Cao, T., Viviano, J. D., Huang, C.-W., Fralick, M., Ghassemi, M., Mamdani, M., Greiner, R., & Bengio, Y. (2021) · Canadian Medical Association Journal · doi:10.1503/cmaj.202066 · Cited by 70 · see citing works
Barish, M., Bolourani, S., Lau, L. F., Shah, S., & Zanos, T. P. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00254-2 · Cited by 58 · see citing works
Evaluating Model Robustness and Stability to Dataset ShiftPreprint
Subbaswamy, A., Adams, R., & Saria, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.15100 · Cited by 49 · see citing works
Implications of the Use of Artificial Intelligence Predictive Models in Health Care Settings
Vaid, A., Sawant, A. S., Suárez‐Fariñas, M., Lee, J., Kaul, S. K., Kovatch, P., Freeman, R., Jiang, J. J., Jayaraman, P., Fayad, Z. A., et al. (2023) · Annals of Internal Medicine · doi:10.7326/m23-0949 · Cited by 46 · see citing works
Amrollahi, F., Shashikumar, S. P., Holder, A. L., & Nemati, S. (2022) · Scientific Reports · doi:10.1038/s41598-022-12497-7 · Cited by 43 · see citing works
An empirical framework for domain generalization in clinical settings
Zhang, H., Dullerud, N., Seyyed-Kalantari, L., Morris, Q., Joshi, S. D., & Ghassemi, M. (2021) · ACM Conference on Health, Inference, and Learning (CHIL) · doi:10.1145/3450439.3451878 · Cited by 42 · see citing works
Bhavani, S. V., Wiley, Z., Verhoef, P. A., Coopersmith, C. M., & Ofotokun, I. (2022) · JAMA · doi:10.1001/jama.2022.12290 · Cited by 42 · see citing works
A Benchmark of Medical Out of Distribution DetectionPreprint
Cao, T., Huang, C.-W., Hui, D. Y.-T., & Cohen, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.04250 · Cited by 39 · see citing works
Racial and Ethnic Bias in Pulse Oximetry and Clinical Outcomes
Valbuena, V. S. M., Merchant, R. M., & Hough, C. L. (2022) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2022.1903 · Cited by 37 · see citing works
Critical Bias in Critical Care Devices
Charpignon, M.-L., Byers, J., Cabral, S., Celi, L. A., Fernandes, C., Gallifant, J., Lough, M. E., Mlombwa, D., Moukheiber, L., Ong, B. A., et al. (2023) · Critical Care Clinics · doi:10.1016/j.ccc.2023.02.005 · Cited by 35 · see citing works
Feng, J., Lee, J., Vesoulis, Z. A., & Li, F. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00479-4 · Cited by 34 · see citing works
Guo, L. L., Morse, K., Aftandilian, C. C., Steinberg, E., Fries, J., Posada, J., Fleming, S. L., Lemmon, J., Jessa, K., Shah, N. H., et al. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02449-8 · Cited by 25 · see citing works
The Compelling Need for Shared Responsibility of AI Oversight: Lessons From Health IT Certification.
Ratwani, R., Classen, D., & Longhurst, C. (2024) · Journal of the American Medical Association (JAMA) · doi:10.1001/jama.2024.12630 · Cited by 24 · see citing works
The Data Addition DilemmaPreprint
Shen, J. H., Raji, I. D., & Chen, I. Y. (2024) · Machine Learning in Health Care · doi:10.48550/arxiv.2408.04154 · Cited by 23 · see citing works
Pfohl, S. R., Zhang, H., Xu, Y., Foryciarz, A., Ghassemi, M., & Shah, N. H. (2022) · Scientific Reports · doi:10.1038/s41598-022-07167-7 · Cited by 22 · see citing works
Continual learning of longitudinal health records
Armstrong, J., & Clifton, D. A. (2022) · 2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI) · doi:10.1109/bhi56158.2022.9926878 · Cited by 21 · see citing works
Adam, G. A., Chang, C.-H. K., Haibe‐Kains, B., & Goldenberg, A. (2020) · Machine Learning for Healthcare Conference · Cited by 15 · see citing works
Proactive Algorithm Monitoring to Ensure Health Equity
Sendak, M., Balu, S., & Hernandez, A. F. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.45022 · Cited by 13 · see citing works
Mind the Performance Gap: Examining Dataset Shift During Prospective Validation
Ötleş, E., Oh, J., Li, B. C., Bochinski, M., Joo, H., Ortwine, J., Shenoy, E. S., Washer, L., Young, V. B., Rao, K., et al. (2021) · PubMed · doi:10.48550/arxiv.2107.13964 · Cited by 13 · see citing works
Identifying and Addressing Bias in Artificial Intelligence
Crowe, B., & Rodriguez, J. A. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.25955 · Cited by 12 · see citing works
Inaccuracy of pulse oximetry in darker-skinned patients is unchanged across 32 years
Tobin, M. J., & Jubran, A. (2022) · European Respiratory Journal · doi:10.1183/13993003.00520-2022 · Cited by 9 · see citing works
Drysdale, E., Dolatabadi, E., Chivers, C., Liu, V. X., Saria, S., Sendak, M., Wiens, J., Brudno, M., Hoyt, A., Mazwi, M., et al. (2019) · doi:10.13140/rg.2.2.30793.70241 · Cited by 9 · see citing works
CyclOps: Cyclical development towards operationalizing ML models for healthPreprint
Krishnan, A., Subasri, V., McKeen, K., Kore, A., Ogidi, F., Alinoori, M., Lalani, N., Dhalla, A., Verma, A. A., Razak, F. A., et al. (2022) · medRxiv · doi:10.1101/2022.12.02.22283021 · Cited by 8 · see citing works
BEDS-Bench: Behavior of EHR-models under Distributional Shift--A BenchmarkPreprint
Avati, A., Seneviratne, M. G., Xue, E., Xu, Z., Lakshminarayanan, B., & Dai, A. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.08189 · Cited by 8 · see citing works
Mishra, B., Sinha, N., & Rehman, H. U. (2017) · Journal of Health Research and Reviews · doi:10.4103/2394-2010.208121 · Cited by 4 · see citing works
Ensembling Shift Detectors: An Extensive Empirical Evaluation
Maggio, S., & Dreyfus-Schmidt, L. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86523-8_22 · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [61]Bellot A, van der Schaar M. Accounting for unobserved confounding in domain generalization. arXiv. Preprint posted online July 21, 2020. doi: 10.48550/arXiv.2007.1065 · DOI
Cited by
Sorted by citations
Al-Ewaidat, O. A., & Naffaa, M. M. (2025) · Rheumato · doi:10.3390/rheumato5040017 · Cited by 24 · see citing works
Monitoring strategies for continuous evaluation of deployed clinical prediction models
Kim, G. Y. E., Corbin, C. K., Grolleau, F., Baiocchi, M., & Chen, J. H. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104854 · Cited by 14 · see citing works
AI for colon cancer: A focus on classification, detection, and predictive modeling
Merabet, A., Saighi, A., Harous, S., Ferradji, M. A., Laboudi, Z., Almaktoom, A. T., Mousavirad, S. J., Elbatal, I., & Mohamed, A. W. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.106115 · Cited by 14 · see citing works
Han, L. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.13705 · Cited by 9 · see citing works
Hamamoto, R., Koyama, T., Takahashi, S., Yasuda, T., Kobayashi, K., Akagi, Y., Kouno, N., Sudo, K., Hirata, M., Sunami, K., et al. (2026) · Journal of Hematology & Oncology · doi:10.1186/s13045-026-01781-y · Cited by 8 · see citing works
Artificial Intelligence in Clinical Oncology: From Productivity Enhancement to Creative Discovery
Kuno, M., Osumi, H., Udagawa, S., Yoshikawa, K., Ooki, A., Shinozaki, E., Ishikawa, T., Oba, J., Yamaguchi, K., & Sakurada, K. (2025) · Current Oncology · doi:10.3390/curroncol32110588 · Cited by 7 · see citing works
Abdalla, N., Arab, R. A. E., Abdrbo, A. A., Almari, M., Ayoub, M. Y., Alsaaideh, B., Dagamseh, M., Almagharbeh, W. T., Abuadas, F. H. A., Mahfouz, M. S. A., et al. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1737957 · Cited by 7 · see citing works
Jiang, Z., Chen, H., Wu, Y., Qin, Y., Pei, C., Zeng, D., Sheng, B., & Wong, T. Y. (2026) · Intelligent Medicine · doi:10.1016/j.imed.2026.01.001 · Cited by 5 · see citing works
Xie, W., Liu, J., & Liu, S. (2026) · Journal of Medical Internet Research · doi:10.2196/91727 · Cited by 6 · see citing works
The future is now: How AI is reshaping spine care
Muehlbauer, E. J., Alvi, M. A., Kennedy, D. J. “. J. ”., & Fehlings, M. G. (2025) · North American Spine Society Journal (NASSJ) · doi:10.1016/j.xnsj.2025.100825 · Cited by 4 · see citing works
Validation is not enough: Longitudinal evidence of post-deployment fragility in clinical AI systems
Kopanitsa, G. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001534 · Cited by 3 · see citing works
Garg, P., Singhal, G., & Singhal, S. S. (2026) · Frontiers in Molecular Biosciences · doi:10.3389/fmolb.2026.1750124 · Cited by 4 · see citing works
Polit, F., Bahmad, H. F., Kassab, M. B., Elajami, M. K., Recine, M. A., Alghamdi, S., & Poppiti, R. (2026) · Frontiers in Oncology · doi:10.3389/fonc.2026.1833926 · Cited by 3 · see citing works
Clinical artificial intelligence—the case for a new physician role
Mahajan, A., & Bates, D. W. (2025) · The Lancet Regional Health - Americas · doi:10.1016/j.lana.2025.101280 · Cited by 2 · see citing works
Kovacheva, V., Kleinlein, R., Wheeler, N., Venkatesh, K. K., Jelovsek, E., Bates, D. W., & Gray, K. J. (2025) · Pregnancy · doi:10.1002/pmf2.70200 · Cited by 2 · see citing works
TRIAGE: Trustworthy Reporting and Assessment for Clinical Gain and Effectiveness of AI Models
Fazilati, F., Rajabi, M. T., Alihosseini, N., Farsani, M. E., Sandid, S. H., Zamani, S., Farahani, M. A., Biriaei, F., Sadeghipour, F., Mirshamsi, M. T., et al. (2026) · Diagnostics · doi:10.3390/diagnostics16050666 · Cited by 5 · see citing works
Zhao, Q., Jiang, W., Zhang, C., Ouyang, M., Wu, N., & Guo, Y. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1863050 · Cited by 2 · see citing works
Kalinich, M., Luccarelli, J., Maria, J. P. S., Moss, F., & Torous, J. B. (2026) · BMJ Mental Health · doi:10.1136/bmjment-2026-302626 · Cited by 2 · see citing works
Arab, R. A. E., Mustafa, M. H., Almagharbeh, W. T., Ayoub, M. Y., Alsanawi, F., Almathen, F., Almosabeh, R., & Talaq, M. A. (2026) · Healthcare · doi:10.3390/healthcare14142052 · Cited by 2 · see citing works
Agentic systems in computational pathology: architectures, evidence, and translational challenges
Lu, X., Li, Q., Gao, Y., Wei, D., Lyu, M., Ma, S., Li, J., Wu, Y., Cai, L., Zhang, T., et al. (2026) · Journal of Translational Medicine · doi:10.1186/s12967-026-08829-0 · Cited by 2 · see citing works
A practical guide to the implementation of AI in orthopaedic research—Part 5: Data management
Zsidai, B., Oettl, F. C., Pruneski, J. A., Pánics, G., Winkler, P. W., Senorski, E. H., Hirschmann, M. T., Yu, Y., FELDT, R. H., Samuelsson, K., et al. (2025) · Journal of Experimental Orthopaedics · doi:10.1002/jeo2.70581 · Cited by 1 · see citing works
Pulick, E., Carey, K. A., Qyli, T., Oguss, M. K., Picart, J. K., Penumalee, L. S., Nezirova, L., Tully, S. T., GILBERT, E. R., Shah, N. S., et al. (2026) · JMIR AI · doi:10.2196/81642 · Cited by 1 · see citing works
Artificial intelligence in emergency medicine critical care
Heslin, S. M., Nocito, R., & Weingart, S. D. (2026) · Clinical and Experimental Emergency Medicine · doi:10.15441/ceem.26.057 · Cited by 1 · see citing works
Alignment of Policy, Practice, and Patient Safety for Trustworthy AI in Radiology
Doo, F. X., Davis, M., Poff, J. A., Lui, Y. W., Haines, K., & Towbin, A. J. (2026) · Radiology Artificial Intelligence · doi:10.1148/ryai.250982 · Cited by 2 · see citing works
From algorithms to accountability: Saudi Arabia's path to AI healthcare regulation
Dhali, M., Awad, T., & Hassan, S. (2026) · Journal of Open Innovation Technology Market and Complexity · doi:10.1016/j.joitmc.2026.100810 · Cited by 1 · see citing works
A roadmap for medical large language models: a review of foundations, applications, and challenges
Sha, Y.-Y., Yu, L., Lin, Z.-H., Kaur, A., Lou, Y., Gornale, S. S., Zhang, T., Wong, L. S., Wang, Z., Yan, Y., et al. (2026) · Military Medical Research · doi:10.1016/j.mmr.2026.100050 · Cited by 2 · see citing works
AI-based augmentation of oncology clinical trials
Villa, A., Eadie, A. L., Synnott, D., Romanò, R., Piffoux, M., Wong, E. Y. T., Scheinerman, N., Tan, D. S. W., de Braud, F., Koopman, M., et al. (2026) · Nature Reviews Clinical Oncology · doi:10.1038/s41571-026-01189-0 · Cited by 3 · see citing works
Khang, T. F., Ho, J. W. K., Hou, L., Song, J., & Wong, L. (2026) · Quantitative Biology · doi:10.1002/qub2.70058 · Cited by 1 · see citing works
Beyond Regulatory Approval: Lifecycle Assurance for Clinical Artificial Intelligence
Shujaat, S. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02466-2 · Cited by 1 · see citing works
Pharma's AI Inflection Point: What Does It Mean for Early Phase Clinical Development?
Issa, A. M. (2025) · Clinical Pharmacology in Drug Development · doi:10.1002/cpdd.1596 · Cited by 1 · see citing works
Jayaraman, S., Kumar, A. R., Palm, E., Fernando, M., Sarabu, C. R., Choma, M. A., Wang, K., Ravi, V., Suarez, M., McCaffrey, P., et al. (2026) · doi:10.2196/preprints.105596 · Cited by 1 · see citing works
Clinical AI in Canadian Community Hospitals: A Framework for Implementation Readiness
Mohiyeddini, C. (2026) · Healthcare Management Forum · doi:10.1177/08404704261494446
Grach, S., Badawi, A., Ahmad, F., & Dolatabadi, E. (2026) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2026.105077
Meslamani, A. Z. A., Jarab, A. S., & Mohammed, E. M. A. (2025) · Bioanalysis · doi:10.1080/17576180.2026.2614302
Albadiri, A. O. O., & Almagtome, A. (2025) · 2025 IEEE 5th International Conference on ICT in Business Industry & Government (ICTBIG) · doi:10.1109/ictbig68706.2025.11323563
Eskandar, K. (2026) · Journal of Addictive Diseases · doi:10.1080/10550887.2025.2607456
Application of Foundation Models in Emergency and Critical Care: A Scoping ReviewPreprint
Kong, Y., Mai, X., Meyer, A., FANG, Y. C., Xu, Z., Song, Z., Gong, C., Wacker, D. A., Heneghan, J. A., Wang, Q., et al. (2026) · Research Square · doi:10.21203/rs.3.rs-8338830/v1
Lee, J. T., Wu, A. J. W., Li, V., Lee, P.-H., Ilyuk, A., Lin, Y.-C., Chen, Y.-C., Chen, H.-H., Niu, K.-Y., Forde, B., et al. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6142287
The portability paradox of foundation models for clinical decision support
Rosen, K. L., Sui, M., Ong, A. Y., & Kvedar, J. C. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02615-4
Deng, J., Patel, S. K., Fung, M., Heybati, K., Layard, B., Wang, B., Rubin, B., Simard, T., Hibbert, B., Anderson, T. J., et al. (2026) · BMJ Open · doi:10.1136/bmjopen-2025-108799
Ambient AI and Measurement Bias in Psychiatric Notes
Bair, H. (2026) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2026.1041
陈, 丽. (2026) · Advances in Clinical Medicine · doi:10.12677/acm.2026.1651913
The Future of Surgical Decision-Making in Critical Care
Fowler, M. J., Lee, A., Nobuhara, C. K., Nassar, A. K., Hindin, D. I., & Hameed, S. M. (2026) · Current Emergency and Hospital Medicine Reports · doi:10.1007/s40138-026-00346-0
Designing Trustworthy and Explainable Clinical Decision Support Systems
Maleki, D. N. M., Zamanifar, A., Farhadi, A., & Sorouri, M. (2026) · Springer eBooks · doi:10.1007/978-3-032-15088-2_5
Song, Y., Shi, W., Yang, C., Yang, X., & Yu, C. (2026) · Hepatology International · doi:10.1007/s12072-026-11100-9
Artificial Intelligence in Rare Diseases: Workflow-Integrated Precision Kidney Care
Thongprayoon, C., Pesce, F., & Cheungpasitporn, W. (2026) · Clinics and Practice · doi:10.3390/clinpract16060101
Drift isn’t a bug, it’s the work: post-deployment surveillance for clinical artificial intelligence
Syailendra, E. A. (2026) · BMJ Digital Health & AI · doi:10.1136/bmjdh-2026-000111
Li, C., Bohman, J., Patil, V., Mcshinsky, R., Yong, C., Burningham, Z. R., & Halwani, A. S. (2026) · Health Informatics Journal · doi:10.1177/14604582261456059
Li, M., & Zhang, Q. (2026) · Archives of Microbiology · doi:10.1007/s00203-026-04942-2
Heslin, S. M. (2026) · Journal of Patient Safety · doi:10.1097/pts.0000000000001545
Research and Scholarly Methods: Using Artificial Intelligence
Trinkley, K. E., & Maw, A. (2026) · JACCP JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY · doi:10.1002/jac5.70239
Medical AI across Data Regimes to Promote Proactive Health
Li, P., Wu, J., Zuo, Q., Zhang, L., & Zhan, Q. (2026) · Health Data Science · doi:10.34133/hds.0495
Alkam, T., Tarshizi, E., & Benschoten, A. H. V. (2026) · Journal of dementia and Alzheimer's disease · doi:10.3390/jdad3040048
Achieving Equity in Generative Artificial Intelligence in Cardiovascular Medicine
Kulkarni, A., & Visweswaran, S. (2026) · JACC Advances · doi:10.1016/j.jacadv.2026.102967
The climate adaptation gap in AMR governance: A converging crisis
Pan, M., Li, X., & Ma, C. (2026) · Integrated Environmental Assessment and Management · doi:10.1093/inteam/vjag085
Jeong, D., Moon, S., & Yu, J. Y. (2026) · Mathematics · doi:10.3390/math14142457
From appraisal to deployment: A governance framework for large language models in emergency medicine
Aykut, A. (2026) · Artificial Intelligence in Emergency Medicine · doi:10.1016/j.aiemed.2026.100029
Bilionis, I., Berrios, R. C., Muñoz, A. D. A., Fernández-Luque, L., & Castillo, C. (2026) · JAMIA Open · doi:10.1093/jamiaopen/ooag125
Zhu, H., zhu, C., Un, S., CHENG, Q., Meng, Z., Wang, Q., Hu, C. H., & Zhu, H. (2026) · Research Square · doi:10.21203/rs.3.rs-10265116/v1
Human-AI Synergy in Clinical Decision-Making: Balancing Innovation, Safety and Patient-Centric Care
Eskandar, K. (2026) · Artificial intelligence · doi:10.5772/intechopen.1014561
Zhu, M., Luo, B., Wu, X., Ying, Z. H. A. O., Wang, S., Hu, Y. Y., Liu, R., ZHANG, C., Ji, L. I., Wang, H., et al. (2026) · Applied Clinical Informatics · doi:10.1055/a-2921-5990 · Cited by 1 · see citing works
Will the EU AI Act facilitate market entry for adaptive medical AI systems?
Niemiec, E. (2026) · Health Policy and Technology · doi:10.1016/j.hlpt.2026.101295
Continuous assurance for AI-driven clinical decision support systems
Al‐Horani, R. A., & Tadesse, A. F. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1870819
Continual Learning without Silent Change in Medication-Support Systems
Kim, D., Park, J., Kang, M.-S., & Jung, H.-W. (2026) · Archives of Pharmacy Practice · doi:10.51847/65q1pxmaoe
Perspective Chapter: Leveraging AI to Enhance Geriatric Care on Medicine and Surgery Floors
Nathan, A., Pagali, S. R., Chaar, M. K. A., Agarwal, S., & Agarwal, N. (2026) · IntechOpen eBooks · doi:10.5772/intechopen.1016908
Yuen, P. T.-Y., Hui, C. C.-Y., Ngan, O. M. Y., Saberi, D., Lau, K. K., & Chan, E. (2026) · doi:10.2196/preprints.109364
From Informed Consent to Informed Dependence: Rethinking Autonomy in AI-Mediated Clinical Care
Lahtinen, M., Salo, E., & Virtanen, J. (2026) · Asian Journal of Ethics in Health and Medicine · doi:10.51847/9ylvg44hby
Danilatou, V., & Arachchillage, D. J. (2026) · Seminars in Thrombosis and Hemostasis · doi:10.1055/a-2943-7750
AI Cautionary Guide: Pitfalls and Strategies for the Use of Machine Learning in Medical Research
Liu, Q., Huang, R., Zhang, H., & Osborne, T. F. (2026) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.70469
Zhang, Z., Wang, Y., Zhang, L., Chen, D., Zhang, W., Li, F., & Zhou, J. (2026) · Med Research · doi:10.1002/mdr2.70088
Cabrera-Rodríguez, R., Reyes-Castañeda, I., Lorenzo-Sánchez, I., Valenzuela-Fernández, A., & Trujillo‐González, R. (2026) · New Microbes and New Infections · doi:10.1016/j.nmni.2026.101843
Avoiding common failures in AI for health and medicine
Salaudeen, O., Zhang, H., Kim, E., Singh, K., & Ghassemi, M. (2026) · Cell · doi:10.1016/j.cell.2026.07.025
Chi, H., Liu, X., Yan, Y., Wang, L., Luo, P., Su, L., Jiang, L., Zhou, X., Huang, G., Jiang, Y., et al. (2026) · Genes & Diseases · doi:10.1016/j.gendis.2026.102480 · Cited by 1 · see citing works
Navarrete-Campos, J., Pradenas, L., Parada, V., & Scherer, R. F. (2026) · Applied Sciences · doi:10.3390/app16199419 · Cited by 1 · see citing works
Laurent, S., Dubois, P., Lefevre, M. A., & Moreau, C. (2026) · Journal of Health Informatics and Digital Systems · doi:10.68159/g088779265
Lenze, N. R., & Goldstein, C. A. (2026) · Sleep Medicine Clinics · doi:10.1016/j.jsmc.2026.08.020
Chapter 11. Model Development and Validation
Quesada, A. J. F., Hernández-González, E. A., Pacheco, R. H., & Pérez, R. (2026) · doi:10.62486/978-9915-9876-8-2.ch11
Health Information Technology, Data, and Algorithms
El‐Azab, S., Moura, I., & Nong, P. (2026) · Wiley eBooks · doi:10.1002/9781394311446.ch13
Introduction to using symbolic regression for interpretable machine learning in healthcare
Ferguson, M., Liu, F., Lin, H., McManus, D. D., & Gerber, B. S. (2026) · Communications Medicine · doi:10.1038/s43856-026-01914-x
Lazzaro, F. D., Barletta, G., Magnanimi, E., Cirillo, B., Frati, P., Marinelli, E., Pugliese, F. R., Paola, L. D., & Giordano, G. G. (2026) · Healthcare · doi:10.3390/healthcare14203382
Adapa, K., Garg, M., Sethi, R., Venketa, A. Y., Chauhan, T., Regi, M., & Mazur, Ł. (2026) · Asia-Pacific Journal of Oncology Nursing · doi:10.1016/j.apjon.2026.101044
Choi, J., Seyam, O., Biswas, D., Sivalokanathan, S., Tai, H.-H., Kong, D., & Farhan, S. (2026) · Sensors · doi:10.3390/s26206503
Clinical AI in 2025–2026 — Six Themes from the SAIL Year in Review
Kim, E., Elia, M. V., Li, M. M., Healey, E., Erdman, L., Chandak, P., Argaw, P., Li, B., Black, K. C., Marwaha, J. S., et al. (2026) · NEJM AI · doi:10.1056/aip2600817
Liu, K., Yang, F., Lyu, X., He, H., & Wen, W. (2026) · Journal of Translational Internal Medicine · doi:10.1515/jtim-2026-0060
Flipping the script: co-creation and local innovation in AI for global health paradigms
Patel, M., Magsanoc-Alikpala, M. J. K., Jacomina, L. E., Pike, L., Iyengar, P., Lee, N. Y., & Dee, E. C. (2026) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2026-102172
Ramalhete, L., Oliveira, V., & Quintas, R. (2026) · AI in Medicine · doi:10.3390/aimed1040028
- Authors
- Subasri, V., Krishnan, A., Kore, A., Dhalla, A., Pandya, D., Wang, B., Malkin, D., Razak, F., Verma, A. A., Goldenberg, A., & Dolatabadi, E.
- Affiliations
- University Health Network (CA)
- Vector Institute (CA)
- University of Toronto (CA)
- St. Michael's Hospital (CA)
- Corresponding author
- Subasri, Vallijah
- ORCID
- Venue
- JAMA Network Open
- Publisher
- American Medical Association (AMA)
- Published
- 2025-06-04
- Volume
- 8
- Issue
- 6
- Pages
- e2513685
- ISSN
- 2574-3805 (ISSN-L)
- PMC
- PMC12138723
- OpenAlex
- W4411025380
- Cited by
- 87
- References
- 75
- Citations by year
- 2026: 75 · 2025: 12
- FWCI
- 34.90
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JAMA Network Open (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Canadian Institutes of Health Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- clinical AI, in-hospital mortality prediction, maximum mean discrepancy, transfer learning, continual learning, hospital type, COVID-19 pandemic, electronic health records, model drift, patient safety, internal medicine ward, brain natriuretic peptide, D-dimer, nursing home admission, academic vs community hospitals
- MeSH terms
- COVID-19, SARS-CoV-2, Adult, Aged, Aged, 80 and over, Artificial Intelligence, Female, Humans, Male, Middle Aged, Ontario, Hospital Mortality, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
