Longitudinal Model Shifts of Machine Learning–Based Clinical Risk Prediction Models: Evaluation Study of Multiple Use Cases Across Different Hospitals
Open accessAbstract
Background: In recent years, machine learning (ML)–based models have been widely used in clinical domains to predict clinical risk events. However, in production, the performances of such models heavily rely on changes in the system and data. The dynamic nature of the system environment, characterized by continuous changes, has significant implications for prediction models, leading to performance degradation and reduced clinical efficacy. Thus, monitoring model shifts and evaluating their impact on prediction models are of utmost importance. Objective: This study aimed to assess the impact of a model shift on ML-based prediction models by evaluating 3 different use cases—delirium, sepsis, and acute kidney injury (AKI)—from 2 hospitals (M and H) with different patient populations and investigate potential model deterioration during the COVID-19 pandemic period. Methods: We trained prediction models using retrospective data from earlier years and examined the presence of a model shift using data from more recent years. We used the area under the receiver operating characteristic curve (AUROC) to evaluate model performance and analyzed the calibration curves over time. We also assessed the influence on clinical decisions by evaluating the alert rate, the rates of over- and underdiagnosis, and the decision curve. Results: The 2 data sets used in this study contained 189,775 and 180,976 medical cases for hospitals M and H, respectively. Statistical analyses (Z test) revealed no significant difference (P>.05) between the AUROCs from the different years for all use cases and hospitals. For example, in hospital M, AKI did not show a significant difference between 2020 (AUROC=0.898) and 2021 (AUROC=0.907, Z=–1.171, P=.242). Similar results were observed in both hospitals and for all use cases (sepsis and delirium) when comparing all the different years. However, when evaluating the calibration curves at the 2 hospitals, model shifts were observed for the delirium and sepsis use cases but not for AKI. Additionally, to investigate the clinical utility of our models, we performed decision curve analysis (DCA) and compared the results across the different years. A pairwise nonparametric statistical comparison showed no differences in the net benefit at the probability thresholds of interest (P>.05). The comprehensive evaluations performed in this study ensured robust model performance of all the investigated models across the years. Moreover, neither performance deteriorations nor alert surges were observed during the COVID-19 pandemic period. Conclusions: Clinical risk prediction models were affected by the dynamic and continuous evolution of clinical practices and workflows. The performance of the models evaluated in this study appeared stable when assessed using AUROCs, showing no significant variations over the years. Additional model shift investigations suggested that a calibration shift was present for certain use cases (delirium and sepsis). However, these changes did not have any impact on the clinical utility of the models based on DCA. Consequently, it is crucial to closely monitor data changes and detect possible model shifts, along with their potential influence on clinical decision-making.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Predictive models may lose calibration and discrimination over time within the same institution, even when patient demographics and nominal case severity remain relatively stable.
“Longitudinal analyses have shown that predictive models may lose calibration and discrimination over time within the same institution, even when patient demographics and nominal case severity remain relatively stable [9].”
From Validation is not enough: Longitudinal evidence of post-deployment fragility in clinical AI systems (2026) · cites this paper as [9]
The Discussion (Principal Findings; page unavailable) reports calibration shifts for the delirium and sepsis models at the hospitals, and the Results (Data Source Analysis; page unavailable) reports a consistent proportion of women and men over time. But discrimination did not significantly decline: AUROCs remained stable across years at both hospitals. The copy also does not establish that case severity remained stable. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856 · see citing works
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450 · see citing works
KDIGO Clinical Practice Guidelines for Acute Kidney Injury
Khwaja, A. (2012) · Nephron Clinical Practice · doi:10.1159/000339789 · Cited by 8,039 · see citing works
A guide to deep learning in healthcare
Esteva, A., Robicquet, A., Ramsundar, B., Kuleshov, V., DePristo, M., Chou, K., Cui, C., Corrado, G., Thrun, S., & Dean, J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0316-z · Cited by 5,312 · see citing works
Rajkomar, A., Dean, J., & Kohane, I. (2019) · New England Journal of Medicine · doi:10.1056/nejmra1814259 · Cited by 4,579 · see citing works
Scalable and accurate deep learning with electronic health records
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0029-1 · Cited by 2,648 · cited 1× · see citing works
“However, most of such applications have been developed and evaluated on retrospective data [6,7], and their performance in live clinical settings remains understudied.”
Scalable and accurate deep learning with electronic health records.
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · eScholarship (California Digital Library) · Cited by 2,242 · see citing works
Advances in Large-Margin Classifiers
(2000) · The MIT Press eBooks · doi:10.7551/mitpress/1113.001.0001 · Cited by 1,902 · see citing works
Transforming classifier scores into accurate multiclass probability estimates
Zadrozny, B., & Elkan, C. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775151 · Cited by 1,280 · see citing works
A simple, step-by-step guide to interpreting decision curve analysis
Vickers, A. J., van Calster, B., & Steyerberg, E. W. (2019) · Diagnostic and Prognostic Research · doi:10.1186/s41512-019-0064-7 · Cited by 1,174 · see citing works
Goldstein, B. A., Navar, A. M., Pencina, M. J., & Ioannidis, J. P. A. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw042 · Cited by 936 · see citing works
Seyyed-Kalantari, L., Zhang, H., McDermott, M. B. A., Chen, I. Y., & Ghassemi, M. (2021) · Nature Medicine · doi:10.1038/s41591-021-01595-0 · Cited by 928 · see citing works
The Clinician and Dataset Shift in Artificial Intelligence
Finlayson, S. G., Subbaswamy, A., Singh, K., Bowers, J., Kupke, A., Zittrain, J., Kohane, I. S., & Saria, S. (2021) · New England Journal of Medicine · doi:10.1056/nejmc2104626 · Cited by 855 · see citing works
Wu, E., Wu, K., Daneshjou, R., Ouyang, D., Ho, D. E., & Zou, J. (2021) · Nature Medicine · doi:10.1038/s41591-021-01312-x · Cited by 494 · see citing works
A tutorial on calibration measurements and calibration models for clinical prediction models
Huang, Y., Li, W., Macheret, F., Gabriel, R. A., & Ohno-Machado, L. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz228 · Cited by 486 · see citing works
Updating methods improved the performance of a clinical prediction model in new patients
Janssen, K. J. M., Moons, K. G. M., Kalkman, C. J., Grobbee, D. E., & Vergouwe, Y. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.04.018 · Cited by 381 · see citing works
Cutillo, C. M., Sharma, K. R., Foschini, L., Kundu, S., Mackintosh, M., Mandl, K. D., MI in Healthcare Workshop Working Group, Beck, T., Collier, E., Colvis, C., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0254-2 · Cited by 337 · see citing works
Calibration drift in regression and machine learning models for acute kidney injury
Davis, S. E., Lasko, T. A., Chen, G., Siew, E. D., & Matheny, M. E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx030 · Cited by 312 · see citing works
There is no such thing as a validated prediction model
Van Calster, B., Steyerberg, E. W., Wynants, L., & van Smeden, M. (2023) · BMC Medicine · doi:10.1186/s12916-023-02779-w · Cited by 286 · see citing works
Andersen, F. A. (1999) · International Journal of Toxicology · doi:10.1177/109158189901800306 · Cited by 277 · see citing works
Decision curve analysis: a technical note
Zhang, Z., Rousson, V., Lee, W., Ferdynus, C., Chen, M., Qian, X., & Guo, Y. (2018) · Annals of Translational Medicine · doi:10.21037/atm.2018.07.02 · Cited by 252 · see citing works
Detection of calibration drift in clinical prediction models to inform model updating
Davis, S. E., Greevy, R. A., Lasko, T. A., Walsh, C. G., & Matheny, M. E. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103611 · Cited by 160 · see citing works
(2015) · doi:10.32685/10.143.2015.937 · Cited by 141 · see citing works
Biomedical Entity Representations with Synonym Marginalization
Sung, M., Jeon, H., Lee, J., & Kang, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.335 · Cited by 139 · see citing works
Henry, K. E., Adams, R., Parent, C., Soleimani, H., Sridharan, A., Johnson, L., Hager, D. N., Cosgrove, S. E., Markowski, A., Klein, E. Y., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01895-z · Cited by 127 · see citing works
Jauk, S., Kramer, D., Großauer, B., Rienmüller, S., Avian, A., Berghold, A., Leodolter, W., & Schulz, S. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa113 · Cited by 84 · see citing works
Minne, L., Eslami, S., de Keizer, N., de Jonge, E., de Rooij, S. E., & Abu-Hanna, A. (2012) · Intensive Care Medicine · doi:10.1007/s00134-011-2390-2 · Cited by 69 · see citing works
Li, Y., Wang, H., & Luo, Y. (2022) · Circulation Heart Failure · doi:10.1161/circheartfailure.122.009473 · Cited by 66 · see citing works
Quantification of Sepsis Model Alerts in 24 US Hospitals Before and During the COVID-19 Pandemic
Wong, A., Cao, J., Lyons, P. G., Dutta, S., Major, V. J., Ötleş, E., & Singh, K. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.35286 · Cited by 57 · see citing works
Rothman, M. J., Levy, M. M., Dellinger, R. P., Jones, S. L., Fogerty, R. L., Voelker, K. G., Gross, B., Marchetti, A. E., & Beals, J. (2016) · Journal of Critical Care · doi:10.1016/j.jcrc.2016.11.037 · Cited by 51 · see citing works
Sun, H., Depraetere, K., Meesseman, L., Silva, P. C., Szymanowsky, R., Fliegenschmidt, J., Hulde, N., von Dossow, V., Vanbiervliet, M., Baerdemaeker, J. D., et al. (2022) · Journal of Medical Internet Research · doi:10.2196/34295 · Cited by 50 · see citing works
Self-citationSepsis in patients hospitalized with coronavirus disease 2019: how often and how severe?
Ramos, F. J. D. S., Freitas, F. G. R., & Machado, F. R. (2021) · Current Opinion in Critical Care · doi:10.1097/mcc.0000000000000861 · Cited by 38 · see citing works
A scalable approach for developing clinical risk prediction applications in different hospitals
Sun, H., Depraetere, K., Meesseman, L., Roo, J. D., Vanbiervliet, M., Baerdemaeker, J. D., Muys, H., von Dossow, V., Hulde, N., & Szymanowsky, R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103783 · Cited by 31 · see citing works
Self-citationNavigating the machine learning pipeline: a scoping review of inpatient delirium prediction models
Strating, T., Hanjani, L. S., Tornvall, I., Hubbard, R. E., & Scott, I. A. (2023) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2023-100767 · Cited by 21 · see citing works
Fliegenschmidt, J., Hulde, N., Preising, M. G., Ruggeri, S., Szymanowsky, R., Meesseman, L., Sun, H., Dahlweid, F.-M., & von Dossow, V. (2022) · European Journal of Anaesthesiology Intensive Care · doi:10.1097/ea9.0000000000000010 · Cited by 7 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- [26]Vaswani A Shazeer N Parmar N Uszkoreit J Jones L Gomez AN Kaiser Ł Polosukhin I Attention is all you need 2017 NIPS 2017: 31st Conference on Neural Information Processing Systems December 4-9, 2017 Long Beach, CA
- [29]Guo C Pleiss G Sun Y Weinberger K On calibration of modern neural networks 2017 ICML'17: 34th International Conference on Machine Learning August 6-11, 2017 Sydney, NSW 1321 1330
Cited by
Sorted by citations
Validation is not enough: Longitudinal evidence of post-deployment fragility in clinical AI systems
Kopanitsa, G. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001534 · Cited by 3 · see citing works
“Longitudinal analyses have shown that predictive models may lose calibration and discrimination over time within the same institution, even when patient demographics and nominal case severity remain relatively stable [9].”
Bridging fairness in machine learning: A legal critique of technical strategies
Liu, J., Luo, X., & Tang, G. H. (2026) · Computer law & security review · doi:10.1016/j.clsr.2026.106339 · Cited by 3 · see citing works
Patient-Centred Explainability in IVF Outcome Prediction
Sivaprasad, A., Reiter, E., McLernon, D. J., Tintarev, N., Bhattacharya, S., & Oren, N. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-00656-1_7 · Cited by 2 · see citing works
Dual-Branch Self-Supervised Contrastive Pre-Training Framework for Sleep Stage Classification
Ouyang, J., Wei, Y., Qian, S., Dahlweid, F.-M., Sun, H., Lai, Z., & Zhang, X. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3628659 · Cited by 2 · see citing works
Self-citationZhang, X., Wang, A., Xu, R., & Liu, D. (2025) · Frontiers in Pediatrics · doi:10.3389/fped.2025.1629795 · Cited by 2 · see citing works
Yang, L., & Wang, B. (2025) · BMC Cardiovascular Disorders · doi:10.1186/s12872-025-05247-w · Cited by 2 · see citing works
Predicting kidney injury after cardiac surgery with cardio-pulmonary bypass using machine learning
Fliegenschmidt, J., Sasono, G., Silva, P. C., Meesseman, L., Rezk, M., Dahlweid, M., Bergmann, A., Hulde, N., & von Dossow, V. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1695494 · Cited by 2 · see citing works
Self-citationBalafkhar, R. M., Baalawi, Y. S., Almashhor, A. M., Alasheq, A. K., Alhebshi, O. H., & Mohaini, M. A. (2026) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2026.105006 · Cited by 2 · see citing works
Beyond Regulatory Approval: Lifecycle Assurance for Clinical Artificial Intelligence
Shujaat, S. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02466-2 · Cited by 1 · see citing works
Ye, Y., Jiang, Z., Shen, L., Lai, Z., Dahlweid, F.-M., Sun, H., & Zhang, X. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.108384
Self-citationOuyang, J., Xiao, P., Chen, J., Dahlweid, F.-M., Chen, Y., Wei, Y., & Lai, Z. (2026) · Array · doi:10.1016/j.array.2026.100718
Self-citationTumor cell specific total mRNA expression informed neural networks predicts cancer progressionPreprint
Paul, A., Lal, J. C., Ji, S., Fong, C., Chen, K., Ding, Y., Li, R., Dai, Y., Tran, Q., Montierth, M. D., et al. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.05.01.722212
Liu, J., Brown, K., Ma, M. J., RoyChoudhury, A., Malin, B., McCoy, A. B., Wright, A., Ancker, J. S., Rosen, T., Han, J. H., et al. (2026) · Research Square · doi:10.21203/rs.3.rs-9733941/v1
MED-XEL: Explainable AI on Ensemble Methods for Early Neurological Risk Forecasting
Sherly, P. S., Velvizhy, P., Matan, P., Kumar, K. A., & Maanav, S. (2026) · International Conference on Artificial Intelligence and Data Engineering (AIDE) · doi:10.1109/aide69088.2026.11544221
Peña, A. D., Clariza, J., & Aballiar-Vista, M. A. (2026) · Transactions of the Institute of Aviation · doi:10.2478/tar-2026-0009
NA-XGB: A Neuroadaptive XGBoost Framework for Real-Time EEG-based Stress Detection on Edge Devices
Ahmed, M., & Preethi, G. (2026) · International Conference on Inventive Research in Computing Applications (ICIRCA) · doi:10.1109/icirca69024.2026.11570269
Post-Deployment Update Triggers for Clinical AI: An Error-Taxonomy Framework for Safe Model Revision
Herrera, L., Rojas, D., & Castro, A. (2024) · Journal of Health Informatics and Digital Systems · doi:10.68159/b329996013
Chi, H., Liu, X., Yan, Y., Wang, L., Luo, P., Su, L., Jiang, L., Zhou, X., Huang, G., Jiang, Y., et al. (2026) · Genes & Diseases · doi:10.1016/j.gendis.2026.102480 · Cited by 1 · see citing works
Zeng, Q., Li, J., Hu, J., & Yang, C. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1952864
Parvin, R. (2026) · Digital Health · doi:10.1177/20552076261501912
- Authors
- Cabanillas Silva, P., Sun, H., Rezk, M., Roccaro-Waldmeyer, D. M., Fliegenschmidt, J., Hulde, N., von Dossow, V., Meesseman, L., Depraetere, K., Stieg, J., Szymanowsky, R., & Dahlweid, F.-M.
- Affiliations
- Jiaxing University (CN)
- University Hospitals of the Ruhr-University of Bochum (DE)
- Corresponding author
- Rezk, Mohamed
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2024-12-13
- Volume
- 26
- Pages
- e51409
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC11681292
- OpenAlex
- W4405363405
- Cited by
- 20
- References
- 35
- Citations by year
- 2026: 14 · 2025: 5 · 2024: 1
- FWCI
- 2.95
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- European Commission (101017549, HORIZON2020, 875406)
- Ruhr-Universität Bochum (100126059)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- clinical prediction model, model drift, delirium prediction, sepsis prediction, acute kidney injury prediction, COVID-19 pandemic, model calibration, clinical decision support alerts, hospital differences, model degradation, decision curve analysis
- MeSH terms
- Machine Learning, COVID-19, Aged, Female, Hospitals, Humans, Longitudinal Studies, Male, Middle Aged, Retrospective Studies, ROC Curve, Risk Assessment, Sepsis, Acute Kidney Injury
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
