The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Multicenter validation of a machine learning model to predict intensive care unit readmission within 48 hours after discharge
Abstract
Background: Intensive care unit (ICU) readmission is a crucial indicator of patient safety. However, discharge decisions often rely on subjective assessment due to a lack of standardized guidelines. We aimed to develop a machine-learning model to predict ICU readmission within 48 h and compare its performance to traditional scoring systems. Methods: We developed an ensemble model, iREAD, that generates a probability score at ICU discharge, representing the likelihood of the patient being readmitted to the ICU within 48 h, using data from Seoul National University Hospital (SNUH) and validated it using the MIMIC-III and eICU-CRD datasets. From September 2007 to August 2021, a total of 70,842 patients were included from SNUH. The MIMIC-III datasets comprised 43,237 patients admitted to ICUs between 2001 and 2012 at Beth Israel Deaconess Medical Center, and the eICU-CRD datasets included 90,271 ICU admissions across 208 hospitals between 2014 and 2015. Patients younger than 18, those who died in ICUs, or who refused life-sustaining treatment were excluded from the final analysis. The model's performance was evaluated using the area under the receiver operating characteristic curve (AUROC) and compared to the traditional scores and conventional machine learning models. Kaplan-Meier analysis was performed to compare the outcome between the high-risk and low-risk groups. Findings: We developed the iREAD, that utilized 30 input features, encompassing demographics, length of stay, vital signs, GCS, and laboratory values. iREAD demonstrated superior performance compared with other models across all cohorts (all P < 0.001). In the internal validation, iREAD achieved AUROCs of 0.771 (95% CI 0.743-0.798), 0.834 (0.821-0.846), and 0.820 (0.808-0.832) for early (≤48 h), late (>48 h), and overall ICU readmissions, respectively. External validations with MIMIC-III and eICU-CRD also showed modest performance with AUROCs of 0.768 (0.748-0.787) and 0.725 (0.712-0.739) for overall readmission in MIMIC-III and eICU-CRD respectively, demonstrating superior performance compared to other models (All P < 0.001; higher than other models). Kaplan-Meier analysis revealed that over 40% of high-risk patients predicted by iREAD were readmitted within 48 h, representing a more than four-fold increase in predictive performance compared to the traditional scores. Interpretation: iREAD demonstrates superior performance in predicting ICU readmission within 48 h after discharge compared to traditional scoring systems or conventional machine learning models in both internal and external validations. While the performance degradation observed in the external validations suggests the need for further prospective validation on diverse patient populations, the robust performance and ability to identify high-risk patients have the potential to guide clinical decision-making. Funding: This work was supported by the Korea Health Technology Research & Development Project through the Korea Health Industry Development Institute, funded by the Ministry of Health and Welfare, Republic of Korea (grant number RS-2021-KH114109).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Fleuren, L. M., Klausch, T. L. T., Zwager, C. L., Schoonmade, L. J., Guo, T., Roggeveen, L. F., Swart, E. L., Girbes, A. R. J., Thoral, P., Ercole, A., et al. (2020) · Intensive Care Medicine · doi:10.1007/s00134-019-05872-y · Cited by 755
Yang, G., Ye, Q., & Xia, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.07.016 · Cited by 717
Fairness of artificial intelligence in healthcare: review and recommendations
Ueda, D., Kakinuma, T., Fujita, S., Kamagata, K., Fushimi, Y., Ito, R., Matsui, Y., Nozaki, T., Nakaura, T., Fujima, N., et al. (2024) · Japanese Journal of Radiology · doi:10.1007/s11604-023-01474-3 · Cited by 607
Early prediction of circulatory failure in the intensive care unit using machine learning
Hyland, S. L., Faltys, M., Hüser, M., Lyu, X., Gumbsch, T., Esteban, C., Bock, C., Horn, M., Moor, M., Rieck, B., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0789-4 · Cited by 477
Use of machine learning to analyse routinely collected intensive care unit data: a systematic review
Shillan, D., Sterne, J. A. C., Champneys, A., & Gibbison, B. (2019) · Critical Care · doi:10.1186/s13054-019-2564-9 · Cited by 251 · cited 1×
“16 19 Similarly, prediction models using machine learning algorithms have been proposed to assess the risk of ICU read-mission before discharging patients from the ICU to”
Barbieri, S., Kemp, J., Perez-Concha, O., Kotwal, S., Gallagher, M., Ritchie, A., & Jorm, L. (2020) · Scientific Reports · doi:10.1038/s41598-020-58053-z · Cited by 86
Hong, N., Liu, C., Gao, J., Han, L., Chang, F., Gong, M., & Su, L. (2022) · JMIR Medical Informatics · doi:10.2196/28781 · Cited by 76
Plotnikoff, K. M., Krewulak, K. D., Hernández, L., Spence, K., Foster, N., Longmore, S., Straus, S. E., Niven, D. J., Parsons Leigh, J., Stelfox, H. T., et al. (2021) · Critical Care · doi:10.1186/s13054-021-03857-2 · Cited by 69
Early prediction of ICU readmissions using classification algorithms
Loreto, M., Lisboa, T., & Moreira, V. P. (2020) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.103636 · Cited by 53
Predictive Modeling for Readmission to Intensive Care: A Systematic Review
Ruppert, M. M., Loftus, T. J., Small, C., Li, H., Ozrazgat-Baslanti, T., Balch, J., Holmes, R., Tighe, P. J., Upchurch, G. R., Efron, P. A., et al. (2023) · Critical Care Explorations · doi:10.1097/cce.0000000000000848 · Cited by 24 · cited 4×
“1,8,12,21,27,45 However, models based on static variables have inherent limitations in capturing trends that re fl ect changes in patient conditions during ICU stays.”
Influential
Research works printed but not linked: 41
- Woldhek, 2017, Readmission of ICU patients: a quality indicator?, J Crit Care, 38, 328, 10.1016/j.jcrc.2016.12.001 · DOI
- Kramer, 2013, The association between ICU readmission rate and patient outcomes, Crit Care Med, 41, 24, 10.1097/CCM.0b013e3182657b8a · DOI · Paper page
- Berenholtz, 2002, Qualitative review of intensive care unit quality indicators, J Crit Care, 17, 1, 10.1053/jcrc.2002.33035 · DOI · Paper page
- Kramer, 2016, Can this patient be safely discharged from the ICU?, Intensive Care Med, 42, 580, 10.1007/s00134-015-4148-8 · DOI
- Stelfox, 2015, A scoping review of patient discharge from intensive care: opportunities and tools to improve care, Chest, 147, 317, 10.1378/chest.13-2965 · DOI
- Bai, 2021, Managing admission and discharge processes in intensive care units, Health Care Manag Sci, 24, 666, 10.1007/s10729-021-09560-6 · DOI
- Badawi, 2012, Readmissions and death after ICU discharge: development and validation of two predictive models, PLoS One, 7, 10.1371/journal.pone.0048758 · DOI
- Frost, 2009, Severity of illness and risk of readmission to intensive care: a meta-analysis, Resuscitation, 80, 505, 10.1016/j.resuscitation.2009.02.015 · DOI
- Timmers, 2012, Patients' characteristics associated with readmission to a surgical intensive care unit, Am J Crit Care, 21, e120, 10.4037/ajcc2012773 · DOI
- Campbell, 2008, Predicting death and readmission after intensive care discharge, Br J Anaesth, 100, 656, 10.1093/bja/aen069 · DOI · Paper page
- Gajic, 2008, The Stability and Workload Index for Transfer score predicts unplanned intensive care unit patient readmission: initial development and validation, Crit Care Med, 36, 676, 10.1097/CCM.0B013E318164E3B0 · DOI · Paper page
- Reini, 2012, The prognostic value of the Modified Early Warning Score in critically ill patients: a prospective, observational study, Eur J Anaesthesiol, 29, 152, 10.1097/EJA.0b013e32835032d8 · DOI
- Jahn, 2019, The predictive performance of SAPS 2 and SAPS 3 in an intermediate care unit for internal medicine at a German university transplant center; a retrospective analysis, PLoS One, 14, 10.1371/journal.pone.0222164 · DOI
- Lim, 2024, Real-time machine learning model to predict short-term mortality in critically ill patients: development and international validation, Crit Care, 28, 76, 10.1186/s13054-024-04866-7 · DOI
- Thoral, 2021, Explainable machine learning on AmsterdamUMCdb for ICU discharge decision support: uniting intensivists and data scientists, Crit Care Explor, 3
- Johnson, 2016, MIMIC-III, a freely accessible critical care database, Sci Data, 3, 10.1038/sdata.2016.35 · DOI · Paper page
- Pollard, 2018, The eICU Collaborative Research Database, a freely available multi-center database for critical care research, Sci Data, 5, 10.1038/sdata.2018.178 · DOI · Paper page
- Moor, 2023, Predicting sepsis using deep learning across international sites: a retrospective development and validation study, eClinicalMedicine, 62, 10.1016/j.eclinm.2023.102124 · DOI · Paper page
- Liu, 2023, Illness severity assessment of older adults in critical illness using machine learning (ELDER-ICU): an international multicentre study with subgroup bias evaluation, Lancet Digit Health, 5, e657, 10.1016/S2589-7500(23)00128-0 · DOI
- Rojas, 2018, Predicting intensive care unit readmission with machine learning using electronic health record data, Ann Am Thorac Soc, 15, 846, 10.1513/AnnalsATS.201710-787OC · DOI · Paper page
- Lee, 2022, Multi-center validation of machine learning model for preoperative prediction of postoperative mortality, NPJ Digit Med, 5, 91, 10.1038/s41746-022-00625-6 · DOI
- Ke, 2017, LightGBM: a highly efficient gradient boosting decision tree, Adv Neural Inf Process Syst, 30, 3149
- Ba, 2016, Layer normalization, arXiv
- Srivastava, 2014, Dropout: a simple way to prevent neural networks from overfitting, J Mach Learn Res, 15, 1929
- Kim, 2023, Early prediction of need for invasive mechanical ventilation in the neonatal intensive care unit using artificial intelligence and electronic health records: a clinical study, BMC Pediatr, 23, 525, 10.1186/s12887-023-04350-1 · DOI
- Cho, 2023, Prospective, multicenter validation of the deep learning-based cardiac arrest risk management system for predicting in-hospital cardiac arrest or unplanned intensive care unit transfer in patients admitted to general wards, Crit Care, 27, 346, 10.1186/s13054-023-04609-0 · DOI
- Cho, 2024, External validation of deep learning-based cardiac arrest risk management system for predicting in-hospital cardiac arrest in patients admitted to general wards based on rapid response system operating and nonoperating periods: a single-center study, Crit Care Med, 52, e110, 10.1097/CCM.0000000000006137 · DOI
- Kim, 2023, Development and validation of deep-learning-based sepsis and septic shock early prediction system (DeepSEPS) using real-world ICU data, J Clin Med, 12, 7156
- Lee, 2021, A multicentre validation study of the deep learning-based early warning score for predicting in-hospital cardiac arrest in patients admitted to general wards, Resuscitation, 163, 78, 10.1016/j.resuscitation.2021.04.013 · DOI
- Subbe, 2001, Validation of a modified early warning score in medical admissions, QJM, 94, 521, 10.1093/qjmed/94.10.521 · DOI · Paper page
- Jones, 2012, NEWSDIG: the national early warning score development and implementation group, Clin Med, 12, 501, 10.7861/clinmedicine.12-6-501 · DOI · Paper page
- Knaus, 1985, APACHE II: a severity of disease classification system, Crit Care Med, 13, 818, 10.1097/00003246-198510000-00009 · DOI · Paper page
- Smith, 2008, A review, and performance evaluation, of single-parameter "track and trigger" systems, Resuscitation, 79, 11, 10.1016/j.resuscitation.2008.05.004 · DOI · Paper page
- Lundberg, 2017, A unified approach to interpreting model predictions, Adv Neural Inf Process Syst, 30, 4768
- Wang, 2019, Machine learning for survival analysis: a survey, ACM Comput Surv, 51, 1, 10.1145/3214306 · DOI · Paper page
- Kastrup, 2013, Predictive ability of the stability and workload index for transfer score to predict unplanned readmissions after ICU discharge, Crit Care Med, 41, 1608, 10.1097/CCM.0b013e31828a217b · DOI
- Lin, 2019, Analysis and prediction of unplanned intensive care unit readmission using recurrent neural networks with long short-term memory, PLoS One, 14
- de Hond, 2023, Predicting readmission or death after discharge from the ICU: external validation and retraining of a machine learning model, Crit Care Med, 51, 291, 10.1097/CCM.0000000000005758 · DOI
- Frost, 2010, Readmission to intensive care: development of a nomogram for individualising risk, Crit Care Resusc, 12, 83
- Jo, 2015, Readmission to medical intensive care units: risk factors and prediction, Yonsei Med J, 56, 543, 10.3349/ymj.2015.56.2.543 · DOI
- Hammer, 2021, A tool to predict readmission to the intensive care unit in surgical critical care patients-the RISC score, J Intensive Care Med, 36, 1296, 10.1177/0885066620949164 · DOI
Metadata
- Authors
- Lim, L., Kim, M., Cho, K., Yoo, D., Sim, D., Ryu, H. G., & Lee, H.-C.
- Venue
- eClinicalMedicine
- Publisher
- Elsevier BV
- Published
- 2025-03
- Volume
- 81
- Pages
- 103112
- ISSN
- 2589-5370
- Cited by
- 23
- References
- 10
- Language
- en
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- creativecommons.org/licenses/by-nc/4.0/ (vor)
- Funders
- Korea Health Industry Development Institute
- Korea Ministry of Health and Welfare (RS-2021-KH114109)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multicenter validation of a machine learning model to predict intensive care unit readmission within 48 hours after discharge. https://app.cochat.ai/papers/10.1016/j.eclinm.2025.103112
- The paper
- Lim, L., Kim, M., Cho, K., Yoo, D., Sim, D., Ryu, H. G., & Lee, H.-C. (2025). Multicenter validation of a machine learning model to predict intensive care unit readmission within 48 hours after discharge. eClinicalMedicine. https://doi.org/10.1016/j.eclinm.2025.103112
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.