Using machine learning methods to predict in-hospital mortality of sepsis patients in the ICU
Open accessThe DOI matches its Crossref record; 54 of the 55 references it cites were found in a registry or an index.
Abstract
Background: Early and accurate identification of sepsis patients with high risk of in-hospital death can help physicians in intensive care units (ICUs) make optimal clinical decisions. This study aimed to develop machine learning-based tools to predict the risk of hospital death of patients with sepsis in ICUs. Methods: The source database used for model development and validation is the medical information mart for intensive care (MIMIC) III. We identified adult sepsis patients using the new sepsis definition Sepsis-3. A total of 86 predictor variables consisting of demographics, laboratory tests and comorbidities were used. We employed the least absolute shrinkage and selection operator (LASSO), random forest (RF), gradient boosting machine (GBM) and the traditional logistic regression (LR) method to develop prediction models. In addition, the prediction performance of the four developed models was evaluated and compared with that of an existent scoring tool – simplified acute physiology score (SAPS) II – using five different performance measures: the area under the receiver operating characteristic curve (AUROC), Brier score, sensitivity, specificity and calibration plot. Results: The records of 16,688 sepsis patients in MIMIC III were used for model training and test. Amongst them, 2949 (17.7%) patients had in-hospital death. The average AUROCs of the LASSO, RF, GBM, LR and SAPS II models were 0.829, 0.829, 0.845, 0.833 and 0.77, respectively. The Brier scores of the LASSO, RF, GBM, LR and SAPS II models were 0.108, 0.109, 0.104, 0.107 and 0.146, respectively. The calibration plots showed that the GBM, LASSO and LR models had good calibration; the RF model underestimated high-risk patients; and SAPS II had the poorest calibration. Conclusion: The machine learning-based models developed in this study had good prediction performance. Amongst them, the GBM model showed the best performance in predicting the risk of in-hospital death. It has the potential to assist physicians in the ICU to perform appropriate clinical interventions for critically ill sepsis patients and thus may help improve the prognoses of sepsis patients in the ICU.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · cited 2×
“Several innovative and pragmatic machine learning methods such as random forest (RF) [27], gradient boosting machine (GBM) [28] and the least absolute shrinkage and selection operator (LASSO) [29] which is a type of linear regression using shrinkage, have been proposed, and these models have good…”
BackgroundMethodThe third international consensus definitions for sepsis and septic shock (Sepsis-3).
Singer, M., Deutschman, C. S., Seymour, C. W., Shankar-Hari, M., Annane, D., Bauer, M., Bellomo, R., Bernard, G. R., Chiche, J.-D., Coopersmith, C. M., et al. (2016) · JAMA · doi:10.1001/jama.2016.0287 · Cited by 28,250
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
APACHE II: A severity of disease classification system
KNAUS, W. A., DRAPER, E. A., WAGNER, D. P., & ZIMMERMAN, J. E. (1985) · Critical Care Medicine · doi:10.1097/00003246-198510000-00009 · Cited by 13,664
APACHE II-A Severity of Disease Classification System: Reply
Knaus, W. A., Draper, E. A., Wagner, D. P., & Zimmerman, J. E. (1986) · Critical Care Medicine · doi:10.1097/00003246-198608000-00028 · Cited by 13,376
Bone, R. C., Balk, R. A., Cerra, F. B., Dellinger, R. P., Fein, A. M., Knaus, W. A., Schein, R. M. H., & Sibbald, W. J. (1992) · Chest · doi:10.1378/chest.101.6.1644 · Cited by 13,198
Vincent, J. L., Moreno, R., Takala, J., Willatts, S., De Mendonça, A., Bruining, H., Reinhart, C. K., Suter, P. M., & Thijs, L. G. (1996) · Intensive Care Medicine · doi:10.1007/bf01709751 · Cited by 11,674 · cited 1×
“In Sepsis-3, instead of checking the SIRS criteria as in previous definitions, infected patients who have a sequential organ failure assessment (SOFA) score [11] higher than 2 are defined as having sepsis.”
MethodAngus, D. C., Linde-Zwirble, W. T., Lidicker, J., Clermont, G., Carcillo, J., & Pinsky, M. R. (2001) · Critical Care Medicine · doi:10.1097/00003246-200107000-00002 · Cited by 8,503
Ensemble methods in machine learning
Dietterich, T. G. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45014-9_1 · Cited by 8,408
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
Le Gall, J.-R. (1993) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.270.24.2957 · Cited by 7,773
APACHE II: a severity of disease classification system.
Knaus, W. A., Draper, E. S., Wagner, D. P., & Zimmerman, J. E. (1985) · PubMed · Cited by 6,034
The epidemiology of sepsis in the United States from 1979 through 2000
Martin, G. S., Mannino, D. M., Eaton, S., & Moss, M. (2003) · New England Journal of Medicine · doi:10.1056/nejmoa022139 · Cited by 5,841
Fleischmann, C., Scherag, A., Adhikari, N. K. J., Hartog, C. S., Tsaganos, T., Schlattmann, P., Angus, D. C., & Reinhart, K. (2016) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201504-0781oc · Cited by 3,548
The APACHE III Prognostic System
Knaus, W. A., Wagner, D. P., Draper, E. A., Zimmerman, J. E., Bergner, M., Bastos, P. G., Sirio, C. A., Murphy, D. J., Lotring, T., Damiano, A., et al. (1991) · Chest · doi:10.1378/chest.100.6.1619 · Cited by 3,439
The APACHE III Prognostic System
Teres, D., & Lemeshow, S. (1992) · Chest · doi:10.1378/chest.102.6.1919b · Cited by 2,880
2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference
Conference, F. T. I. S. D., Levy, M. M., Fink, M. P., Marshall, J. C., Abraham, E., Angus, D., Cook, D., Cohen, J., Opal, S. M., Vincent, J.-L., et al. (2003) · Intensive Care Medicine · doi:10.1007/s00134-003-1662-x · Cited by 2,871
An empirical comparison of supervised learning algorithms
Caruana, R., & Niculescu-Mizil, A. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143865 · Cited by 2,780
Zimmerman, J. E., Kramer, A. A., McNair, D. S., & Malila, F. M. (2006) · Critical Care Medicine · doi:10.1097/01.ccm.0000215112.84523.f0 · Cited by 1,764
Jiménez‐Valverde, A. (2012) · Global Ecology and Biogeography · doi:10.1111/j.1466-8238.2011.00683.x · Cited by 803
Taylor, R. A., Pare, J. R., Venkatesh, A. K., Mowafi, H., Melnick, E. R., Fleischman, W., & Hall, M. K. (2016) · Academic Emergency Medicine · doi:10.1111/acem.12876 · Cited by 491
Masson, S., Caironi, P., Spanuth, E., Thomae, R., Panigada, M., Sangiorgi, G., Fumagalli, R., Mauri, T., Isgrò, S., Fanizza, C., et al. (2014) · Critical Care · doi:10.1186/cc13183 · Cited by 240
Lee, H.-C., Yoon, H.-K., Nam, K., Cho, Y. J., Kim, T. K., Kim, W. H., & Bahk, J.-H. (2018) · Journal of Clinical Medicine · doi:10.3390/jcm7100322 · Cited by 194
Lin, K., Hu, Y., & Kong, G. (2019) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.02.002 · Cited by 172
Self-citationEarly natural killer cell counts in blood predict mortality in severe sepsis
Andaluz‐Ojeda, D., Iglesias, V., Bobillo, F., Almansa, R., Rico, L., Gandía, F., Loma, A. M., Nieto, C., Diego, R., Ramos, E., et al. (2011) · Critical Care · doi:10.1186/cc10501 · Cited by 123
Data mining for indicators of early mortality in a database of clinical records
Richards, G., Rayward-Smith, V. J., Sönksen, P. H., Carey, S., & Weng, C. (2001) · Artificial Intelligence in Medicine · doi:10.1016/s0933-3657(00)00110-x · Cited by 118
Evaluation prediction techniques to achievement an optimal biomedical analysis
Al-Janabi, S., & Mahdi, M. A. (2019) · International Journal of Grid and Utility Computing · doi:10.1504/ijguc.2019.102021 · Cited by 81
Real-time mortality prediction in the Intensive Care Unit.
Johnson, A. E. W., & Mark, R. G. (2017) · PubMed · Cited by 73
Zygun, D. A., Laupland, K. B., Fick, G. H., Sandham, J. D., & Doig, C. J. (2005) · Canadian Journal of Anesthesia/Journal canadien d anesthésie · doi:10.1007/bf03016068 · Cited by 65
Pregernig, A., Müller, M. M., Held, U., & Beck‐Schimmer, B. (2019) · Annals of Intensive Care · doi:10.1186/s13613-019-0600-1 · Cited by 64
Performance targets and the Brier score
Roulston, M. S. (2007) · Meteorological Applications · doi:10.1002/met.21 · Cited by 63
Osborn, T. M., Phillips, G. S., Lemeshow, S. A., Townsend, S. R., Schorr, C., Levy, M. M., & Dellinger, R. P. (2014) · Critical Care Medicine · doi:10.1097/ccm.0000000000000416 · Cited by 63
Scalable Predictive Analysis in Critically Ill Patients Using a Visual Open Data Analysis Platform
Poucke, S. V., Zhang, Z., Schmitz, M., Vukicevic, M., Laenen, M. V., Celi, L. A., & Deyne, C. D. (2016) · PLOS ONE · doi:10.1371/journal.pone.0145791 · Cited by 62
Ou, L., Chen, J., Hillman, K., Flabouris, A., Parr, M. J., Assareh, H., & Bellomo, R. (2017) · Critical Care · doi:10.1186/s13054-016-1596-7 · Cited by 62
Designing risk prediction models for ambulatory no-shows across different specialties and clinics
Ding, X., Gellad, Z. F., Mather, R. C. “., Barth, P., Poon, E. G., Newman, M. F., & Goldstein, B. A. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy002 · Cited by 61
Baxt, W. G. (1994) · Cancer Letters · doi:10.1016/0304-3835(94)90090-6 · Cited by 59
Gurm, H. S., Kooiman, J., Lalonde, T. A., Grines, C. L., Share, D. A., & Seth, M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0096385 · Cited by 57
Zhang, Z., & Hong, Y. (2017) · Oncotarget · doi:10.18632/oncotarget.17870 · Cited by 57
Newgard, C. D., Lewis, R. J., & Jolly, B. T. (2002) · Annals of Emergency Medicine · doi:10.1067/mem.2002.123549 · Cited by 55
Evaluation prediction techniques to achieve optimal biomedical analysis
Mahdi, M. A., & Al_Janabi, S. (2019) · International Journal of Grid and Utility Computing · doi:10.1504/ijguc.2019.10020511 · Cited by 52
Mahdi, M. A., & Al-Janabi, S. (2019) · Lecture notes in networks and systems · doi:10.1007/978-3-030-23672-4_23 · Cited by 49
Model for predicting short-term mortality of severe sepsis
Adrie, C., Français, A., Alvarez-Gonzalez, A., Mounier, R., Azoulay, É., Zahar, J., Clec’h, C., Goldgran-Tolédano, D., Hammer, L., Descorps-Declere, A., et al. (2009) · Critical Care · doi:10.1186/cc7881 · Cited by 47
Clinical Evaluation of Sepsis-1 and Sepsis-3 in the ICU
Fang, X., Wang, Z., Yang, J., Cai, H., Yao, Z., Li, K., & Fang, Q. (2017) · CHEST Journal · doi:10.1016/j.chest.2017.06.037 · Cited by 43
Fang, W., Douglas, I. S., Chen, Y., Lin, C.-Y., Kao, H.-C., Fang, Y.-T., Huang, C.-H., Chang, Y., Huang, K., Wang, Y.-H., et al. (2017) · PLoS ONE · doi:10.1371/journal.pone.0187088 · Cited by 38
Venables, W. N., & Ripley, B. D. (2002) · Statisctics and computing/Statistics and computing · doi:10.1007/978-0-387-21706-2_9 · Cited by 35
Sepsis mortality prediction with the Quotient Basis Kernel
Ribas, V. J., Vellido, A., Romero, E. E., & Ruiz‐Rodríguez, J. C. (2014) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2014.03.004 · Cited by 33
Khwannimit, B., Bhurayanontachai, R., & Vattanavanit, V. (2016) · Shock · doi:10.1097/shk.0000000000000818 · Cited by 30
Incidence and Definition of Sepsis and Associated Organ Dysfunction
Dremsizov, T. T., Kellum, J. A., & Angus, D. C. (2004) · The International Journal of Artificial Organs · doi:10.1177/039139880402700503 · Cited by 20
Temporal Pattern Discovery for Accurate Sepsis Diagnosis in ICU PatientsPreprint
Sheetrit, E., Nissim, N., Klimov, D., Fuchs, L., Elovici, Y., & D, Y. S. M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.01720 · Cited by 12
Machine Learning for Predicting Sepsis In‐hospital Mortality: An Important Start
Scott, H. F., & Colborn, K. L. (2016) · Academic Emergency Medicine · doi:10.1111/acem.13009 · Cited by 11
Pediatric Sepsis From Start to Finish*
Zimmerman, J. J. (2015) · Pediatric Critical Care Medicine · doi:10.1097/pcc.0000000000000389 · Cited by 5
1485: USING CLINICAL FEATURES AND BIOMARKERS TO PREDICT 60-DAY MORTALITY OF SEPSIS PATIENTS
Xie, Y., Yabes, J. G., Parker, R., & Clermont, G. (2017) · Critical Care Medicine · doi:10.1097/01.ccm.0000529487.11010.78 · Cited by 1
Cited by
Hu, C., Li, L., Huang, W., Wu, T., Xu, Q., Liu, J., & Hu, B. (2022) · Infectious Diseases and Therapy · doi:10.1007/s40121-022-00628-6 · Cited by 160
Sheu, R.-K., & Pardeshi, M. S. (2022) · Sensors · doi:10.3390/s22208068 · Cited by 129
Syed, M., Syed, S., Sexton, K., Syeda, H. B., Garza, M., Zozus, M., Syed, F., Begum, S., Syed, A. U., Sanford, J., et al. (2021) · Informatics · doi:10.3390/informatics8010016 · Cited by 68
Evaluating machine learning models for sepsis prediction: A systematic review of methodologies
Deng, H., Sun, M., Wang, Y., Zeng, J., Yuan, T., Li, T., Li, D.-H., Chen, W., Zhou, P., Wang, Q., et al. (2021) · iScience · doi:10.1016/j.isci.2021.103651 · Cited by 84
Rahmatinejad, Z., Dehghani, T., Hoseini, B., Rahmatinejad, F., Lotfata, A., Reihani, H., & Eslami, S. (2024) · Scientific Reports · doi:10.1038/s41598-024-54038-4 · Cited by 74
Metadata
- Authors
- Kong, G., Lin, K., & Hu, Y.
- Affiliations
- Peking University (CN)
- Corresponding author
- Kong, Guilan
- Venue
- BMC Medical Informatics and Decision Making
- Published
- 2020-10-02
- Volume
- 20
- Issue
- 1
- Pages
- 251-251
- ISSN
- 1472-6947 (ISSN-L)
- PMC
- PMC7531110
- OpenAlex
- W3090266264
- Cited by
- 180
- References
- 55
- Citations by year
- 2026: 25 · 2025: 38 · 2024: 32 · 2023: 29 · 2022: 33 · 2021: 22 · 2020: 1
- FWCI
- 15.24
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- BMC Medical Informatics and Decision Making (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Natural Science Foundation of China (81771938, 91846101)
- Peking University (BMU2018MX020, PKU2017LCX05, BMU2020JI011)
- Topics
- Health Sciences › Medicine › Critical Care and Intensive Care Medicine › Sepsis Diagnosis and Treatment
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- sepsis, in-hospital mortality, mortality prediction, intensive care unit, machine learning, gradient boosting, random forest, LASSO, logistic regression, SAPS II, MIMIC-III, Sepsis-3, prognostic model, model calibration
- MeSH terms
- Machine Learning, Adult, Aged, Aged, 80 and over, Critical Care, Female, Humans, Intensive Care Units, Male, Middle Aged, Models, Theoretical, Critical Illness, Hospital Mortality, Sepsis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
