Mortality prediction in intensive care units with the Super ICU Learner Algorithm (SICULA): a population-based study.
Open accessAbstract
Background Improved mortality prediction for patients in intensive care units is a big challenge. Many severity scores have been proposed, but findings of validation studies have shown that they are not adequately calibrated. The Super ICU Learner Algorithm (SICULA), an ensemble machine learning technique that uses multiple learning algorithms to obtain better prediction performance, does at least as well as the best member of its library. We aimed to assess whether the Super Learner could provide a new mortality prediction algorithm for patients in intensive care units, and to assess its performance compared with other scoring systems.Methods From January, 2001, to December, 2008, we used the Multiparameter Intelligent Monitoring in Intensive Care II (MIMIC-II) database (version 26) including all patients admitted to an intensive care unit at the Beth Israel Deaconess Medical Centre, Boston, MA, USA. We assessed the calibration, discrimination, and risk classification of predicted hospital mortality based on Super Learner compared with SAPS-II, APACHE-II, and SOFA. We calculated performance measures with cross-validation to avoid making biased assessments. Our proposed score was then externally validated on a dataset of 200 randomly selected patients admitted at the intensive care unit of Hôpital Européen Georges-Pompidou, Paris, France, between Sept 1, 2013, and June, 30, 2014. The primary outcome was hospital mortality. The explanatory variables were the same as those included in the SAPS II score.Findings 24,508 patients were included, with median SAPS-II of 38 (IQR 27-51) and median SOFA of 5 (IQR 2-8). 3002 of 24,508 (12%) patients died in the Beth Israel Deaconess Medical Centre. We produced two sets of predictions based on the Super Learner; the first based on the 17 variables as they appear in the SAPS-II score (SL1), and the second, on the original, untransformed variables (SL2). The two versions yielded average predicted probabilities of death of 0·12 (IQR 0·02-0·16) and 0·13 (0·01-0·19), whereas the corresponding value for SOFA was 0·12 (0·05-0·15) and for SAPS-II 0·30 (0·08-0·48). The cross-validated area under the receiver operating characteristic curve (AUROC) for SAPS-II was 0·78 (95% CI 0·77-0·78) and 0·71 (0·70-0·72) for SOFA. Super Learner had an AUROC of 0·85 (0·84-0·85) when the explanatory variables were categorised as in SAPS-II, and of 0·88 (0·87-0·89) when the same explanatory variables were included without any transformation. Additionally, Super Learner showed better calibration properties than previous score systems. On the external validation dataset, the AUROC was 0·94 (0·90-0·98) and calibration properties were good.Interpretation Compared with conventional severity scores, Super Learner offers improved performance for predicting hospital mortality in patients in intensive care units. A user-friendly implementation is available online and should be useful for clinicians seeking to validate our score.Funding Fulbright Foundation, Assistance Publique-Hôpitaux de Paris, Doris Duke Clinical Scientist Development Award, and the NIH.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Super Learner ranks candidate algorithms by performance and applies an optimal weight to each, creating ensemble algorithms that can accurately predict transfusion requirements and mortality among trauma patients.
“To capitalize on this phenomenon, Super Learner ranks a set of candidate algorithms20,28,38,39 by their performance and applies an optimal weight to each, creating ensemble algorithms that can accurately predict transfusion requirements and mortality among trauma patients.20,28,38,39”
From Artificial Intelligence and Surgical Decision-making. (2020) · cites this paper as [38]
The Introduction and Prediction Algorithms section describe ranking candidate algorithms by predictive performance and combining them with optimal weights. The Results report accurate ICU mortality prediction, but the study included a mixed ICU population rather than evaluating trauma patients separately, and it did not assess transfusion requirements (Results > Patients; Patients and Methods > Data Collection and Patients). (LLM, conf=0.90)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Knaus, W. A., Draper, E. A., Wagner, D. P., & Zimmerman, J. E. (1985) · Critical Care Medicine · doi:10.1097/00003246-198510000-00009 · Cited by 15,333 · see citing works
Goldberger, A. L., Amaral, L. A. N., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., & Stanley, H. E. (2000) · Circulation · doi:10.1161/01.cir.101.23.e215 · Cited by 14,809 · see citing works
APACHE II-A Severity of Disease Classification System
Knaus, W. A., Draper, E. A., Wagner, D. P., & Zimmerman, J. E. (1986) · Critical Care Medicine · doi:10.1097/00003246-198608000-00028 · Cited by 13,379 · see citing works
Vincent, J. L., Moreno, R., Takala, J., Willatts, S., De Mendonça, A., Bruining, H., Reinhart, C. K., Suter, P. M., & Thijs, L. G. (1996) · Intensive Care Medicine · doi:10.1007/bf01709751 · Cited by 11,674 · see citing works
Le Gall, J.-R. (1993) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.270.24.2957 · Cited by 7,773 · see citing works
Pencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493 · see citing works
APACHE II: a severity of disease classification system.
Knaus, W. A., Draper, E. S., Wagner, D. P., & Zimmerman, J. E. (1985) · PubMed · Cited by 6,034 · see citing works
The APACHE III Prognostic System
Knaus, W. A., Wagner, D. P., Draper, E. A., Zimmerman, J. E., Bergner, M., BASTOS, P. G., Sirio, C., Murphy, D. J., Lotring, T., Damiano, A., et al. (1991) · CHEST Journal · doi:10.1378/chest.100.6.1619 · Cited by 3,524 · see citing works
ROCR: visualizing classifier performance in R
Sing, T., Sander, O., Beerenwinkel, N., & Lengauer, T. (2005) · Computer applications in the biosciences · doi:10.1093/bioinformatics/bti623 · Cited by 3,172 · see citing works
The APACHE III Prognostic System
Teres, D., & Lemeshow, S. A. (1992) · CHEST Journal · doi:10.1378/chest.102.6.1919b · Cited by 2,880 · see citing works
Internal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746 · see citing works
Strobl, C., Malley, J., & Tutz, G. (2009) · Psychological Methods · doi:10.1037/a0016973 · Cited by 2,583 · see citing works
APACHE—acute physiology and chronic health evaluation: a physiologically based classification system
Knaus, W. A., Zimmerman, J. E., Wagner, D. P., Draper, E. A., & Lawrence, D. E. (1981) · Critical Care Medicine · doi:10.1097/00003246-198108000-00008 · Cited by 2,190 · see citing works
Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction
Cook, N. R. (2007) · Circulation · doi:10.1161/circulationaha.106.672402 · Cited by 2,123 · see citing works
van der Laan, M. J., Polley, E., & Hubbard, A. (2007) · Statistical Applications in Genetics and Molecular Biology · doi:10.2202/1544-6115.1309 · Cited by 1,472 · see citing works
Moreno, R. P., Metnitz, P., Almeida, E., Jordan, B., Bauer, P., Campos, R. A., Iapichino, G., Edbrooke, D. L., Capuzzo, M., Gall, J. L., et al. (2005) · Intensive Care Medicine · doi:10.1007/s00134-005-2763-5 · Cited by 1,414 · see citing works
Greenland, P. (2007) · Statistics in Medicine · doi:10.1002/sim.2976 · Cited by 1,300 · see citing works
A simplified acute physiology score for ICU patients
Gall, J. L., Loirat, P., Alpérovitch, A., GLASER, P., Granthil, C., Mathieu, D., MERCIER, P., THOMAS, R., & Villers, D. (1984) · Critical Care Medicine · doi:10.1097/00003246-198411000-00012 · Cited by 1,247 · see citing works
Saeed, M., Villarroel, M., Reisner, A. T., Clifford, G., Lehman, L.-W., Moody, G., Heldt, T., Kyaw, T. H., Moody, B., & Mark, R. G. (2011) · Critical Care Medicine · doi:10.1097/ccm.0b013e31820a92c6 · Cited by 1,162 · see citing works
Lemeshow, S. A. (1993) · JAMA · doi:10.1001/jama.1993.03510200084037 · Cited by 1,030 · see citing works
Kramer, A. A., & Zimmerman, J. E. (2007) · Critical Care Medicine · doi:10.1097/01.ccm.0000275267.64078.b0 · Cited by 886 · see citing works
Halpern, N. A., & Pastores, S. M. (2009) · Critical Care Medicine · doi:10.1097/ccm.0b013e3181b090d0 · Cited by 818 · see citing works
Statistical Evaluation of Prognostic versus Diagnostic Models: Beyond the ROC Curve
Cook, N. R. (2007) · Clinical Chemistry · doi:10.1373/clinchem.2007.096529 · Cited by 691 · see citing works
The APACHE III Prognostic System
Knaus, W. A. (1992) · CHEST Journal · doi:10.1016/s0012-3692(16)40910-4 · Cited by 626 · see citing works
Two further applications of a model for binary regression
COX, D. R. (1958) · Biometrika · doi:10.1093/biomet/45.3-4.562 · Cited by 556 · see citing works
Changes in hospital mortality for United States intensive care unit admissions from 1988 to 2012
Zimmerman, J. E., Kramer, A. A., & Knaus, W. A. (2013) · Critical Care · doi:10.1186/cc12695 · Cited by 533 · see citing works
Automated de-identification of free-text medical records
Neamatullah, I., Douglass, M. M., Lehman, L.-W. H., Reisner, A., Villarroel, M., Long, W. J., Szolovits, P., Moody, G. B., Mark, R. G., & Clifford, G. D. (2008) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-8-32 · Cited by 413 · see citing works
Validation techniques for logistic regression models
Miller, M. E., Hui, S. L., & Tierney, W. M. (1991) · Statistics in Medicine · doi:10.1002/sim.4780100805 · Cited by 322 · see citing works
Dybowski, R., Gant, V, Weller, P., & Chang, R. (1996) · The Lancet · doi:10.1016/s0140-6736(96)90609-1 · Cited by 240 · see citing works
Super Learning: An Application to the Prediction of HIV-1 Drug Resistance
Sinisi, S. E., Polley, E. C., Petersen, M. L., Rhee, S., & van der Laan, M. J. (2007) · Statistical Applications in Genetics and Molecular Biology · doi:10.2202/1544-6115.1240 · Cited by 238 · see citing works
Self-citationvan der Laan, M. J., & Dudoit, S. (2003) · Collection of Biostatistics Research Archive · Cited by 205 · see citing works
Harrison, D., Brady, A., Parry, G., Carpenter, J., & Rowan, K. (2006) · Critical Care Medicine · doi:10.1097/01.ccm.0000216702.94014.75 · Cited by 185 · see citing works
Mortality prediction using SAPS II: an update for French intensive care units
Gall, J. R. L., Neumann, A., Hémery, F., et al. (2005) · Critical Care · doi:10.1186/cc3821 · Cited by 174 · see citing works
Asymptotics of cross-validated risk estimation in estimator selection and performance assessment
Dudoit, S., & van der Laan, M. J. (2005) · Statistical Methodology · doi:10.1016/j.stamet.2005.02.003 · Cited by 168 · see citing works
Clermont, G., Angus, D. C., DiRusso, S. M., Griffin, M. F., & Linde‐Zwirble, W. T. (2001) · Critical Care Medicine · doi:10.1097/00003246-200102000-00012 · Cited by 165 · see citing works
Open-access MIMIC-II database for intensive care research
Lee, J., Scott, D., Villarroel, M., Clifford, G. D., Saeed, M., & Mark, R. G. (2011) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/iembs.2011.6092050 · Cited by 154 · see citing works
Beck, D. H., Smith, G. B., Pappachan, J., & Millar, B. (2003) · Intensive Care Medicine · doi:10.1007/s00134-002-1607-9 · Cited by 149 · see citing works
Kim, S., Kim, W., & Park, R. W. (2011) · Healthcare Informatics Research · doi:10.4258/hir.2011.17.4.232 · Cited by 132 · see citing works
Sakr, Y. L., Krauss, C., Amaral, A. C. K.-B., et al. (2008) · British Journal of Anaesthesia · doi:10.1093/bja/aen291 · Cited by 128 · see citing works
Bertolini, G., D’Amico, R., Nardi, D., Tinazzi, A., & Apolone, G. (2000) · PubMed · Cited by 123 · see citing works
Recent innovations in intensive care unit risk-prediction models
Rosenberg, A. L. (2002) · Current Opinion in Critical Care · doi:10.1097/00075198-200208000-00009 · Cited by 103 · see citing works
Caution when using prognostic models: A prospective comparison of 3 recent prognostic models
Nassar, A. P., Mocelin, A. O., Nunes, A. L. B., Giannini, F. P., Brauer, L., Andrade, F. M., & Dias, C. A. (2011) · Journal of Critical Care · doi:10.1016/j.jcrc.2011.08.016 · Cited by 86 · see citing works
Greenland, S. (2007) · Statistics in Medicine · doi:10.1002/sim.2995 · Cited by 80 · see citing works
Aegerter, P., Boumendil, A., Retbi, A., et al. (2005) · Intensive Care Medicine · doi:10.1007/s00134-005-2557-9 · Cited by 79 · see citing works
SAPS 3 admission score: an external validation in a general intensive care population
Ledoux, D., Canivet, J.-L., Preiser, J., et al. (2008) · Intensive Care Medicine · doi:10.1007/s00134-008-1187-4 · Cited by 70 · see citing works
Austrian validation and customization of the SAPS 3 Admission Score
on behalf of the ASDI Study Group, Metnitz, B., Schaden, E., Moreno, R. P., Gall, J. L., Bauer, P., & Metnitz, P. (2008) · Intensive Care Medicine · doi:10.1007/s00134-008-1286-2 · Cited by 64 · see citing works
GiViTI, Poole, D., Rossi, C., Latronico, N., Rossi, G., Finazzi, S., & Bertolini, G. (2012) · Intensive Care Medicine · doi:10.1007/s00134-012-2578-0 · Cited by 63 · see citing works
Cost effectiveness of adult intensive care in the UK
Ridley, S., & Morris, S. (2007) · Anaesthesia · doi:10.1111/j.1365-2044.2007.04997.x · Cited by 62 · see citing works
Patient specific predictions in the intensive care unit using a Bayesian ensemblePreprint
Johnson, A. E. W., Dunkley, N., Mayaud, L., Tsanas, A., Kramer, A. A., & Clifford, G. D. (2012) · Cited by 54 · see citing works
Severe sepsis mortality prediction with relevance vector machines
Ribas, V. J., Lopez, J. C., Ruiz‐Sanmartín, A., et al. (2011) · doi:10.1109/iembs.2011.6089906 · Cited by 40 · see citing works
Predicting ICU survival: A meta-level approach
Gortzis, L. G., Sakellaropoulos, F., Ilias, Ι., Stamoulis, K., & Dimopoulou, I. (2008) · BMC Health Services Research · doi:10.1186/1472-6963-8-157 · Cited by 31 · see citing works
PhysioNet 2012 Challenge: Predicting mortality of ICU patients using a cascaded SVM-GLM paradigm
Citi, L., & Barbieri, R. (2012) · Open Access at Essex (University of Essex) · Cited by 31 · see citing works
Using VLAD scores to have a look insight ICU performance: towards a modelling of the errors
Foltran, F., Berchialla, P., Giunta, F., et al. (2010) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2009.01240.x · Cited by 8 · see citing works
Goldman, H. B. (2007) · Current Urology Reports · doi:10.1007/s11934-007-0068-7 · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [50]Van der Laan MJ Rose S Targeted Learning: Causal Inference for Observational and Experimental Data Springer 2011
Cited by
Sorted by citations
Artificial Intelligence and Surgical Decision-making.
Loftus, T. J., Tighe, P. J., Filiberto, A. C., Efron, P. A., Brakenridge, S. C., Mohr, A. M., Rashidi, P., Upchurch, G. R., & Bihorac, A. (2020) · JAMA Surgery · doi:10.1001/jamasurg.2019.4917 · Cited by 611 · see citing works
“To capitalize on this phenomenon, Super Learner ranks a set of candidate algorithms 20,28,38,39 by their performance and applies an optimal weight to each, creating ensemble algorithms that can accurately predict transfusion requirements and mortality among trauma patients.”
BackgroundArmoundas, A. A., Narayan, S. M., Arnett, D. K., Spector-Bagdady, K., Bennett, D. A., Celi, L. A., Friedman, P. A., Gollob, M. H., Hall, J. L., Kwitek, A. E., et al. (2024) · Circulation · doi:10.1161/cir.0000000000001201 · Cited by 347 · see citing works
An algorithm based on deep learning for predicting in-hospital cardiac arrest
Kwon, J., Lee, Y., Lee, Y., Lee, S., & Park, J. (2018) · Journal of the American Heart Association · doi:10.1161/jaha.118.008678 · Cited by 299 · see citing works
Big data hurdles in precision medicine and precision public health
Prosperi, M., Min, J. S., Bian, J., & Modave, F. (2018) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-018-0719-2 · Cited by 200 · see citing works
Awad, A., Bader-El-Den, M., McNicholas, J., & Briggs, J. (2017) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2017.10.002 · Cited by 208 · see citing works
Luedtke, A., Sadikova, E., & Kessler, R. C. (2019) · Clinical Psychological Science · doi:10.1177/2167702618815466 · Cited by 143 · see citing works
Ethical considerations about artificial intelligence for prognostication in intensive care
Beil, M., Proft, I., van Heerden, D., Sviri, S., & van Heerden, P. V. (2019) · Intensive Care Medicine Experimental · doi:10.1186/s40635-019-0286-6 · Cited by 124 · see citing works
Patient length of stay and mortality prediction: A survey
Awad, A., Bader–El–Den, M., & McNicholas, J. (2017) · Health Services Management Research · doi:10.1177/0951484817696212 · Cited by 128 · see citing works
Gallifant, J., Zhang, J., del Pilar Arias Lopez, M., Zhu, T., Camporota, L., Celi, L. A., & Formenti, F. (2022) · British Journal of Anaesthesia · doi:10.1016/j.bja.2021.09.025 · Cited by 72 · see citing works
External validation of AI-based scoring systems in the ICU: a systematic review and meta-analysis
Rockenschaub, P., Akay, E. M., Carlisle, B. G., Hilbert, A., Wendland, J., Meyer-Eschenbach, F., Näher, A.-F., Frey, D., & Madai, V. I. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02830-7 · Cited by 58 · see citing works
Deasy, J., Liò, P., & Ercole, A. (2020) · Scientific Reports · doi:10.1038/s41598-020-79142-z · Cited by 52 · see citing works
21st Century critical care medicine: an overview
Padte, S., Samala Venkata, V., Mehta, P., Tawfeeq, S., Kashyap, R., & Surani, S. (2024) · World Journal of Critical Care Medicine · doi:10.5492/wjccm.v13.i1.90176 · Cited by 26 · see citing works
Boussi Rahmouni, H., Hassine, N. B. E. H., Chouchen, M., Ceylan, H. İ., Muntean, R. I., Bragazzi, N. L., & Dergaa, I. (2025) · Healthcare · doi:10.3390/healthcare13202553 · Cited by 25 · see citing works
Cui, X., Chang, Y., Yang, C., Cong, Z., Wang, B., & Leng, Y. (2022) · Journal of Personalized Medicine · doi:10.3390/jpm13010050 · Cited by 19 · see citing works
- Authors
- Pirracchio, R., Petersen, M. L., Carone, M., Rigon, M. R., Chevret, S., & van der Laan, M. J.
- Affiliations
- Inserm (FR)
- University of California, Berkeley (US)
- University of Washington (US)
- Venue
- The Lancet Respiratory Medicine
- Publisher
- Elsevier BV
- Published
- 2015-01
- Volume
- 3
- Issue
- 1
- Pages
- 42-52
- ISSN
- 2213-2600 (ISSN-L)
- PMC
- PMC4321691
- OpenAlex
- W2150765167
- Cited by
- 374
- References
- 54
- Citations by year
- 2026: 23 · 2025: 25 · 2024: 37 · 2023: 37 · 2022: 44 · 2021: 47 · 2020: 48 · 2019: 45 · 2018: 34 · 2017: 16 · 2016: 11 · 2015: 6 · 2014: 1
- FWCI
- 8.56
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- The Lancet Respiratory Medicine (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Funders
- Assistance Publique - Hôpitaux de Paris
- Assistance publique-Hôpitaux de Paris
- National Institute of Allergy and Infectious Diseases (R01AI074345)
- Topics
- Health Sciences › Medicine › Epidemiology › Sepsis Diagnosis and Treatment
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- mortality prediction, intensive care unit, ensemble learning, super learner, in-hospital mortality, severity scoring, SAPS II, APACHE II score, SOFA score, MIMIC II database, external validation, risk classification, discrimination and calibration, risk stratification, intensive care unit admission criteria
- MeSH terms
- Machine Learning, Aged, Aged, 80 and over, Algorithms, Boston, Female, Humans, Intensive Care Units, Male, Middle Aged, Risk, ROC Curve, Severity of Illness Index, Reproducibility of Results, Hospital Mortality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
