Development and evaluation of an automated machine learning algorithm for in-hospital mortality risk adjustment among critical care patients.
The DOI matches its Crossref record; 28 of the 29 references it cites were found in a registry or an index.
Abstract
Objectives Risk adjustment algorithms for ICU mortality are necessary for measuring and improving ICU performance. Existing risk adjustment algorithms are not widely adopted. Key barriers to adoption include licensing and implementation costs as well as labor costs associated with human-intensive data collection. Widespread adoption of electronic health records makes automated risk adjustment feasible. Using modern machine learning methods and open source tools, we developed and evaluated a retrospective risk adjustment algorithm for in-hospital mortality among ICU patients. The Risk of Inpatient Death score can be fully automated and is reliant upon data elements that are generated in the course of usual hospital processes. Setting One hundred thirty-one ICUs in 53 hospitals operated by Tenet Healthcare. Patients A cohort of 237,173 ICU patients discharged between January 2014 and December 2016. Design The data were randomly split into training (36 hospitals), and validation (17 hospitals) data sets. Feature selection and model training were carried out using the training set while the discrimination, calibration, and accuracy of the model were assessed in the validation data set. Measurements and main results Model discrimination was evaluated based on the area under receiver operating characteristic curve; accuracy and calibration were assessed via adjusted Brier scores and visual analysis of calibration curves. Seventeen features, including a mix of clinical and administr…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Machine learning has been used to accurately predict sepsis, in-hospital mortality, and acute kidney injury using intraoperative time-series data.9,21,27,32”
From Artificial Intelligence and Surgical Decision-making. (2020) · cites this paper as [21]
Citations
Cites
APACHE II: A severity of disease classification system
KNAUS, W. A., DRAPER, E. A., WAGNER, D. P., & ZIMMERMAN, J. E. (1985) · Critical Care Medicine · doi:10.1097/00003246-198510000-00009 · Cited by 13,664
APACHE II-A Severity of Disease Classification System: Reply
Knaus, W. A., Draper, E. A., Wagner, D. P., & Zimmerman, J. E. (1986) · Critical Care Medicine · doi:10.1097/00003246-198608000-00028 · Cited by 13,376
Machine learning: Trends, perspectives, and prospects
Jordan, M. I., & Mitchell, T. M. (2015) · Science · doi:10.1126/science.aaa8415 · Cited by 9,955
Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author)
Breiman, L. (2001) · Statistical Science · doi:10.1214/ss/1009213726 · Cited by 4,340
A simplified acute physiology score for ICU patients
GALL, J.-R. L., LOIRAT, P., ALPEROVITCH, A., GLASER, P., GRANTHIL, C., MATHIEU, D., MERCIER, P., THOMAS, R., & VILLERS, D. (1984) · Critical Care Medicine · doi:10.1097/00003246-198411000-00012 · Cited by 1,251
Kramer, A. A., & Zimmerman, J. E. (2007) · Critical Care Medicine · doi:10.1097/01.ccm.0000275267.64078.b0 · Cited by 886 · cited 1×
“The Hosmer-Lemeshow test is often used to assess the calibration of risk; however, this test can be unreliable when the sample size is large (22, 23).”
MethodHalpern, N. A., & Pastores, S. M. (2010) · Critical Care Medicine · doi:10.1097/ccm.0b013e3181b090d0 · Cited by 817 · cited 3×
“At the time of this analysis, Tenet Healthcare operated 79 acute care hospitals across the United States.”
BackgroundNates, J. L., Nunnally, M., Kleinpell, R., Blosser, S., Goldner, J., Birriel, B., Fowler, C. S., Byrum, D., Miles, W. S., Bailey, H., et al. (2016) · Critical Care Medicine · doi:10.1097/ccm.0000000000001856 · Cited by 683 · cited 1×
“This feature selection process was carried out in three stages: first, an environmental scan of existing risk adjustment algorithms and of published ICU admission criteria (15); second, consultation with one of the authors (D.”
MethodStandardizing the power of the Hosmer–Lemeshow goodness of fit test in large data sets
Paul, P., Pennell, M. L., & Lemeshow, S. (2013) · Statistics in Medicine · doi:10.1002/sim.5525 · Cited by 432 · cited 1×
“The Hosmer-Lemeshow test is often used to assess the calibration of risk; however, this test can be unreliable when the sample size is large (22, 23).”
MethodHiggins, T. L., Teres, D., Copes, W. S., Nathanson, B. H., Stark, M., & Kramer, A. A. (2007) · Critical Care Medicine · doi:10.1097/01.ccm.0000257337.63529.9f · Cited by 333
Research works printed but not linked: 1
- Lewis-kraus, 2016, The Great A.I. Awakening., N Y Times
Cited by
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Artificial Intelligence and Surgical Decision-making.
Loftus, T. J., Tighe, P. J., Filiberto, A. C., Efron, P. A., Brakenridge, S. C., Mohr, A. M., Rashidi, P., Upchurch, G. R., & Bihorac, A. (2020) · JAMA Surgery · doi:10.1001/jamasurg.2019.4917 · Cited by 650
“9,21,27,32 Each machine learning algorithm has distinct advantages and disadvantages for different tasks such that performance depends on fit between algorithm and task.”
BackgroundWang, K., Tian, J., Zheng, C., Yang, H., Ren, J., Liu, Y., Han, Q., & Zhang, Y. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104813 · Cited by 361
Use of machine learning to analyse routinely collected intensive care unit data: a systematic review
Shillan, D., Sterne, J. A. C., Champneys, A., & Gibbison, B. (2019) · Critical Care · doi:10.1186/s13054-019-2564-9 · Cited by 251
“Six studies used data on more than 100,000 patients: one in 2015, one in 2017, and four in 2018 [2, 12 – 16].”
BackgroundMethod
Metadata
- Authors
- Delahanty, R. J., Kaufman, D., & Jones, S. S.
- Affiliations
- Tenet Healthcare (US)
- The Intensivist Group/Sound Physicians, Tacoma, WA.
- ORCID
- Jones, Spencer S.
- Venue
- Critical Care Medicine
- Published
- 2018-06
- Volume
- 46
- Issue
- 6
- Pages
- e481-e488
- ISSN
- 0090-3493 (ISSN-L)
- OpenAlex
- W2793981925
- Cited by
- 105
- References
- 29
- Citations by year
- 2026: 5 · 2025: 10 · 2024: 17 · 2023: 18 · 2022: 12 · 2021: 13 · 2020: 11 · 2019: 14 · 2018: 5
- FWCI
- 6.06
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Critical Care Medicine (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Critical Care and Intensive Care Medicine › Sepsis Diagnosis and Treatment
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- risk adjustment, in-hospital mortality, intensive care unit, electronic health records, clinical prediction model, discrimination and calibration, Tenet Healthcare
- MeSH terms
- Unsupervised Machine Learning, Algorithms, Female, Humans, Intensive Care Units, Male, Middle Aged, Models, Statistical, Hospital Mortality, Risk Adjustment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Development and evaluation of an automated machine learning algorithm for in-hospital mortality risk adjustment among critical care patients. https://app.cochat.ai/papers/10.1097/ccm.0000000000003011
- The paper
- Delahanty, R. J., Kaufman, D., & Jones, S. S. (2018). Development and evaluation of an automated machine learning algorithm for in-hospital mortality risk adjustment among critical care patients. Critical Care Medicine. https://doi.org/10.1097/ccm.0000000000003011
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.