CoChat
Sign Up
Cited by 638Machine Learning in HealthcareArtificial Intelligence in HealthcareSepsis Diagnosis and Treatment

Multicenter Comparison of Machine Learning Methods and Conventional Regression for Predicting Clinical Deterioration on the Wards

Open access
Churpek, Matthew M., Yuen, Trevor C., Winslow, Christopher, Meltzer, David O., Kattan, Michael W., Edelson, Dana P.·2016·Critical Care Medicine
DOI 10.1097/ccm.0000000000001571PMC PMC4736499
Open this paper in CoChat — free account

Abstract

OBJECTIVE: Machine learning methods are flexible prediction algorithms that may be more accurate than conventional regression. We compared the accuracy of different techniques for detecting clinical deterioration on the wards in a large, multicenter database. DESIGN: Observational cohort study. SETTING: Five hospitals, from November 2008 until January 2013. PATIENTS: Hospitalized ward patients INTERVENTIONS: None MEASUREMENTS AND MAIN RESULTS: Demographic variables, laboratory values, and vital signs were utilized in a discrete-time survival analysis framework to predict the combined outcome of cardiac arrest, intensive care unit transfer, or death. Two logistic regression models (one using linear predictor terms and a second utilizing restricted cubic splines) were compared to several different machine learning methods. The models were derived in the first 60% of the data by date and then validated in the next 40%. For model derivation, each event time window was matched to a non-event window. All models were compared to each other and to the Modified Early Warning score, a commonly cited early warning score, using the area under the receiver operating characteristic curve (AUC). A total of 269,999 patients were admitted, and 424 cardiac arrests, 13,188 intensive care unit transfers, and 2,840 deaths occurred in the study. In the validation dataset, the random forest model was the most accurate model (AUC, 0.80 [95% CI, 0.80-0.80]). The logistic regression model with spline predictors was more accurate than the model utilizing linear predictors (AUC, 0.77 vs 0.74; p < 0.01), and all models were more accurate than the MEWS (AUC, 0.70 [95% CI, 0.70-0.70]). CONCLUSIONS: In this multicenter study, we found that several machine learning methods more accurately predicted clinical deterioration than logistic regression. Use of detection algorithms derived from these techniques may result in improved identification of critically ill patients on the wards.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1097/ccm.0000000000001571

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Models using readily available EHR data can accurately predict clinical deterioration and sepsis hours before they occur.

    “Two prominent examples relevant to critical care are models using readily available EHR data that can accurately predict clinical deterioration and sepsis hours before they occur [28–33]; these models exemplify real-time CDSS that alert clinicians and prompt evaluation, testing, and interventions [33].”

    From Use of artificial intelligence in critical care: opportunities and obstacles (2024) · cites this paper as [29]

    The Predictor variables and Model development sections describe routinely collected EHR data used to predict ward deterioration within the following eight-hour interval; the Results report a validation AUC of 0.80. The paper does not evaluate a model predicting sepsis hours in advance. Its Discussion describes sepsis alerts from another study and says its own models could be implemented with an EHR, rather than reporting that they operated as real-time alerts. Page numbers are not available in our copy. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • The CONCERN intervention is distinct from other multifactorial machine learning clinical decision support implementations in its modeling approach, integration, and evaluation.

    “While there are other existing multifactorial machine learning clinical decision support implementations that share similarities to ours27–29, the CONCERN intervention is distinct in its modeling approach by using metadata of nursing documentation as early predictors of patient deterioration in acute and intensive care settings, being fully integrated as a real-time EWS into the EHR and undergoing”

    From Real-time surveillance system for patient deterioration: a pragmatic cluster-randomized controlled trial (2025) · cites this paper as [29]

    The Methods (Predictor variables) and Discussion describe a multifactorial machine-learning model for predicting ward deterioration, supporting the citation as an example of a similar modeling approach. But the Discussion says the models *can be implemented* with an EHR; it does not report an implemented clinical decision-support system. The claim about how CONCERN differs is the citing authors’ own comparison, not a finding attributed to this paper. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • AI has been applied in various domains of healthcare including cancer research, cardiology, diabetes, mental health, identification of prognosis, Alzheimer's disease, identification of difference in clinical groups, cardiovascular disease, and stroke-related studies.

    “AI has been put in application in various domains of healthcare [83–87] which includes cancer research [88], cardiology [89], diabetes [90], mental health [91], identification of prognosis [92], identification of Alzheimer's disease [93], identification of difference in the clinical groups [94], identification of cardiovascular disease [39], stroke-related studies [95], etc.”

    From Applications of Artificial Intelligence and Big Data Analytics in m-Health: A Healthcare System Perspective (2020) · cites this paper as [83]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 29

  1. [2]Iwashyna TJ Liu V What’s so different about big data? A primer for clinicians trained to think epidemiologically Ann Am Thorac Soc 2014 11 7 1130 1135 25102315 PMC4214055
  2. [3]Bell RM Koren Y Lessons from the Netflix prize challenge ACM SIGKDD Explorations Newsletter 2007 9 2 75 79
  3. [4]Knaus WA Wagner DP Draper EA The APACHE III prognostic system. Risk prediction of hospital mortality for critically ill hospitalized adults Chest 12 1991 100 6 1619 1636 1959406
  4. [5]Le Gall JR Lemeshow S Saulnier F A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study JAMA 1993 270 24 2957 2963 8254858
  5. [7]Pinsky MR Dubrawski A Gleaning knowledge from data in the intensive care unit Am J Respir Crit Care Med 2014 190 6 606 610 25068389 PMC4214111
  6. [8]Churpek MM Yuen TC Winslow C Multicenter development and validation of a risk stratification tool for ward patients Am J Respir Crit Care Med 2014 190 6 649 655 25089847 PMC4214112
  7. [9]He H Garcia EA Learning from imbalanced data Knowledge and Data Engineering, IEEE Transactions 2009 21 9 1263 1284
  8. [10]Kuhn M Johnson K Applied predictive modeling New York, NY Springer 2013 http://dx.doi.org/10.1007/978-1-4614-6849-3 · DOI · Paper page
  9. [12]Hastie T Tibshirani R Friedman J The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Second Edition (Springer Series in Statistics) 2 New York, NY Springer 2009
  10. [13]Subbe CP Kruger M Rutherford P Gemmel L Validation of a modified Early Warning Score in medical admissions QJM 2001 94 10 521 526 11588210
  11. [14]Smith GB Prytherch DR Meredith P Schmidt PE Featherstone PI The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death Resuscitation 2013 84 4 465 470 23295778
  12. [15]Steyerberg EW Clinical prediction models a practical approach to development, validation, and updating Statistics for biology and health New York Springer 2009 http://dx.doi.org/10.1007/978-0-387-77244-8 · DOI · Paper page
  13. [18]Pirracchio R Petersen ML Carone M Rigon MR Chevret S van der Laan MJ Mortality prediction in intensive care units with the Super ICU Learner Algorithm (SICULA): a population-based study The Lancet. Respiratory medicine 1 2015 3 1 42 52 25466337 PMC4321691
  14. [19]Mao Y Chen Y Hackmann G Medical Data Mining for Early Deterioration Warning in General Hospital Wards Paper presented at: Data Mining Workshops (ICDMW), 2011 IEEE 11th International Conference on 2011
  15. [20]Badriyah T Briggs JS Meredith P Decision-tree early warning score (DTEWS) validates the design of the National Early Warning Score (NEWS) Resuscitation 2014 85 3 418 423 24361673
  16. [21]Kattan MW Comparison of Cox regression with other methods for determining prediction models and nomograms The Journal of urology 2003 170 6 Pt 2 S6 9 discussion S10 14610404
  17. [22]Austin PC Lee DS Steyerberg EW Tu JV Regression trees for predicting mortality in patients with cardiovascular disease: what improvement is achieved by using ensemble-based methods? Biometrical journal. Biometrische Zeitschrift 9 2012 54 5 657 673 22777999 PMC3470596
  18. [23]van der Ploeg T Smits M Dippel DW Hunink M Steyerberg EW Prediction of intracranial findings on CT-scans by alternative modelling techniques BMC medical research methodology 2011 11 143 22026551 PMC3212831
  19. [24]Wolpert D The supervised learning no-free-lunch theorems Proceedings of the 6th Online World Conference of Soft Computing in Industrial Applications 2001 10 24
  20. [25]Van Belle VM Van Calster B Timmerman D A mathematical model for interpretable clinical decision support with applications in gynecology PloS one 2012 7 3 e34312 22479598 PMC3315538
  21. [26]Churpek MM Yuen TC Huber MT Park SY Hall JB Edelson DP Predicting cardiac arrest on the wards: a nested case-control study Chest 5 2012 141 5 1170 1176 22052772 PMC3342781
  22. [27]Escobar GJ LaGuardia JC Turk BJ Ragins A Kipnis P Draper D Early detection of impending physiologic deterioration among patients who are not in intensive care: development of predictive models using data from an automated electronic medical record J Hosp Med 2012 7 5 388 395 22447632
  23. [28]Sawyer AM Deal EN Labelle AJ Implementation of a real-time computerized sepsis alert in nonintensive care unit patients Crit Care Med 2011 39 3 469 473 21169824
  24. [29]Bailey TC Chen Y Mao Y A trial of a real-time alert for clinical deterioration in patients hospitalized on general medical wards J Hosp Med 2013 8 5 236 242 23440923
  25. [30]Herasevich V Tsapenko M Kojicic M Limiting ventilator-induced lung injury through individual electronic medical record surveillance Crit Care Med 2011 39 1 34 39 20959788
  26. [31]Hravnak M Devita MA Clontz A Edwards L Valenta C Pinsky MR Cardiorespiratory instability before and after implementing an integrated monitoring system Crit Care Med 2011 39 1 65 72 20935559 PMC3673290
  27. [32]Hravnak M Chen L Dubrawski A Bose E Pinsky MR Temporal distribution of instability events in continuously monitored step-down unit patients: implications for Rapid Response Systems Resuscitation 2015 89 99 105 25637693 PMC4363221
  28. [33]Biganzoli E Boracchi P Mariani L Marubini E Feed forward neural networks for the analysis of censored survival data: a partial logistic regression approach Stat Med 1998 17 10 1169 1186 9618776
  29. [35]Zupan B Demsar J Kattan MW Beck JR Bratko I Machine learning for survival analysis: a case study on recurrence of prostate cancer Artif Intell Med 2000 20 1 59 75 11185421

Cited by

Sorted by citations

Authors
Churpek, M. M., Yuen, T. C., Winslow, C., Meltzer, D. O., Kattan, M. W., & Edelson, D. P.
Affiliations
  • Cleveland Clinic (US)
  • University of Chicago (US)
  • Endeavor Health (US)
ORCID
Venue
Critical Care Medicine
Publisher
Ovid Technologies (Wolters Kluwer Health)
Published
2016-02
Volume
44
Issue
2
Pages
368-374
ISSN
0090-3493 (ISSN-L)
DOI
10.1097/ccm.0000000000001571
PMC
PMC4736499
OpenAlex
W2323525554
Cited by
638
References
2
Citations by year
2026: 58 · 2025: 76 · 2024: 67 · 2023: 63 · 2022: 92 · 2021: 83 · 2020: 76 · 2019: 47 · 2018: 50 · 2017: 20 · 2016: 5
FWCI
33.50
Type
journal article
Language
en
Open access
Locations
  • Critical Care Medicine (publishedVersion)
  • PubMed (publishedVersion)
  • Europe PMC (PubMed Central) (acceptedVersion)
  • PubMed Central (acceptedVersion)
Funders
  • National Institutes of Health (UL1TR000430)
  • National Institute on Aging (K24AG031326)
  • National Heart, Lung, and Blood Institute (K08HL121080)
  • National Center for Research Resources (UL1RR024999)
  • National Center for Advancing Translational Sciences
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Health Sciences › Medicine › Epidemiology › Sepsis Diagnosis and Treatment
Keywords
clinical deterioration prediction, machine learning, random forest, logistic regression, restricted cubic splines, early warning scores, Modified Early Warning Score, cardiac arrest, ICU transfer, in-hospital mortality, vital signs, laboratory parameters, discrete-time survival analysis
MeSH terms
Machine Learning, Age Factors, Heart Arrest, Humans, Intensive Care Units, ROC Curve, Socioeconomic Factors, Time Factors, Models, Statistical, Cohort Studies, Logistic Models, Survival Analysis, Neural Networks, Computer, Risk Assessment, Diagnostic Techniques and Procedures, Vital Signs, Support Vector Machine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review