CoChat
Sign Up
Cited by 19Machine Learning in HealthcareSepsis Diagnosis and TreatmentArtificial Intelligence in Healthcare

External Validation and Comparison of a General Ward Deterioration Index Between Diversely Different Health Systems

Open access
Cummings, Brandon C., Blackmer, Joseph M., Motyka, Jonathan R., Farzaneh, Negar, Cao, Loc, Bisco, Erin L., Glassbrook, James D., Roebuck, Michael D. +7 more·2023·Critical Care Medicine
DOI 10.1097/ccm.0000000000005837PMC PMC10187626
Open this paper in CoChat — free account

Abstract

OBJECTIVES: Implementing a predictive analytic model in a new clinical environment is fraught with challenges. Dataset shifts such as differences in clinical practice, new data acquisition devices, or changes in the electronic health record (EHR) implementation mean that the input data seen by a model can differ significantly from the data it was trained on. Validating models at multiple institutions is therefore critical. Here, using retrospective data, we demonstrate how Predicting Intensive Care Transfers and other UnfoReseen Events (PICTURE), a deterioration index developed at a single academic medical center, generalizes to a second institution with significantly different patient population. DESIGN: PICTURE is a deterioration index designed for the general ward, which uses structured EHR data such as laboratory values and vital signs. SETTING: The general wards of two large hospitals, one an academic medical center and the other a community hospital. SUBJECTS: The model has previously been trained and validated on a cohort of 165,018 general ward encounters from a large academic medical center. Here, we apply this model to 11,083 encounters from a separate community hospital. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: The hospitals were found to have significant differences in missingness rates (> 5% difference in 9/52 features), deterioration rate (4.5% vs 2.5%), and racial makeup (20% non-White vs 49% non-White). Despite these differences, PICTURE’s performance was consistent (area under the receiver operating characteristic curve [AUROC], 0.870; 95% CI, 0.861–0.878), area under the precision-recall curve (AUPRC, 0.298; 95% CI, 0.275–0.320) at the first hospital; AUROC 0.875 (0.851–0.902), AUPRC 0.339 (0.281–0.398) at the second. AUPRC was standardized to a 2.5% event rate. PICTURE also outperformed both the Epic Deterioration Index and the National Early Warning Score at both institutions. CONCLUSIONS: Important differences were observed between the two institutions, including data availability and demographic makeup. PICTURE was able to identify general ward patients at risk of deterioration at both hospitals with consistent performance (AUROC and AUPRC) and compared favorably to existing metrics.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1097/ccm.0000000000005837

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • CDS tools directly influence patient care, with sepsis alerts prompting earlier antibiotic administration and fluid resuscitation, and deterioration scores used to trigger rapid response evaluations, intensified monitoring, and escalation of care.

    “These tools directly influence patient care, with sepsis alerts prompting earlier antibiotic administration and fluid resuscitation, and deterioration scores used to trigger rapid response evaluations, intensified monitoring, and escalation of care.[3][4][5]”

    From A Systematic Review and Meta-analysis of Externally Validated Epic Clinical Decision Support Tools (2026) · cites this paper as [4]

    Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • Models using readily available EHR data can accurately predict clinical deterioration and sepsis hours before they occur.

    “Two prominent examples relevant to critical care are models using readily available EHR data that can accurately predict clinical deterioration and sepsis hours before they occur [28–33]; these models exemplify real-time CDSS that alert clinicians and prompt evaluation, testing, and interventions [33].”

    From Use of artificial intelligence in critical care: opportunities and obstacles (2024) · cites this paper as [31]

  • Based on simulations using a previously published model, the bias in observed model performance estimates increases as the model becomes more successful in improving clinical outcomes.

    “Based on simulations using a previously published model,4 the resulting bias in observed model performance estimates increases as the model becomes more successful in improving clinical outcomes (Fig. 1).”

    From Challenges in the postmarket surveillance of clinical prediction models. (2025) · cites this paper as [4]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 24

  1. [1]Allen J Currey J Jones D: Development and validation of the medical emergency team-risk prediction model for clinical deterioration in acute hospital patients, at time of an emergency admission. Crit Care Med 2022; 50: 1588– 1598 35866655
  2. [2]Saab A Abi Khalil C Jammal M: Early prediction of all-cause clinical deterioration in general wards patients: Development and validation of a biomarker-based machine learning model derived from rapid response team activations. J Patient Saf 2022; 18: 578– 586 35985042
  3. [3]Reardon PM Seely AJE Fernando SM: Can early warning systems enhance detection of high risk patients by rapid response teams? J Intensive Care Med 2021; 36: 542– 549 33530838
  4. [4]Fernandes M Mendes R Vieira SM: Predicting intensive care unit admission among patients presenting to the emergency department using machine learning and natural language processing. PLoS One 2020; 15: e0229331 32126097 PMC7053743
  5. [5]Churpek MM Yuen TC Winslow C: Multicenter comparison of machine learning methods and conventional regression for predicting clinical deterioration on the wards. Crit Care Med 2016; 44: 368– 374 26771782 PMC4736499
  6. [6]Kipnis P Turk BJ Wulf DA: Development and validation of an electronic medical record-based alert score for detection of inpatient deterioration outside the ICU. J Biomed Inform 2016; 64: 10– 19 27658885 PMC5510648
  7. [7]Desautels T Calvert J Hoffman J: Using transfer learning for improved mortality prediction in a data-scarce hospital setting. Biomed Inform Insights 2017; 9: 1178222617712994 28638239 PMC5470861
  8. [8]Desautels T Das R Calvert J: Prediction of early unplanned intensive care unit readmission in a UK tertiary care hospital: A cross-sectional machine learning approach. BMJ Open 2017; 7: e017199 PMC5640090 28918412
  9. [9]Alvarez CA Clark CA Zhang S: Predicting out of intensive care unit cardiopulmonary arrest or death using electronic medical record data. BMC Med Inform Decis Mak 2013; 13: 28 23442316 PMC3599266
  10. [10]Green M Lander H Snyder A: Comparison of the between the flags calling criteria to the MEWS, NEWS and the electronic Cardiac Arrest Risk Triage (eCART) score for the identification of deteriorating ward patients. Resuscitation 2018; 123: 86– 91 29169912 PMC6556215
  11. [11]Escobar GJ LaGuardia JC Turk BJ: Early detection of impending physiologic deterioration among patients who are not in intensive care: Development of predictive models using data from an automated electronic medical record. J Hosp Med 2012; 7: 388– 395 22447632
  12. [12]Churpek MM Yuen TC Park SY: Using electronic health record data to develop and validate a prediction model for adverse outcomes in the wards*. Crit Care Med 2014; 42: 841– 848 24247472 PMC3959228
  13. [13]Churpek MM Yuen TC Winslow C: Multicenter development and validation of a risk stratification tool for ward patients. Am J Respir Crit Care Med 2014; 190: 649– 655 25089847 PMC4214112
  14. [14]Chen L Ogundele O Clermont G: Dynamic and personalized risk forecast in step-down units. Implications for monitoring paradigms. Ann Am Thorac Soc 2017; 14: 384– 391 28033032 PMC5427723
  15. [16]Coiera E: The last mile: Where artificial intelligence meets reality. J Med Internet Res 2019; 21: e16323 31702559 PMC7351266
  16. [19]Subbaswamy A Saria S: From development to deployment: Dataset shift, causality, and shift-stable models in health AI. Biostatistics 2020; 21: 345– 352 31742354
  17. [20]Ramspek CL Jager KJ Dekker FW: External validation of prognostic models: What, why, how, when and where? Clin Kidney J 2021; 14: 49– 58 33564405 PMC7857818
  18. [21]Siontis GCM Tzoulaki I Castaldi PJ: External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination. J Clin Epidemiol 2015; 68: 25– 34 25441703
  19. [22]Cummings BC Ansari S Motyka JR: Predicting intensive care transfers and other unforeseen events: Analytic model validation study and comparison to existing methods. JMIR Med Informat 2021; 9: e25066 PMC8061893 33818393
  20. [23]Gillies CE Taylor DF Cummings BC: Demonstrating the consequences of learning missingness patterns in early warning systems for preventative health care: A novel simulation and solution. J Biomed Inform 2020; 110: 103528 32795506
  21. [24]Fleishon HB Itri JN Boland GW: Academic medical centers and community hospitals integration: Trends and strategies. J Am Coll Radiol 2017; 14: 45– 51 27815052
  22. [25]Hagberg A Swart P Schult D: Exploring network structure, dynamics, and function using NetworkX. Proceedings of the 7th Python in Science Conference. Pasadena, CA, 2008, pp 11– 15
  23. [26]Linnen DT Escobar GJ Hu X: Statistical modeling and aggregate-weighted scoring systems in prediction of mortality and ICU transfer: A systematic review. J Hosp Med 2019; 14: 161– 169 30811322 PMC6628701
  24. [27]McDonald CJ Huff SM Suico JG: LOINC, a universal standard for identifying laboratory observations: A 5-year update. Clin Chem 2003; 49: 624– 633 12651816

Cited by

Sorted by citations

Authors
Cummings, B. C., Blackmer, J. M., Motyka, J. R., Farzaneh, N., Cao, L., Bisco, E. L., Glassbrook, J. D., Roebuck, M. D., Gillies, C. E., Admon, A. J., Medlin, R. P., Singh, K., Sjoding, M. W., Ward, K. R., & Ansari, S.
Affiliations
  • University of Michigan (US)
  • Hurley Medical Center (US)
Corresponding author
Cummings, Brandon C.
ORCID
Venue
Critical Care Medicine
Publisher
Ovid Technologies (Wolters Kluwer Health)
Published
2023-03-16
Volume
51
Issue
6
Pages
775-786
ISSN
0090-3493 (ISSN-L)
DOI
10.1097/ccm.0000000000005837
PMC
PMC10187626
OpenAlex
W4327625974
Cited by
19
References
5
Citations by year
2026: 5 · 2025: 5 · 2024: 6 · 2023: 3
FWCI
1.79
Type
journal article
Language
en
Open access
Licence
Locations
  • Critical Care Medicine (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
Funders
  • Teva Pharmaceutical Industries
  • University of Michigan
  • Regeneron Pharmaceuticals
  • Flatiron Health
  • National Institutes of Health (K12TR004374)
  • College of Pharmacy, University of Michigan
  • Michigan Institute for Data Science, University of Michigan
  • National Heart, Lung, and Blood Institute (R01HL158626, K08HL155407)
  • National Center for Advancing Translational Sciences
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Epidemiology › Sepsis Diagnosis and Treatment
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
deterioration index, images, external validation, model generalization, distribution shift, community hospitals, academic medical centers, electronic health record data, vital signs, laboratory parameters, early warning scores, precision-recall curve, missing data, clinical decision support
MeSH terms
Critical Care, Hospitals, Community, Humans, Patients' Rooms, Retrospective Studies, ROC Curve

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review