CoChat
Sign Up
Cited by 315Healthcare Technology and Patient MonitoringMachine Learning in HealthcareSepsis Diagnosis and Treatment

Automated Identification of Adults at Risk for In-Hospital Clinical Deterioration

Open access
Escobar, Gabriel J., Liu, Vincent X., Schuler, Alejandro, Lawson, Brian, Greene, John D., Kipnis, Patricia·2020·New England Journal of Medicine
DOI 10.1056/nejmsa2001090PMC PMC7787261
Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: Hospitalized adults whose condition deteriorates while they are in wards (outside the intensive care unit [ICU]) have considerable morbidity and mortality. Early identification of patients at risk for clinical deterioration has relied on manually calculated scores. Outcomes after an automated detection of impending clinical deterioration have not been widely reported. METHODS: On the basis of a validated model that uses information from electronic medical records to identify hospitalized patients at high risk for clinical deterioration (which permits automated, real-time risk-score calculation), we developed an intervention program involving remote monitoring by nurses who reviewed records of patients who had been identified as being at high risk; results of this monitoring were then communicated to rapid-response teams at hospitals. We compared outcomes (including the primary outcome, mortality within 30 days after an alert) among hospitalized patients (excluding those in the ICU) whose condition reached the alert threshold at hospitals where the system was operational (intervention sites, where alerts led to a clinical response) with outcomes among patients at hospitals where the system had not yet been deployed (comparison sites, where a patient's condition would have triggered a clinical response after an alert had the system been operational). Multivariate analyses adjusted for demographic characteristics, severity of illness, and burden of coexisting conditions. RESULTS: The program was deployed in a staggered fashion at 19 hospitals between August 1, 2016, and February 28, 2019. We identified 548,838 non-ICU hospitalizations involving 326,816 patients. A total of 43,949 hospitalizations (involving 35,669 patients) involved a patient whose condition reached the alert threshold; 15,487 hospitalizations were included in the intervention cohort, and 28,462 hospitalizations in the comparison cohort. Mortality within 30 days after an alert was lower in the intervention cohort than in the comparison cohort (adjusted relative risk, 0.84, 95% confidence interval, 0.78 to 0.90; P<0.001). CONCLUSIONS: The use of an automated predictive model to identify high-risk patients for whom interventions by rapid-response teams could be implemented was associated with decreased mortality. (Funded by the Gordon and Betty Moore Foundation and others.).

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1056/nejmsa2001090

Cites

39 of 39 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Translation of these approaches in clinical intervention studies has improved outcomes.

    “Translation of these approaches in clinical intervention studies has improved outcomes [16, 34, 35].”

    From Use of artificial intelligence in critical care: opportunities and obstacles (2024) · cites this paper as [34]

    The Results (Adjusted Analyses; page not provided) report that the intervention was associated with lower mortality and ICU admission, shorter hospital stays, and more favorable status after 30 days. The Discussion cautions that the study cannot establish what caused those improvements, so “has improved outcomes” states the effect more definitively than this source supports. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Pragmatic randomized trials and quasi-experimental designs offer advantages over pre-post designs or comparisons against historical controls.

    “Pragmatic randomized trials and quasi-experimental designs offer advantages over pre-post designs or comparisons against historical controls [34, 52].”

    From Use of artificial intelligence in critical care: opportunities and obstacles (2024) · cites this paper as [34]

    The Discussion says this study lacked the methodological rigor of a randomized controlled trial, supporting an advantage for randomization. The Methods describe intervention and comparison episodes with adjustment for secular trends, but the paper does not compare pragmatic randomized trials or quasi-experimental designs with pre-post designs or historical controls. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Other EWSs have influenced in-hospital mortality, sepsis or LOS, but not across the breadth of outcomes observed in this study.

    “Other EWSs have influenced in-hospital mortality, sepsis or LOS, but not across the breadth of outcomes we observed4,5,26,27.”

    From Real-time surveillance system for patient deterioration: a pragmatic cluster-randomized controlled trial (2025) · cites this paper as [27]

    The Discussion (page not provided) reports that other early-warning or rapid-response systems were associated with shorter hospital stays or improved mortality, while some showed no change in mortality or ICU admission. It does not address sepsis or establish that those systems affected fewer outcomes than the citing study observed. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Ease of use and integration into the workflow have been noted as important factors influencing adoption.

    “Additionally, ease of use and integration into the workflow have been noted as important factors influencing adoption 26,28,47,48 .”

    From Factors driving provider adoption of the TREWS machine learning-based early warning system and its effects on sepsis treatment timing (2022) · cites this paper as [26]

    The Discussion (page unavailable) says that automation and the lack of a required immediate staff response facilitated the system’s incorporation into a rapid-response program built around standardized workflows. This supports the workflow-integration point, but the paper does not explicitly assess ease of use or establish it as a factor influencing adoption. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • The CONCERN intervention is distinct from other multifactorial machine learning clinical decision support implementations in its modeling approach, integration, and evaluation.

    “While there are other existing multifactorial machine learning clinical decision support implementations that share similarities to ours27–29, the CONCERN intervention is distinct in its modeling approach by using metadata of nursing documentation as early predictors of patient deterioration in acute and intensive care settings, being fully integrated as a real-time EWS into the EHR and undergoing”

    From Real-time surveillance system for patient deterioration: a pragmatic cluster-randomized controlled trial (2025) · cites this paper as [27]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 6

  1. [19]Escobar GJ, Liu V, Kim YS, Early detection of impending deterioration outside the ICU: a difference-in-differences (DiD) study. Am J Respir Crit Care Med 2016; 93: A7614. abstract.
  2. [29]Harrell F Jr. Regression modeling strategies with applications to linear models, logistic regression, and survival analysis. New York: Springer-Verlag, 2010.
  3. [32]Kalbfleisch JD, Prentice RL. The statistical analysis of failure time data. 2nd ed. New York: John Wiley, 2011.
  4. [33]Austin PC, Fine JP. Practical recommendations for reporting Fine-Gray model analyses for competing risk data. Stat Med 2017; 36: 4391– 400. 28913837 PMC5698744
  5. [34]Harrell FE Jr. Semiparametric modeling of health care cost and resource utilization. Presented at the 24th Annual Midwest Biopharmaceutical Statistics Workshop, Muncie, IN, May 21–23 2001.
  6. [38]Collett D Modelling survival data in medical research. 3rd ed. Boca Raton, FL: Chapman and Hall/CRC Press, 2014.

Cited by

Sorted by citations

Related

Authors
Escobar, G. J., Liu, V. X., Schuler, A., Lawson, B., Greene, J. D., & Kipnis, P.
Affiliations
Kaiser Permanente (US)
ORCID
Venue
New England Journal of Medicine
Publisher
Massachusetts Medical Society
Published
2020-11-12
Volume
383
Issue
20
Pages
1951-1960
ISSN
  • 0028-4793 (ISSN-L)
  • 1533-4406
DOI
10.1056/nejmsa2001090
PMC
PMC7787261
OpenAlex
W3106297998
Cited by
314
References
39
Citations by year
2026: 52 · 2025: 63 · 2024: 51 · 2023: 47 · 2022: 56 · 2021: 42 · 2020: 3
FWCI
33.81
Type
journal article
Language
en
Open access
Licence
www.nejmgroup.org/legal/terms-of-use.htm (vor)
Locations
  • New England Journal of Medicine (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
Funders
  • Sidney Garfield Memorial Fund
  • Gordon and Betty Moore Foundation (2663, 2663.01)
  • The Permanente Medical Group, Inc.
  • Kaiser Foundation Hospitals, Inc.
  • National Institutes of Health (K23GM112018, R35GM128672)
  • Agency for Healthcare Research and Quality (1R01HS018480-01)
Topics
  • Health Sciences › Medicine › Surgery › Healthcare Technology and Patient Monitoring
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Epidemiology › Sepsis Diagnosis and Treatment
Keywords
clinical deterioration, early warning systems, rapid response teams, remote patient monitoring, 30-day mortality, electronic health records, hospitalized adults
MeSH terms
Alert Fatigue, Health Personnel, Laboratory Critical Values, Clinical Deterioration, Adult, Aged, Automation, Female, Hospitalization, Humans, Length of Stay, Male, Middle Aged, Models, Theoretical, Nursing Staff, Hospital, Patient Readmission, Telemetry, Multivariate Analysis, Hospital Mortality, Risk Assessment, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review