CoChat
Sign Up
Cited by 111COVID-19 Clinical Research StudiesMachine Learning in HealthcareSepsis Diagnosis and Treatment

The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.

Evaluating a Widely Implemented Proprietary Deterioration Index Model among Hospitalized Patients with COVID-19

Open access
Singh, Karandeep, Valley, Thomas S., Tang, Shengpu, Li, Benjamin Y., Kamran, Fahad, Sjoding, Michael W., Wiens, Jenna, Otles, Erkin, Donnelly, John P., Wei, Melissa Y., McBride, Jonathon P., Cao, Jie, Penoza, Carleen, Ayanian, John Z., Nallamothu, Brahmajee K.·2021·Annals of the American Thoracic Society
DOI 10.1513/annalsats.202006-698ocPMC PMC8328366

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Rationale: The Epic Deterioration Index (EDI) is a proprietary prediction model implemented in over 100 U.S. hospitals that was widely used to support medical decision-making during the coronavirus disease (COVID-19) pandemic. The EDI has not been independently evaluated, and other proprietary models have been shown to be biased against vulnerable populations. Objectives: To independently evaluate the EDI in hospitalized patients with COVID-19 overall and in disproportionately affected subgroups. Methods: We studied adult patients admitted with COVID-19 to units other than the intensive care unit at a large academic medical center from March 9 through May 20, 2020. We used the EDI, calculated at 15-minute intervals, to predict a composite outcome of intensive care unit–level care, mechanical ventilation, or in-hospital death. In a subset of patients hospitalized for at least 48 hours, we also evaluated the ability of the EDI to identify patients at low risk of experiencing this composite outcome during their remaining hospitalization. Results: Among 392 COVID-19 hospitalizations meeting inclusion criteria, 103 (26%) met the composite outcome. The median age of the cohort was 64 (interquartile range, 53–75) with 168 (43%) Black patients and 169 (43%) women. The area under the receiver-operating characteristic curve of the EDI was 0.79 (95% confidence interval, 0.74–0.84). EDI predictions did not differ by race or sex. When exploring clinically relevant thresholds of the EDI, we found patients who met or exceeded an EDI of 68.8 made up 14% of the study cohort and had a 74% probability of experiencing the composite outcome during their hospitalization with a sensitivity of 39% and a median lead time of 24 hours from when this threshold was first exceeded. Among the 286 patients hospitalized for at least 48 hours who had not experienced the composite outcome, 14 (13%) never exceeded an EDI of 37.9, with a negative predictive value of 90% and a sensitivity above this threshold of 91%. Conclusions: We found the EDI identifies small subsets of high-risk and low-risk patients with COVID-19 with good discrimination, although its clinical use as an early warning system is limited by low sensitivity. These findings highlight the importance of independent evaluation of proprietary models before widespread operational use among patients with COVID-19.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1513/annalsats.202006-698oc

Cites

16 of 16 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1. High-risk analysis plot showing the relationship between the Epic Deterioration Index (EDI) threshold and the sensitivity, specificity, PPV, and NPV, with a histogram demonstrating the distribution of maximum EDI per patient. The high-risk area (≥68.8) is shaded in orange. NPV = negative predictive value; PPV = positive predictive value.
Figure 1. High-risk analysis plot showing the relationship between the Epic Deterioration Index (EDI) threshold and the sensitivity, specificity, PPV, and NPV, with a histogram demonstrating the distribution of maximum EDI per patient. The high-risk area (≥68.8) is shaded in orange. NPV = negative predictive value; PPV = positive predictive value.CC BY-NC-ND · PubMed Central
Figure 2. Distribution of alert times based on exceeding the high-risk threshold (Epic Deterioration Index ≥ 68.8), demonstrating (A) all alerts and (B) alerts in the 24 hours before the outcome, with the first alert highlighted in red. Each point represents a hypothetical alert; no actual alerts were generated.
Figure 2. Distribution of alert times based on exceeding the high-risk threshold (Epic Deterioration Index ≥ 68.8), demonstrating (A) all alerts and (B) alerts in the 24 hours before the outcome, with the first alert highlighted in red. Each point represents a hypothetical alert; no actual alerts were generated.CC BY-NC-ND · PubMed Central
Figure 3. Calibration curve comparing deciles of all predicted Epic Deterioration Index (EDI), rescaled to 0 to 1, with the observed risk, with a line demonstrating ideal calibration (solid) and a histogram of predicted EDI.
Figure 3. Calibration curve comparing deciles of all predicted Epic Deterioration Index (EDI), rescaled to 0 to 1, with the observed risk, with a line demonstrating ideal calibration (solid) and a histogram of predicted EDI.CC BY-NC-ND · PubMed Central
Figure 4. Low-risk analysis plot showing the relationship between the Epic Deterioration Index (EDI) score threshold in the first 48 hours and the sensitivity, specificity, PPV, and NPV, with a histogram demonstrating the distribution of maximum EDI per patient. The low-risk area (<37.9) is shaded in green. NPV = negative predictive value; PPV = positive predictive value.
Figure 4. Low-risk analysis plot showing the relationship between the Epic Deterioration Index (EDI) score threshold in the first 48 hours and the sensitivity, specificity, PPV, and NPV, with a histogram demonstrating the distribution of maximum EDI per patient. The low-risk area (<37.9) is shaded in green. NPV = negative predictive value; PPV = positive predictive value.CC BY-NC-ND · PubMed Central
Figure 5. Four example patients, of whom two experienced an adverse outcome, with shaded risk thresholds. Green (Epic Deterioration Index [EDI] < 37.9) represents low risk, yellow represents intermediate risk (EDI ≥ 37.9 to < 68.8), and orange (EDI ≥ 68.8) represents high risk. EDI scores recorded after the primary outcome are shown in the top panels but were not used in the model validation. The blue line represents transfer to an intensive care unit, and the red line represents the onset of mechanical ventilation.
Figure 5. Four example patients, of whom two experienced an adverse outcome, with shaded risk thresholds. Green (Epic Deterioration Index [EDI] < 37.9) represents low risk, yellow represents intermediate risk (EDI ≥ 37.9 to < 68.8), and orange (EDI ≥ 68.8) represents high risk. EDI scores recorded after the primary outcome are shown in the top panels but were not used in the model validation. The blue line represents transfer to an intensive care unit, and the red line represents the onset of mechanical ventilation.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 16 references

Research works printed but not linked: 1

  1. Robin, pROC: display and analyze ROC curves, 10.32614/CRAN.package.pROC · DOI

Cited by

Sorted by citations

See all 111

Related

Metadata

Authors
Singh, K., Valley, T. S., Tang, S., Li, B. Y., Kamran, F., Sjoding, M. W., Wiens, J., Otles, E., Donnelly, J. P., Wei, M. Y., McBride, J. P., Cao, J., Penoza, C., Ayanian, J. Z., & Nallamothu, B. K.
Affiliations
  • University of Michigan (US)
  • Department of Industrial and Operations Engineering, University of Michigan College of Engineering, Ann Arbor, Michigan; and
  • Michigan Medicine (US)
Corresponding author
Singh, Karandeep
ORCID
Venue
Annals of the American Thoracic Society
Publisher
Oxford University Press (OUP)
Published
2021-07-01
Volume
18
Issue
7
Pages
1129-1137
ISSN
  • 2325-6621 (ISSN-L)
  • 2329-6933
DOI
10.1513/annalsats.202006-698oc
PMC
PMC8328366
OpenAlex
W3114626119
Cited by
111
References
16
Citations by year
2026: 12 · 2025: 15 · 2024: 16 · 2023: 16 · 2022: 32 · 2021: 16
FWCI
2.23
Type
journal article
Language
en
Open access
Licence
Locations
  • Annals of the American Thoracic Society (publishedVersion, cc-by-nc-nd)
  • medRxiv (acceptedVersion)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • PubMed Central (submittedVersion)
Funders
  • National Institutes of Health
  • Michigan Institute for Data Science, University of Michigan
  • National Institute on Aging (K23AG056638)
  • National Heart, Lung, and Blood Institute (K01HL136687, L30 HL133988, K23HL140165, K12HL138039)
  • National Institute of General Medical Sciences (T32GM007863)
  • U.S. National Library of Medicine (R01LM013325)
Topics
  • Health Sciences › Medicine › Infectious Diseases › COVID-19 Clinical Research Studies
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Critical Care and Intensive Care Medicine › Sepsis Diagnosis and Treatment
Keywords
COVID-19 pandemic, model evaluation, early warning systems, hospitalized patients, intensive care unit admission, mechanical ventilation, in-hospital mortality, racial and sex disparities, model discrimination, model bias, clinical decision support, health equity, Black patients, women, vulnerable populations

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review