CoChat
Sign Up
Cited by 74Sepsis Diagnosis and TreatmentMachine Learning in HealthcareAcute Kidney Injury Research

Evaluation of sepsis prediction models before onset of treatment

Kamran, Fahad, Tjandra, Donna, Heiler, Andrew, Virzi, Jessica, Singh, Karandeep, King, Jessie E., Valley, Thomas S., Wiens, Jenna·2024·NEJM AI
DOI 10.1056/aioa2300032
Open this paper in CoChat — free account

Abstract

BackgroundTimely interventions, such as antibiotics and intravenous fluids, have been associated with reduced mortality in patients with sepsis. Artificial intelligence (AI) models that accurately predict risk of sepsis onset could speed the delivery of these interventions. Although sepsis models generally aim to predict its onset, clinicians might recognize and treat sepsis before the sepsis definition is met. Predictions occurring after sepsis is clinically recognized (i.e., after treatment begins) may be of limited utility. Researchers have not previously investigated the accuracy of sepsis risk predictions that are made before treatment begins. Thus, we evaluate the discriminative performance of AI sepsis predictions made throughout a hospitalization relative to the time of treatment. MethodsWe used a large retrospective inpatient cohort from the University of Michigan's academic medical center (2018–2020) to evaluate the Epic sepsis model (ESM). The ability of the model to predict sepsis, both before sepsis criteria are met and before indications of treatment plans for sepsis, was evaluated in terms of the area under the receiver operating characteristic curve (AUROC). Indicators of a treatment plan were identified through electronic data capture and included the receipt of antibiotics, fluids, blood culture, and/or lactate measurement. The definition of sepsis was a composite of the Centers for Disease Control and Prevention's surveillance criteria and the severe sepsis and septic shock management bundle definition. ResultsThe study included 77,582 hospitalizations. Sepsis occurred in 3766 hospitalizations (4.9%). ESM achieved an AUROC of 0.62 (95% confidence interval [CI], 0.61 to 0.63) when including predictions before sepsis criteria were met and in some cases, after clinical recognition. When excluding predictions after clinical recognition, the AUROC dropped to 0.47 (95% CI, 0.46 to 0.48). ConclusionsWe evaluate a sepsis risk prediction model to measure its ability to predict sepsis before clinical recognition. Our work has important implications for future work in model development and evaluation, with the goal of maximizing the clinical utility of these models. (Funded by Cisco Research and others.)

Integrity signals

Registry

Crossref · 10.1056/aioa2300032

Cites

42 of 42 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Kamran, F., Tjandra, D., Heiler, A., Virzi, J., Singh, K., King, J. E., Valley, T. S., & Wiens, J.
Affiliations
  • University of Michigan (US)
  • Michigan Medicine (US)
Corresponding author
Wiens, Jenna
ORCID
Venue
NEJM AI
Publisher
Massachusetts Medical Society
Published
2024-02-22
Volume
1
Issue
3
Pages
AIoa2300032-AIoa2300032
ISSN
2836-9386 (ISSN-L)
DOI
10.1056/aioa2300032
OpenAlex
W4391598924
Cited by
74
References
42
Citations by year
2026: 26 · 2025: 30 · 2024: 17 · 2022: 1
FWCI
22.28
Type
journal article
Language
en
Open access
closed
Locations
  • NEJM AI (publishedVersion)
  • PubMed (publishedVersion)
Topics
  • Health Sciences › Medicine › Epidemiology › Sepsis Diagnosis and Treatment
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Nephrology › Acute Kidney Injury Research
Keywords
sepsis, sepsis risk prediction, artificial intelligence, clinical utility, discriminative performance, sepsis definitions, septic shock, hospitalization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review