CoChat
Sign Up
Cited by 109Sepsis Diagnosis and TreatmentMachine Learning in HealthcareArtificial Intelligence in Healthcare

Machine learning for early detection of sepsis: an internal and temporal validation study

Open access
Bedoya, Armando D, Futoma, Joseph, Clement, Meredith E, Corey, Kristin, Brajer, Nathan, Lin, Anthony, Simons, Morgan G, Gao, Michael +5 more·2020·JAMIA Open
DOI 10.1093/jamiaopen/ooaa006PMC PMC7382639
Open this paper in CoChat — free accountOpen PDF

Abstract

Objective: Determine if deep learning detects sepsis earlier and more accurately than other models. To evaluate model performance using implementation-oriented metrics that simulate clinical practice. Materials and Methods: We trained internally and temporally validated a deep learning model (multi-output Gaussian process and recurrent neural network [MGP–RNN]) to detect sepsis using encounters from adult hospitalized patients at a large tertiary academic center. Sepsis was defined as the presence of 2 or more systemic inflammatory response syndrome (SIRS) criteria, a blood culture order, and at least one element of end-organ failure. The training dataset included demographics, comorbidities, vital signs, medication administrations, and labs from October 1, 2014 to December 1, 2015, while the temporal validation dataset was from March 1, 2018 to August 31, 2018. Comparisons were made to 3 machine learning methods, random forest (RF), Cox regression (CR), and penalized logistic regression (PLR), and 3 clinical scores used to detect sepsis, SIRS, quick Sequential Organ Failure Assessment (qSOFA), and National Early Warning Score (NEWS). Traditional discrimination statistics such as the C-statistic as well as metrics aligned with operational implementation were assessed. Results: The training set and internal validation included 42 979 encounters, while the temporal validation set included 39 786 encounters. The C-statistic for predicting sepsis within 4 h of onset was 0.88 for the MGP–RNN compared to 0.836 for RF, 0.849 for CR, 0.822 for PLR, 0.756 for SIRS, 0.619 for NEWS, and 0.481 for qSOFA. MGP–RNN detected sepsis a median of 5 h in advance. Temporal validation assessment continued to show the MGP–RNN outperform all 7 clinical risk score and machine learning comparisons. Conclusions: We developed and validated a novel deep learning model to detect sepsis. Using our data elements and feature set, our modeling approach outperformed other machine learning methods and clinical scores.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamiaopen/ooaa006

Cites

40 of 40 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The Sepsis Watch model has high predictive value within the first hour of ED presentation.

    “For example, despite the model’s high predictive value within the first hour of ED presentation [44], nurses sometimes waited for more clinical information to populate in the medical record before calling physicians about patients flagged as being at high-risk for sepsis.”

    From Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study (2020) · cites this paper as [44]

    The Results report that the model performed best across metrics at almost all points during ED encounters, and Figure 3 presents its performance by hour after presentation. Our copy does not establish that its positive predictive value was high specifically within the first hour; the Discussion notes low positive predictive value at high sensitivities. The reported first-hour sepsis cases are not a measure of the model’s predictive performance. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 8

  1. [7]Cortes-Puch I, Hartog CS. Opening the debate on the new sepsis definition change is not necessarily progress: revision of the sepsis definition should be based on new scientific insights. Am J Respir Crit Care Med 2016; 194: 16– 8. 27166972 PMC4960634
  2. [12]Futoma J, Hariharan S, Heller K. Learning to detect sepsis with a multitask Gaussian process RNN classifier. In: ICML'17: proceedings of the 34th International Conference on Machine Learning - Volume 70; August 2017: 1174– 82.
  3. [13]Futoma J, Hariharan S, Heller K, et al An improved multi-output Gaussian process RNN with real-time validation for early sepsis detection. In: Finale D-V, Jim F, David K, Rajesh R, Byron W, Jenna W, eds. Proceedings of the 2nd Machine Learning for Healthcare Conference. Proceedings of Machine Learning Research: PMLR; 2017: 243– 54.
  4. [15]Lin AL, Sendak M, Bedoya AD, et al Evaluating sepsis definitions for clinical decision support against a definition for epidemiological disease surveillance. bioRxiv 2019: 648907.
  5. [21]Lipton ZC, Kale DC, Elkan C, Wetzel RC. Learning to diagnose with LSTM recurrent neural networks. CoRR 2015; abs/1511.0.
  6. [22]Lipton ZC, Kale DC, Wetzel RC. Directly modeling missing data in sequences with RNNs: improved classification of clinical time series. In: Machine Learning for Healthcare Conference; 2016: 253– 70.
  7. [27]Bonilla E, Chai KM, Williams C. Multi-Task Gaussian Process Prediction. In: proceedings of the 20th Anniversary Conference on Neural Information Processing Systems; 2008: 153– 60.
  8. [45]Lin C, Zhang Y, Ivy J, et al Early diagnosis and prediction of sepsis shock by combining static and dynamic information using convolutional-LSTM. In: 2018 IEEE International Conference on Healthcare Informatics (ICHI). New York, NY: IEEE; 2018: 219– 28.

Cited by

Sorted by citations

Related

Authors
Bedoya, A. D., Futoma, J., Clement, M. E., Corey, K., Brajer, N., Lin, A., Simons, M. G., Gao, M., Nichols, M., Balu, S., Heller, K., Sendak, M., & O’Brien, C.
Affiliations
  • Duke University (US)
  • Harvard University (US)
  • Department of Medicine, Durham, North Carolina, USA
Corresponding author
Bedoya, Armando D
ORCID
Venue
JAMIA Open
Publisher
Oxford University Press (OUP)
Published
2020-04-11
Volume
3
Issue
2
Pages
252-260
ISSN
2574-2531 (ISSN-L)
DOI
10.1093/jamiaopen/ooaa006
PMC
PMC7382639
OpenAlex
W3015460073
Cited by
109
References
40
Citations by year
2026: 13 · 2025: 24 · 2024: 18 · 2023: 25 · 2022: 15 · 2021: 12 · 2020: 2
FWCI
9.68
Type
journal article
Language
en
Open access
Licence
Locations
  • JAMIA Open (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
  • Duke Institute for Health Innovation
  • HHS | NIH
  • National Institute of Allergy and Infectious Diseases
  • NIAID (T32-AI007392)
  • National Defense Science and Engineering Graduate
  • NDSEG
  • Duke Institute for Health Innovation Clinical Research & Innovation Scholarship
  • NSF Faculty Early Career Development Program
Topics
  • Health Sciences › Medicine › Epidemiology › Sepsis Diagnosis and Treatment
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
sepsis detection, early sepsis diagnosis, deep learning, multi-output Gaussian processes, recurrent neural networks, temporal validation, internal validation, hospitalized adults, systemic inflammatory response syndrome, qSOFA, National Early Warning Score, clinical risk score, concordance index

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review