CoChat
Sign Up
Cited by 118Sepsis Diagnosis and TreatmentMachine Learning in HealthcareAcute Kidney Injury Research

Predicting sepsis using deep learning across international sites: a retrospective development and validation study

Open access
Moor, Michael, Bennett, Nicolas, Plečko, Drago, Horn, Max, Rieck, Bastian, Meinshausen, Nicolai, Bühlmann, Peter, Borgwardt, Karsten·2023·eClinicalMedicine
DOI 10.1016/j.eclinm.2023.102124PMC PMC10425671
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: When sepsis is detected, organ damage may have progressed to irreversible stages, leading to poor prognosis. The use of machine learning for predicting sepsis early has shown promise, however international validations are missing. Methods: This was a retrospective, observational, multi-centre cohort study. We developed and externally validated a deep learning system for the prediction of sepsis in the intensive care unit (ICU). Our analysis represents the first international, multi-centre in-ICU cohort study for sepsis prediction using deep learning to our knowledge. Our dataset contains 136,478 unique ICU admissions, representing a refined and harmonised subset of four large ICU databases comprising data collected from ICUs in the US, the Netherlands, and Switzerland between 2001 and 2016. Using the international consensus definition Sepsis-3, we derived hourly-resolved sepsis annotations, amounting to 25,694 (18.8%) patient stays with sepsis. We compared our approach to clinical baselines as well as machine learning baselines and performed an extensive internal and external statistical validation within and across databases, reporting area under the receiver-operating-characteristic curve (AUC). Findings: Averaged over sites, our model was able to predict sepsis with an AUC of 0.846 (95% confidence interval [CI], 0.841-0.852) on a held-out validation cohort internal to each site, and an AUC of 0.761 (95% CI, 0.746-0.770) when validating externally across sites. Given access to a small fine-tuning set (10% per site), the transfer to target sites was improved to an AUC of 0.807 (95% CI, 0.801-0.813). Our model raised 1.4 false alerts per true alert and detected 80% of the septic patients 3.7 h (95% CI, 3.0-4.3) prior to the onset of sepsis, opening a vital window for intervention. Interpretation: By monitoring clinical and laboratory measurements in a retrospective simulation of a real-time prediction scenario, a deep learning system for the detection of sepsis generalised to previously unseen ICU cohorts, internationally. Funding: This study was funded by the Personalized Health and Related Technologies (PHRT) strategic focus area of the ETH domain.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.eclinm.2023.102124

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 24

  1. [1]Dellinger R.P. Levy M.M. Rhodes A. Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock: 2012 Crit Care Med 41 2013 580 637 23353941
  2. [2]Hotchkiss R.S. Moldawer L.L. Opal S.M. Reinhart K. Turnbull I.R. Vincent J.-L. Sepsis and septic shock Nat Rev Dis Primers 2 2016 16045 PMC5538252 28117397
  3. [3]Kaukonen K.-M. Bailey M. Suzuki S. Pilcher D. Bellomo R. Mortality related to severe sepsis and septic shock among critically ill patients in Australia and New Zealand, 2000-2012 JAMA 311 2014 1308 1316 24638143
  4. [4]Ferrer R. Martin-Loeches I. Phillips G. Empiric antibiotic treatment reduces mortality in severe sepsis and septic shock from the first hour: results from a guideline-based performance improvement program Crit Care Med 42 2014 1749 1755 24717459
  5. [5]Pruinelli L. Westra B.L. Yadav P. Delay within the 3-hour surviving sepsis campaign guideline on mortality for patients with severe sepsis and septic shock∗ Crit Care Med 46 2018 500 505 29298189 PMC5851815
  6. [7]Osthoff M. Gürtler N. Bassetti S. Impact of MALDI-TOF-MS-based identification directly from positive blood cultures on patient management: a controlled clinical trial Clin Microbiol Infect 23 2017 78 85 27569710
  7. [8]Pickering B.W. Gajic O. Ahmed A. Herasevich V. Keegan M.T. Data utilization for medical decision making at the time of patient admission to ICU Crit Care Med 41 2013 1502 1510 23528804
  8. [11]Levy M.M. Fink M.P. Marshall J.C. 2001 SCCM/ESICM/ACCP/ATS/SIS international sepsis definitions conference Crit Care Med 31 2003 1250 1256 12682500
  9. [12]Singer M. Deutschman C.S. Seymour C.W. The third international consensus definitions for sepsis and septic shock (Sepsis-3) JAMA 315 2016 801 810 26903338 PMC4968574
  10. [13]Moor M. Rieck B. Horn M. Jutzeler C.R. Borgwardt K. Early prediction of sepsis in the ICU using machine learning: a systematic review Front Med 8 2021 607952 PMC8193357 34124082
  11. [17]Johnson A.E.W. Pollard T.J. Shen L. MIMIC-III, a freely accessible critical care database Sci Data 3 2016 160035 PMC4878278 27219127
  12. [18]Pollard T.J. Johnson A.E.W. Raffa J.D. Celi L.A. Mark R.G. Badawi O. The eICU Collaborative Research Database, a freely available multi-center database for critical care research Sci Data 5 2018 180178 PMC6132188 30204154
  13. [19]Vaswani A. Shazeer N. Parmar N. Attention is all you need Guyon I. Luxburg U.V. Bengio S. Advances in neural information processing systems 30 2017 Curran Associates, Inc. 5998 6008
  14. [20]Cho K. van Merrienboer B. Bahdanau D. Bengio Y. On the properties of neural machine translation: encoder-decoder approaches. arXiv [cs.CL] 2014 http://arxiv.org/abs/1409.1259 · arXiv
  15. [21]Ke G. Meng Q. Finley T. LightGBM: a highly efficient gradient boosting decision tree Guyon I. Luxburg U.V. Bengio S. Advances in neural information processing systems 2017 Curran Associates, Inc.
  16. [23]Jones M. NEWSDIG: the national early warning score development and implementation group Clin Med 12 2012 501 503 PMC5922584 23342400
  17. [24]Subbe C.P. Kruger M. Rutherford P. Gemmel L. Validation of a modified early warning score in medical admissions QJM 94 2001 521 526 11588210
  18. [25]Bone R.C. Balk R.A. Cerra F.B. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis Chest 101 1992 1644 1655 1303622
  19. [27]Wardi G. Carlile M. Holder A. Shashikumar S. Hayden S.R. Nemati S. Predicting progression to septic shock in the emergency department using an externally generalizable machine-learning algorithm Ann Emerg Med 77 2021 395 406 33455840 PMC8554871
  20. [28]Shin Y. Cho K.-J. Lee Y. Multicenter validation of a deep-learning-based pediatric early-warning system for prediction of deterioration events Acute Crit Care 37 2022 654 666 36442471 PMC9732200
  21. [30]Leone M. Asfar P. Radermacher P. Vincent J.-L. Martin C. Optimizing mean arterial pressure in septic shock: a critical reappraisal of the literature Crit Care 19 2015 101 25888071 PMC4355573
  22. [31]Genga K.R. Russell J.A. Update of sepsis in the intensive care unit J Innate Immun 9 2017 441 455 28697503 PMC6738902
  23. [33]Shashikumar S.P. Wardi G. Malhotra A. Nemati S. Artificial intelligence sepsis prediction algorithm learns to say “I don’t know” NPJ Digit Med 4 2021 134 34504260 PMC8429719
  24. [38]Bennett N. Plečko D. Ukor I.-F. Meinshausen N. Bühlmann P. ricu: R’s interface to intensive care data GigaScience 12 2023 p.giad041 PMC10268223 37318234

Cited by

Sorted by citations

Authors
Moor, M., Bennett, N., Plečko, D., Horn, M., Rieck, B., Meinshausen, N., Bühlmann, P., & Borgwardt, K.
Affiliations
  • SIB Swiss Institute of Bioinformatics (CH)
  • ETH Zurich (CH)
Corresponding author
Borgwardt, Karsten
ORCID
Venue
eClinicalMedicine
Publisher
Elsevier BV
Published
2023-08
Volume
62
Pages
102124
ISSN
2589-5370 (ISSN-L)
DOI
10.1016/j.eclinm.2023.102124
PMC
PMC10425671
OpenAlex
W4385762558
Cited by
118
References
13
Citations by year
2026: 51 · 2025: 42 · 2024: 22 · 2023: 3
FWCI
23.56
Type
journal article
Language
en
Open access
Licence
Locations
  • EClinicalMedicine (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Repository for Publications and Research Data (ETH Zurich) (submittedVersion, cc-by-nc-nd)
  • ETH Zürich Research Collection
Funders
Alfried Krupp von Bohlen und Halbach-Stiftung
Topics
  • Health Sciences › Medicine › Epidemiology › Sepsis Diagnosis and Treatment
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Nephrology › Acute Kidney Injury Research
Keywords
sepsis prediction, deep learning, intensive care unit, Sepsis-3, early sepsis detection, cross-site generalization, model transfer, false alarms, clinical decision support, multicenter validation, international validation, model interpretability, explainable AI

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review