Early Prediction of Sepsis From Clinical Data: The PhysioNet/Computing in Cardiology Challenge 2019
Open accessAbstract
Objectives: Sepsis is a major public health concern with significant morbidity, mortality, and healthcare expenses. Early detection and antibiotic treatment of sepsis improve outcomes. However, although professional critical care societies have proposed new clinical criteria that aid sepsis recognition, the fundamental need for early detection and treatment remains unmet. In response, researchers have proposed algorithms for early sepsis detection, but directly comparing such methods has not been possible because of different patient cohorts, clinical variables and sepsis criteria, prediction tasks, evaluation metrics, and other differences. To address these issues, the PhysioNet/Computing in Cardiology Challenge 2019 facilitated the development of automated, open-source algorithms for the early detection of sepsis from clinical data. Design: Participants submitted containerized algorithms to a cloud-based testing environment, where we graded entries for their binary classification performance using a novel clinical utility-based evaluation metric. We designed this scoring function specifically for the Challenge to reward algorithms for early predictions and penalize them for late or missed predictions and for false alarms. Setting: ICUs in three separate hospital systems. We shared data from two systems publicly and sequestered data from all three systems for scoring. Patients: We sourced over 60,000 ICU patients with up to 40 clinical variables for each hour of a patient’s ICU stay. We applied Sepsis-3 clinical criteria for sepsis onset. Interventions: None. Measurements and Main Results: A total of 104 groups from academia and industry participated, contributing 853 submissions. Furthermore, 90 abstracts based on Challenge entries were accepted for presentation at Computing in Cardiology. Conclusions: Diverse computational approaches predict the onset of sepsis several hours before clinical recognition, but generalizability to different hospital systems remains a challenge.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Models using readily available EHR data can accurately predict clinical deterioration and sepsis hours before they occur.
“Two prominent examples relevant to critical care are models using readily available EHR data that can accurately predict clinical deterioration and sepsis hours before they occur [28–33]; these models exemplify real-time CDSS that alert clinicians and prompt evaluation, testing, and interventions [33].”
From Use of artificial intelligence in critical care: opportunities and obstacles (2024) · cites this paper as [33]
The Methods (Challenge Objective; page unavailable) describe algorithms using routinely available clinical data to predict sepsis 6–12 hours before onset. The Results (page unavailable) evaluate those predictions, but report that performance varied across hospital systems. This source does not establish accurate prediction of clinical deterioration more generally, or that these algorithms operated as real-time alerts prompting clinical action. (LLM, conf=0.84)
Experimental — accuracy not yet measuredThe incorporation of temporal variability features represents a methodological advancement over models that rely primarily on point-in-time measurements.
“The incorporation of temporal variability features represents a methodological advancement over models that rely primarily on point-in-time measurements28.”
From Machine learning predicts sepsis deterioration trajectories. (2026) · cites this paper as [28]
The Methods (Challenge Data; page unavailable) describe hourly measurements and condensing multiple measurements within an hour to a median. The Discussion (page unavailable) calls the evaluation metric novel, but neither passage addresses temporal variability features or compares them with models using point-in-time measurements. Searches for those terms and for time-series features found no passage making the claimed comparison. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 17 · Cited by 383
Research works printed but not linked: 13
- [1]Seymour CW Liu VX Iwashyna TJ Assessment of clinical criteria for sepsis: For the third international consensus definitions for sepsis and septic shock (sepsis-3). JAMA 2016; 315: 762– 774 26903335 PMC5433435
- [2]Singer M Deutschman CS Seymour CW The third international consensus definitions for sepsis and septic shock (sepsis-3). JAMA 2016; 315: 801– 810 26903338 PMC4968574
- [3]Shankar-Hari M Phillips GS Levy ML Developing a new definition and assessing new clinical criteria for septic shock: For the third international consensus definitions for sepsis and septic shock (sepsis-3). JAMA 2016; 315: 775– 787 26903336 PMC4910392
- [6]Paoli CJ Reynolds MA Sinha M Epidemiology and costs of sepsis in the United States—an analysis based on timing of diagnosis and severity level. Crit Care Med 2018; 46: 1889 30048332 PMC6250243
- [7]Angus DC Linde-Zwirble WT Lidicker J Epidemiology of severe sepsis in the United States: Analysis of incidence, outcome, and associated costs of care. Crit Care Med 2001; 29: 1303– 1310 11445675
- [9]Moran JL Myburgh JA Syres GA The outcome of patients with sepsis and septic shock presenting to emergency departments in Australia and New Zealand. Crit Care Resusc 2007; 9: 8 17352661
- [10]Stoller J Halpin L Weis M Epidemiology of severe sepsis: 2008-2012. J Crit Care 2016; 31: 58– 62 26601855
- [12]Liu VX Fielding-Singh V Greene JD The timing of early antibiotics and hospital mortality in sepsis. Am J Respir Crit Care Med 2017; 196: 856– 863 28345952 PMC5649973
- [14]Prescott HC Iwashyna TJ Improving sepsis treatment by embracing diagnostic uncertainty. Ann Am Thorac Soc 2019; 16: 426– 429 30883190 PMC6441693
- [16]Henry KE Hager DN Pronovost PJ A targeted real-time early warning score (TREWScore) for septic shock. Sci Transl Med 2015; 7: 299ra122– 299ra122 26246167
- [17]Nemati S Holder A Razmi F An interpretable machine learning model for accurate prediction of sepsis in the ICU. Crit Care Med 2018; 46: 547– 553 29286945 PMC5851825
- [18]Cheng LF Prasad N Engelhardt BE An optimal policy for patient laboratory tests in intensive care units. In: Pacific Symposium on Biocomputing. Singapore, 2019, pp World Scientific, 320– 331 PMC6417830 30864333
- [19]Aboukhalil A Nielsen L Saeed M Reducing false alarm rates for critical arrhythmias using the arterial blood pressure waveform. J Biomed Inform 2008; 41: 442– 451 18440873 PMC2504518
- Authors
- Reyna, M. A., Josef, C. S., Jeter, R., Shashikumar, S. P., Westover, M. B., Nemati, S., Clifford, G. D., & Sharma, A.
- Affiliations
- Emory University (US)
- Georgia Institute of Technology (US)
- Massachusetts General Hospital (US)
- Corresponding author
- Reyna, Matthew A.
- ORCID
- Venue
- Critical Care Medicine
- Published
- 2020-02
- Volume
- 48
- Issue
- 2
- Pages
- 210-217
- ISSN
- 0090-3493 (ISSN-L)
- PMC
- PMC6964870
- OpenAlex
- W2995282027
- Cited by
- 383
- References
- 17
- Citations by year
- 2026: 69 · 2025: 58 · 2024: 57 · 2023: 41 · 2022: 33 · 2021: 47 · 2020: 33 · 2019: 45
- FWCI
- 29.93
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Critical Care Medicine (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health (5u24ca215109-05, UL1TR002378)
- National Cancer Institute (U24CA215109,)
- National Institute of General Medical Sciences (R01GM104987, T32GM095442)
- National Institute of Neurological Disorders and Stroke (R01NS102190, R01NS107291)
- National Institute of Environmental Health Sciences (K01ES025445)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Epidemiology › Sepsis Diagnosis and Treatment
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Surgery › Healthcare Technology and Patient Monitoring
- Keywords
- sepsis, early detection, sepsis prediction, intensive care unit, clinical data, Sepsis-3 criteria, algorithm generalizability, false alarms, binary classification
- MeSH terms
- Algorithms, Female, Humans, Intensive Care Units, Male, Severity of Illness Index, Time Factors, United States, Sepsis, Early Diagnosis, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
