CoChat
Sign Up
Cited by 21Sepsis Diagnosis and TreatmentMachine Learning in HealthcareArtificial Intelligence in Healthcare

Evaluating sepsis watch generalizability through multisite external validation of a sepsis machine learning model.

Open access
Valan, Bruno, Prakash, Anusha, Ratliff, William, Gao, Michael, Muthya, Srikanth, Thomas, Ajit, Eaton, Jennifer L., Gardner, Matt +7 more·2025·npj Digital Medicine
DOI 10.1038/s41746-025-01664-5PMC PMC12159134
Open this paper in CoChat — free accountOpen PDF

Abstract

Sepsis accounts for a substantial portion of global deaths and healthcare costs. The objective of this reproducibility study is to validate Duke Health's Sepsis Watch ML model, in a new community healthcare setting and assess its performance and clinical utility in early sepsis detection at Summa Health's emergency departments. The study analyzed the model's ability to predict sepsis using a combination of static and dynamic patient data using 205,005 encounters between 2020 and 2021 from 101,584 unique patients. 54.7% (n = 112,223) patients were female and the average age was 50 (IQR [38,71]). The AUROC ranged from 0.906 to 0.960, and the AUPRC ranged from 0.177 to 0.252 across the four sites. Ultimately, the reproducibility of the Sepsis Watch model in a community health system setting confirmed its strong and robust performance and portability across different geographical and demographic contexts with little variation.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41746-025-01664-5

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The overall AUROC of the new model was generally lower than that reported in other frontier sepsis prediction models, such as COMPOSER, Targeted Real-time Early Warning System, and SepsisWatch.

    “While the overall performance (eg, AUROC) of the new model was generally lower than that reported in other frontier sepsis prediction models, such as COMPOSER,18 Targeted Real-time Early Warning System,19 and SepsisWatch,20”

    From Multicenter Prospective Validation of an Updated Proprietary Sepsis Prediction Model (2026) · cites this paper as [20]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 34

  1. [1]Levy, M. M. et al. 2001 sccm/esicm/accp/ats/sis international sepsis definitions conference. Crit. Care Med. 31, 1250–1256 (2003). 12682500 · DOI
  2. [2]Liu, V. et al. Hospital deaths in patients with sepsis from 2 independent cohorts. JAMA 312, 90 (2014). 24838355 · DOI
  3. [3]Rudd, K. E. et al. Global, regional, and national sepsis incidence and mortality, 1990–2017: analysis for the global burden of disease study. Lancet 395, 200–211 (2020). 31954465 PMC6970225 · DOI · Paper page
  4. [4]Buchman, T. G. et al. Sepsis among medicare beneficiaries: 1. the burdens of sepsis, 2012–2018*. Crit. Care Med. 48, 276–288 (2020). 32058366 PMC7017943 · DOI
  5. [5]Paoli, C. J., Reynolds, M. A., Sinha, M., Gitlin, M. & Crouser, E. Epidemiology and costs of sepsis in the united states–an analysis based on timing of diagnosis and severity level*. Crit. Care Med. 46, 1889–1897 (2018). 30048332 PMC6250243 · DOI
  6. [6]Rhodes, A. et al. Surviving sepsis campaign: international guidelines for management of sepsis and septic shock: 2016. Intensive Care Med. 43, 304–377 (2017). 28101605 · DOI · Paper page
  7. [7]Cohen, J. et al. Sepsis: a roadmap for future research. Lancet Infect. Dis. 15, 581–614 (2015). 25932591 · DOI
  8. [8]Kumar, A. et al. Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock*. Crit. Care Med. 34, 1589–1596 (2006). 16625125 · DOI · Paper page
  9. [9]Ferrer, R. et al. Empiric antibiotic treatment reduces mortality in severe sepsis and septic shock from the first hour: Results from a guideline-based performance improvement program*. Crit. Care Med. 42, 1749–1755 (2014). 24717459 · DOI
  10. [10]Liu, V. X. et al. The timing of early antibiotics and hospital mortality in sepsis. Am. J. Respir. Crit. Care Med. 196, 856–863 (2017). 28345952 PMC5649973 · DOI · Paper page
  11. [11]Peltan, I. D. et al. Ed door-to-antibiotic time and long-term mortality in sepsis. Chest 155, 938–946 (2019). 30779916 PMC6533450 · DOI · Paper page
  12. [13]Townsend, S. R. et al. Effects of compliance with the early management bundle (sep-1) on mortality changes among medicare beneficiaries with sepsis. CHEST 161, 392–406 (2022). 34364867 · DOI · Paper page
  13. [14]Barbash, I. J. et al. Treatment patterns and clinical outcomes after the introduction of the medicare sepsis performance measure (sep-1). Ann. Intern. Med. 174, 927–935 (2021). 33872042 PMC8844885 · DOI · Paper page
  14. [15]Henry, K. E., Hager, D. N., Pronovost, P. J. & Saria, S. A targeted real-time early warning score (trewscore) for septic shock. Sci. Transl. Med.7, 10.1126/scitranslmed.aab3719 (2015). 26246167 · DOI · Paper page
  15. [16]Desautels, T. et al. Prediction of sepsis in the intensive care unit with minimal electronic health record data: a machine learning approach. JMIR Med. Inform. 4, e28 (2016). 27694098 PMC5065680 · DOI · Paper page
  16. [17]Horng, S. et al. Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning. PLOS ONE 12, e0174708 (2017). 28384212 PMC5383046 · DOI · Paper page
  17. [19]Tarabichi, Y. et al. Improving timeliness of antibiotic administration using a provider and pharmacist facing sepsis early warning system in the emergency department setting: a randomized controlled quality improvement initiative*. Crit. Care Med. 50, 418–427 (2021). 34415866 · DOI · Paper page
  18. [21]Shimabukuro, D. W., Barton, C. W., Feldman, M. D., Mataraso, S. J. & Das, R. Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: a randomised clinical trial. BMJ Open Respir. Res. 4, e000234 (2017). 29435343 PMC5687546 · DOI · Paper page
  19. [22]Nelson, J. L., Smith, B. L., Jared, J. D. & Younger, J. G. Prospective trial of real-time electronic surveillance to expedite early care of severe sepsis. Ann. Emerg. Med. 57, 500–504 (2011). 21227543 · DOI · Paper page
  20. [23]Lyons, P. G. et al. Factors associated with variability in the performance of a proprietary sepsis prediction model across 9 networked hospitals in the us. JAMA Intern. Med. 183, 611 (2023). 37010858 PMC10071393 · DOI · Paper page
  21. [24]Giannini, H. M. et al. A machine learning algorithm to predict severe sepsis and septic shock: development, implementation, and impact on clinical practice*. Crit. Care Med. 47, 1485–1492 (2019). 31389839 PMC8635476 · DOI · Paper page
  22. [25]Brajer, N. et al. Prospective and external evaluation of a machine learning model to predict in-hospital mortality of adults at time of admission. JAMA Netw. Open 3, e1920733 (2020). 32031645 PMC12068827 · DOI
  23. [26]Bedoya, A. D. et al. Minimal impact of implemented early warning score and best practice alert for patient deterioration*. Crit. Care Med. 47, 49–55 (2019). 30247239 PMC6298839 · DOI · Paper page
  24. [29]Sendak, M. et al, Development and Validation of ML-DQA – a Machine Learning Data Quality Assurance Framework for Healthcare. In Proc. 7th Machine Learning for Healthcare Conference 741–759 https://proceedings.mlr.press/v182/sendak22a.html (PMLR, 2022).
  25. [30]Topiwala, R., Patel, K., Twigg, J., Rhule, J. & Meisenberg, B. Retrospective observational study of the clinical performance characteristics of a machine learning approach to early sepsis identification. Crit. Care Explor. 1, e0046 (2019). 32166288 PMC7063939 · DOI · Paper page
  26. [32]Moor, M. et al. Predicting sepsis using deep learning across international sites: a retrospective development and validation study. eClinicalMedicine 62, 102124 (2023). 37588623 PMC10425671 · DOI · Paper page
  27. [33]Chen, I. Y. et al. Ethical machine learning in healthcare. Annu. Rev. Biomed. Data Sci. 4, 123–144 (2021). 34396058 PMC8362902 · DOI · Paper page
  28. [35]Russo, E., Singh, H. & Gregory, M. E. Electronic health record alert-related workload as a predictor of burnout in primary care providers. Appl. Clin. Inform. 08, 686–697 (2017). PMC6220682 28678892 · DOI · Paper page
  29. [36]Borowski, M. et al. Medical device alarms. Biomedizinische Tech./Biomed. Eng. 56, 73–83 (2011). 21366502 · DOI
  30. [37]Johnson, A. E. W. et al. A comparative analysis of sepsis identification methods in an electronic database*. Crit. Care Med. 46, 494–499 (2018). 29303796 PMC5851804 · DOI
  31. [38]Yu, S. C. et al. Comparison of sepsis definitions as automated criteria. Crit. Care Med. 49, e433–e443 (2021). 33591014 · DOI · Paper page
  32. [39]Krishnamoorthy, M., Sjoding, M. W. & Wiens, J. Off-label use of artificial intelligence models in healthcare. Nat. Med. 30, 1525–1527 (2024). 38519769 · DOI
  33. [40]Wang, H., Li, Y., Naidech, A. & Luo, Y. Comparison between machine learning methods for mortality prediction for sepsis patients with different social determinants. BMC Med. Inform. Decision Making22, 10.1186/s12911-022-01871-0 (2022). PMC9204861 35710407 · DOI
  34. [41]Futoma, J. et al. An improved multi-output gaussian process rnn with real-time validation for early sepsis detection. In Proc. 2nd Machine Learning for Healthcare Conference (eds Doshi-Velez, F et al.), 243–254 (PMLR, 2017). https://proceedings.mlr.press/v68/futoma17a.html.

Cited by

Sorted by citations

Authors
Valan, B., Prakash, A., Ratliff, W., Gao, M., Muthya, S., Thomas, A., Eaton, J. L., Gardner, M., Nichols, M., Revoir, M., Tart, D., O’Brien, C., Patel, M., Balu, S., & Sendak, M.
Affiliations
  • Duke University (US)
  • Summa Health System (US)
  • Duke University Hospital (US)
Corresponding author
Sendak, Mark
ORCID
Venue
npj Digital Medicine
Publisher
Springer Science and Business Media LLC
Published
2025-06-11
Volume
8
Issue
1
Pages
350-350
ISSN
2398-6352 (ISSN-L)
DOI
10.1038/s41746-025-01664-5
PMC
PMC12159134
OpenAlex
W4411221282
Cited by
21
References
7
Citations by year
2026: 18 · 2025: 3
FWCI
14.55
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
National Institutes of Health (R42 GM144999-03)
Topics
  • Health Sciences › Medicine › Epidemiology › Sepsis Diagnosis and Treatment
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
sepsis, machine learning models, early sepsis detection, external validation, model generalization, community health systems, emergency department, model reproducibility, sepsis prediction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review