A Machine Learning Algorithm to Predict Severe Sepsis and Septic Shock: Development, Implementation, and Impact on Clinical Practice*
Open accessAbstract
Objectives: Develop and implement a machine learning algorithm to predict severe sepsis and septic shock and evaluate the impact on clinical practice and patient outcomes. Design: Retrospective cohort for algorithm derivation and validation, pre-post impact evaluation. Setting: Tertiary teaching hospital system in Philadelphia, PA. Patients: All non-ICU admissions; algorithm derivation July 2011 to June 2014 (n = 162,212); algorithm validation October to December 2015 (n = 10,448); silent versus alert comparison January 2016 to February 2017 (silent n = 22,280; alert n = 32,184). Interventions: A random-forest classifier, derived and validated using electronic health record data, was deployed both silently and later with an alert to notify clinical teams of sepsis prediction. Measurement and Main Result: Patients identified for training the algorithm were required to have International Classification of Diseases, 9th Edition codes for severe sepsis or septic shock and a positive blood culture during their hospital encounter with either a lactate greater than 2.2 mmol/L or a systolic blood pressure less than 90 mm Hg. The algorithm demonstrated a sensitivity of 26% and specificity of 98%, with a positive predictive value of 29% and positive likelihood ratio of 13. The alert resulted in a small statistically significant increase in lactate testing and IV fluid administration. There was no significant difference in mortality, discharge disposition, or transfer to ICU, although there was a reduction in time-to-ICU transfer. Conclusions: Our machine learning algorithm can predict, with low sensitivity but high specificity, the impending occurrence of severe sepsis and septic shock. Algorithm-generated predictive alerts modestly impacted clinical measures. Next steps include describing clinical perception of this tool and optimizing algorithm design and delivery.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
It has been difficult for predictive CDSSs to achieve impactful adoption.
“It has been difficult for predictive CDSSs to achieve impactful adoption [10,13,14].”
From Factors influencing clinician trust in predictive clinical decision support systems for in-hospital deterioration: qualitative descriptive study (2022) · cites this paper as [14]
The Discussion reports that the sepsis prediction alert was implemented across a multi-hospital system but had limited impact on clinical practice and outcomes. That supports difficulty achieving impact for this alert, but the paper does not establish that impactful adoption has been difficult for predictive CDSSs generally; it also describes widespread implementation as feasible. (LLM, conf=0.83)
Experimental — accuracy not yet measuredFew machine learning models have been implemented as part of clinical care.
“Despite the rise in competing models and products for early sepsis detection [5], few machine learning models have been implemented as part of clinical care [6-8].”
From Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study (2020) · cites this paper as [6]
The Introduction (page unavailable) says that sepsis-prediction machine learning algorithms had performed well on retrospective data, but clinical implementation was largely confined to ICU populations; it gives limited examples and says large-scale non-ICU implementation had not been reported. This supports the point for sepsis-prediction models, but not the broader claim about machine learning models generally. (LLM, conf=0.82)
Experimental — accuracy not yet measuredInterventional studies included two pre-post implementation studies (in-hospital) and one ICU randomized controlled trial.
“The interventional studies included two pre-post implementation studies (in-hospital) [30, 31] and one ICU randomized controlled trial [32].”
From Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy (2020) · cites this paper as [30]
Only three interventional studies have been identified in this review, which were carried out in different clinical settings and show mixed results.
“Only three interventional studies have been identified in this review, which were carried out in different clinical settings and show mixed results [31, 32, 51].”
From Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy (2020) · cites this paper as [51]
The identified trajectory patterns represent fundamental aspects of sepsis pathophysiology rather than artifacts of local care patterns.
“This demonstrates the generalizability of our approach and suggests that the identified trajectory patterns represent fundamental aspects of sepsis pathophysiology rather than artifacts of local care patterns24,25.”
From Machine learning predicts sepsis deterioration trajectories. (2026) · cites this paper as [24]
The Discussion (Algorithm Design and Future Algorithm Optimization; page unavailable) describes a sepsis prediction algorithm and says its criteria were not externally validated. Although it mentions a patient’s sepsis trajectory and changing practice patterns, it does not identify trajectory patterns or assess whether they reflect sepsis pathophysiology rather than local care. Searches for trajectory patterns, generalizability, and local care patterns found no passage addressing that distinction. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The third international consensus definitions for sepsis and septic shock (Sepsis-3).
Singer, M., Deutschman, C. S., Seymour, C. W., Shankar-Hari, M., Annane, D., Bauer, M., Bellomo, R., Bernard, G. R., Chiche, J.-D., Coopersmith, C. M., et al. (2016) · JAMA · doi:10.1001/jama.2016.0287 · Cited by 28,250 · see citing works
Deo, R. C. (2015) · Circulation · doi:10.1161/circulationaha.115.001593 · Cited by 3,466 · see citing works
Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009-2014.
Rhee, C., Dantes, R., Epstein, L., Murphy, D. J., Seymour, C. W., Iwashyna, T. J., Kadri, S. S., Angus, D. C., Danner, R. L., Fiore, A. E., et al. (2017) · JAMA · doi:10.1001/jama.2017.13836 · Cited by 1,884 · see citing works
Unintended consequences of machine learning in medicine
Cabitza, F., Rasoini, R., & Gensini, G. F. (2017) · JAMA · doi:10.1001/jama.2017.7797 · Cited by 1,066 · see citing works
The timing of early antibiotics and hospital mortality in sepsis.
Liu, V. X., Fielding-Singh, V., Greene, J. D., Baker, J. M., Iwashyna, T. J., Bhattacharya, J., & Escobar, G. J. (2017) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201609-1848oc · Cited by 1,006 · see citing works
with the HITEC Investigators, Ancker, J. S., Edwards, A., Nosal, S., Hauser, D., Mauer, E., & Kaushal, R. (2017) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-017-0430-8 · Cited by 709 · see citing works
Churpek, M. M., Yuen, T. C., Winslow, C., Meltzer, D. O., Kattan, M. W., & Edelson, D. P. (2016) · Critical Care Medicine · doi:10.1097/ccm.0000000000001571 · Cited by 638 · see citing works
A targeted real-time early warning score (TREWScore) for septic shock.
Henry, K. E., Hager, D. N., Pronovost, P. J., & Saria, S. (2015) · Science Translational Medicine · doi:10.1126/scitranslmed.aab3719 · Cited by 598 · see citing works
Taylor, R. A., Pare, J. R., Venkatesh, A. K., Mowafi, H. O., Melnick, E. R., Fleischman, W. M., & Hall, M. K. (2015) · Academic Emergency Medicine · doi:10.1111/acem.12876 · Cited by 496 · see citing works
Shimabukuro, D. W., Barton, C. W., Feldman, M. D., Mataraso, S. J., & Das, R. (2017) · BMJ Open Respiratory Research · doi:10.1136/bmjresp-2017-000234 · Cited by 382 · see citing works
Mao, Q., Jay, M., Hoffman, J. L., Calvert, J., Barton, C., Shimabukuro, D., Shieh, L., Chettipally, U., Fletcher, G., Kerem, Y., et al. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-017833 · Cited by 336 · see citing works
Horng, S., Sontag, D. A., Halpern, Y., Jernite, Y., Shapiro, N. I., & Nathanson, L. A. (2017) · PLOS ONE · doi:10.1371/journal.pone.0174708 · Cited by 310 · see citing works
Multicenter Development and Validation of a Risk Stratification Tool for Ward Patients
Churpek, M. M., Yuen, T. C., Winslow, C. J., Robicsek, A. A., Meltzer, D. O., Gibbons, R. D., & Edelson, D. P. (2014) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201406-1022oc · Cited by 258 · see citing works
Bellomo, R., Ackerman, M., Bailey, M., et al. (2012) · Critical Care Medicine · doi:10.1097/ccm.0b013e318255d9a0 · Cited by 227 · see citing works
Escobar, G. J., LaGuardia, J. C., Turk, B. J., Ragins, A. I., Kipnis, P., & Draper, D. (2012) · Journal of Hospital Medicine · doi:10.1002/jhm.1929 · Cited by 185 · see citing works
Development, implementation, and impact of an automated early warning and response system for sepsis
Umscheid, C. A., Betesh, J., VanZandbergen, C., Hanish, A., Tait, G., Mikkelsen, M. E., French, B. C., & Fuchs, B. D. (2014) · Journal of Hospital Medicine · doi:10.1002/jhm.2259 · Cited by 167 · see citing works
Self-citationGinestra, J. C., Giannini, H. M., Schweickert, W. D., Meadows, L., Lynch, M. J., Pavan, K., Chivers, C. J., Draugelis, M., Donnelly, P. J., Fuchs, B. D., et al. (2019) · Critical Care Medicine · doi:10.1097/ccm.0000000000003803 · Cited by 159 · see citing works
Self-citationMcCoy, A., & Das, R. (2017) · BMJ Open Quality · doi:10.1136/bmjoq-2017-000158 · Cited by 148 · see citing works
Murphy, D. R., Reis, B., Sittig, D. F., & Singh, H. (2012) · The American Journal of Medicine · doi:10.1016/j.amjmed.2011.07.029 · Cited by 142 · see citing works
Marafino, B. J., Park, M., Davies, J. M., Thombley, R. L., Luft, H. S., Sing, D. C., Kazi, D. S., DeJong, C., Boscardin, W. J., Dean, M. L., et al. (2018) · JAMA Network Open · doi:10.1001/jamanetworkopen.2018.5097 · Cited by 120 · see citing works
Inclusion of Unstructured Clinical Text Improves Early Prediction of Death or Prolonged ICU Stay*
Weissman, G. E., Hubbard, R. A., Ungar, L., Harhay, M. O., Greene, C. S., Himes, B. E., & Halpern, S. D. (2018) · Critical Care Medicine · doi:10.1097/ccm.0000000000003148 · Cited by 89 · see citing works
Wellner, B., Grand, J., Canzone, E., Coarr, M., Brady, P. W., Simmons, J. M., Kirkendall, E. S., Dean, N., Kleinman, M. E., & Sylvester, P. (2017) · JMIR Medical Informatics · doi:10.2196/medinform.8680 · Cited by 78 · see citing works
Guidi, J. L., Clark, K., Upton, M. T., Faust, H., Umscheid, C. A., Lane-Fall, M. B., Mikkelsen, M. E., Schweickert, W. D., Vanzandbergen, C. A., Betesh, J., et al. (2015) · Annals of the American Thoracic Society · doi:10.1513/annalsats.201503-129oc · Cited by 43 · see citing works
Self-citationBirnbaum, A. J., Bijur, P. E., Kuperman, G. J., Gennis, P., & Berger, T. (2010) · Applied Clinical Informatics · doi:10.4338/aci-2010-09-ra-0054 · Cited by 37 · see citing works
Toward a Two-Tier Clinical Warning System for Hospitalized Patients
Hackmann, G. (2010) · PubMed · doi:10.7936/k70v8b1g · Cited by 34 · see citing works
Real-Time Automated Sampling of Electronic Medical Records Predicts Hospital Mortality
Khurana, H. S., Khurana, H. S., Groves, R. H., et al. (2016) · The American Journal of Medicine · doi:10.1016/j.amjmed.2016.02.037 · Cited by 30 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [2]Torio CM, Andrews RM: National Inpatient Hospital Costs: The Most Expensive Conditions by Payer, 2011: Statistical Brief #160 [Internet]. In: Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Rockville (MD): Agency for Healthcare Research and Quality (US); 2006. [ cited 2017 Oct 3] Available from: http://www.ncbi.nlm.nih.gov/books/NBK169005/ 24199255
- [28]Ryan DP, Daley BJ, Wong K,: Prediction of ICU in-hospital mortality using a deep Boltzmann machine and dropout neural net [Internet]. IEEE; 2013. p. 1– 4.[ cited 2018 Apr 4] Available from: http://ieeexplore.ieee.org/document/6618491/
Cited by
Sorted by citations
Wong, A., Otles, E., Donnelly, J. P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., et al. (2021) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2021.2626 · Cited by 995 · see citing works
Fleuren, L. M., Klausch, T. L. T., Zwager, C. L., Schoonmade, L. J., Guo, T., Roggeveen, L. F., Swart, E. L., Girbes, A. R. J., Thoral, P., Ercole, A., et al. (2020) · Intensive Care Medicine · doi:10.1007/s00134-019-05872-y · Cited by 755 · see citing works
“Only three interventional studies have been identified in this review, which were carried out in different clinical settings and show mixed results [31, 32, 51].”
BackgroundRole of Artificial Intelligence Applications in Real-Life Clinical Practice: Systematic Review
Yin, J., Ngiam, K. Y., & Teo, H. H. (2021) · Journal of Medical Internet Research · doi:10.2196/25759 · Cited by 419 · see citing works
Adams, R., Henry, K. E., Sridharan, A., Soleimani, H., Zhan, A., Rawat, N., Johnson, L., Hager, D. N., Cosgrove, S. E., Markowski, A., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01894-0 · Cited by 365 · see citing works
Sendak, M. P., Ratliff, W., Sarro, D., Alderton, E., Futoma, J., Gao, M., Nichols, M., Revoir, M., Yashar, F., Miller, C., et al. (2020) · JMIR Medical Informatics · doi:10.2196/15182 · Cited by 287 · see citing works
A Path for Translation of Machine Learning Products into Healthcare Delivery
(2020) · EMJ Innovations · doi:10.33590/emjinnov/19-00172 · Cited by 172 · see citing works
Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study
Sandhu, S., Lin, A. L., Brajer, N., Sperling, J., Ratliff, W., Bedoya, A. D., Balu, S., O'Brien, C., & Sendak, M. P. (2020) · Journal of Medical Internet Research · doi:10.2196/22421 · Cited by 168 · see citing works
Impact of a deep learning sepsis prediction model on quality of care and survival
Boussina, A., Shashikumar, S. P., Malhotra, A., Owens, R. L., El-Kareh, R., Longhurst, C. A., Quintero, K., Donahue, A., Chan, T. C., Nemati, S., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-023-00986-6 · Cited by 170 · see citing works
Ginestra, J. C., Giannini, H. M., Schweickert, W. D., Meadows, L., Lynch, M. J., Pavan, K., Chivers, C. J., Draugelis, M., Donnelly, P. J., Fuchs, B. D., et al. (2019) · Critical Care Medicine · doi:10.1097/ccm.0000000000003803 · Cited by 159 · see citing works
Self-citationHenry, K. E., Adams, R., Parent, C., Soleimani, H., Sridharan, A., Johnson, L., Hager, D. N., Cosgrove, S. E., Markowski, A., Klein, E. Y., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01895-z · Cited by 127 · see citing works
Schwartz, J. M., George, M., Rossetti, S. C., Dykes, P. C., Minshall, S. R., Lucas, E., & Cato, K. D. (2022) · JMIR Human Factors · doi:10.2196/33960 · Cited by 88 · see citing works
“It has been difficult for predictive CDSSs to achieve impactful adoption [10,13,14].”
Backgroundvan der Vegt, A. H., Scott, I. A., Dermawan, K., Schnetler, R. J., Kalke, V. R., & Lane, P. J. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad075 · Cited by 72 · see citing works
“Related to the timing of alerts (D7), decisions about what actions clinicians should take for alerts involving patients showing no symptoms or signs of sepsis proved problematic (D15), as unclear roles and responsibilities constituted potential barriers to adoption (B3, B5).(33,44)”
BackgroundMethodJauk, S., Kramer, D., Avian, A., Berghold, A., Leodolter, W., & Schulz, S. (2021) · Journal of Medical Systems · doi:10.1007/s10916-021-01727-6 · Cited by 59 · see citing works
Joshi, M., Mecklai, K., Rozenblum, R., & Samal, L. (2022) · JAMIA Open · doi:10.1093/jamiaopen/ooac022 · Cited by 46 · see citing works
Hassan, N., Slight, R., Weiand, D., Vellinga, A., Morgan, G., Aboushareb, F., & Slight, S. P. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104457 · Cited by 42 · see citing works
Wardi, G., Owens, R., Josef, C., Malhotra, A., Longhurst, C., & Nemati, S. (2023) · Critical Care Medicine · doi:10.1097/ccm.0000000000005894 · Cited by 35 · see citing works
Enhancing sepsis management through machine learning techniques: A review
Ocampo-Quintero, N., Vidal-Cortés, P., del Río Carbajo, L., Fdez-Riverola, F., Reboiro-Jato, M., & Glez-Peña, D. (2022) · Medicina Intensiva · doi:10.1016/j.medin.2020.04.003 · Cited by 20 · see citing works
Wan, Y.-K. J., Wright, M. C., McFarland, M. M., Dishman, D., Nies, M. A., Rush, A., Madaras-Kelly, K., Jeppesen, A., & Del Fiol, G. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad203 · Cited by 11 · see citing works
Fixler, A., Oliaro, B., Frieden, M., Girardo, C., Winterbottom, F. A., Fort, L. B., & Hill, J. (2023) · Ochsner Journal · doi:10.31486/toj.22.0098 · Cited by 11 · see citing works
Machine learning predicts sepsis deterioration trajectories.
Zhang, R., Long, F., Zhao, Z., Wu, J., Tan, R., Xu, W., Li, L., Long, Y., & Qu, H. (2026) · NPJ Digital Medicine · doi:10.1038/s41746-026-02565-x · Cited by 6 · see citing works
“This demonstrates the generalizability of our approach and suggests that the identi fi ed trajectory patterns represent fundamental aspects of sepsis pathophysiology rather than artifacts of local care patterns 24,25 .”
Zarantonello, F., De Cassai, A., Pettenuzzo, T., Sella, N., Mormando, G., Bolzon, A., Fulvio, G. A., Bertoncello, C. A., & Boscolo, A. (2026) · Korean journal of anesthesiology · doi:10.4097/kja.251142 · Cited by 3 · see citing works
Catling, F. J., Nagendran, M., Festor, P., Bien, Z., Harris, S., Faisal, A. A., Gordon, A. C., & Komorowski, M. (2023) · doi:10.1101/2023.08.26.23294666 · Cited by 3 · see citing works
- Authors
- Giannini, H. M., Ginestra, J. C., Chivers, C., Draugelis, M., Hanish, A., Schweickert, W. D., Fuchs, B. D., Meadows, L., Lynch, M., Donnelly, P. J., Pavan, K., Fishman, N. O., Hanson, C. W., & Umscheid, C. A.
- Affiliations
- Hospital of the University of Pennsylvania (US)
- University of Pennsylvania Health System (US)
- Penn Presbyterian Medical Center (US)
- ORCID
- Venue
- Critical Care Medicine
- Published
- 2019-11
- Volume
- 47
- Issue
- 11
- Pages
- 1485-1492
- ISSN
- 0090-3493 (ISSN-L)
- PMC
- PMC8635476
- OpenAlex
- W2968919852
- Cited by
- 269
- References
- 26
- Citations by year
- 2026: 32 · 2025: 40 · 2024: 40 · 2023: 46 · 2022: 40 · 2021: 32 · 2020: 34 · 2019: 4
- FWCI
- 20.93
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Critical Care Medicine (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institutes of Health (UL1TR000003)
- National Center for Research Resources (UL1RR024134)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Epidemiology › Sepsis Diagnosis and Treatment
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- severe sepsis, septic shock, sepsis prediction, machine learning, random forest, clinical decision support alerts, electronic health records, intravenous fluid therapy, ICU transfer, patient outcomes
- MeSH terms
- Machine Learning, Algorithms, Diagnosis, Computer-Assisted, Hospitals, Teaching, Humans, Retrospective Studies, Sensitivity and Specificity, Shock, Septic, Cohort Studies, Sepsis, Decision Support Systems, Clinical, Electronic Health Records, Text Messaging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
