CoChat
Sign Up
Cited by 80Sepsis Diagnosis and TreatmentMachine Learning in HealthcareAcute Kidney Injury Research

Effect of a sepsis prediction algorithm on patient mortality, length of stay and readmission: a prospective multicentre clinical outcomes evaluation of real-world patient data from US hospitals

Open access
Burdick, Hoyt, Pino, Eduardo, Gabel-Comeau, Denise, McCoy, Andrea, Gu, Carol, Roberts, Jonathan, Le, Sidney, Slote, Joseph +4 more·2020·BMJ Health & Care Informatics
DOI 10.1136/bmjhci-2019-100109PMC PMC7245419
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Severe sepsis and septic shock are among the leading causes of death in the USA. While early prediction of severe sepsis can reduce adverse patient outcomes, sepsis remains one of the most expensive conditions to diagnose and treat. Objective: The purpose of this study was to evaluate the effect of a machine learning algorithm for severe sepsis prediction on in-hospital mortality, hospital length of stay and 30-day readmission. Design: Prospective clinical outcomes evaluation. Setting: Evaluation was performed on a multiyear, multicentre clinical data set of real-world data containing 75 147 patient encounters from nine hospitals across the continental USA, ranging from community hospitals to large academic medical centres. Participants: Analyses were performed for 17 758 adult patients who met two or more systemic inflammatory response syndrome criteria at any point during their stay (‘sepsis-related’ patients). Interventions: Machine learning algorithm for severe sepsis prediction. Outcome measures: In-hospital mortality, length of stay and 30-day readmission rates. Results: Hospitals saw an average 39.5% reduction of in-hospital mortality, a 32.3% reduction in hospital length of stay and a 22.7% reduction in 30-day readmission rate for sepsis-related patient stays when using the machine learning algorithm in clinical outcomes analysis. Conclusions: Reductions of in-hospital mortality, hospital length of stay and 30-day readmissions were observed in real-world clinical use of the machine learning-based algorithm. The predictive algorithm may be successfully used to improve sepsis-related outcomes in live clinical settings. Trial registration number: NCT03960203

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1136/bmjhci-2019-100109

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 62

  1. [1]Angus DC, Linde-Zwirble WT, Lidicker J, et al Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care. Crit Care Med 2001; 29: 1303– 10. 10.1097/00003246-200107000-00002 11445675 · DOI · Paper page
  2. [2]Stevenson EK, Rubenstein AR, Radin GT, et al Two decades of mortality trends among patients with severe sepsis: a comparative meta-analysis*. Crit Care Med 2014; 42: 625– 31. 10.1097/CCM.0000000000000026 24201173 PMC4313930 · DOI
  3. [3]Torio CM, Celeste M, Andrews RM "National inpatient hospital costs: the most expensive conditions by payer 2011", 2013. 24199255
  4. [4]Torio C, Moore B National Inpatient Hospital Costs: The Most Expensive Conditions by Payer, 2013. HCUP Statistical Brief #204. Agency for Healthcare Research and Quality, Rockville, MD, 2016 Available: http://www.hcup-us.ahrq.gov/reports/statbriefs/sb204-MostExpensive-Hospital-Conditions.pdf 27359025
  5. [5]Dellinger RP, Levy MM, Rhodes A, et al Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock, 2012. Intensive Care Med 2013; 39: 165– 228. 10.1007/s00134-012-2769-8 23361625 PMC7095153 · DOI
  6. [6]Lagu T, Rothberg MB, Shieh M-S, et al Hospitalizations, costs, and outcomes of severe sepsis in the United States 2003 to 2007. Crit Care Med 2012; 40: 754– 61. 10.1097/CCM.0b013e318232db65 21963582 · DOI · Paper page
  7. [7]Gaieski DF, Edwards JM, Kallan MJ, et al Benchmarking the incidence and mortality of severe sepsis in the United States. Crit Care Med 2013; 41: 1167– 74. 10.1097/CCM.0b013e31827c09f8 23442987 · DOI
  8. [8]Levy MM, Fink MP, Marshall JC, et al 2001 sccm/esicm/accp/ats/sis international sepsis definitions conference. Crit Care Med 2003; 31: 1250– 6. 10.1097/01.CCM.0000050454.01978.3B 12682500 · DOI
  9. [9]Singer M, Deutschman CS, Seymour CW, et al The third International consensus definitions for sepsis and septic shock (sepsis-3). JAMA 2016; 315: 801– 10. 10.1001/jama.2016.0287 26903338 PMC4968574 · DOI · Paper page
  10. [10]Shankar-Hari M, Phillips GS, Levy ML, et al Developing a new definition and assessing new clinical criteria for septic shock: for the third International consensus definitions for sepsis and septic shock (Sepsis-3). JAMA 2016; 315: 775– 87. 10.1001/jama.2016.0289 26903336 PMC4910392 · DOI
  11. [12]Moore LJ, Moore FA Early diagnosis and evidence-based care of surgical sepsis. J Intensive Care Med 2013; 28: 107– 17. 10.1177/0885066611408690 21747125 · DOI
  12. [13]Kenzaka T, Okayama M, Kuroki S, et al Importance of vital signs to the early diagnosis and severity of sepsis: association between vital signs and sequential organ failure assessment score in patients with sepsis. Intern Med 2012; 51: 871– 6. 10.2169/internalmedicine.51.6951 22504241 · DOI
  13. [14]Vincent JL, Moreno R, Takala J, et al The SOFA (sepsis-related organ failure assessment) score to describe organ dysfunction/failure. on behalf of the Working group on sepsis-related problems of the European Society of intensive care medicine. Intensive Care Med 1996; 22: 707– 10. 10.1007/bf01709751 8844239 · DOI · Paper page
  14. [15]Jaimes F, Garcés J, Cuervo J, et al The systemic inflammatory response syndrome (SIRS) to identify infected patients in the emergency room. Intensive Care Med 2003; 29: 1368– 71. 10.1007/s00134-003-1874-0 12830377 · DOI
  15. [16]Subbe CP, Slater A, Menon D, et al Validation of physiological scoring systems in the accident and emergency department. Emerg Med J 2006; 23: 841– 5. 10.1136/emj.2006.035816 17057134 PMC2464409 · DOI
  16. [17]McLymont N, Glover GW Scoring systems for the characterization of sepsis and associated outcomes. Ann Transl Med 2016; 4: 527. 10.21037/atm.2016.12.53 28149888 PMC5233540 · DOI
  17. [18]Churpek MM, Snyder A, Han X, et al Quick sepsis-related organ failure assessment, systemic inflammatory response syndrome, and early warning scores for detecting clinical deterioration in infected patients outside the intensive care unit. Am J Respir Crit Care Med 2017; 195: 906– 11. 10.1164/rccm.201604-0854OC 27649072 PMC5387705 · DOI · Paper page
  18. [19]Usman OA, Usman AA, Ward MA Comparison of SIRS, qSOFA, and news for the early identification of sepsis in the emergency department. Am J Emerg Med 2019; 37: 1490– 7. 10.1016/j.ajem.2018.10.058 30470600 · DOI
  19. [20]Johnson AEW, Aboab J, Raffa JD, et al A comparative analysis of sepsis identification methods in an electronic Database*. Crit Care Med 2018; 46: 494– 9. 10.1097/CCM.0000000000002965 29303796 PMC5851804 · DOI
  20. [22]van der Woude SW, van Doormaal FF, Hutten BA, et al Classifying sepsis patients in the emergency department using SIRS, qSOFA or MEWS. Neth J Med 2018; 76: 158– 66. 29845938
  21. [23]Churpek MM, Zadravecz FJ, Winslow C, et al Incidence and prognostic value of the systemic inflammatory response syndrome and organ dysfunctions in ward patients. Am J Respir Crit Care Med 2015; 192: 958– 64. 10.1164/rccm.201502-0275OC 26158402 PMC4642209 · DOI
  22. [24]Kaukonen K-M, Bailey M, Pilcher D, et al Systemic inflammatory response syndrome criteria in defining severe sepsis. N Engl J Med 2015; 372: 1629– 38. 10.1056/NEJMoa1415236 25776936 · DOI
  23. [25]Finkelsztein EJ, Jones DS, Ma KC, et al Comparison of qSOFA and SIRS for predicting adverse outcomes of patients with suspicion of sepsis outside the intensive care unit. Crit Care 2017; 21: 73. 10.1186/s13054-017-1658-5 28342442 PMC5366240 · DOI
  24. [26]Roney JK, Whitley BE, Maples JC, et al Modified early warning scoring (MEWS): evaluating the evidence for tool inclusion of sepsis screening criteria and impact on mortality and failure to rescue. J Clin Nurs 2015; 24: 3343– 54. 10.1111/jocn.12952 26780181 · DOI
  25. [27]Lie KC, Lau C-Y, Van Vinh Chau N, et al Utility of SOFA score, management and outcomes of sepsis in Southeast Asia: a multinational multicenter prospective observational study. J Intensive Care 2018; 6: 9. 10.1186/s40560-018-0279-7 29468069 PMC5813360 · DOI
  26. [28]Innocenti F, Tozzi C, Donnini C, et al SOFA score in septic patients: incremental prognostic value over age, comorbidities, and parameters of sepsis severity. Intern Emerg Med 2018; 13: 405– 12. 10.1007/s11739-017-1629-5 28188577 · DOI
  27. [29]Office of the National Coordinator for Health Information Technology 'Office-based Physician Electronic Health Record Adoption,' Health IT Quick-Stat #50, 2016 Available: http://www.webcitation.org/6rmdNMHPW 42525813
  28. [34]Phua J, Ngerng W, See K, et al Characteristics and outcomes of culture-negative versus culture-positive severe sepsis. Crit Care 2013; 17: R202. 10.1186/cc12896 24028771 PMC4057416 · DOI · Paper page
  29. [35]Lamantia MA, Stewart PW, Platts-Mills TF, et al Predictive value of initial triage vital signs for critically ill older adults. West J Emerg Med 2013; 14: 453– 60. 10.5811/westjem.2013.5.13411 24106542 PMC3789908 · DOI
  30. [36]Girard TD, Opal SM, Ely EW Insights into severe sepsis in older patients: from epidemiology to evidence-based management. Clin Infect Dis 2005; 40: 719– 27. 10.1086/427876 15714419 · DOI
  31. [39]Mao Q, Jay M, Hoffman JL, et al Multicentre validation of a sepsis prediction algorithm using only vital sign data in the emergency department, general ward and ICU. BMJ Open 2018; 8: e017833. 10.1136/bmjopen-2017-017833 PMC5829820 29374661 · DOI · Paper page
  32. [40]Shimabukuro DW, Barton CW, Feldman MD, et al Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: a randomised clinical trial. BMJ Open Respir Res 2017; 4: e000234. 10.1136/bmjresp-2017-000234 PMC5687546 29435343 · DOI · Paper page
  33. [41]McCoy A, Das R Reducing patient mortality, length of stay and readmissions through machine learning-based sepsis prediction in the emergency department, intensive care unit and hospital floor units. BMJ Open Qual 2017; 6: e000158. 10.1136/bmjoq-2017-000158 PMC5699136 29450295 · DOI · Paper page
  34. [42]Burdick H, Pino E, Gabel-Comeau D, et al Evaluating a sepsis prediction machine learning algorithm using minimal electronic health record data in the emergency department and intensive care unit. bioRxiv: 224014.
  35. [44]Desautels T, Calvert J, Hoffman J, et al Prediction of sepsis in the intensive care unit with minimal electronic health record data: a machine learning approach. JMIR Med Inform 2016; 4: e28. 10.2196/medinform.5909 27694098 PMC5065680 · DOI · Paper page
  36. [45]Calvert J, Desautels T, Chettipally U, et al High-Performance detection and early prediction of septic shock for alcohol-use disorder patients. Ann Med Surg 2016; 8: 50– 5. 10.1016/j.amsu.2016.04.023 PMC4960347 27489621 · DOI · Paper page
  37. [46]Chen T, Guestrin C XGBoost: a scalable tree boosting system. Paper presented at the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016.
  38. [47]Shao J, Zhong B Last observation carry-forward and last observation analysis. Stat Med 2003; 22: 2429– 41. 10.1002/sim.1519 12872300 · DOI
  39. [48]Ali MW, Talukder E Analysis of longitudinal binary data with missing data due to dropouts. J Biopharm Stat 2005; 15: 993– 1007. 10.1080/10543400500266692 16279357 · DOI
  40. [49]Mohamadlou H, Lynn-Palevsky A, Barton C, et al Prediction of acute kidney injury with a machine learning algorithm using electronic health record data. Can J Kidney Health Dis 2018; 5: 205435811877632. 10.1177/2054358118776326 PMC6080076 30094049 · DOI
  41. [50]Fang X, Wang Z, Yang J, et al Clinical evaluation of Sepsis-1 and Sepsis-3 in the ICU. Chest 2018; 153: 1169– 76. 10.1016/j.chest.2017.06.037 28711593 · DOI
  42. [51]Burdick H, Pino E, Gabel-Comeau D, et al Effect of a sepsis prediction algorithm on patient mortality, length of stay and readmission. bioRxiv 2018; 1: 457465. PMC7245419 32354696
  43. [52]Mayr FB, Yende S, Angus DC Epidemiology of severe sepsis. Virulence 2014; 5: 4– 11. 10.4161/viru.27372 24335434 PMC3916382 · DOI
  44. [53]Umscheid CA, Betesh J, VanZandbergen C, et al Development, implementation, and impact of an automated early warning and response system for sepsis. J Hosp Med 2015; 10: 26– 31. 10.1002/jhm.2259 25263548 PMC4410778 · DOI
  45. [55]Rivers E, Nguyen B, Havstad S, et al Early goal-directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med 2001; 345: 1368– 77. 10.1056/NEJMoa010307 11794169 · DOI · Paper page
  46. [57]Giannini HM, Ginestra JC, Chivers C, et al A machine learning algorithm to predict severe sepsis and septic shock: development, implementation, and impact on clinical practice. Crit Care Med 2019; 47: 1485– 92. 10.1097/CCM.0000000000003891 31389839 PMC8635476 · DOI · Paper page
  47. [58]Taylor RA, Pare JR, Venkatesh AK, et al Prediction of in-hospital mortality in emergency department patients with sepsis: a local big data-driven, machine learning approach. Acad Emerg Med 2016; 23: 269– 78. 10.1111/acem.12876 26679719 PMC5884101 · DOI
  48. [59]Wulff A, Montag S, Marschollek M, et al Clinical decision-support systems for detection of systemic inflammatory response syndrome, sepsis, and septic shock in critically ill patients: a systematic review. Methods Inf Med 2019: 1– 15. 31499571
  49. [60]Nachimuthu SK, Haug PJ Early detection of sepsis in the emergency department using dynamic Bayesian networks. AMIA Annu Symp Proc 2012; 2012: 653– 62. 23304338 PMC3540576
  50. [61]Henry KE, Hager DN, Pronovost PJ, et al A targeted real-time early warning score (TREWScore) for septic shock. Sci Transl Med 2015; 7: 299ra122– 299. 10.1126/scitranslmed.aab3719 26246167 · DOI · Paper page
  51. [62]Nemati S, Holder A, Razmi F, et al An interpretable machine learning model for accurate prediction of sepsis in the ICU. Crit Care Med 2018; 46: 547– 53. 10.1097/CCM.0000000000002936 29286945 PMC5851825 · DOI · Paper page
  52. [63]Dyagilev K, Saria S Learning (predictive) risk scores in the presence of censoring due to interventions. Mach Learn 2016; 102: 323– 48. 10.1007/s10994-015-5527-7 · DOI
  53. [64]Stanculescu I, Williams CK, Freer Y, editors A hierarchical switching linear dynamical system applied to the detection of sepsis in neonatal condition monitoring. UAI, 2014.
  54. [65]Stanculescu I, Williams CKI, Freer Y Autoregressive hidden Markov models for the early detection of neonatal sepsis. IEEE J Biomed Health Inform 2014; 18: 1560– 70. 10.1109/JBHI.2013.2294692 25192568 · DOI
  55. [66]Harrison AM, Thongprayoon C, Kashyap R, et al Developing the surveillance algorithm for detection of failure to recognize and treat severe sepsis. Mayo Clin Proc 2015; 90: 166– 75. 10.1016/j.mayocp.2014.11.014 25576199 PMC6571011 · DOI
  56. [67]Nelson JL, Smith BL, Jared JD, et al Prospective trial of real-time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med 2011; 57: 500– 4. 10.1016/j.annemergmed.2010.12.008 21227543 · DOI · Paper page
  57. [68]Umscheid CA, Betesh J, VanZandbergen C, et al Development, implementation, and impact of an automated early warning and response system for sepsis. J Hosp Med 2015; 10: 26– 31. 10.1002/jhm.2259 25263548 PMC4410778 · DOI
  58. [69]Austrian JS, Jamin CT, Doty GR, et al Impact of an emergency department electronic sepsis surveillance system on patient mortality and length of stay. J Am Med Inform Assoc 2018; 25: 523– 9. 10.1093/jamia/ocx072 29025165 PMC7646853 · DOI
  59. [70]Tiru B, DiNino EK, Orenstein A, et al The economic and humanistic burden of severe sepsis. Pharmacoeconomics 2015; 33: 925– 37. 10.1007/s40273-015-0282-y 25935211 · DOI
  60. [71]Ferrer R, Martin-Loeches I, Phillips G, et al Empiric antibiotic treatment reduces mortality in severe sepsis and septic shock from the first hour: results from a guideline-based performance improvement program. Crit Care Med 2014; 42: 1749– 55. 10.1097/CCM.0000000000000330 24717459 · DOI
  61. [72]Calvert J, Hoffman J, Barton C, et al Cost and mortality impact of an algorithm-driven sepsis prediction system. J Med Econ 2017; 20: 646– 51. 10.1080/13696998.2017.1307203 28294646 · DOI
  62. [73]Cabitza F, Rasoini R, Gensini GF Unintended consequences of machine learning in medicine. JAMA 2017; 318: 517– 8. 10.1001/jama.2017.7797 28727867 · DOI · Paper page

Cited by

Sorted by citations

Authors
Burdick, H., Pino, E., Gabel-Comeau, D., McCoy, A., Gu, C., Roberts, J., Le, S., Slote, J., Pellegrini, E., Green-Saxena, A., Hoffman, J., & Das, R.
Affiliations
  • Cabell Huntington Hospital (US)
  • Cape May Regional Medical Center, Cape May Court House, New Jersey, USA
  • Dascena (United States) (US)
  • Dascena Inc, Oakland, California, USA
Corresponding author
Hoffman, Jana
ORCID
Venue
BMJ Health & Care Informatics
Publisher
BMJ
Published
2020-04
Volume
27
Issue
1
Pages
e100109
ISSN
2632-1009 (ISSN-L)
DOI
10.1136/bmjhci-2019-100109
PMC
PMC7245419
OpenAlex
W3023171249
Cited by
80
References
7
Citations by year
2026: 14 · 2025: 14 · 2024: 11 · 2023: 15 · 2022: 17 · 2021: 7 · 2020: 2
FWCI
7.35
Type
journal article
Language
en
Open access
Licence
Locations
  • BMJ Health & Care Informatics (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
National Center for Advancing Translational Sciences (1R43TR002221, 1R43TR002309)
Topics
  • Health Sciences › Medicine › Epidemiology › Sepsis Diagnosis and Treatment
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Nephrology › Acute Kidney Injury Research
Keywords
severe sepsis, septic shock, sepsis prediction, machine learning algorithms, in-hospital mortality, length of hospital stay, 30-day readmission, systemic inflammatory response syndrome, clinical outcomes, real-world data
MeSH terms
Adult, Aged, Algorithms, Female, Forecasting, Humans, Length of Stay, Male, Middle Aged, Patient Readmission, Prospective Studies, United States, Databases, Factual, Hospital Mortality, Sepsis, Young Adult

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review