Predicting Progression to Septic Shock in the Emergency Department Using an Externally Generalizable Machine-Learning Algorithm
Open accessThe DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Abstract
Study objective: Machine-learning algorithms allow improved prediction of sepsis syndromes in the emergency department (ED), using data from electronic medical records. Transfer learning, a new subfield of machine learning, allows generalizability of an algorithm across clinical sites. We aim to validate the Artificial Intelligence Sepsis Expert for the prediction of delayed septic shock in a cohort of patients treated in the ED and demonstrate the feasibility of transfer learning to improve external validity at a second site. Methods: This was an observational cohort study using data from greater than 180,000 patients from 2 academic medical centers between 2014 and 2019, using multiple definitions of sepsis. The Artificial Intelligence Sepsis Expert algorithm was trained with 40 input variables at the development site to predict delayed septic shock (occurring greater than 4 hours after ED triage) at various prediction windows. We then validated the algorithm at a second site, using transfer learning to demonstrate generalizability of the algorithm. Results: We identified 9,354 patients with severe sepsis, of whom 723 developed septic shock at least 4 hours after triage. The Artificial Intelligence Sepsis Expert algorithm demonstrated excellent area under the receiver operating characteristic curve (>0.8) at 8 and 12 hours for the prediction of delayed septic shock. Transfer learning significantly improved the test characteristics of the Artificial Intelligence Sepsis Expert algorithm and yielded comparable performance at the validation site. Conclusion: The Artificial Intelligence Sepsis Expert algorithm accurately predicted the development of delayed septic shock. The use of transfer learning allowed significantly improved external validity and generalizability at a second site. Future prospective studies are indicated to evaluate the clinical utility of this model.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697
Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning
Kermany, D. S., Goldbaum, M., Cai, W., Valentim, C. C. S., Liang, H., Baxter, S. L., McKeown, A., Yang, G., Wu, X., Yan, F., et al. (2018) · Cell · doi:10.1016/j.cell.2018.02.010 · Cited by 4,842
Surviving Sepsis Campaign: International guidelines for management of sepsis and septic shock: 2016.
Rhodes, A., Evans, L. E., Alhazzani, W., Levy, M. M., Antonelli, M., Ferrer, R., Kumar, A., Sevransky, J. E., Sprung, C. L., Nunnally, M. E., et al. (2017) · Critical Care Medicine · doi:10.1097/ccm.0000000000002255 · Cited by 3,652
Deo, R. C. (2015) · Circulation · doi:10.1161/circulationaha.115.001593 · Cited by 3,466
Scalable and accurate deep learning with electronic health records
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0029-1 · Cited by 2,648 · cited 1×
“34–36 Previously, decision rules used in emergency medicine have shown impressive test characteristics at the institutions or regions that developed them, but attempts at external validation have yielded lower sensitivity and specificity, limiting general use.”
BackgroundIncidence and trends of sepsis in US hospitals using clinical vs claims data, 2009-2014.
Rhee, C., Dantes, R., Epstein, L., Murphy, D. J., Seymour, C. W., Iwashyna, T. J., Kadri, S. S., Angus, D. C., Danner, R. L., Fiore, A. E., et al. (2017) · JAMA · doi:10.1001/jama.2017.13836 · Cited by 1,884
Privacy in the age of medical big data
Price, W. N., & Cohen, I. G. (2019) · Nature Medicine · doi:10.1038/s41591-018-0272-7 · Cited by 1,522
Smith, G. B., Prytherch, D. R., Meredith, P., Schmidt, P. E., & Featherstone, P. I. (2013) · Resuscitation · doi:10.1016/j.resuscitation.2012.12.016 · Cited by 1,077
Goldstein, B. A., Navar, A. M., Pencina, M. J., & Ioannidis, J. P. A. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw042 · Cited by 936
Cruz Rivera, S., Liu, X., Chan, A.-W., Denniston, A. K., Calvert, M. J., The SPIRIT-AI and CONSORT-AI Working Group, SPIRIT-AI and CONSORT-AI Steering Group, Darzi, A., Holmes, C., Yau, C., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-1037-7 · Cited by 933
Cited by
Impact of a deep learning sepsis prediction model on quality of care and survival
Boussina, A., Shashikumar, S. P., Malhotra, A., Owens, R. L., El-Kareh, R., Longhurst, C. A., Quintero, K., Donahue, A., Chan, T. C., Nemati, S., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-023-00986-6 · Cited by 170
Self-citationMoor, M., Bennett, N., Plečko, D., Horn, M., Rieck, B., Meinshausen, N., Bühlmann, P., & Borgwardt, K. (2023) · eClinicalMedicine · doi:10.1016/j.eclinm.2023.102124 · Cited by 118
Wardi, G., Owens, R., Josef, C., Malhotra, A., Longhurst, C., & Nemati, S. (2023) · Critical Care Medicine · doi:10.1097/ccm.0000000000005894 · Cited by 35
Self-citationWang, Y., Gao, Z., Zhang, Y., Lu, Z., & Sun, F. (2025) · Internal and Emergency Medicine · doi:10.1007/s11739-024-03732-2 · Cited by 29
“With the flexibility to incorporate diverse clinical criteria and data attributes [29], ML can construct predictive models for various medical scenarios, including the prognosis of cardiac arrest in septic patients [30, 31], forecasting multiple organ function in elderly individuals [32, 33],…”
Transferability and interpretability of the sepsis prediction models in the intensive care unit
Chen, Q., Li, R., Lin, C., Lai, C., Chen, D., Qu, H., Huang, Y., Lu, W., Tang, Y., & Li, L. (2022) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-022-02090-3 · Cited by 28
Metadata
- Authors
- Wardi, G., Carlile, M., Holder, A., Shashikumar, S., Hayden, S. R., & Nemati, S.
- Venue
- Annals of Emergency Medicine
- Publisher
- Elsevier BV
- Published
- 2021-04
- Volume
- 77
- Issue
- 4
- Pages
- 395-406
- ISSN
- 0196-0644 (ISSN-L)
- OpenAlex
- W3120766804
- Cited by
- 96
- References
- 22
- FWCI
- 6.14
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Predicting Progression to Septic Shock in the Emergency Department Using an Externally Generalizable Machine-Learning Algorithm. https://app.cochat.ai/papers/10.1016/j.annemergmed.2020.11.007
- The paper
- Wardi, G., Carlile, M., Holder, A., Shashikumar, S., Hayden, S. R., & Nemati, S. (2021). Predicting Progression to Septic Shock in the Emergency Department Using an Externally Generalizable Machine-Learning Algorithm. Annals of Emergency Medicine. https://doi.org/10.1016/j.annemergmed.2020.11.007
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.