Bringing the Promise of Artificial Intelligence to Critical Care: What the Experience With Sepsis Analytics Can Teach Us
Open accessAbstract
In 1985, development of a computer system called “Deep Thought” began at Carnegie Mellon University with the lofty objective of developing an autonomous system capable of outperforming the world’s top chess grandmasters. Later renamed “Deep Blue,” this chess-playing expert system defeated world champion Gary Kasparov in 1997 in a six-game match. However, it was not until 2017 that a deep artificial neural network algorithm known as “AlphaZero” achieved super-human performance in several challenging games, including Chess, Shogi, and Go (1). Such triumphs in computer-based technologies are common today as artificial intelligence (AI) applications, such as ChatGPT and DALL-E, are mimicking human capabilities, even passing medical board examinations (2). The term AI is used to describe the general ability of computers to emulate various characteristics of human intelligence, including pattern recognition, inference, and sequential decision-making, among others. Machine learning (ML) is a subset of AI that can learn the complex interactions or temporal relationships among multivariate risk factors without the need to hand-craft such features via expert knowledge (3). Retrospective studies have demonstrated ML applications are particularly useful for their diagnostic and prognostic capabilities leveraging vast quantity of data available in the ICU (4,5). Certain ML algorithms have approached human performance at narrow tasks such as predicting resuscitation strategies in sepsis (6), need for mechanical ventilation (7), mortality in critically ill patients (8), and ICU length of stay (9). Sepsis is an attractive target for ML approaches as it is an inherently complex, common, costly, and deadly condition. Prediction of sepsis is the most common ML application described, although recent advances include approaches to optimize therapeutics and resuscitation strategies (6,10). Given the potential to improve patient-centered outcomes and excitement about newer analytic approaches, it is no surprise that the number of ML algorithms aimed to improve sepsis care is increasing at a rapid rate. However, errors in sepsis prediction are often highlighted both in anecdotal and health system-wide failures that can be traced to poor implementation approaches, rudimentary ML algorithms, application of algorithms outside their intended use, or without proper maintenance. Noting these criticisms, what can be done at this point to demonstrate value of these predictive models? We believe that a revised focus on data enrichment, proper implementation, and rigorous testing is required to bring the promise of AI to the ICU. DATA AVAILABILITY AND AUGMENTATION—THINKING OUTSIDE THE EHR Timely data are needed for any model to improve sepsis care and is the basis of any predictive model as shown in Figure 1. To date, most ML algorithms in clinical use are limited to input features that are limited to data available in the electronic health record (EHR), such as vital signs, demographic data, laboratory results, and occasionally imaging studies. Importantly, the frequency of EHR measurements is commonly a function of level of care, workflow practices, and patients’ severity of illness (11–13). This is particularly important for patients in the hospital wards, where data are sparse and delays in sepsis identification are common (12). In other words, the most data and accurate predictions are occurring where patients are already known to have, or be at risk for, sepsis. Thus, it has been suggested that such systems are essentially looking over clinician’s shoulders. In other words, these models are using clinical behavior (e.g., ordering of a serum lactate level) as the expression of preexisting intuition and suspicion to generate a prediction (14). There are several potential solutions, none of which has been widely studied to date in a prospective setting. First, data enrichment, the process of incorporating updated data elements from sources outside the electronic health record, could be used. For example, multimodal data from bedside monitors, IV pumps, mechanical ventilators, and imaging studies could all be collected. Development of wearable biopatches may allow for incorporation of near instantaneous data. Second, under most existing protocols, AI is not actively involved in data generation. The use of smart laboratories—diagnostic studies suggested or even ordered by an AI system at times of particularly low predictive certainty—and/or additional nursing assessments may help improve accuracy of these algorithms, although the challenge is to “choose wisely” and keeps costs and workflow impediments minimized. Testing of this approach is indicated prior to widespread implementation to ensure that any costs—whether it be direct financial costs, cognitive burden on provider, medicolegal risk, or patient discomfort—is minimized while adding value to care. Although the infrastructure of this approach does require coordination between various
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
AI algorithms can analyze vast amounts of data to identify early signs of sepsis or other critical conditions, allowing for more proactive and effective interventions.
“AI algorithms can analyze vast amounts of data to identify early signs of sepsis or other critical conditions, allowing for more proactive and effective interventions [5].”
From Leading with AI in critical care nursing: challenges, opportunities, and the human factor (2024) · cites this paper as [5]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. (2018) · Science · doi:10.1126/science.aar6404 · Cited by 3,562 · cited 2× · see citing works
“Although hardware improvements were also needed to achieve super-human performance, the latest iteration of such systems (1) had to embrace more powerful ML algorithms and were programmed to learn continually.”
BackgroundThe “All of Us” Research Program
The All of Us Research Program Investigators (2019) · New England Journal of Medicine · doi:10.1056/nejmsr1809937 · Cited by 2,440 · see citing works
Komorowski, M., Celi, L. A., Badawi, O., Gordon, A. C., & Faisal, A. A. (2018) · Nature Medicine · doi:10.1038/s41591-018-0213-5 · Cited by 1,290 · see citing works
Wong, A., Otles, E., Donnelly, J. P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., et al. (2021) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2021.2626 · Cited by 995 · see citing works
Encyclopedia of Machine Learning and Data Mining
Webb, G. I., & Sammut, C. (2017) · Springer eBooks · doi:10.1007/978-1-4899-7687-1 · Cited by 926 · see citing works
The “All of Us” Research Program
Sullivan, F., McKinstry, B., & Vasishta, S. (2019) · New England Journal of Medicine · doi:10.1056/nejmc1912496 · Cited by 855 · see citing works
Agniel, D., Kohane, I. S., & Weber, G. M. (2018) · BMJ · doi:10.1136/bmj.k1479 · Cited by 391 · see citing works
Early Prediction of Sepsis From Clinical Data: The PhysioNet/Computing in Cardiology Challenge 2019
Reyna, M. A., Josef, C. S., Jeter, R., Shashikumar, S. P., Westover, M. B., Nemati, S., Clifford, G. D., & Sharma, A. (2020) · Critical Care Medicine · doi:10.1097/ccm.0000000000004145 · Cited by 383 · see citing works
Self-citationShimabukuro, D. W., Barton, C. W., Feldman, M. D., Mataraso, S. J., & Das, R. (2017) · BMJ Open Respiratory Research · doi:10.1136/bmjresp-2017-000234 · Cited by 382 · see citing works
Adams, R., Henry, K. E., Sridharan, A., Soleimani, H., Zhan, A., Rawat, N., Johnson, L., Hager, D. N., Cosgrove, S. E., Markowski, A., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01894-0 · Cited by 365 · see citing works
Leisman, D. E., Harhay, M. O., Lederer, D. J., Abramson, M., Adjei, A. A., Bakker, J., Ballas, Z. K., Barreiro, E., Bell, S. C., Bellomo, R., et al. (2020) · Critical Care Medicine · doi:10.1097/ccm.0000000000004246 · Cited by 271 · see citing works
Giannini, H. M., Ginestra, J. C., Chivers, C., Draugelis, M., Hanish, A., Schweickert, W. D., Fuchs, B. D., Meadows, L., Lynch, M., Donnelly, P. J., et al. (2019) · Critical Care Medicine · doi:10.1097/ccm.0000000000003891 · Cited by 269 · see citing works
Creating a Learning Health System through Rapid-Cycle, Randomized Testing
Horwitz, L. I., Kuznetsova, M., & Jones, S. A. (2019) · New England Journal of Medicine · doi:10.1056/nejmsb1900856 · Cited by 250 · see citing works
“Big Data” in the Intensive Care Unit. Closing the Data Loop
Celi, L. A., Mark, R. G., Stone, D. J., & Montgomery, R. A. (2013) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201212-2311ed · Cited by 209 · see citing works
Awad, A., Bader-El-Den, M., McNicholas, J., & Briggs, J. (2017) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2017.10.002 · Cited by 208 · see citing works
Artificial Intelligence and Health Technology Assessment: Anticipating a New Level of Complexity
Alami, H., Lehoux, P., Auclair, Y., de Guise, M., Gagnon, M., Shaw, J., Roy, D. A., Fleet, R. P., Ahmed, M. A. A., & Fortin, J. (2020) · Journal of Medical Internet Research · doi:10.2196/17707 · Cited by 180 · see citing works
Bridging the implementation gap of machine learning in healthcare
Seneviratne, M. G., Shah, N. H., & Chu, L. (2020) · BMJ Innovations · doi:10.1136/bmjinnov-2019-000359 · Cited by 173 · see citing works
Ginestra, J. C., Giannini, H. M., Schweickert, W. D., Meadows, L., Lynch, M. J., Pavan, K., Chivers, C. J., Draugelis, M., Donnelly, P. J., Fuchs, B. D., et al. (2019) · Critical Care Medicine · doi:10.1097/ccm.0000000000003803 · Cited by 159 · see citing works
Beaulieu‐Jones, B. K., Yuan, W., Brat, G. A., Beam, A. L., Weber, G. M., Ruffin, M., & Kohane, I. S. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00426-3 · Cited by 157 · see citing works
Artificial intelligence sepsis prediction algorithm learns to say “I don’t know”
Shashikumar, S. P., Wardi, G., Malhotra, A., & Nemati, S. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00504-6 · Cited by 130 · see citing works
Self-citationAlgorithm Change Protocols in the Regulation of Adaptive Machine Learning–Based Medical Devices
Gilbert, S., Fenech, M. T., Hirsch, M. C., Upadhyay, S., Biasiucci, A., & Starlinger, J. (2021) · Journal of Medical Internet Research · doi:10.2196/30545 · Cited by 106 · see citing works
Wardi, G., Carlile, M., Holder, A. L., Shashikumar, S. P., Hayden, S. R., & Nemati, S. (2020) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2020.11.007 · Cited by 101 · see citing works
Self-citationKanjilal, S., Oberst, M., Boominathan, S., Zhou, H. H., Hooper, D. C., & Sontag, D. A. (2020) · Science Translational Medicine · doi:10.1126/scitranslmed.aay5067 · Cited by 94 · see citing works
Houthooft, R., Ruyssinck, J., van der Herten, J., et al. (2014) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2014.12.009 · Cited by 90 · see citing works
Tarabichi, Y., Cheng, A., Bar-Shain, D., McCrate, B. M., Reese, L. H., Emerman, C., Siff, J., Wang, C., Kaelber, D. C., Watts, B., et al. (2022) · Critical Care Medicine · doi:10.1097/ccm.0000000000005267 · Cited by 76 · see citing works
The silent trial - the bridge between bench-to-bedside clinical AI applications
Kwong, J. C. C., Erdman, L., Khondker, A., Skreta, M., Goldenberg, A., McCradden, M. D., Lorenzo, A. J., & Rickard, M. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.929508 · Cited by 73 · cited 1× · see citing works
“These silent trials can result in improvement in a model’s predictive ability and also markedly reduce false alarms (17).”
BackgroundShashikumar, S. P., Wardi, G., Paul, P., Paul, P., Carlile, M., Brenner, L. N., Hibbert, K. A., North, C. M., Mukerji, S. S., Robbins, G. K., et al. (2020) · CHEST Journal · doi:10.1016/j.chest.2020.12.009 · Cited by 70 · see citing works
Self-citationMedicare's Enduring Struggle to Define “Reasonable and Necessary” Care
Neumann, P. J., & Chambers, J. D. (2012) · New England Journal of Medicine · doi:10.1056/nejmp1208386 · Cited by 59 · see citing works
Kasthurirathne, S. N., Mamlin, B. W., Kumara, H., Grieve, G., & Biondich, P. (2015) · Journal of Medical Systems · doi:10.1007/s10916-015-0356-6 · Cited by 52 · see citing works
Rethinking Algorithm Performance Metrics for Artificial Intelligence in Diagnostic Medicine
Reyna, M. A., Nsoesie, E. O., & Clifford, G. D. (2022) · JAMA · doi:10.1001/jama.2022.10561 · Cited by 49 · see citing works
Preparing a New Generation of Clinicians for the Era of Big Data.
Moskowitz, A. L., McSparron, J. I., Stone, D. J., & Celi, L. A. (2015) · PubMed · Cited by 36 · see citing works
Clinical Predictive Modeling Development and Deployment through FHIR Web Services.
Khalilia, M. A., Choi, M., Henderson, A., Iyengar, S., Braunstein, M. L., & Sun, J. (2015) · PubMed Central · Cited by 35 · see citing works
Comparison of Sepsis Definitions as Automated Criteria
Yu, S. C., Betthauser, K. D., Gupta, A., Lyons, P. G., Lai, A. M., Kollef, M. H., Payne, P. R. O., & Michelson, A. P. (2021) · Critical Care Medicine · doi:10.1097/ccm.0000000000004875 · Cited by 29 · see citing works
Bradwell, K. R., Wooldridge, J. T., Amor, B., Bennett, T. D., Anand, A., Bremer, C., Yoo, Y. J., Qian, Z., Johnson, S., Pfaff, E., et al. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac054 · Cited by 29 · see citing works
A Locally Optimized Data-Driven Tool to Predict Sepsis-Associated Vasopressor Use in the ICU
Holder, A. L., Shashikumar, S. P., Wardi, G., Buchman, T. G., & Nemati, S. (2021) · Critical Care Medicine · doi:10.1097/ccm.0000000000005175 · Cited by 24 · see citing works
Self-citationHITECH to 21st century cures: clinician burden and evolving health IT policy
Gettinger, A., & Zayas‐Cabán, T. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa330 · Cited by 18 · see citing works
Rush, B. N. M., Stone, D. J., & Celi, L. A. (2018) · Critical Care Medicine · doi:10.1097/ccm.0000000000002892 · Cited by 17 · see citing works
Quantifying Electronic Health Record Data: A Potential Risk for Cognitive Overload
Gal, D. B., Han, B. S., Longhurst, C., Scheinker, D., & Shin, A. Y. (2021) · Hospital Pediatrics · doi:10.1542/hpeds.2020-002402 · Cited by 16 · see citing works
Self-citationRogers, P., Boussina, A. E., Shashikumar, S. P., Wardi, G., Longhurst, C., & Nemati, S. (2022) · Journal of Medical Internet Research · doi:10.2196/43486 · Cited by 13 · see citing works
Self-citationDeveloping a highly‐reliable learning health system
El-Kareh, R. E., Brenner, D. A., & Longhurst, C. (2022) · Learning Health Systems · doi:10.1002/lrh2.10351 · Cited by 11 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Impact of a deep learning sepsis prediction model on quality of care and survival
Boussina, A., Shashikumar, S. P., Malhotra, A., Owens, R. L., El-Kareh, R., Longhurst, C. A., Quintero, K., Donahue, A., Chan, T. C., Nemati, S., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-023-00986-6 · Cited by 170 · see citing works
Self-citationSokol, K., Fackler, J., & Vogt, J. E. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01725-9 · Cited by 105 · see citing works
Leading with AI in critical care nursing: challenges, opportunities, and the human factor
Hassan, E. A., & El-Ashry, A. M. (2024) · BMC Nursing · doi:10.1186/s12912-024-02363-4 · Cited by 72 · see citing works
- Authors
- Wardi, G., Owens, R., Josef, C., Malhotra, A., Longhurst, C., & Nemati, S.
- Affiliations
- UC San Diego Health System (US)
- Healcisio Inc., San Diego, CA.
- Department of Biomedical Informatics, University of California, San Diego, La Jolla, CA.
- Corresponding author
- Wardi, Gabriel
- Venue
- Critical Care Medicine
- Published
- 2023-04-26
- Volume
- 51
- Issue
- 8
- Pages
- 985-991
- ISSN
- 0090-3493 (ISSN-L)
- PMC
- PMC10335736
- OpenAlex
- W4367047646
- Cited by
- 35
- References
- 42
- Citations by year
- 2026: 6 · 2025: 13 · 2024: 15 · 2023: 1
- FWCI
- 6.96
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Critical Care Medicine (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health (K23GM146092)
- Topics
- Health Sciences › Medicine › Epidemiology › Sepsis Diagnosis and Treatment
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- machine learning, data quality, electronic health records, data augmentation, model validation, clinical decision support, sepsis resuscitation, ICU mortality prediction
- MeSH terms
- Artificial Intelligence, Critical Care, Humans, Sepsis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
