CoChat

Deployment of machine learning algorithms to predict sepsis: systematic review and application of the SALIENT clinical AI implementation framework

Open access
van der Vegt, Anton H, Scott, Ian A, Dermawan, Krishna, Schnetler, Rudolf J, Kalke, Vikrant R, Lane, Paul J · 2023 · Journal of the American Medical Informatics Association

The DOI matches its Crossref record; 67 of the 75 references it cites were found in a registry or an index; 9 of the 13 sources its claims cite were checked for support.

Abstract

OBJECTIVE: To retrieve and appraise studies of deployed artificial intelligence (AI)-based sepsis prediction algorithms using systematic methods, identify implementation barriers, enablers, and key decisions and then map these to a novel end-to-end clinical AI implementation framework. MATERIALS AND METHODS: Systematically review studies of clinically applied AI-based sepsis prediction algorithms in regard to methodological quality, deployment and evaluation methods, and outcomes. Identify contextual factors that influence implementation and map these factors to the SALIENT implementation framework. RESULTS: The review identified 30 articles of algorithms applied in adult hospital settings, with 5 studies reporting significantly decreased mortality post-implementation. Eight groups of algorithms were identified, each sharing a common algorithm. We identified 14 barriers, 26 enablers, and 22 decision points which were able to be mapped to the 5 stages of the SALIENT implementation framework. DISCUSSION: Empirical studies of deployed sepsis prediction algorithms demonstrate their potential for improving care and reducing mortality but reveal persisting gaps in existing implementation guidance. In the examined publications, key decision points reflecting real-word implementation experience could be mapped to the SALIENT framework and, as these decision points appear to be AI-task agnostic, this framework may also be applicable to non-sepsis algorithms. The mapping clarified where and when barriers, enablers, and key decisions arise within the end-to-end AI implementation process. CONCLUSIONS: A systematic review of real-world implementation studies of sepsis prediction algorithms was used to validate an end-to-end staged implementation framework that has the ability to account for key factors that warrant attention in ensuring successful deployment, and which extends on previous AI implementation frameworks.

Overview

CoChat Summary

TL;DR

This systematic review of real-world sepsis prediction MLAs found 30 articles on adult hospital deployments and concluded that these systems may reduce mortality, but a definitive causal link was not demonstrated. The authors also mapped 14 barriers, 26 enablers, and 22 decision points onto the SALIENT implementation framework to show how end-to-end AI deployment factors fit into a staged model.

Key findings

  • The review included 30 articles (19 primary articles plus 11 linked papers) on adult hospital settings; the studies were published between 2015 and 2022.
  • Across the included studies, the authors identified 14 unique barriers, 26 unique enablers, and 22 decision points; 17 of the decision points were identified by at least 2 groups.
  • 5 studies reported significantly decreased mortality post-implementation; all 9 papers reporting mortality found decreased mortality, but significance was limited to those 5 studies.
  • The most common barriers were lack of clinician trust, alert fatigue, and dismissal of alerts; the most common enabler was frequent communication to raise awareness of the MLA.

Methods

  • Systematic review conducted according to PRISMA; searched PubMed/Medline, EMBASE, Scopus, Web of Science, and clinicaltrials.gov for English-language studies from January 1, 2012 to June 23, 2022.
  • Included studies had to deploy MLAs to adult hospitalized patients with formally defined sepsis, use live or near-live data, and report at least one performance metric.
  • Extracted implementation barriers, enablers, and decision points were mapped to the SALIENT framework; mortality studies also underwent risk-of-bias assessment using ROBINS-I or RoB 2.

Limitations

  • The paper states limitations of the evidence base: small number of empirical deployed studies, under-reporting of post-implementation performance metrics, focus on adult hospital settings, and potential publication bias from under-reporting of other implementation studies.
  • The authors also note that many studies were non-randomized and subject to confounding from co-interventions such as sepsis awareness programs, and that post-implementation performance metrics were infrequently reported and non-standardized.
  • Evident from the text: most studies were from the US, so generalizability beyond that setting is limited (evident in the study characteristics).

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1093/jamia/ocad075

Cites

67 of 67 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1. Flowchart for study selection.
Figure 1. Flowchart for study selection.CC BY · PubMed Central
Figure 2. Evaluation results for algorithm performance, adoption, clinical process, and mortality improvement for each MLA group. Figure includes risk of bias (RoB) assessment for studies reporting mortality assessment (M = moderate [some concerns for RoB-2], S = serious, C = critical). Black numbered squares denote the cited paper for that result. AUC: area under the receiver operating curve; Sens: sensitivity; Spec: specificity; PPV/NPV: positive/negative predictive value. Solid shaded up arrow: significant improvement, whereas hollow up arrow: non-significant improvement.
Figure 2. Evaluation results for algorithm performance, adoption, clinical process, and mortality improvement for each MLA group. Figure includes risk of bias (RoB) assessment for studies reporting mortality assessment (M = moderate [some concerns for RoB-2], S = serious, C = critical). Black numbered squares denote the cited paper for that result. AUC: area under the receiver operating curve; Sens: sensitivity; Spec: specificity; PPV/NPV: positive/negative predictive value. Solid shaded up arrow: significant improvement, whereas hollow up arrow: non-significant improvement.CC BY · PubMed Central
Figure 3. The number of barriers (red badge), enablers (green badge), and decision points (blue badge), denoted by the number within the badge, mapped to each stage and component of the SALIENT AI implementation framework. Implementation stages are labeled in the title row from left to right. The color-coded solution components are developed in row B, and consist of Clinical workflow (blue), AI model (yellow), data pipeline (green), and human-computer interface (red). The components are integrated in rows C and D, and rows F and G describe cross-stage elements required throughout the entire implementation process, such as governance and quality and safety assurance. w/flow: workflow; dev/test: development & test; HCI: human computer interface.
Figure 3. The number of barriers (red badge), enablers (green badge), and decision points (blue badge), denoted by the number within the badge, mapped to each stage and component of the SALIENT AI implementation framework. Implementation stages are labeled in the title row from left to right. The color-coded solution components are developed in row B, and consist of Clinical workflow (blue), AI model (yellow), data pipeline (green), and human-computer interface (red). The components are integrated in rows C and D, and rows F and G describe cross-stage elements required throughout the entire implementation process, such as governance and quality and safety assurance. w/flow: workflow; dev/test: development & test; HCI: human computer interface.CC BY · PubMed Central

Verification

Claims the paper makes

3 Valid · 0 Not valid · 0 Not checked · 12 Own results and interpretations

  • It may state two separate claims as one. 12

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Cited by

See all 72

Metadata

Authors
van der Vegt, A. H., Scott, I. A., Dermawan, K., Schnetler, R. J., Kalke, V. R., & Lane, P. J.
Affiliations
  • Queensland Digital Health Centre, The University of Queensland , Brisbane, Queensland, Australia
  • Princess Alexandra Hospital (AU)
  • The University of Queensland (AU)
  • Queensland Health (AU)
Venue
Journal of the American Medical Informatics Association
Published
2023-05-12
Volume
30
Issue
7
Pages
1349-1361
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
OpenAlex
W4376270103
Cited by
72
References
75
Citations by year
2026: 24 · 2025: 25 · 2024: 11 · 2023: 12
FWCI
2.80
Type
journal article
Language
en
Open access
Locations
  • Journal of the American Medical Informatics Association (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
  • Queensland Government
  • Advanced Queensland Industry Research Fellowship
Topics
  • Health Sciences › Medicine › Critical Care and Intensive Care Medicine › Sepsis Diagnosis and Treatment
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
sepsis prediction, machine learning algorithms, implementation barriers and facilitators, sepsis mortality, algorithm deployment
MeSH terms
Machine Learning, Adult, Algorithms, Artificial Intelligence, Humans, Sepsis, Empirical Research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.