Deployment of machine learning algorithms to predict sepsis: systematic review and application of the SALIENT clinical AI implementation framework
Open accessThe DOI matches its Crossref record; 67 of the 75 references it cites were found in a registry or an index; 9 of the 13 sources its claims cite were checked for support.
Abstract
OBJECTIVE: To retrieve and appraise studies of deployed artificial intelligence (AI)-based sepsis prediction algorithms using systematic methods, identify implementation barriers, enablers, and key decisions and then map these to a novel end-to-end clinical AI implementation framework. MATERIALS AND METHODS: Systematically review studies of clinically applied AI-based sepsis prediction algorithms in regard to methodological quality, deployment and evaluation methods, and outcomes. Identify contextual factors that influence implementation and map these factors to the SALIENT implementation framework. RESULTS: The review identified 30 articles of algorithms applied in adult hospital settings, with 5 studies reporting significantly decreased mortality post-implementation. Eight groups of algorithms were identified, each sharing a common algorithm. We identified 14 barriers, 26 enablers, and 22 decision points which were able to be mapped to the 5 stages of the SALIENT implementation framework. DISCUSSION: Empirical studies of deployed sepsis prediction algorithms demonstrate their potential for improving care and reducing mortality but reveal persisting gaps in existing implementation guidance. In the examined publications, key decision points reflecting real-word implementation experience could be mapped to the SALIENT framework and, as these decision points appear to be AI-task agnostic, this framework may also be applicable to non-sepsis algorithms. The mapping clarified where and when barriers, enablers, and key decisions arise within the end-to-end AI implementation process. CONCLUSIONS: A systematic review of real-world implementation studies of sepsis prediction algorithms was used to validate an end-to-end staged implementation framework that has the ability to account for key factors that warrant attention in ensuring successful deployment, and which extends on previous AI implementation frameworks.
Overview
CoChat Summary
TL;DR
This systematic review of real-world sepsis prediction MLAs found 30 articles on adult hospital deployments and concluded that these systems may reduce mortality, but a definitive causal link was not demonstrated. The authors also mapped 14 barriers, 26 enablers, and 22 decision points onto the SALIENT implementation framework to show how end-to-end AI deployment factors fit into a staged model.
Key findings
- The review included 30 articles (19 primary articles plus 11 linked papers) on adult hospital settings; the studies were published between 2015 and 2022.
- Across the included studies, the authors identified 14 unique barriers, 26 unique enablers, and 22 decision points; 17 of the decision points were identified by at least 2 groups.
- 5 studies reported significantly decreased mortality post-implementation; all 9 papers reporting mortality found decreased mortality, but significance was limited to those 5 studies.
- The most common barriers were lack of clinician trust, alert fatigue, and dismissal of alerts; the most common enabler was frequent communication to raise awareness of the MLA.
Methods
- Systematic review conducted according to PRISMA; searched PubMed/Medline, EMBASE, Scopus, Web of Science, and clinicaltrials.gov for English-language studies from January 1, 2012 to June 23, 2022.
- Included studies had to deploy MLAs to adult hospitalized patients with formally defined sepsis, use live or near-live data, and report at least one performance metric.
- Extracted implementation barriers, enablers, and decision points were mapped to the SALIENT framework; mortality studies also underwent risk-of-bias assessment using ROBINS-I or RoB 2.
Limitations
- The paper states limitations of the evidence base: small number of empirical deployed studies, under-reporting of post-implementation performance metrics, focus on adult hospital settings, and potential publication bias from under-reporting of other implementation studies.
- The authors also note that many studies were non-randomized and subject to confounding from co-interventions such as sepsis awareness programs, and that post-implementation performance metrics were infrequently reported and non-standardized.
- Evident from the text: most studies were from the US, so generalizability beyond that setting is limited (evident in the study characteristics).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures

![Figure 2. Evaluation results for algorithm performance, adoption, clinical process, and mortality improvement for each MLA group. Figure includes risk of bias (RoB) assessment for studies reporting mortality assessment (M = moderate [some concerns for RoB-2], S = serious, C = critical). Black numbered squares denote the cited paper for that result. AUC: area under the receiver operating curve; Sens: sensitivity; Spec: specificity; PPV/NPV: positive/negative predictive value. Solid shaded up arrow: significant improvement, whereas hollow up arrow: non-significant improvement.](https://figures.cochat.ai/paper-figures/335a66efe8784a43b0da8b8f017ba02a/deployment-of-machine-learning-algorithms-to-predict-sepsis-systematic-review-figure-2.jpg)

Verification
Claims the paper makes
3 Valid · 0 Not valid · 0 Not checked · 12 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Eight groups of algorithms were identified, each sharing a common algorithm.”
§ Deployment of machine learning algorithms to predict sepsis: systematic review and application of the SALIENT clinical AI implementation framework > Abstract > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All barriers and enablers could be mapped to the SALIENT AI implantation framework (see Figure 3).”
§ Deployment of machine learning algorithms to predict sepsis: systematic review and application of the SALIENT clinical AI implementation framework > RESULTS > Identification of implementation barriers, enablers, and decision points and mapping to SALIENT AI implementation framework
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A systematic review of real-world implementation studies of sepsis prediction algorithms was used to validate an end-to-end staged implementation framework”
§ Deployment of machine learning algorithms to predict sepsis: systematic review and application of the SALIENT clinical AI implementation framework > Abstract > Conclusions
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 12
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“More complex systems have incorporated gradient boosting, neural networks, and ensemble methods to identify sepsis risk hours before clinical recognition, providing a larger window for interventions [46].”
From Artificial Intelligence to Facilitate SEP-1 Measure Compliance and Fluid Management in Sepsis (2026) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Across studies, post-implementation performance was inconsistently reported and often declined compared to pre-deployment metrics, reflecting the impact of data and model drift due to evolving clinical practices and data variability, underscoring the need for continuous monitoring, recalibration, and lifecycle evaluation to maintain model reliability in real-world settings.”
From Artificial Intelligence to Facilitate SEP-1 Measure Compliance and Fluid Management in Sepsis (2026) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Implementation barriers for sepsis AI cluster around trust and interpretability, workflow integration, technical readiness and generalizability, governance and safety, leadership alignment, and the resources required for sustainment (Table 4) [10,33,34,46,48,77,78].”
From Artificial Intelligence to Facilitate SEP-1 Measure Compliance and Fluid Management in Sepsis (2026) · cites this paper as [46]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“A systematic review of machine learning sepsis prediction implementations found that only a minority demonstrated statistically significant reductions in mortality, and many reported primarily algorithm performance and process improvements rather than patient-centered outcomes [46].”
From Artificial Intelligence to Facilitate SEP-1 Measure Compliance and Fluid Management in Sepsis (2026) · cites this paper as [46]
Citations
Cites
RoB 2: a revised tool for assessing risk of bias in randomised trials
Sterne, J. A. C., Savović, J., Page, M. J., Elbers, R. G., Blencowe, N. S., Boutron, I., Cates, C. J., Cheng, H.-Y., Corbett, M. S., Eldridge, S. M., et al. (2019) · BMJ · doi:10.1136/bmj.l4898 · Cited by 34,787
PRISMA-P Group, Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-1 · Cited by 26,223
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496
Bone, R. C., Balk, R. A., Cerra, F. B., Dellinger, R. P., Fein, A. M., Knaus, W. A., Schein, R. M. H., & Sibbald, W. J. (1992) · Chest · doi:10.1378/chest.101.6.1644 · Cited by 13,198 · cited 1×
“Sepsis has been defined as a “life-threatening organ dysfunction caused by a dysregulated host response to infection”.(2,3) Early recognition and treatment of sepsis can reduce mortality, and rule-based surveillance systems for detecting sepsis in hospital settings can improve outcomes.”
BackgroundRudd, K. E., Johnson, S. C., Agesa, K. M., Shackelford, K. A., Tsoi, D., Kievlan, D. R., Colombara, D. V., Ikuta, K. S., Kissoon, N., Finfer, S., et al. (2020) · The Lancet · doi:10.1016/s0140-6736(19)32989-7 · Cited by 7,971 · cited 1×
“Sepsis accounts for nearly 20% of deaths worldwide, killing over 11 million people in 2017.(1) Sepsis has been defined as a “life-threatening organ dysfunction caused by a dysregulated host response to infection”.”
BackgroundBioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Seymour, C. W., Liu, V. X., Iwashyna, T. J., Brunkhorst, F. M., Rea, T. D., Scherag, A., Rubenfeld, G., Kahn, J. M., Shankar-Hari, M., Singer, M., et al. (2016) · JAMA · doi:10.1001/jama.2016.0288 · Cited by 3,939
Greenhalgh, T., Wherton, J., Papoutsi, C., Lynch, J., Hughes, G., A'Court, C., Hinder, S., Fahy, N., Procter, R., & Shaw, S. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8775 · Cited by 2,887
Time to treatment and mortality during mandated emergency care for sepsis.
Seymour, C. W., Gesten, F., Prescott, H. C., Friedrich, M. E., Iwashyna, T. J., Phillips, G. S., Lemeshow, S., Osborn, T., Terry, K. M., & Levy, M. M. (2017) · New England Journal of Medicine · doi:10.1056/nejmoa1703058 · Cited by 2,225 · cited 1×
“Early recognition and treatment of sepsis can reduce mortality, and rule-based surveillance systems for detecting sepsis in hospital settings can improve outcomes.(4,5) More recently, sepsis prediction algorithms employing artificial intelligence (AI), herein called machine learning algorithms (MLAs), that can detect…”
BackgroundWong, A., Otles, E., Donnelly, J. P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., et al. (2021) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2021.2626 · Cited by 995 · cited 1×
“Most (66%) post-implementation studies did not report real-world MLA performance (Figure 2), and of the 3 that did, one reported improved MLA performance(58) while the other 2 showed marked declines,(34,50) and similarly for the external validation study of the EPIC tool (Group G).(51) Table 1.”
ResultFleuren, L. M., Klausch, T. L. T., Zwager, C. L., Schoonmade, L. J., Guo, T., Roggeveen, L. F., Swart, E. L., Girbes, A. R. J., Thoral, P., Ercole, A., et al. (2020) · Intensive Care Medicine · doi:10.1007/s00134-019-05872-y · Cited by 755
Liu, X., Rivera, S. C., Moher, D., Calvert, M. J., & Denniston, A. K. (2020) · BMJ · doi:10.1136/bmj.m3164 · Cited by 702
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · BMJ · doi:10.1136/bmj-2022-070904 · Cited by 665
A targeted real-time early warning score (TREWScore) for septic shock.
Henry, K. E., Hager, D. N., Pronovost, P. J., & Saria, S. (2015) · Science Translational Medicine · doi:10.1126/scitranslmed.aab3719 · Cited by 598
Desautels, T., Calvert, J., Hoffman, J., Jay, M., Kerem, Y., Shieh, L., Shimabukuro, D., Chettipally, U., Feldman, M. D., Barton, C., et al. (2016) · JMIR Medical Informatics · doi:10.2196/medinform.5909 · Cited by 509 · cited 1×
“Deciding which features to input and how simple (eg, vital signs only) or complex (eg, waveform data and laboratory results) they are was seen to influence model generalizability (D6) to different care locations.(38,57,58,62) Group (B) supported different variables for different sites, claiming flexibility,(62) althou…”
BackgroundTaylor, R. A., Pare, J. R., Venkatesh, A. K., Mowafi, H., Melnick, E. R., Fleischman, W., & Hall, M. K. (2016) · Academic Emergency Medicine · doi:10.1111/acem.12876 · Cited by 491
Role of Artificial Intelligence Applications in Real-Life Clinical Practice: Systematic Review
Yin, J., Ngiam, K. Y., & Teo, H. H. (2021) · Journal of Medical Internet Research · doi:10.2196/25759 · Cited by 419
Shimabukuro, D. W., Barton, C. W., Feldman, M. D., Mataraso, S. J., & Das, R. (2017) · BMJ Open Respiratory Research · doi:10.1136/bmjresp-2017-000234 · Cited by 382 · cited 5×
“Deciding which features to input and how simple (eg, vital signs only) or complex (eg, waveform data and laboratory results) they are was seen to influence model generalizability (D6) to different care locations.(38,57,58,62) Group (B) supported different variables for different sites, claiming flexibility,(62) althou…”
BackgroundResultInfluentialAdams, R., Henry, K. E., Sridharan, A., Soleimani, H., Zhan, A., Rawat, N., Johnson, L., Hager, D. N., Cosgrove, S. E., Markowski, A., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01894-0 · Cited by 365 · cited 6×
“The prime purpose for implementing sepsis MLAs varied which in turn determined how they were trained and evaluated (D3),(43) with evaluation metrics and success criteria varying depending on whether increasing sepsis care bundle compliance,(50) providing a sepsis detection and management system,(41) reducing anti-micr…”
BackgroundMethodInfluentialMao, Q., Jay, M., Hoffman, J. L., Calvert, J., Barton, C., Shimabukuro, D., Shieh, L., Chettipally, U., Fletcher, G., Kerem, Y., et al. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-017833 · Cited by 336
Sendak, M. P., Ratliff, W., Sarro, D., Alderton, E., Futoma, J., Gao, M., Nichols, M., Revoir, M., Yashar, F., Miller, C., et al. (2020) · JMIR Medical Informatics · doi:10.2196/15182 · Cited by 287 · cited 4×
“Similarly, whether to develop and implement the MLA in-house or use an external vendor (D10), which involved weighing up the capability to future-proof the organization for future AI solutions(41) versus implementation and maintenance challenges arising from separate ownership of the input data and the AI model.(41) T…”
BackgroundMethodInfluentialDamiani, E., Donati, A., Serafini, G., Rinaldi, L., Adrario, E., Pelaia, P., Busani, S., & Girardis, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0125827 · Cited by 275
Giannini, H. M., Ginestra, J. C., Chivers, C., Draugelis, M., Hanish, A., Schweickert, W. D., Fuchs, B. D., Meadows, L., Lynch, M., Donnelly, P. J., et al. (2019) · Critical Care Medicine · doi:10.1097/ccm.0000000000003891 · Cited by 269 · cited 2×
“Related to the timing of alerts (D7), decisions about what actions clinicians should take for alerts involving patients showing no symptoms or signs of sepsis proved problematic (D15), as unclear roles and responsibilities constituted potential barriers to adoption (B3, B5).(33,44)”
BackgroundMethodPrediction of sepsis patients using machine learning approach: a meta-analysis
Islam, M. M., Nasrin, T., Walther, B. A., Wu, C.-C., Yang, H.-C., & Li, Y.-C. (2019) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2018.12.027 · Cited by 259 · cited 1×
“More recently, sepsis prediction algorithms employing artificial intelligence (AI), herein called machine learning algorithms (MLAs), that can detect evolving sepsis in patients earlier than rule-based methods, have proliferated.(9,10) Most MLA studies assess performance based on static training and testing data colle…”
BackgroundClinical Implications and Challenges of Artificial Intelligence and Deep Learning
Stead, W. W. (2018) · JAMA · doi:10.1001/jama.2018.11029 · Cited by 250
A computational approach to early sepsis detection
Calvert, J. S., Price, D. A., Chettipally, U. K., Barton, C. W., Feldman, M. D., Hoffman, J. L., Jay, M., & Das, R. (2016) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2016.05.003 · Cited by 244 · cited 1×
“95, which was similar to pre-implementation studies.(55,59,60) In regard to pre-deployment silent or shadow trials evaluating algorithm performance against conventional clinical judgment in a live-data environment (D22),(33) 3 groups (A, D, E) conducted such trials for 6, 3 and an unknown number of months, respectivel…”
Early Prediction of Sepsis in the ICU Using Machine Learning: A Systematic Review
Moor, M., Rieck, B., Horn, M., Jutzeler, C. R., & Borgwardt, K. (2021) · Frontiers in Medicine · doi:10.3389/fmed.2021.607952 · Cited by 231
Evaluation framework to guide implementation of AI systems into healthcare settings
Reddy, S., Rogers, W., Makinen, V.-P., Coiera, E., Brown, P., Wenzel, M., Weicken, E., Ansari, S., Mathur, P., Casey, A., et al. (2021) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2021-100444 · Cited by 230
Gama, F., Tyskbo, D., Nygren, J., Barlow, J., Reed, J., & Svedberg, P. (2022) · Journal of Medical Internet Research · doi:10.2196/32215 · Cited by 216
Henry, K. E., Kornfield, R., Sridharan, A., Linton, R. C., Groh, C., Wang, T., Wu, A., Mutlu, B., & Saria, S. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00597-7 · Cited by 207 · cited 1×
“Group (E) utilized an improvement cycle to decide on the alert threshold at each local implementation site in improving clinician adoption.(47,59) Related to the timing of alerts (D7), decisions about what actions clinicians should take for alerts involving patients showing no symptoms or signs of sepsis proved proble…”
Method"The human body is a black box": supporting clinical decision-making with deep learning
Sendak, M., Elish, M. C., Gao, M., Futoma, J., Ratliff, W., Nichols, M., Bedoya, A., Balu, S., & O'Brien, C. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372827 · Cited by 196 · cited 1×
“Decisions had to be made (D9) about how to access the data: direct from the Electronic Health Record (EHR), which could entail partnering with the vendor, or indirectly from a real-time data warehouse or various feeder systems.(43) Similarly, whether to develop and implement the MLA in-house or use an external vendor…”
BackgroundIntegrating a Machine Learning System Into Clinical Workflows: Qualitative Study
Sandhu, S., Lin, A. L., Brajer, N., Sperling, J., Ratliff, W., Bedoya, A. D., Balu, S., O'Brien, C., & Sendak, M. P. (2020) · Journal of Medical Internet Research · doi:10.2196/22421 · Cited by 168 · cited 4×
“Five (A, C, E, F, G) groups chose the former, whereas Sandhu et al found physicians preferred the latter, having a nurse contact them directly, often in-person, rather than by means of EHR-generated alerts which imposed greater cognitive load and interruptions.(42) However, the same physicians still saw nurse contacts…”
BackgroundInfluentialFernando, S. M., Rochwerg, B. N., & SEELY, A. J. E. (2018) · Canadian Medical Association Journal · doi:10.1503/cmaj.170149 · Cited by 163 · cited 1×
“Sepsis has been defined as a “life-threatening organ dysfunction caused by a dysregulated host response to infection”.(2,3) Early recognition and treatment of sepsis can reduce mortality, and rule-based surveillance systems for detecting sepsis in hospital settings can improve outcomes.”
BackgroundGinestra, J. C., Giannini, H. M., Schweickert, W. D., Meadows, L., Lynch, M. J., Pavan, K., Chivers, C. J., Draugelis, M., Donnelly, P. J., Fuchs, B. D., et al. (2019) · Critical Care Medicine · doi:10.1097/ccm.0000000000003803 · Cited by 159 · cited 3×
“Predicting onset of sepsis as early as possible involved trade-offs between: (1) alerts that were too early, where clinicians may not have known what to do, and therefore dismissed the alerts(33,41,42,44,47,50,53); and (2) alerts that were too late for patients for whom clinicians already suspected sepsis and had init…”
BackgroundMcCoy, A., & Das, R. (2017) · BMJ Open Quality · doi:10.1136/bmjoq-2017-000158 · Cited by 148 · cited 5×
“Group (B) supported multiple alerts for the same patient, but incorporated a snooze feature to suppress alerts within 6 h of the first alert.(59,61) Whether to include more information about what caused alerts, versus just firing alerts alone (D20) had implications as to how the algorithm was trained.”
BackgroundMethodInfluentialHenry, K. E., Adams, R., Parent, C., Soleimani, H., Sridharan, A., Johnson, L., Hager, D. N., Cosgrove, S. E., Markowski, A., Klein, E. Y., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01895-z · Cited by 144 · cited 4×
“The decision by groups (E, F) to enable clinicians to feedback whether they thought the alert represented sepsis or something else (D19) enabled implementation teams to evaluate clinical utility, and provide feedback to clinicians about missed sepsis cases, which incentivized greater adoption.(42,48)”
BackgroundMethodInfluentialClinical applications of artificial intelligence in sepsis: A narrative review
Schinkel, M., Paranjape, K., Nannan Panday, R. S., Skyttberg, N., & Nanayakkara, P. W. B. (2019) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2019.103488 · Cited by 130 · cited 1×
“More recently, sepsis prediction algorithms employing artificial intelligence (AI), herein called machine learning algorithms (MLAs), that can detect evolving sepsis in patients earlier than rule-based methods, have proliferated.(9,10) Most MLA studies assess performance based on static training and testing data colle…”
BackgroundDesigning Medical Informatics Research and Library--Resource Projects to Increase What Is Learned
Stead, W. W., Haynes, R. B., Fuller, S., Friedman, C. P., Travis, L. E., Beck, J. R., Fenichel, C. H., Chandrasekaran, B., Buchanan, B. G., Abola, E. E., et al. (1994) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1994.95236134 · Cited by 127 · cited 1×
“The recently reported Decide-AI research reporting guidelines depict key stages of algorithm development, evaluation, and implementation,(22) (Figure 1) which, in the companion paper to this work,(21) were mapped to Stead et al’s multi-stage approach to translating medical informatics interventions from the lab to the…”
MethodMachine learning for early detection of sepsis: an internal and temporal validation study
Bedoya, A. D., Futoma, J., Clement, M. E., Corey, K., Brajer, N., Lin, A., Simons, M. G., Gao, M., Nichols, M., Balu, S., et al. (2020) · JAMIA Open · doi:10.1093/jamiaopen/ooaa006 · Cited by 109 · cited 1×
“The required level of data pipeline sophistication, including data imputation (D11) and transformation, also necessitated trade-offs between engineering effort versus model performance (D12),(41) with group (D) having to remove a data imputation pipeline because of its complexity.(40)”
BackgroundImplementation frameworks for end-to-end clinical AI: derivation of the SALIENT framework
van der Vegt, A. H., Scott, I. A., Dermawan, K., Schnetler, R. J., Kalke, V. R., & Lane, P. J. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad088 · Cited by 109
Self-citationBakken, S., & Ruland, C. M. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3085 · Cited by 92
Schwartz, J. M., George, M., Rossetti, S. C., Dykes, P. C., Minshall, S. R., Lucas, E., & Cato, K. D. (2022) · JMIR Human Factors · doi:10.2196/33960 · Cited by 88
Reducing the global burden of sepsis
Dugani, S. B., Veillard, J. H. M., & Kissoon, N. “. (2017) · Canadian Medical Association Journal · doi:10.1503/cmaj.160798 · Cited by 86 · cited 1×
“Early recognition and treatment of sepsis can reduce mortality, and rule-based surveillance systems for detecting sepsis in hospital settings can improve outcomes.(4,5) More recently, sepsis prediction algorithms employing artificial intelligence (AI), herein called machine learning algorithms (MLAs), that can detect…”
BackgroundMachine Learning in Health Care: A Critical Appraisal of Challenges and Opportunities
Sendak, M., Gao, M., Nichols, M., Lin, A., & Balu, S. (2019) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.5334/egems.287 · Cited by 84
Burdick, H., Pino, E., Gabel-Comeau, D., McCoy, A., Gu, C., Roberts, J., Le, S., Slote, J., Pellegrini, E., Green-Saxena, A., et al. (2020) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2019-100109 · Cited by 80 · cited 4×
“The prime purpose for implementing sepsis MLAs varied which in turn determined how they were trained and evaluated (D3),(43) with evaluation metrics and success criteria varying depending on whether increasing sepsis care bundle compliance,(50) providing a sepsis detection and management system,(41) reducing anti-micr…”
BackgroundMethodInfluentialAn Improved Multi-Output Gaussian Process RNN with Real-Time Validation for Early Sepsis Detection
Futoma, J., Hariharan, S., Sendak, M., Brajer, N., Clement, M. E., Bedoya, A., O’Brien, C. L., & Heller, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.05894 · Cited by 65
High-performance detection and early prediction of septic shock for alcohol-use disorder patients
Calvert, J., Desautels, T., Chettipally, U., Barton, C., Hoffman, J., Jay, M., Mao, Q., Mohamadlou, H., & Das, R. (2016) · Annals of Medicine & Surgery · doi:10.1016/j.amsu.2016.04.023 · Cited by 62
Developing the Surveillance Algorithm for Detection of Failure to Recognize and Treat Severe Sepsis
Harrison, A. M., Thongprayoon, C., Kashyap, R., Chute, C. G., Gajic, O., Pickering, B. W., & Herasevich, V. (2015) · Mayo Clinic Proceedings · doi:10.1016/j.mayocp.2014.11.014 · Cited by 59
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · European Journal of Clinical Investigation · doi:10.1111/eci.12376 · Cited by 58
Burdick, H., Pino, E., Gabel-Comeau, D., Gu, C., Roberts, J., Le, S., Slote, J., Saber, N., Pellegrini, E., Green‐Saxena, A., et al. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01284-x · Cited by 58
Implementation of an Artificial Intelligence Algorithm for sepsis detection
Gonçalves, L. S., Amaro, M. L. D. M., Romero, A. D. L. M., Schamne, F. K., Fressatto, J. L., & Bezerra, C. W. (2020) · Revista Brasileira de Enfermagem · doi:10.1590/0034-7167-2018-0421 · Cited by 53
Reed, J. E., Howe, C., Doyle, C., & Bell, D. (2019) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzy160 · Cited by 48
Wulff, A., Montag, S., Marschollek, M., & Jack, T. (2019) · Methods of Information in Medicine · doi:10.1055/s-0039-1695717 · Cited by 47
Joshi, M., Mecklai, K., Rozenblum, R., & Samal, L. (2022) · JAMIA Open · doi:10.1093/jamiaopen/ooac022 · Cited by 46 · cited 1×
“Limitations relate to the small number of empirical studies of deployed sepsis-prediction algorithms, underreporting of post-implementation performance metrics, focus on adult hospital settings, and potential publication bias from underreporting of other sepsis MLA implementation studies.(18) Although risk of bias for…”
BackgroundBrown, S. M., Jones, J., Kuttler, K. G., Keddington, R. K., Allen, T. L., & Haug, P. (2016) · BMC Emergency Medicine · doi:10.1186/s12873-016-0095-0 · Cited by 45
Faisal, M., Scally, A., Richardson, D., Beatson, K., Howes, R., Speed, K., & Mohammed, M. A. (2018) · Critical Care Medicine · doi:10.1097/ccm.0000000000002967 · Cited by 44 · cited 1×
“Importantly, the concern here is diagnosing sepsis (ie, using a diagnostic predictive algorithm) rather than predicting the likelihood of sepsis occurring in a patient before the inflammatory process begins (ie, a prognostic predictive algorithm).(66) Fourth, how algorithm predictions are presented to clinicians, and…”
BackgroundHassan, N., Slight, R., Weiand, D., Vellinga, A., Morgan, G., Aboushareb, F., & Slight, S. P. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104457 · Cited by 42 · cited 1×
“Third, the choice of outcome definition, in this case sepsis, is critical as it can directly influence algorithm performance measures, particularly specificity.(6) Definitions of sepsis varied from initial systemic inflammatory process (eg, Sepsis-1 definition)(3) to multi-organ dysfunction (eg, Sepsis-3 definition)(6…”
BackgroundAckermann, K., Baker, J., Green, M., Fullick, M., Varinli, H., Westbrook, J. I., & Li, L. (2021) · Journal of Medical Internet Research · doi:10.2196/31083 · Cited by 40
Clinical prediction rules: a systematic review of healthcare provider opinions and preferences
Kennedy, G., & Gallego, B. (2019) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2018.12.003 · Cited by 39 · cited 1×
“Fourth, how algorithm predictions are presented to clinicians, and the extent to which they are accompanied by additional information or even recommendations are key determinants of clinician acceptance.(67)”
BackgroundCost and mortality impact of an algorithm-driven sepsis prediction system
Calvert, J. S., Hoffman, J. L., Barton, C., Shimabukuro, D. W., Ries, M. H., Chettipally, U. K., Kerem, Y., Jay, M., Mataraso, S. J., & Das, R. (2017) · Journal of Medical Economics · doi:10.1080/13696998.2017.1307203 · Cited by 38 · cited 1×
“Deciding which features to input and how simple (eg, vital signs only) or complex (eg, waveform data and laboratory results) they are was seen to influence model generalizability (D6) to different care locations.(38,57,58,62) Group (B) supported different variables for different sites, claiming flexibility,(62) althou…”
BackgroundBurdick, H., Pino, E., Gabel-Comeau, D., Gu, C., Huang, H., Lynn-Palevsky, A., & Das, R. (2017) · doi:10.1101/224014 · Cited by 20 · cited 2×
“Different thresholds were chosen according to the anticipated impacts on clinical processes and clinician workload and adoption.(60,61)”
MethodScherer, J. D. S., Pereira, J. S., Debastiani, M. S., & Bica, C. G. (2022) · Revista Brasileira de Enfermagem · doi:10.1590/0034-7167-2021-0586 · Cited by 19
Evaluation of the effectiveness of an automated sepsis predictive tool on patient outcomes
Schootman, M., Wiskow, C., Loux, T., Meyer, L., Powell, S., Gandhi, A., & Lacasse, A. (2022) · Journal of Critical Care · doi:10.1016/j.jcrc.2022.154061 · Cited by 17 · cited 1×
“Only 2 trials controlled for these differences in before-after cohorts.(46,52) Implementation evaluation Eighty-five distinct metrics were identified across 26 (87%) papers, grouped into 5 evaluation categories: (1) algorithm performance; (2) algorithm adoption; (3) clinical process effects; (4) patient outcome effect…”
BackgroundTopiwala, R., Patel, K., Twigg, J., Rhule, J., & Meisenberg, B. (2019) · Critical Care Explorations · doi:10.1097/cce.0000000000000046 · Cited by 16 · cited 2×
“Similar results were seen for 2 independent implementations of the same algorithm (Group B).(34,58) Third, the choice of outcome definition, in this case sepsis, is critical as it can directly influence algorithm performance measures, particularly specificity.”
BackgroundResultLipatov, K., Daniels, C. E., Park, J. G., Elmer, J., Hanson, A. C., Madsen, B. E., Clements, C. M., Gajic, O., Pickering, B. W., & Herasevich, V. (2022) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2021.09.086 · Cited by 11 · cited 5×
“Despite group (F) reporting high adoption rates (77% to 84%) and significantly improved rates of sepsis care bundle compliance, post-implementation the MLA specificity dropped markedly, from 96% to 80%, and there was no significant change in all-cause mortality.(50)”
BackgroundMethodResultInfluential
Cited by
Nair, M., Svedberg, P., Larsson, I., & Nygren, J. M. (2024) · PLOS ONE · doi:10.1371/journal.pone.0305949 · Cited by 179
A Nationwide Network of Health AI Assurance Laboratories
Shah, N. H., Halamka, J. D., Saria, S., Pencina, M., Tazbaz, T., Tripathi, M., Callahan, A., Hildahl, H., & Anderson, B. (2024) · JAMA · doi:10.1001/jama.2023.26930 · Cited by 114
Implementation frameworks for end-to-end clinical AI: derivation of the SALIENT framework
van der Vegt, A. H., Scott, I. A., Dermawan, K., Schnetler, R. J., Kalke, V. R., & Lane, P. J. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad088 · Cited by 109
Self-citationLi, F., Wang, S., Gao, Z., Qing, M., Pan, S. L., Liu, Y., & Hu, C. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2024.1510792 · Cited by 119
Achieving large-scale clinician adoption of AI-enabled decision support
Scott, I. A., van der Vegt, A., Lane, P., McPhail, S., & Magrabi, F. (2024) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2023-100971 · Cited by 65
Self-citation
Metadata
- Authors
- van der Vegt, A. H., Scott, I. A., Dermawan, K., Schnetler, R. J., Kalke, V. R., & Lane, P. J.
- Affiliations
- Queensland Digital Health Centre, The University of Queensland , Brisbane, Queensland, Australia
- Princess Alexandra Hospital (AU)
- The University of Queensland (AU)
- Queensland Health (AU)
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2023-05-12
- Volume
- 30
- Issue
- 7
- Pages
- 1349-1361
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC10280361
- OpenAlex
- W4376270103
- Cited by
- 72
- References
- 75
- Citations by year
- 2026: 24 · 2025: 25 · 2024: 11 · 2023: 12
- FWCI
- 2.80
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Queensland Government
- Advanced Queensland Industry Research Fellowship
- Topics
- Health Sciences › Medicine › Critical Care and Intensive Care Medicine › Sepsis Diagnosis and Treatment
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- sepsis prediction, machine learning algorithms, implementation barriers and facilitators, sepsis mortality, algorithm deployment
- MeSH terms
- Machine Learning, Adult, Algorithms, Artificial Intelligence, Humans, Sepsis, Empirical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.