The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Implementation approaches and barriers for rule-based and machine learning-based sepsis risk prediction tools: a qualitative study
Open accessAbstract
Objective: Many options are currently available for sepsis surveillance clinical decision support (CDS) from electronic medical record (EMR) vendors, third party, and homegrown models drawing on rule-based (RB) and machine learning (ML) algorithms. This study explores sepsis CDS implementation from the perspective of implementation leads by describing the motivations, tool choices, and implementation experiences of a diverse group of implementers. Materials and Methods: Semi-structured interviews were conducted with and a questionnaire was administered to 21 hospital leaders overseeing CDS implementation at 15 US medical centers. Participants were recruited via convenience sampling. Responses were coded by 2 coders with consensus approach and inductively analyzed for themes. Results: Use of sepsis CDS is motivated in part by quality metrics for sepsis patients. Choice of tool is driven by ease of integration, customization capability, and perceived predictive potential. Implementation processes for these CDS tools are complex, time-consuming, interdisciplinary undertakings resulting in heterogeneous choice of tools and workflow integration. To improve clinician acceptance, implementers addressed both optimization of the alerts as well as clinician understanding and buy in. More distrust and confusion was reported for ML models, as compared to RB models. Respondents described a variety of approaches to overcome implementation barriers; these approaches related to alert firing, content, integration, and buy-in. Discussion: While there are shared socio-technical challenges of implementing CDS for both RB and ML models, attention to user education, support, expectation management, and dissemination of effective practices may improve feasibility and effectiveness of ML models in quality improvement efforts. Conclusion: Further implementation science research is needed to determine real world efficacy of these tools. Clinician acceptance is a significant barrier to sepsis CDS implementation. Successful implementation of less clinically intuitive ML models may require additional attention to user confusion and distrust.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Limitations relate to the small number of empirical studies of deployed sepsis-prediction algorithms, under-reporting of post-implementation performance metrics, focus on adult hospital settings, and potential publication bias from under-reporting of other sepsis MLA implementation studies.18”
From Deployment of machine learning algorithms to predict sepsis: systematic review and application of the SALIENT clinical AI implementation framework (2023) · cites this paper as [18]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Implementation barriers for sepsis AI cluster around trust and interpretability, workflow integration, technical readiness and generalizability, governance and safety, leadership alignment, and the resources required for sustainment (Table 4) [10,33,34,46,48,77,78].”
From Artificial Intelligence to Facilitate SEP-1 Measure Compliance and Fluid Management in Sepsis (2026) · cites this paper as [77]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Other studies have examined implementation issues when integrating machine learning models into clinical practice and reported numerous socio-technical factors such as the unfamiliarity of AI among clinicians and lack of AI education, trust in the accuracy and utility of predictive models, and IT support among other issues that affected this process (Joshi et al., 2022;Sandhu et al., 2020).”
From The application and use of artificial intelligence in cancer nursing: A systematic review (2024) · cites this paper as Joshi et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“Their study found that implementation was complex and resource-intensive, with major barriers including alert optimization, workflow integration, and especially clinician acceptance. There were greater distrust, confusion, and interpretability challenges for machine learning models, highlighting the need for improved user education, transparency, and implementation strategies.”
From Artificial Intelligence to Facilitate SEP-1 Measure Compliance and Fluid Management in Sepsis (2026) · cites this paper as [77]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
S., N. L. (2017) · doi:10.1177/1609406917733847 · Cited by 14,051
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
Saturation in qualitative research: exploring its conceptualization and operationalization
Saunders, B., Sim, J., Kingstone, T., Baker, S., Waterfield, J., Bartlam, B., Burroughs, H., & Jinks, C. (2018) · Quality & Quantity · doi:10.1007/s11135-017-0574-8 · Cited by 12,033
Seymour, C. W., Liu, V. X., Iwashyna, T. J., Brunkhorst, F. M., Rea, T. D., Scherag, A., Rubenfeld, G., Kahn, J. M., Shankar-Hari, M., Singer, M., et al. (2016) · JAMA · doi:10.1001/jama.2016.0288 · Cited by 3,939 · cited 1×
“Some prospective studies have shown significant reduction in time to blood culture and antibiotics without significant reductions in mortality.(9,12,16) While others have largely been negative, including 2 randomized controlled trials.”
BackgroundExplainable machine-learning predictions for the prevention of hypoxaemia during surgery
Lundberg, S. M., Nair, B., Vavilala, M. S., Horibe, M., Eisses, M. J., Adams, T., Liston, D. E., Low, D. K.-W., Newman, S.-F., Kim, J., et al. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0304-0 · Cited by 1,967
Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009-2014.
Rhee, C., Dantes, R., Epstein, L., Murphy, D. J., Seymour, C. W., Iwashyna, T. J., Kadri, S. S., Angus, D. C., Danner, R. L., Fiore, A. E., et al. (2017) · JAMA · doi:10.1001/jama.2017.13836 · Cited by 1,884
Wong, A., Otles, E., Donnelly, J. P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., et al. (2021) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2021.2626 · Cited by 995
Fleuren, L. M., Klausch, T. L. T., Zwager, C. L., Schoonmade, L. J., Guo, T., Roggeveen, L. F., Swart, E. L., Girbes, A. R. J., Thoral, P., Ercole, A., et al. (2020) · Intensive Care Medicine · doi:10.1007/s00134-019-05872-y · Cited by 755 · cited 1×
“This approach was chosen as qualitative methods can provide “rich descriptions of complex phenomena” typical of healthcare organization and IT implementation.(20) A convenience sample was developed of hospital leaders from various hospitals nationwide with criteria intended to include implementors of CDS tools from 3…”
MethodIntroducing artificial intelligence training in medical education
Paranjape, K., Schinkel, M., Nannan Panday, R., Car, J., & Nanayakkara, P. (2019) · JMIR Medical Education · doi:10.2196/16048 · Cited by 634
Desautels, T., Calvert, J., Hoffman, J., Jay, M., Kerem, Y., Shieh, L., Shimabukuro, D., Chettipally, U., Feldman, M. D., Barton, C., et al. (2016) · JMIR Medical Informatics · doi:10.2196/medinform.5909 · Cited by 509
Cited by
Hogg, H. D. J., Al-Zubaidy, M., Technology Enhanced Macular Services Study Reference Group, Talks, J., Denniston, A. K., Kelly, C. J., Malawana, J., Papoutsi, C., Teare, M. D., Keane, P. A., et al. (2023) · Journal of Medical Internet Research · doi:10.2196/39742 · Cited by 163
Rahimi, A. K., Pienaar, O., Ghadimi, M., Canfell, O. J., Pole, J. D., Shrapnel, S., van der Vegt, A. H., & Sullivan, C. M. (2024) · Journal of Medical Internet Research · doi:10.2196/49655 · Cited by 164
Trust in and Acceptance of Artificial Intelligence Applications in Medicine: Mixed Methods Study
Shevtsova, D., Ahmed, A., Boot, I. W. A., Sanges, C., Hudecek, M., Jacobs, J. J. L., Hort, S., & Vrijhoef, H. J. M. (2024) · JMIR Human Factors · doi:10.2196/47031 · Cited by 124
Implementation frameworks for end-to-end clinical AI: derivation of the SALIENT framework
van der Vegt, A. H., Scott, I. A., Dermawan, K., Schnetler, R. J., Kalke, V. R., & Lane, P. J. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad088 · Cited by 109
Finkelstein, J., Gabriel, A. S., Schmer, S., Truong, T., & Dunn, A. S. (2024) · Journal of Medical Systems · doi:10.1007/s10916-024-02104-9 · Cited by 76
Metadata
- Authors
- Joshi, M., Mecklai, K., Rozenblum, R., & Samal, L.
- Affiliations
- Stanford Medicine (US)
- Harvard University (US)
- Brigham and Women's Hospital (US)
- Corresponding author
- Joshi, Mugdha
- Venue
- JAMIA Open
- Publisher
- Oxford University Press (OUP)
- Published
- 2022-04-06
- Volume
- 5
- Issue
- 2
- Pages
- ooac022-ooac022
- ISSN
- 2574-2531 (ISSN-L)
- PMC
- PMC9030109
- OpenAlex
- W4223913070
- Cited by
- 46
- References
- 40
- Citations by year
- 2026: 9 · 2025: 17 · 2024: 12 · 2023: 7 · 2022: 1
- FWCI
- 6.93
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- JAMIA Open (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Critical Care and Intensive Care Medicine › Sepsis Diagnosis and Treatment
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- sepsis risk prediction, clinical decision support, rule-based models, machine learning models, implementation barriers, clinician acceptance, electronic health records, sepsis surveillance, trust in AI, quality improvement, socio-technical challenges
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Implementation approaches and barriers for rule-based and machine learning-based sepsis risk prediction tools: a qualitative study. https://app.cochat.ai/papers/10.1093/jamiaopen/ooac022
- The paper
- Joshi, M., Mecklai, K., Rozenblum, R., & Samal, L. (2022). Implementation approaches and barriers for rule-based and machine learning-based sepsis risk prediction tools: a qualitative study. JAMIA Open. https://doi.org/10.1093/jamiaopen/ooac022
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.