The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index; 25 of the 34 sources its claims cite were checked for support.
Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study
Open accessPublished version of a preprint
Abstract
Background: Machine learning models have the potential to improve diagnostic accuracy and management of acute conditions. Despite growing efforts to evaluate and validate such models, little is known about how to best translate and implement these products as part of routine clinical care. Objective: This study aims to explore the factors influencing the integration of a machine learning sepsis early warning system (Sepsis Watch) into clinical workflows. Methods: We conducted semistructured interviews with 15 frontline emergency department physicians and rapid response team nurses who participated in the Sepsis Watch quality improvement initiative. Interviews were audio recorded and transcribed. We used a modified grounded theory approach to identify key themes and analyze qualitative data. Results: A total of 3 dominant themes emerged: perceived utility and trust, implementation of Sepsis Watch processes, and workforce considerations. Participants described their unfamiliarity with machine learning models. As a result, clinician trust was influenced by the perceived accuracy and utility of the model from personal program experience. Implementation of Sepsis Watch was facilitated by the easy-to-use tablet application and communication strategies that were developed by nurses to share model outputs with physicians. Barriers included the flow of information among clinicians and gaps in knowledge about the model itself and broader workflow processes. Conclusions: This study generated insights into how frontline clinicians perceived machine learning models and the barriers to integrating them into clinical workflows. These findings can inform future efforts to implement machine learning interventions in real-world settings and maximize the adoption of these interventions.
Overview
CoChat Summary
TL;DR
This qualitative study interviewed 15 frontline clinicians to identify factors affecting integration of a machine learning sepsis early warning system, Sepsis Watch, into emergency department workflows. It found three dominant themes—perceived utility and trust, implementation processes, and workforce considerations—with clinician trust shaped by personal experience, model transparency limits, and human communication.
Key findings
- Three dominant themes emerged: perceived utility and trust, implementation of Sepsis Watch processes, and workforce considerations, with 10 corresponding subthemes.
- RRT nurses, the primary users, spoke positively about Sepsis Watch accuracy and the easy-to-use color-coded application; ED physicians reported less trust, citing algorithm components such as blood cultures being too heavily weighted, missed sepsis cases, and false positives.
- Implementation was facilitated by the tablet/web application and nurse-developed communication strategies, but barriers included information flow challenges, workflow interruptions, and gaps in knowledge about the black box model.
- The program created a new Sepsis Watch Nurse role; nurses valued it and identified good clinical judgment, sepsis knowledge, critical care experience, and communication skills as important, while strong computer skills were not considered necessary.
Methods
- Semistructured interviews were conducted with 15 participants—7 ED attending physicians and 8 RRT nurses—from January 2019 to April 2019, starting 4 months into Sepsis Watch implementation at Duke University Hospital; interviews averaged 35 minutes and continued until thematic saturation.
- A modified grounded theory approach was used with three coding phases, and data were analyzed in NVivo 12; interview guides were informed by the situational awareness model and piloted among 3 clinicians.
- Sepsis Watch used a deep learning model trained on 42,000 inpatient encounters and 32 million data points to predict likelihood of meeting a sepsis phenotype within the subsequent 4 hours, with outputs displayed as black, red, orange, or yellow cards.
Limitations
- The paper states that findings should be generalized with caution to settings differing in organizational structure, capacity, and professional norms; the study focuses on workflow integration rather than the multi-year planning and stakeholder engagement process; only frontline clinicians were interviewed, excluding leadership, administrators, data scientists, and patients; and including clinicians involved in program development may have biased findings positively.
- The paper also states that further studies with larger samples and survey approaches are needed to quantify adoption factors and examine variation by clinician characteristics.
- Evident from the text: this was a single-site academic hospital study using convenience sampling and qualitative self-report, and it did not measure patient outcomes.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
15 Valid · 1 Not valid · 3 Not checked · 87 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Similarly, only frontline clinicians were interviewed despite the large number of stakeholders involved in project development and maintenance (eg, organizational leadership, hospital administrators, data scientists) or impacted directly by the program (ie, patients; [45,46]).”
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Discussion > Limitations
What the sources say
- Disputedcites [45] sourceExperimental — accuracy not yet measured
The Results (page unavailable) say pediatric patients and their guardians completed an evaluation questionnaire; they were not excluded from the study’s feedback. The paper reports a randomized trial with patient questionnaires, not interviews limited to frontline clinicians. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
- Disputedcites [46] sourceExperimental — accuracy not yet measured
The Methods (Protocol) and Results (Participant follow-up) say patients were contacted and interviewed about their preferred screening model. The Discussion calls for future research into clinician and organizational stakeholder perspectives, rather than reporting interviews with frontline clinicians. The supplied passages do not give page numbers. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Unfortunately, many sepsis CDS systems fail to improve outcomes because of poor diagnostic accuracy and program implementation [23-25].”
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Introduction
What the sources say
- Partly supportedcites [23] sourceExperimental — accuracy not yet measured
The Abstract conclusions and Discussion report that the implemented NEWS alert had no appreciable impact on clinical outcomes, performed poorly, and was generally ignored by nursing staff. The paper concerns one early-warning system for patient deterioration, not many sepsis-specific CDS systems, so it does not support that scope of the claim. (LLM, conf=0.88)
- Supportedcites [24] source
The Development and Implementation section reports that rigorous trials of sepsis alerts, including a system with CDS, did not improve clinically significant outcomes. The Barriers section explains that inaccurate alerts can render a system ineffective and that poorly planned implementation can undermine its impact on outcomes. Together, these support the citing sentence’s broad summary. (LLM, conf=0.86)
- Partly supportedcites [25] sourceExperimental — accuracy not yet measured
The Background reports that some sepsis-alert studies found no improvement in outcomes and says inconsistent findings may reflect differences in alert algorithms and downstream care. The Discussion identifies low alert specificity and failures in subsequent clinical action as potential barriers. It does not establish that poor diagnostic accuracy and implementation caused many sepsis CDS systems to fail, as the claim states. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“As a result, little evidence exists on the optimal integration of these sepsis models and other machine learning models into clinical workflows [9].”
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Introduction
What the sources say
- Supportedcites [9] source
In “Machine Learning Applied to Clinical Decision Support Tasks” (page unavailable), the paper says recommendations for design and implementation are only beginning to emerge and that there is no single best workflow. In “2. Evaluate and Validate” (page unavailable), it describes clinical integration as difficult and reports only one small randomized trial among the products reviewed. These passages support the claim that evidence on optimal workflow integration, including for sepsis models, is limited. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Clinician preferences were incorporated to optimize the ease of use and utility for end users [39].”
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Discussion
What the sources say
- Partly supportedcites [39] sourceExperimental — accuracy not yet measured
The Discussion (§4.1; page not provided) recommends designing health IT to be useful and not difficult to use, supporting the stated design goal. It does not say that clinician preferences were incorporated into Sepsis Watch or establish that this design step occurred. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Despite the rise in competing models and products for early sepsis detection [5], few machine learning models have been implemented as part of clinical care [6-8].”
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Introduction
What the sources say
- Supportedcites [5] source
The Conclusion states that clinical implementation of sepsis-prediction models is currently scarce. The Discussion’s “Clinical model performance” section also identifies only three interventional studies, supporting the claim that few models have been implemented in clinical care. (LLM, conf=0.94)
- Partly supportedcites [6] sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) says that sepsis-prediction machine learning algorithms had performed well on retrospective data, but clinical implementation was largely confined to ICU populations; it gives limited examples and says large-scale non-ICU implementation had not been reported. This supports the point for sepsis-prediction models, but not the broader claim about machine learning models generally. (LLM, conf=0.82)
- Supportedcites [7] source
The Background (page not provided) says prospective clinical studies of EHR tools for sepsis have most often used rules-based approaches, while the machine-learning studies it lists were retrospective or pilots. The Conclusion describes this paper's implementation of a machine-learning sepsis algorithm in clinical care. Together, these support the claim in its sepsis-detection context. (LLM, conf=0.86)
- Partly supportedcites [8] sourceExperimental — accuracy not yet measured
The Abstract Introduction says that few electronic sepsis-monitoring methods offer predictive capabilities and that no severe-sepsis prediction system had previously been validated in a randomised study. The Conclusion describes this paper's own clinical trial of a machine-learning sepsis algorithm. These passages support limited clinical evaluation of predictive sepsis systems, but do not establish that few machine-learning models generally have been implemented in clinical care. (LLM, conf=0.79)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
47 more claims · 3 with their cited sources checked
Create a free account to see all 106 claims and every source verdict.
- It may state two separate claims as one. 27
- Probably not a substantial claim. 30
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Other studies have examined implementation issues when integrating machine learning models into clinical practice and reported numerous socio-technical factors such as the unfamiliarity of AI among clinicians and lack of AI education, trust in the accuracy and utility of predictive models, and IT support among other issues that affected this process (Joshi et al., 2022;Sandhu et al., 2020).”
From The application and use of artificial intelligence in cancer nursing: A systematic review (2024) · cites this paper as Sandhu et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Implementation studies similarly emphasize that pairing alerts with escalation protocols is essential to translate prediction into improved process measures and outcomes.”
From Artificial Intelligence to Facilitate SEP-1 Measure Compliance and Fluid Management in Sepsis (2026) · cites this paper as [48]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Implementation barriers for sepsis AI cluster around trust and interpretability, workflow integration, technical readiness and generalizability, governance and safety, leadership alignment, and the resources required for sustainment (Table 4) [10,33,34,46,48,77,78].”
From Artificial Intelligence to Facilitate SEP-1 Measure Compliance and Fluid Management in Sepsis (2026) · cites this paper as [48]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI-enabled early warning systems and EHR-integrated clinical decision support can improve surveillance, risk stratification, and the timeliness of key sepsis processes required by SEP-1 [45,48,49].”
From Artificial Intelligence to Facilitate SEP-1 Measure Compliance and Fluid Management in Sepsis (2026) · cites this paper as [48]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“An early warning system, Sepsis Watch, can pair model output with rapid response team nurse review and escalation pathways, reframing AI output as a trigger for needed responses rather than an interruptive alert, which at times may be ignored.”
From Artificial Intelligence to Facilitate SEP-1 Measure Compliance and Fluid Management in Sepsis (2026) · cites this paper as [48]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Methods > Study Design
“we recruited 7 ED physicians (n=7; Table 1) and 8 RRT nurses (n=8; Table 2)”
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Methods > Study Design
“Table: Table 1 Characteristics of rapid response team nurse participants.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Results > Implementation of Sepsis Watch Processes
“RRT physicians reported that at the start of the program, the ED physicians were unfamiliar with the purpose of the program, the role of the RRT nurse, and the flow of the call.”
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Methods > Study Design
“we recruited 7 ED physicians (n=7; Table 1) and 8 RRT nurses (n=8; Table 2)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Methods > Study Design
“| ED attending 7 | 6 years | 3 years | No |”
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Methods > Study Design
“| Participant | Experience as attending physician at pilot site | Experience as attending physician | Was the participant involved in program development? |”
Citations
Cites
User Acceptance of Information Technology: Toward A Unified View1
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003) · MIS Quarterly · doi:10.2307/30036540 · Cited by 45,035
(2014) · Cited by 15,747
Research Design: Qualitative, Quantitative, and Mixed Methods Approaches
ภู่พันธ์, ศ. (2018) · วารสารวิธีวิทยาการวิจัย (Journal of Research Methodology: JRM) · Cited by 13,071
Saturation in qualitative research: exploring its conceptualization and operationalization
Saunders, B., Sim, J., Kingstone, T., Baker, S., Waterfield, J., Bartlam, B., Burroughs, H., & Jinks, C. (2018) · Quality & Quantity · doi:10.1007/s11135-017-0574-8 · Cited by 12,033 · cited 1×
“In the qualitative analytic process, data collection was terminated when no new themes or insights emerged (ie, thematic saturation) across both participant groups [36].”
BackgroundToward a Theory of Situation Awareness in Dynamic Systems
Endsley, M. R. (1995) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872095779049543 · Cited by 8,875
Kumar, A., Roberts, D., Wood, K. E., Light, B., Parrillo, J. E., Sharma, S., Suppes, R., Feinstein, D., Zanotti, S., Taiberg, L., et al. (2006) · Critical Care Medicine · doi:10.1097/01.ccm.0000217961.75225.e9 · Cited by 6,109
The Technology Acceptance Model: Its past and its future in health care
Holden, R. J., & Karsh, B.-T. (2010) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2009.07.002 · Cited by 2,669
Time to treatment and mortality during mandated emergency care for sepsis.
Seymour, C. W., Gesten, F., Prescott, H. C., Friedrich, M. E., Iwashyna, T. J., Phillips, G. S., Lemeshow, S., Osborn, T., Terry, K. M., & Levy, M. M. (2017) · New England Journal of Medicine · doi:10.1056/nejmoa1703058 · Cited by 2,225
Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009-2014.
Rhee, C., Dantes, R., Epstein, L., Murphy, D. J., Seymour, C. W., Iwashyna, T. J., Kadri, S. S., Angus, D. C., Danner, R. L., Fiore, A. E., et al. (2017) · JAMA · doi:10.1001/jama.2017.13836 · Cited by 1,884
Artificial Intelligence and Human Trust in Healthcare: Focus on Clinicians
Asan, O., Bayrak, A. E., & Choudhury, A. (2020) · Journal of Medical Internet Research · doi:10.2196/15154 · Cited by 825
Research works printed but not linked: 2
- TorioCMMooreBJNational Inpatient Hospital Costs: The Most Expensive Conditions by Payer, 2013: Statistical Brief #204Healthcare Cost and Utilization Project (HCUP)2020-11-13https://europepmc.org/abstract/med/27359025
- Futoma, J, arXivstat
Cited by
Lambert, S. I., Madi, M., Sopka, S., Lenes, A., Stange, H., Buszello, C.-P., & Stephan, A. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00852-5 · Cited by 404
Artificial Intelligence -based technologies in nursing: A scoping literature review of the evidence
von Gerich, H., Moen, H., Block, L. J., Chu, C. H., DeForest, H., Hobensack, M., Michalowski, M., Mitchell, J., Nibber, R., Olalia, M. A., et al. (2022) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2021.104153 · Cited by 360
An interdisciplinary review of AI and HRM: Challenges and future directions
Pan, Y., & Froese, F. J. (2023) · Human Resource Management Review · doi:10.1016/j.hrmr.2022.100924 · Cited by 214
Rony, M. K. K., Kayesh, I., Bala, S. D., Akter, F., & Parvin, M. R. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e25718 · Cited by 190
Nair, M., Svedberg, P., Larsson, I., & Nygren, J. M. (2024) · PLOS ONE · doi:10.1371/journal.pone.0305949 · Cited by 179
Metadata
- Authors
- Sandhu, S., Lin, A. L., Brajer, N., Sperling, J., Ratliff, W., Bedoya, A. D., Balu, S., O'Brien, C., & Sendak, M. P.
- Affiliations
- Duke University (US)
- Corresponding author
- Sandhu, Sahil
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2020-11-19
- Volume
- 22
- Issue
- 11
- Pages
- e22421
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC7714645
- OpenAlex
- W3094937636
- Cited by
- 168
- References
- 50
- Citations by year
- 2026: 37 · 2025: 35 · 2024: 38 · 2023: 25 · 2022: 23 · 2021: 10
- FWCI
- 4.29
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Critical Care and Intensive Care Medicine › Sepsis Diagnosis and Treatment
- Keywords
- machine learning integration, clinical workflow, implementation barriers, clinician trust, perceived usefulness, emergency physicians, rapid response teams, nurse-physician communication, clinical decision support, technology adoption
- MeSH terms
- Machine Learning, Humans, Qualitative Research, Workflow
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study. https://app.cochat.ai/papers/10.2196/22421
- The paper
- Sandhu, S., Lin, A. L., Brajer, N., Sperling, J., Ratliff, W., Bedoya, A. D., Balu, S., O'Brien, C., & Sendak, M. P. (2020). Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study. Journal of Medical Internet Research. https://doi.org/10.2196/22421
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
