Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study
Open accessPublished version of a preprint
The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index; 25 of the 34 sources its claims cite were checked for support.
Abstract
Background: Machine learning models have the potential to improve diagnostic accuracy and management of acute conditions. Despite growing efforts to evaluate and validate such models, little is known about how to best translate and implement these products as part of routine clinical care. Objective: This study aims to explore the factors influencing the integration of a machine learning sepsis early warning system (Sepsis Watch) into clinical workflows. Methods: We conducted semistructured interviews with 15 frontline emergency department physicians and rapid response team nurses who participated in the Sepsis Watch quality improvement initiative. Interviews were audio recorded and transcribed. We used a modified grounded theory approach to identify key themes and analyze qualitative data. Results: A total of 3 dominant themes emerged: perceived utility and trust, implementation of Sepsis Watch processes, and workforce considerations. Participants described their unfamiliarity with machine learning models. As a result, clinician trust was influenced by the perceived accuracy and utility of the model from personal program experience. Implementation of Sepsis Watch was facilitated by the easy-to-use tablet application and communication strategies that were developed by nurses to share model outputs with physicians. Barriers included the flow of information among clinicians and gaps in knowledge about the model itself and broader workflow processes. Conclusions: This study generated insights into how frontline clinicians perceived machine learning models and the barriers to integrating them into clinical workflows. These findings can inform future efforts to implement machine learning interventions in real-world settings and maximize the adoption of these interventions.
Overview
CoChat Summary
TL;DR
This qualitative study interviewed 15 frontline clinicians to identify factors affecting integration of a machine learning sepsis early warning system, Sepsis Watch, into emergency department workflows. It found three dominant themes—perceived utility and trust, implementation processes, and workforce considerations—with clinician trust shaped by personal experience, model transparency limits, and human communication.
Key findings
- Three dominant themes emerged: perceived utility and trust, implementation of Sepsis Watch processes, and workforce considerations, with 10 corresponding subthemes.
- RRT nurses, the primary users, spoke positively about Sepsis Watch accuracy and the easy-to-use color-coded application; ED physicians reported less trust, citing algorithm components such as blood cultures being too heavily weighted, missed sepsis cases, and false positives.
- Implementation was facilitated by the tablet/web application and nurse-developed communication strategies, but barriers included information flow challenges, workflow interruptions, and gaps in knowledge about the black box model.
- The program created a new Sepsis Watch Nurse role; nurses valued it and identified good clinical judgment, sepsis knowledge, critical care experience, and communication skills as important, while strong computer skills were not considered necessary.
Methods
- Semistructured interviews were conducted with 15 participants—7 ED attending physicians and 8 RRT nurses—from January 2019 to April 2019, starting 4 months into Sepsis Watch implementation at Duke University Hospital; interviews averaged 35 minutes and continued until thematic saturation.
- A modified grounded theory approach was used with three coding phases, and data were analyzed in NVivo 12; interview guides were informed by the situational awareness model and piloted among 3 clinicians.
- Sepsis Watch used a deep learning model trained on 42,000 inpatient encounters and 32 million data points to predict likelihood of meeting a sepsis phenotype within the subsequent 4 hours, with outputs displayed as black, red, orange, or yellow cards.
Limitations
- The paper states that findings should be generalized with caution to settings differing in organizational structure, capacity, and professional norms; the study focuses on workflow integration rather than the multi-year planning and stakeholder engagement process; only frontline clinicians were interviewed, excluding leadership, administrators, data scientists, and patients; and including clinicians involved in program development may have biased findings positively.
- The paper also states that further studies with larger samples and survey approaches are needed to quantify adoption factors and examine variation by clinician characteristics.
- Evident from the text: this was a single-site academic hospital study using convenience sampling and qualitative self-report, and it did not measure patient outcomes.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
15 Valid · 1 Not valid · 3 Not checked · 87 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Similarly, only frontline clinicians were interviewed despite the large number of stakeholders involved in project development and maintenance (eg, organizational leadership, hospital administrators, data scientists) or impacted directly by the program (ie, patients; [45,46]).”
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Discussion > Limitations
What the sources say
- Disputedcites [45] sourceExperimental — accuracy not yet measured
The Results (page unavailable) say pediatric patients and their guardians completed an evaluation questionnaire; they were not excluded from the study’s feedback. The paper reports a randomized trial with patient questionnaires, not interviews limited to frontline clinicians. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
- Disputedcites [46] sourceExperimental — accuracy not yet measured
The Methods (Protocol) and Results (Participant follow-up) say patients were contacted and interviewed about their preferred screening model. The Discussion calls for future research into clinician and organizational stakeholder perspectives, rather than reporting interviews with frontline clinicians. The supplied passages do not give page numbers. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Unfortunately, many sepsis CDS systems fail to improve outcomes because of poor diagnostic accuracy and program implementation [23-25].”
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Introduction
What the sources say
- Partly supportedcites [23] sourceExperimental — accuracy not yet measured
The Abstract conclusions and Discussion report that the implemented NEWS alert had no appreciable impact on clinical outcomes, performed poorly, and was generally ignored by nursing staff. The paper concerns one early-warning system for patient deterioration, not many sepsis-specific CDS systems, so it does not support that scope of the claim. (LLM, conf=0.88)
- Supportedcites [24] source
The Development and Implementation section reports that rigorous trials of sepsis alerts, including a system with CDS, did not improve clinically significant outcomes. The Barriers section explains that inaccurate alerts can render a system ineffective and that poorly planned implementation can undermine its impact on outcomes. Together, these support the citing sentence’s broad summary. (LLM, conf=0.86)
- Partly supportedcites [25] sourceExperimental — accuracy not yet measured
The Background reports that some sepsis-alert studies found no improvement in outcomes and says inconsistent findings may reflect differences in alert algorithms and downstream care. The Discussion identifies low alert specificity and failures in subsequent clinical action as potential barriers. It does not establish that poor diagnostic accuracy and implementation caused many sepsis CDS systems to fail, as the claim states. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“As a result, little evidence exists on the optimal integration of these sepsis models and other machine learning models into clinical workflows [9].”
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Introduction
What the sources say
- Supportedcites [9] source
In “Machine Learning Applied to Clinical Decision Support Tasks” (page unavailable), the paper says recommendations for design and implementation are only beginning to emerge and that there is no single best workflow. In “2. Evaluate and Validate” (page unavailable), it describes clinical integration as difficult and reports only one small randomized trial among the products reviewed. These passages support the claim that evidence on optimal workflow integration, including for sepsis models, is limited. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Clinician preferences were incorporated to optimize the ease of use and utility for end users [39].”
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Discussion
What the sources say
- Partly supportedcites [39] sourceExperimental — accuracy not yet measured
The Discussion (§4.1; page not provided) recommends designing health IT to be useful and not difficult to use, supporting the stated design goal. It does not say that clinician preferences were incorporated into Sepsis Watch or establish that this design step occurred. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Despite the rise in competing models and products for early sepsis detection [5], few machine learning models have been implemented as part of clinical care [6-8].”
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Introduction
What the sources say
- Supportedcites [5] source
The Conclusion states that clinical implementation of sepsis-prediction models is currently scarce. The Discussion’s “Clinical model performance” section also identifies only three interventional studies, supporting the claim that few models have been implemented in clinical care. (LLM, conf=0.94)
- Partly supportedcites [6] sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) says that sepsis-prediction machine learning algorithms had performed well on retrospective data, but clinical implementation was largely confined to ICU populations; it gives limited examples and says large-scale non-ICU implementation had not been reported. This supports the point for sepsis-prediction models, but not the broader claim about machine learning models generally. (LLM, conf=0.82)
- Supportedcites [7] source
The Background (page not provided) says prospective clinical studies of EHR tools for sepsis have most often used rules-based approaches, while the machine-learning studies it lists were retrospective or pilots. The Conclusion describes this paper's implementation of a machine-learning sepsis algorithm in clinical care. Together, these support the claim in its sepsis-detection context. (LLM, conf=0.86)
- Partly supportedcites [8] sourceExperimental — accuracy not yet measured
The Abstract Introduction says that few electronic sepsis-monitoring methods offer predictive capabilities and that no severe-sepsis prediction system had previously been validated in a randomised study. The Conclusion describes this paper's own clinical trial of a machine-learning sepsis algorithm. These passages support limited clinical evaluation of predictive sepsis systems, but do not establish that few machine-learning models generally have been implemented in clinical care. (LLM, conf=0.79)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
47 more claims · 3 with their cited sources checked
Create a free account to see all 106 claims and every source verdict.
- It may state two separate claims as one. 27
- Probably not a substantial claim. 30
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Other studies have examined implementation issues when integrating machine learning models into clinical practice and reported numerous socio-technical factors such as the unfamiliarity of AI among clinicians and lack of AI education, trust in the accuracy and utility of predictive models, and IT support among other issues that affected this process (Joshi et al., 2022;Sandhu et al., 2020).”
From The application and use of artificial intelligence in cancer nursing: A systematic review (2024) · cites this paper as Sandhu et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Implementation studies similarly emphasize that pairing alerts with escalation protocols is essential to translate prediction into improved process measures and outcomes.”
From Artificial Intelligence to Facilitate SEP-1 Measure Compliance and Fluid Management in Sepsis (2026) · cites this paper as [48]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Implementation barriers for sepsis AI cluster around trust and interpretability, workflow integration, technical readiness and generalizability, governance and safety, leadership alignment, and the resources required for sustainment (Table 4) [10,33,34,46,48,77,78].”
From Artificial Intelligence to Facilitate SEP-1 Measure Compliance and Fluid Management in Sepsis (2026) · cites this paper as [48]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI-enabled early warning systems and EHR-integrated clinical decision support can improve surveillance, risk stratification, and the timeliness of key sepsis processes required by SEP-1 [45,48,49].”
From Artificial Intelligence to Facilitate SEP-1 Measure Compliance and Fluid Management in Sepsis (2026) · cites this paper as [48]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“An early warning system, Sepsis Watch, can pair model output with rapid response team nurse review and escalation pathways, reframing AI output as a trigger for needed responses rather than an interruptive alert, which at times may be ignored.”
From Artificial Intelligence to Facilitate SEP-1 Measure Compliance and Fluid Management in Sepsis (2026) · cites this paper as [48]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Methods > Study Design
“we recruited 7 ED physicians (n=7; Table 1) and 8 RRT nurses (n=8; Table 2)”
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Methods > Study Design
“Table: Table 1 Characteristics of rapid response team nurse participants.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Results > Implementation of Sepsis Watch Processes
“RRT physicians reported that at the start of the program, the ED physicians were unfamiliar with the purpose of the program, the role of the RRT nurse, and the flow of the call.”
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Methods > Study Design
“we recruited 7 ED physicians (n=7; Table 1) and 8 RRT nurses (n=8; Table 2)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Methods > Study Design
“| ED attending 7 | 6 years | 3 years | No |”
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Methods > Study Design
“| Participant | Experience as attending physician at pilot site | Experience as attending physician | Was the participant involved in program development? |”
Citations
Cites
User Acceptance of Information Technology: Toward A Unified View1
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003) · MIS Quarterly · doi:10.2307/30036540 · Cited by 45,035
(2014) · Cited by 15,747
Research Design: Qualitative, Quantitative, and Mixed Methods Approaches
ภู่พันธ์, ศ. (2018) · วารสารวิธีวิทยาการวิจัย (Journal of Research Methodology: JRM) · Cited by 13,070
Saturation in qualitative research: exploring its conceptualization and operationalization
Saunders, B., Sim, J., Kingstone, T., Baker, S., Waterfield, J., Bartlam, B., Burroughs, H., & Jinks, C. (2018) · Quality & Quantity · doi:10.1007/s11135-017-0574-8 · Cited by 12,033 · cited 1×
“In the qualitative analytic process, data collection was terminated when no new themes or insights emerged (ie, thematic saturation) across both participant groups [36].”
BackgroundToward a Theory of Situation Awareness in Dynamic Systems
Endsley, M. R. (1995) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872095779049543 · Cited by 8,875
Kumar, A., Roberts, D., Wood, K. E., Light, B., Parrillo, J. E., Sharma, S., Suppes, R., Feinstein, D., Zanotti, S., Taiberg, L., et al. (2006) · Critical Care Medicine · doi:10.1097/01.ccm.0000217961.75225.e9 · Cited by 6,109 · cited 1×
“The reasons for low compliance include the lack of a gold standard for sepsis diagnosis and the difficulty of rapidly mobilizing resources needed to treat individuals suspected of having sepsis [15,16].”
BackgroundThe Technology Acceptance Model: Its past and its future in health care
Holden, R. J., & Karsh, B.-T. (2010) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2009.07.002 · Cited by 2,669
Time to treatment and mortality during mandated emergency care for sepsis.
Seymour, C. W., Gesten, F., Prescott, H. C., Friedrich, M. E., Iwashyna, T. J., Phillips, G. S., Lemeshow, S., Osborn, T., Terry, K. M., & Levy, M. M. (2017) · New England Journal of Medicine · doi:10.1056/nejmoa1703058 · Cited by 2,225
Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009-2014.
Rhee, C., Dantes, R., Epstein, L., Murphy, D. J., Seymour, C. W., Iwashyna, T. J., Kadri, S. S., Angus, D. C., Danner, R. L., Fiore, A. E., et al. (2017) · JAMA · doi:10.1001/jama.2017.13836 · Cited by 1,884 · cited 1×
“A prominent use case for machine learning in health care is sepsis, a leading cause of death in US hospitals [1], which accounts for 1.7 million hospitalizations [2] and costs the US health system US $23 billion annually [3].”
BackgroundInfluentialArtificial Intelligence and Human Trust in Healthcare: Focus on Clinicians
Asan, O., Bayrak, A. E., & Choudhury, A. (2020) · Journal of Medical Internet Research · doi:10.2196/15154 · Cited by 825
Fleuren, L. M., Klausch, T. L. T., Zwager, C. L., Schoonmade, L. J., Guo, T., Roggeveen, L. F., Swart, E. L., Girbes, A. R. J., Thoral, P., Ercole, A., et al. (2020) · Intensive Care Medicine · doi:10.1007/s00134-019-05872-y · Cited by 755 · cited 1×
“Despite the rise in competing models and products for early sepsis detection [5], few machine learning models have been implemented as part of clinical care [6-8].”
BackgroundFerrer, R. (2008) · JAMA · doi:10.1001/jama.299.19.2294 · Cited by 741
The Surviving Sepsis Campaign Bundle: 2018 Update
Levy, M. M., Evans, L. E., & Rhodes, A. (2018) · Critical Care Medicine · doi:10.1097/ccm.0000000000003119 · Cited by 679
Rhee, C., Jones, T. M., Hamad, Y., Pande, A., Varon, J., O’Brien, C., Anderson, D. J., Warren, D. K., Dantes, R. B., Epstein, L., et al. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2018.7571 · Cited by 584
Self-citationNguyen, H. B., Corbett, S. W., Steele, R., Banta, J., Clark, R. T., Hayes, S. R., Edwards, J., Cho, T. W., & Wittlake, W. A. (2007) · Critical Care Medicine · doi:10.1097/01.ccm.0000259463.33848.3d · Cited by 555
Shimabukuro, D. W., Barton, C. W., Feldman, M. D., Mataraso, S. J., & Das, R. (2017) · BMJ Open Respiratory Research · doi:10.1136/bmjresp-2017-000234 · Cited by 382
Blease, C., Kaptchuk, T. J., Bernstein, M. H., Mandl, K. D., Halamka, J. D., & DesRoches, C. M. (2019) · Journal of Medical Internet Research · doi:10.2196/12802 · Cited by 346
Machine learning and medical education
Kolachalama, V. B., & Garg, P. S. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0061-1 · Cited by 320
Physician perspectives on integration of artificial intelligence into diagnostic pathology
Sarwar, S., Dent, A., Faust, K., Richer, M., Djuric, U., Van Ommeren, R., & Diamandis, P. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0106-0 · Cited by 297
Laï, M.-C., Brian, M., & Mamzer, M.-F. (2020) · Journal of Translational Medicine · doi:10.1186/s12967-019-02204-y · Cited by 294
Sendak, M. P., Ratliff, W., Sarro, D., Alderton, E., Futoma, J., Gao, M., Nichols, M., Revoir, M., Yashar, F., Miller, C., et al. (2020) · JMIR Medical Informatics · doi:10.2196/15182 · Cited by 287 · cited 2×
“According to our institutional definition for sepsis, over 20% of adults admitted through the DUH ED develop sepsis [31], and nearly 68% of sepsis occurs within the first 24 hours of hospital encounter [32].”
BackgroundSelf-citationPresenting machine learning model information to clinical end users with model facts labels
Sendak, M. P., Gao, M., Brajer, N., & Balu, S. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0253-3 · Cited by 279 · cited 1×
“Model developers should develop clear product labels to help clinicians understand when and how to appropriately incorporate machine learning model outputs into clinical decisions [43].”
BackgroundSelf-citationGiannini, H. M., Ginestra, J. C., Chivers, C., Draugelis, M., Hanish, A., Schweickert, W. D., Fuchs, B. D., Meadows, L., Lynch, M., Donnelly, P. J., et al. (2019) · Critical Care Medicine · doi:10.1097/ccm.0000000000003891 · Cited by 269 · cited 1×
“Despite the rise in competing models and products for early sepsis detection [5], few machine learning models have been implemented as part of clinical care [6-8].”
BackgroundPrediction of sepsis patients using machine learning approach: a meta-analysis
Islam, M. M., Nasrin, T., Walther, B. A., Wu, C.-C., Yang, H.-C., & Li, Y.-C. (2019) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2018.12.027 · Cited by 259 · cited 1×
“Machine learning algorithms have been shown to outperform traditional screening scores in early sepsis detection [4].”
BackgroundArtificial intelligence and the future of psychiatry: Insights from a global physician survey
Doraiswamy, P. M., Blease, C., & Bodner, K. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2019.101753 · Cited by 253
Parshuram, C. S., Dryden-Palmer, K., Farrell, C., Gottesman, R., Gray, M., Hutchison, J. S., Helfaer, M., Hunt, E. A., Joffe, A. R., Lacroix, J., et al. (2018) · JAMA · doi:10.1001/jama.2018.0948 · Cited by 229
Keel, S., Lee, P. Y., Scheetz, J., Li, Z., Kotowicz, M. A., MacIsaac, R. J., & He, M. (2018) · Scientific Reports · doi:10.1038/s41598-018-22612-2 · Cited by 220
Lin, H., Li, R., Liu, Z., Chen, J., Yang, Y., Chen, H., Lin, Z., Lai, W., Long, E., Wu, X., et al. (2019) · EClinicalMedicine · doi:10.1016/j.eclinm.2019.03.001 · Cited by 199
"The human body is a black box": supporting clinical decision-making with deep learning
Sendak, M., Elish, M. C., Gao, M., Futoma, J., Ratliff, W., Nichols, M., Bedoya, A., Balu, S., & O'Brien, C. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372827 · Cited by 196
Self-citationA Path for Translation of Machine Learning Products into Healthcare Delivery
(2020) · EMJ Innovations · doi:10.33590/emjinnov/19-00172 · Cited by 172
Self-citationImplementation of a real-time computerized sepsis alert in nonintensive care unit patients*
Sawyer, A. M., Deal, E. N., Labelle, A. J., Witt, C., Thiel, S. W., Heard, K., Reichley, R. M., Micek, S. T., & Kollef, M. H. (2011) · Critical Care Medicine · doi:10.1097/ccm.0b013e318205df85 · Cited by 169 · cited 1×
“CDS sepsis early warning systems leverage electronic health information to continuously stratify patients for the risk of sepsis and alert clinicians [17-22].”
BackgroundDevelopment, implementation, and impact of an automated early warning and response system for sepsis
Umscheid, C. A., Betesh, J., VanZandbergen, C., Hanish, A., Tait, G., Mikkelsen, M. E., French, B., & Fuchs, B. D. (2015) · Journal of Hospital Medicine · doi:10.1002/jhm.2259 · Cited by 168
Minimal Impact of Implemented Early Warning Score and Best Practice Alert for Patient Deterioration*
Bedoya, A. D., Clement, M. E., Phelan, M., Steorts, R. C., O’Brien, C., & Goldstein, B. A. (2019) · Critical Care Medicine · doi:10.1097/ccm.0000000000003439 · Cited by 162
Self-citationGinestra, J. C., Giannini, H. M., Schweickert, W. D., Meadows, L., Lynch, M. J., Pavan, K., Chivers, C. J., Draugelis, M., Donnelly, P. J., Fuchs, B. D., et al. (2019) · Critical Care Medicine · doi:10.1097/ccm.0000000000003803 · Cited by 159 · cited 1×
“Similarly, 2 previous studies examining predictive alerts for sepsis suggested that the perceived utility decreased when the model frequently identified clinically stable patients [27,28].”
ResultVan de Velde, S., Heselmans, A., Delvaux, N., Brandt, L., Marco-Ruiz, L., Spitaels, D., Cloetens, H., Kortteisto, T., Roshanov, P., Kunnamo, I., et al. (2018) · Implementation Science · doi:10.1186/s13012-018-0790-1 · Cited by 156
McCoy, A., & Das, R. (2017) · BMJ Open Quality · doi:10.1136/bmjoq-2017-000158 · Cited by 148
Prospective Trial of Real-Time Electronic Surveillance to Expedite Early Care of Severe Sepsis
Nelson, J. L., Smith, B. L., Jared, J. D., & Younger, J. G. (2011) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2010.12.008 · Cited by 122
Downing, N. L., Rolnick, J., Poole, S. F., Hall, E., Wessels, A. J., Heidenreich, P., & Shieh, L. (2019) · BMJ Quality & Safety · doi:10.1136/bmjqs-2018-008765 · Cited by 112
Machine learning for early detection of sepsis: an internal and temporal validation study
Bedoya, A. D., Futoma, J., Clement, M. E., Corey, K., Brajer, N., Lin, A., Simons, M. G., Gao, M., Nichols, M., Balu, S., et al. (2020) · JAMIA Open · doi:10.1093/jamiaopen/ooaa006 · Cited by 109 · cited 1×
“For example, despite the model’s high predictive value within the first hour of ED presentation [44], nurses sometimes waited for more clinical information to populate in the medical record before calling physicians about patients flagged as being at high-risk for sepsis.”
BackgroundSelf-citationGarcia, S., & Miller, S. (2019) · American Mathematical Society eBooks · doi:10.1090/mbk/121/79 · Cited by 109
Manaktala, S., & Claypool, S. R. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw056 · Cited by 89
National Performance on the Medicare SEP-1 Sepsis Quality Measure
Barbash, I. J., Davis, B., & Kahn, J. M. (2019) · Critical Care Medicine · doi:10.1097/ccm.0000000000003613 · Cited by 86
An Improved Multi-Output Gaussian Process RNN with Real-Time Validation for Early Sepsis Detection
Futoma, J., Hariharan, S., Sendak, M., Brajer, N., Clement, M. E., Bedoya, A., O’Brien, C. L., & Heller, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.05894 · Cited by 65
Self-citationThe Stakes of Uncertainty: Developing and Integrating Machine Learning in Clinical Care
ELISH, M. C. (2018) · Ethnographic Praxis in Industry Conference Proceedings · doi:10.1111/1559-8918.2018.01213 · Cited by 59 · cited 1×
“…of literature on both physician perspectives of sepsis CDS implementation [26-28] and sepsis machine learning algorithms, more research is needed to understand clinician views on black box machine learning models that do not explain their predictions in a way that humans can understand [29,30].”
BackgroundArabi, Y. M., Al-Dorzi, H. M., Alamry, A., Hijazi, R., Alsolamy, S., Al Salamah, M., Tamim, H. M., Al-Qahtani, S., Al-Dawood, A., Marini, A. M., et al. (2017) · Annals of Intensive Care · doi:10.1186/s13613-017-0280-7 · Cited by 52
Development and Implementation of Sepsis Alert Systems
Harrison, A. M., Gajic, O., Pickering, B. W., & Herasevich, V. (2016) · Clinics in Chest Medicine · doi:10.1016/j.ccm.2016.01.004 · Cited by 46 · cited 1×
“Unfortunately, many sepsis CDS systems fail to improve outcomes because of poor diagnostic accuracy and program implementation [23-25].”
BackgroundGuidi, J. L., Clark, K., Upton, M. T., Faust, H., Umscheid, C. A., Lane-Fall, M. B., Mikkelsen, M. E., Schweickert, W. D., Vanzandbergen, C. A., Betesh, J., et al. (2015) · Annals of the American Thoracic Society · doi:10.1513/annalsats.201503-129oc · Cited by 43
Sepsis guideline implementation: benefits, pitfalls and possible solutions
Kissoon, N. (2014) · Critical Care · doi:10.1186/cc13774 · Cited by 25
Clinician Perceptions of an Early Warning System on Patient Safety
de Vries, A., Draaisma, J. M. T., & Fuijkschot, J. (2017) · Hospital Pediatrics · doi:10.1542/hpeds.2016-0138 · Cited by 21
Lin, A. L., Sendak, M., Bedoya, A., Clement, M. E., Brajer, N., Futoma, J., Bosworth, H. B., Heller, K., & O’Brien, C. L. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/648907 · Cited by 2
Self-citation
Research works printed but not linked: 2
- TorioCMMooreBJNational Inpatient Hospital Costs: The Most Expensive Conditions by Payer, 2013: Statistical Brief #204Healthcare Cost and Utilization Project (HCUP)2020-11-13https://europepmc.org/abstract/med/27359025
- Futoma, J, arXivstat
Cited by
Lambert, S. I., Madi, M., Sopka, S., Lenes, A., Stange, H., Buszello, C.-P., & Stephan, A. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00852-5 · Cited by 404
Artificial Intelligence -based technologies in nursing: A scoping literature review of the evidence
von Gerich, H., Moen, H., Block, L. J., Chu, C. H., DeForest, H., Hobensack, M., Michalowski, M., Mitchell, J., Nibber, R., Olalia, M. A., et al. (2022) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2021.104153 · Cited by 360
An interdisciplinary review of AI and HRM: Challenges and future directions
Pan, Y., & Froese, F. J. (2023) · Human Resource Management Review · doi:10.1016/j.hrmr.2022.100924 · Cited by 214
Rony, M. K. K., Kayesh, I., Bala, S. D., Akter, F., & Parvin, M. R. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e25718 · Cited by 190
“The notion of AI foreseeing potential patient deteriorations before they manifest sparks excitement, hinting at the possibility of timelier interventions and improved clinical results [44].”
BackgroundNair, M., Svedberg, P., Larsson, I., & Nygren, J. M. (2024) · PLOS ONE · doi:10.1371/journal.pone.0305949 · Cited by 179
Metadata
- Authors
- Sandhu, S., Lin, A. L., Brajer, N., Sperling, J., Ratliff, W., Bedoya, A. D., Balu, S., O'Brien, C., & Sendak, M. P.
- Affiliations
- Duke University (US)
- Corresponding author
- Sandhu, Sahil
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2020-11-19
- Volume
- 22
- Issue
- 11
- Pages
- e22421
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC7714645
- OpenAlex
- W3094937636
- Cited by
- 168
- References
- 50
- Citations by year
- 2026: 37 · 2025: 35 · 2024: 38 · 2023: 25 · 2022: 23 · 2021: 10
- FWCI
- 4.29
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Critical Care and Intensive Care Medicine › Sepsis Diagnosis and Treatment
- Keywords
- machine learning integration, clinical workflow, implementation barriers, clinician trust, perceived usefulness, emergency physicians, rapid response teams, nurse-physician communication, clinical decision support, technology adoption
- MeSH terms
- Machine Learning, Humans, Qualitative Research, Workflow
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
