Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study
Open accessPublished version of a preprint
The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index; 25 of the 34 sources its claims cite were checked for support.
Abstract
Background: Machine learning models have the potential to improve diagnostic accuracy and management of acute conditions. Despite growing efforts to evaluate and validate such models, little is known about how to best translate and implement these products as part of routine clinical care. Objective: This study aims to explore the factors influencing the integration of a machine learning sepsis early warning system (Sepsis Watch) into clinical workflows. Methods: We conducted semistructured interviews with 15 frontline emergency department physicians and rapid response team nurses who participated in the Sepsis Watch quality improvement initiative. Interviews were audio recorded and transcribed. We used a modified grounded theory approach to identify key themes and analyze qualitative data. Results: A total of 3 dominant themes emerged: perceived utility and trust, implementation of Sepsis Watch processes, and workforce considerations. Participants described their unfamiliarity with machine learning models. As a result, clinician trust was influenced by the perceived accuracy and utility of the model from personal program experience. Implementation of Sepsis Watch was facilitated by the easy-to-use tablet application and communication strategies that were developed by nurses to share model outputs with physicians. Barriers included the flow of information among clinicians and gaps in knowledge about the model itself and broader workflow processes. Conclusions: This study generated insights into how frontline clinicians perceived machine learning models and the barriers to integrating them into clinical workflows. These findings can inform future efforts to implement machine learning interventions in real-world settings and maximize the adoption of these interventions.
Overview
CoChat Summary
TL;DR
This qualitative study interviewed 15 frontline clinicians to identify factors affecting integration of a machine learning sepsis early warning system, Sepsis Watch, into emergency department workflows. It found three dominant themes—perceived utility and trust, implementation processes, and workforce considerations—with clinician trust shaped by personal experience, model transparency limits, and human communication.
Key findings
- Three dominant themes emerged: perceived utility and trust, implementation of Sepsis Watch processes, and workforce considerations, with 10 corresponding subthemes.
- RRT nurses, the primary users, spoke positively about Sepsis Watch accuracy and the easy-to-use color-coded application; ED physicians reported less trust, citing algorithm components such as blood cultures being too heavily weighted, missed sepsis cases, and false positives.
- Implementation was facilitated by the tablet/web application and nurse-developed communication strategies, but barriers included information flow challenges, workflow interruptions, and gaps in knowledge about the black box model.
- The program created a new Sepsis Watch Nurse role; nurses valued it and identified good clinical judgment, sepsis knowledge, critical care experience, and communication skills as important, while strong computer skills were not considered necessary.
Methods
- Semistructured interviews were conducted with 15 participants—7 ED attending physicians and 8 RRT nurses—from January 2019 to April 2019, starting 4 months into Sepsis Watch implementation at Duke University Hospital; interviews averaged 35 minutes and continued until thematic saturation.
- A modified grounded theory approach was used with three coding phases, and data were analyzed in NVivo 12; interview guides were informed by the situational awareness model and piloted among 3 clinicians.
- Sepsis Watch used a deep learning model trained on 42,000 inpatient encounters and 32 million data points to predict likelihood of meeting a sepsis phenotype within the subsequent 4 hours, with outputs displayed as black, red, orange, or yellow cards.
Limitations
- The paper states that findings should be generalized with caution to settings differing in organizational structure, capacity, and professional norms; the study focuses on workflow integration rather than the multi-year planning and stakeholder engagement process; only frontline clinicians were interviewed, excluding leadership, administrators, data scientists, and patients; and including clinicians involved in program development may have biased findings positively.
- The paper also states that further studies with larger samples and survey approaches are needed to quantify adoption factors and examine variation by clinician characteristics.
- Evident from the text: this was a single-site academic hospital study using convenience sampling and qualitative self-report, and it did not measure patient outcomes.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
15 Valid · 1 Not valid · 3 Not checked · 87 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Similarly, only frontline clinicians were interviewed despite the large number of stakeholders involved in project development and maintenance (eg, organizational leadership, hospital administrators, data scientists) or impacted directly by the program (ie, patients; [45,46]).”
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Discussion > Limitations
What the sources say
- Disputedcites [45] sourceExperimental — accuracy not yet measured
The Results (page unavailable) say pediatric patients and their guardians completed an evaluation questionnaire; they were not excluded from the study’s feedback. The paper reports a randomized trial with patient questionnaires, not interviews limited to frontline clinicians. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
- Disputedcites [46] sourceExperimental — accuracy not yet measured
The Methods (Protocol) and Results (Participant follow-up) say patients were contacted and interviewed about their preferred screening model. The Discussion calls for future research into clinician and organizational stakeholder perspectives, rather than reporting interviews with frontline clinicians. The supplied passages do not give page numbers. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Unfortunately, many sepsis CDS systems fail to improve outcomes because of poor diagnostic accuracy and program implementation [23-25].”
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Introduction
What the sources say
- Partly supportedcites [23] sourceExperimental — accuracy not yet measured
The Abstract conclusions and Discussion report that the implemented NEWS alert had no appreciable impact on clinical outcomes, performed poorly, and was generally ignored by nursing staff. The paper concerns one early-warning system for patient deterioration, not many sepsis-specific CDS systems, so it does not support that scope of the claim. (LLM, conf=0.88)
- Supportedcites [24] source
The Development and Implementation section reports that rigorous trials of sepsis alerts, including a system with CDS, did not improve clinically significant outcomes. The Barriers section explains that inaccurate alerts can render a system ineffective and that poorly planned implementation can undermine its impact on outcomes. Together, these support the citing sentence’s broad summary. (LLM, conf=0.86)
- Partly supportedcites [25] sourceExperimental — accuracy not yet measured
The Background reports that some sepsis-alert studies found no improvement in outcomes and says inconsistent findings may reflect differences in alert algorithms and downstream care. The Discussion identifies low alert specificity and failures in subsequent clinical action as potential barriers. It does not establish that poor diagnostic accuracy and implementation caused many sepsis CDS systems to fail, as the claim states. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“As a result, little evidence exists on the optimal integration of these sepsis models and other machine learning models into clinical workflows [9].”
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Introduction
What the sources say
- Supportedcites [9] source
In “Machine Learning Applied to Clinical Decision Support Tasks” (page unavailable), the paper says recommendations for design and implementation are only beginning to emerge and that there is no single best workflow. In “2. Evaluate and Validate” (page unavailable), it describes clinical integration as difficult and reports only one small randomized trial among the products reviewed. These passages support the claim that evidence on optimal workflow integration, including for sepsis models, is limited. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Clinician preferences were incorporated to optimize the ease of use and utility for end users [39].”
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Discussion
What the sources say
- Partly supportedcites [39] sourceExperimental — accuracy not yet measured
The Discussion (§4.1; page not provided) recommends designing health IT to be useful and not difficult to use, supporting the stated design goal. It does not say that clinician preferences were incorporated into Sepsis Watch or establish that this design step occurred. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Despite the rise in competing models and products for early sepsis detection [5], few machine learning models have been implemented as part of clinical care [6-8].”
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Introduction
What the sources say
- Supportedcites [5] source
The Conclusion states that clinical implementation of sepsis-prediction models is currently scarce. The Discussion’s “Clinical model performance” section also identifies only three interventional studies, supporting the claim that few models have been implemented in clinical care. (LLM, conf=0.94)
- Partly supportedcites [6] sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) says that sepsis-prediction machine learning algorithms had performed well on retrospective data, but clinical implementation was largely confined to ICU populations; it gives limited examples and says large-scale non-ICU implementation had not been reported. This supports the point for sepsis-prediction models, but not the broader claim about machine learning models generally. (LLM, conf=0.82)
- Supportedcites [7] source
The Background (page not provided) says prospective clinical studies of EHR tools for sepsis have most often used rules-based approaches, while the machine-learning studies it lists were retrospective or pilots. The Conclusion describes this paper's implementation of a machine-learning sepsis algorithm in clinical care. Together, these support the claim in its sepsis-detection context. (LLM, conf=0.86)
- Partly supportedcites [8] sourceExperimental — accuracy not yet measured
The Abstract Introduction says that few electronic sepsis-monitoring methods offer predictive capabilities and that no severe-sepsis prediction system had previously been validated in a randomised study. The Conclusion describes this paper's own clinical trial of a machine-learning sepsis algorithm. These passages support limited clinical evaluation of predictive sepsis systems, but do not establish that few machine-learning models generally have been implemented in clinical care. (LLM, conf=0.79)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
47 more claims · 3 with their cited sources checked
Create a free account to see all 106 claims and every source verdict.
- It may state two separate claims as one. 27
- Probably not a substantial claim. 30
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Other studies have examined implementation issues when integrating machine learning models into clinical practice and reported numerous socio-technical factors such as the unfamiliarity of AI among clinicians and lack of AI education, trust in the accuracy and utility of predictive models, and IT support among other issues that affected this process (Joshi et al., 2022;Sandhu et al., 2020).”
From The application and use of artificial intelligence in cancer nursing: A systematic review (2024) · cites this paper as Sandhu et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Implementation studies similarly emphasize that pairing alerts with escalation protocols is essential to translate prediction into improved process measures and outcomes.”
From Artificial Intelligence to Facilitate SEP-1 Measure Compliance and Fluid Management in Sepsis (2026) · cites this paper as [48]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Implementation barriers for sepsis AI cluster around trust and interpretability, workflow integration, technical readiness and generalizability, governance and safety, leadership alignment, and the resources required for sustainment (Table 4) [10,33,34,46,48,77,78].”
From Artificial Intelligence to Facilitate SEP-1 Measure Compliance and Fluid Management in Sepsis (2026) · cites this paper as [48]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI-enabled early warning systems and EHR-integrated clinical decision support can improve surveillance, risk stratification, and the timeliness of key sepsis processes required by SEP-1 [45,48,49].”
From Artificial Intelligence to Facilitate SEP-1 Measure Compliance and Fluid Management in Sepsis (2026) · cites this paper as [48]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“An early warning system, Sepsis Watch, can pair model output with rapid response team nurse review and escalation pathways, reframing AI output as a trigger for needed responses rather than an interruptive alert, which at times may be ignored.”
From Artificial Intelligence to Facilitate SEP-1 Measure Compliance and Fluid Management in Sepsis (2026) · cites this paper as [48]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Methods > Study Design
“we recruited 7 ED physicians (n=7; Table 1) and 8 RRT nurses (n=8; Table 2)”
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Methods > Study Design
“Table: Table 1 Characteristics of rapid response team nurse participants.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Results > Implementation of Sepsis Watch Processes
“RRT physicians reported that at the start of the program, the ED physicians were unfamiliar with the purpose of the program, the role of the RRT nurse, and the flow of the call.”
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Methods > Study Design
“we recruited 7 ED physicians (n=7; Table 1) and 8 RRT nurses (n=8; Table 2)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Methods > Study Design
“| ED attending 7 | 6 years | 3 years | No |”
§ Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study > Methods > Study Design
“| Participant | Experience as attending physician at pilot site | Experience as attending physician | Was the participant involved in program development? |”
Citations
Cites
User Acceptance of Information Technology: Toward A Unified View1
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003) · MIS Quarterly · doi:10.2307/30036540 · Cited by 45,035
(2014) · Cited by 15,747
Research Design: Qualitative, Quantitative, and Mixed Methods Approaches
ภู่พันธ์, ศ. (2018) · วารสารวิธีวิทยาการวิจัย (Journal of Research Methodology: JRM) · Cited by 13,070
Saturation in qualitative research: exploring its conceptualization and operationalization
Saunders, B., Sim, J., Kingstone, T., Baker, S., Waterfield, J., Bartlam, B., Burroughs, H., & Jinks, C. (2018) · Quality & Quantity · doi:10.1007/s11135-017-0574-8 · Cited by 12,033 · cited 1×
“In the qualitative analytic process, data collection was terminated when no new themes or insights emerged (ie, thematic saturation) across both participant groups [36].”
BackgroundToward a Theory of Situation Awareness in Dynamic Systems
Endsley, M. R. (1995) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872095779049543 · Cited by 8,875
Kumar, A., Roberts, D., Wood, K. E., Light, B., Parrillo, J. E., Sharma, S., Suppes, R., Feinstein, D., Zanotti, S., Taiberg, L., et al. (2006) · Critical Care Medicine · doi:10.1097/01.ccm.0000217961.75225.e9 · Cited by 6,109 · cited 1×
“The reasons for low compliance include the lack of a gold standard for sepsis diagnosis and the difficulty of rapidly mobilizing resources needed to treat individuals suspected of having sepsis [15,16].”
BackgroundThe Technology Acceptance Model: Its past and its future in health care
Holden, R. J., & Karsh, B.-T. (2010) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2009.07.002 · Cited by 2,669
Time to treatment and mortality during mandated emergency care for sepsis.
Seymour, C. W., Gesten, F., Prescott, H. C., Friedrich, M. E., Iwashyna, T. J., Phillips, G. S., Lemeshow, S., Osborn, T., Terry, K. M., & Levy, M. M. (2017) · New England Journal of Medicine · doi:10.1056/nejmoa1703058 · Cited by 2,225
Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009-2014.
Rhee, C., Dantes, R., Epstein, L., Murphy, D. J., Seymour, C. W., Iwashyna, T. J., Kadri, S. S., Angus, D. C., Danner, R. L., Fiore, A. E., et al. (2017) · JAMA · doi:10.1001/jama.2017.13836 · Cited by 1,884 · cited 1×
“A prominent use case for machine learning in health care is sepsis, a leading cause of death in US hospitals [1], which accounts for 1.7 million hospitalizations [2] and costs the US health system US $23 billion annually [3].”
BackgroundInfluentialArtificial Intelligence and Human Trust in Healthcare: Focus on Clinicians
Asan, O., Bayrak, A. E., & Choudhury, A. (2020) · Journal of Medical Internet Research · doi:10.2196/15154 · Cited by 825
Research works printed but not linked: 2
- TorioCMMooreBJNational Inpatient Hospital Costs: The Most Expensive Conditions by Payer, 2013: Statistical Brief #204Healthcare Cost and Utilization Project (HCUP)2020-11-13https://europepmc.org/abstract/med/27359025
- Futoma, J, arXivstat
Cited by
Lambert, S. I., Madi, M., Sopka, S., Lenes, A., Stange, H., Buszello, C.-P., & Stephan, A. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00852-5 · Cited by 404
Artificial Intelligence -based technologies in nursing: A scoping literature review of the evidence
von Gerich, H., Moen, H., Block, L. J., Chu, C. H., DeForest, H., Hobensack, M., Michalowski, M., Mitchell, J., Nibber, R., Olalia, M. A., et al. (2022) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2021.104153 · Cited by 360
An interdisciplinary review of AI and HRM: Challenges and future directions
Pan, Y., & Froese, F. J. (2023) · Human Resource Management Review · doi:10.1016/j.hrmr.2022.100924 · Cited by 214
Rony, M. K. K., Kayesh, I., Bala, S. D., Akter, F., & Parvin, M. R. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e25718 · Cited by 190
“The notion of AI foreseeing potential patient deteriorations before they manifest sparks excitement, hinting at the possibility of timelier interventions and improved clinical results [44].”
BackgroundNair, M., Svedberg, P., Larsson, I., & Nygren, J. M. (2024) · PLOS ONE · doi:10.1371/journal.pone.0305949 · Cited by 179
Hogg, H. D. J., Al-Zubaidy, M., Technology Enhanced Macular Services Study Reference Group, Talks, J., Denniston, A. K., Kelly, C. J., Malawana, J., Papoutsi, C., Teare, M. D., Keane, P. A., et al. (2023) · Journal of Medical Internet Research · doi:10.2196/39742 · Cited by 163
Vo, V., Chen, G., Aquino, Y. S. J., Carter, S. M., Do, Q. N., & Woode, M. E. (2023) · Social Science & Medicine · doi:10.1016/j.socscimed.2023.116357 · Cited by 154
Applications of Artificial Intelligence in Nursing Care: A Systematic Review
Martinez-Ortigosa, A., Martinez-Granados, A., Gil-Hernández, E., Rodriguez-Arrastia, M., Ropero-Padilla, C., & Roman, P. (2023) · Journal of Nursing Management · doi:10.1155/2023/3219127 · Cited by 131
Rahimi, A. K., Pienaar, O., Ghadimi, M., Canfell, O. J., Pole, J. D., Shrapnel, S., van der Vegt, A. H., & Sullivan, C. M. (2024) · Journal of Medical Internet Research · doi:10.2196/49655 · Cited by 164
Schwartz, J. M., George, M., Rossetti, S. C., Dykes, P. C., Minshall, S. R., Lucas, E., & Cato, K. D. (2022) · JMIR Human Factors · doi:10.2196/33960 · Cited by 88
“For example, Sandhu et al [7] reported that “even when physicians did not trust a model output, they still reported paying closer attention to a patient’s clinical progression,” aligning with our category CDSSs provide reason to probe further.”
BackgroundInfluentialHwang, G.-J., Tang, K.-Y., & Tu, Y.-F. (2024) · Interactive Learning Environments · doi:10.1080/10494820.2022.2086579 · Cited by 87
Rajashekar, N. C., Shin, Y. E., Pu, Y., Chung, S., You, K., Giuffre, M., Chan, C. E., Saarinen, T., Hsiao, A., Sekhon, J., et al. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642024 · Cited by 85
University of California, S. F., Adler-Milstein, J., Aggarwal, N., University of Wisconsin-Madison, Ahmed, M., National Academy of Medicine, Castner, J., Incorporated, C., Evans, B. J., University of Florida, et al. (2022) · NAM Perspectives · doi:10.31478/202209c · Cited by 82
Self-citationZając, H. D., Li, D., Dai, X., Carlsen, J. F., Kensing, F., & Andersen, T. O. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3582430 · Cited by 74
Trust in AI-assisted health systems and AI’s trust in humans
Sagona, M., Dai, T., Macis, M., & Darden, M. (2025) · npj Health Systems · doi:10.1038/s44401-025-00016-5 · Cited by 71
van der Vegt, A. H., Scott, I. A., Dermawan, K., Schnetler, R. J., Kalke, V. R., & Lane, P. J. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad075 · Cited by 72
“Five (A, C, E, F, G) groups chose the former, whereas Sandhu et al found physicians preferred the latter, having a nurse contact them directly, often in-person, rather than by means of EHR-generated alerts which imposed greater cognitive load and interruptions.(42) However, the same physicians still saw nurse contacts…”
BackgroundInfluentialBarwise, A. K., Curtis, S., Diedrich, D. A., & Pickering, B. W. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad224 · Cited by 86
Giddings, R., Joseph, A., Callender, T., Janes, S. M., van der Schaar, M., Sheringham, J., & Navani, N. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00241-8 · Cited by 80
Abdelwanis, M., Simsekler, M. C. E., Gabor, A. F., Sleptchenko, A., & Omar, M. (2025) · Safety Science · doi:10.1016/j.ssci.2025.107028 · Cited by 97
Adoption of Machine Learning Systems for Medical Diagnostics in Clinics: Qualitative Interview Study
Pumplun, L., Fecho, M., Wahl, N., Peters, F., & Buxmann, P. (2021) · Journal of Medical Internet Research · doi:10.2196/29301 · Cited by 57
Health Care Professionals’ Experience of Using AI: Systematic Review With Narrative Synthesis
Ayorinde, A., Mensah, D. O., Walsh, J., Ghosh, I., Ibrahim, S. A., Hogg, J., Peek, N., & Griffiths, F. (2024) · Journal of Medical Internet Research · doi:10.2196/55766 · Cited by 55
Tulk Jesso, S., Kelliher, A., Sanghavi, H., Martin, T., & Henrickson Parker, S. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.830345 · Cited by 54
Preti, L. M., Ardito, V., Compagni, A., Petracca, F., & Cappellaro, G. (2024) · Journal of Medical Internet Research · doi:10.2196/55897 · Cited by 62
“The most recurrent themes within this construct revolved around ease of use and intuitive design [33,45,49,62].”
InfluentialCastaldo, R., Cavaliere, C., Soricelli, A., Salvatore, M., Pecchia, L., & Franzese, M. (2021) · Journal of Medical Internet Research · doi:10.2196/22394 · Cited by 56
The application and use of artificial intelligence in cancer nursing: A systematic review
O'Connor, S., Vercell, A., Wong, D., Yorke, J., Fallatah, F. A., Cave, L., & Anny Chen, L.-Y. (2024) · European Journal of Oncology Nursing · doi:10.1016/j.ejon.2024.102510 · Cited by 51
Choi, A., Choi, S. Y., Chung, K., Chung, H. S., Song, T., Choi, B., & Kim, J. H. (2023) · Scientific Reports · doi:10.1038/s41598-023-35617-3 · Cited by 47
A normative framework for artificial intelligence as a sociotechnical system in healthcare
McCradden, M. D., Joshi, S. D., Anderson, J. A., & London, A. J. (2023) · Patterns · doi:10.1016/j.patter.2023.100864 · Cited by 74
Evaluating GPT-4-based ChatGPT’s Clinical Potential on the NEJM Quiz
Ueda, D., Walston, S. L., Matsumoto, T., Deguchi, R., Tatekawa, H., & Miki, Y. (2023) · doi:10.1101/2023.05.04.23289493 · Cited by 45
Evaluating GPT-4-based ChatGPT's clinical potential on the NEJM quiz
Ueda, D., Walston, S. L., Matsumoto, T., Deguchi, R., Tatekawa, H., & Miki, Y. (2024) · BMC Digital Health · doi:10.1186/s44247-023-00058-5 · Cited by 44
Matthiesen, S., Diederichsen, S. Z., Hansen, M. K. H., Villumsen, C., Lassen, M. C. H., Jacobsen, P. K., Risum, N., Winkel, B. G., Philbert, B. T., Svendsen, J. H., et al. (2021) · JMIR Human Factors · doi:10.2196/26964 · Cited by 45
“While AI algorithms have been validated and have been shown to have similar or higher accuracy than humans, recent studies of AI deployment in clinical settings report that professional autonomy, workflow, and local sociotechnical factors have impacts on how accuracy is perceived and used in clinical practice [24,43,4…”
BackgroundKashyap, S., Morse, K. E., Patel, B., & Shah, N. H. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab154 · Cited by 43
Chan, S. L., Lee, J. W., Ong, M. E. H., Siddiqui, F. J., Graves, N., Ho, A. F. W., & Liu, N. (2023) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2023.02.001 · Cited by 49
Bhat, J. A., Feng, X., Mir, Z. A., Raina, A., & Siddique, K. H. M. (2023) · Physiologia Plantarum · doi:10.1111/ppl.13969 · Cited by 55
Advancing primary care with Artificial Intelligence and Machine Learning
Yang, Z., Silcox, C., Sendak, M., Rose, S., Rehkopf, D., Phillips, R., Peterson, L., Marino, M., Maier, J., Lin, S., et al. (2022) · Healthcare · doi:10.1016/j.hjdsi.2021.100594 · Cited by 38
Self-citationArtificial intelligence-driven transformative applications in disease diagnosis technology
Zhou, J., Park, S., Dong, S., Tang, X., & Wei, X. (2025) · Medical Review · doi:10.1515/mr-2024-0097 · Cited by 55
Evans, R. P., Bryant, L. D., Russell, G., & Absolom, K. L. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105342 · Cited by 41
Nair, M., Nygren, J., Nilsén, P., Gama, F., Neher, M., Larsson, I., & Svedberg, P. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1550459 · Cited by 49
Singer, S. J., Kellogg, K. C., Galper, A. B., & Viola, D. (2021) · Health Care Management Review · doi:10.1097/hmr.0000000000000324 · Cited by 32
Predicting Sepsis Mortality in a Population-Based National Database: Machine Learning Approach
Park, J. Y., Hsu, T., Hu, J., Chen, C., Hsu, W., Lee, M., Ho, J., & Lee, C. (2022) · Journal of Medical Internet Research · doi:10.2196/29982 · Cited by 39
Gokhale, S., Taylor, D., Gill, J., Hu, Y., Zeps, N., Lequertier, V., Prado, L., Teede, H. J., & Enticott, J. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1192969 · Cited by 48
Integrating artificial intelligence into an ophthalmologist’s workflow: obstacles and opportunities
Taribagil, P., Hogg, H. D. J., Balaskas, K., & Keane, P. A. (2023) · Expert Review of Ophthalmology · doi:10.1080/17469899.2023.2175672 · Cited by 40
Zhong, T., Zhuang, Z., Dong, X., Wong, K., Wong, W., Wang, J., He, D., & Liu, S. (2021) · JMIR Medical Informatics · doi:10.2196/29226 · Cited by 26
Recommendations to promote fairness and inclusion in biomedical AI research and clinical use
Griffin, A. C., Wang, K., Leung, T. I., & Facelli, J. C. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104693 · Cited by 29
de Almeida, J. G., Messiou, C., Withey, S. J., Matos, C., Koh, D.-M., & Papanikolaou, N. (2025) · European Radiology · doi:10.1007/s00330-025-11654-6 · Cited by 23
Parsons, C. S., Zuiderwijk, A., Orchard, N. A., Oosterhoff, J. H. F., & de Reuver, M. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-03237-8 · Cited by 39
Hwang, G., Chang, P.-Y., Tseng, W.-Y., Chou, C.-A., Wu, C.-H., & Tu, Y. (2022) · CIN Computers Informatics Nursing · doi:10.1097/cin.0000000000000897 · Cited by 31
Ed‐Driouch, C., Mars, F., Gourraud, P., & Dumas, C. (2022) · Sensors · doi:10.3390/s22218313 · Cited by 31
Wieben, A. M., Alreshidi, B. G., Douthit, B. J., Sileo, M., Vyas, P., Steege, L. M. B., & Gilmore‐Bykovskyi, A. L. (2024) · Journal of Nursing Scholarship · doi:10.1111/jnu.13001 · Cited by 36
Maniscalco, M., Candia, C., Fuschillo, S., Ambrosino, P., Paris, D., & Motta, A. (2024) · Journal of Breath Research · doi:10.1088/1752-7163/ad7a9a · Cited by 28
Investigation of a Web-Based Explainable AI Screening for Prolonged Grief Disorder
She, W. J., Ang, C. S., Neimeyer, R. A., Burke, L. A., Zhang, Y., Jatowt, A., Kawai, Y., Hu, J., Rauterberg, M., Prigerson, H. G., et al. (2022) · IEEE Access · doi:10.1109/access.2022.3163311 · Cited by 32
Cheng, R. K.-H., Aggarwal, A. K., Chakraborty, A., Harish, V., McGowan, M., Roy, A., Szulewski, A., & Nolan, B. (2024) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2024.05.020 · Cited by 27
Morris, A. H., Horvat, C. M., Stagg, B. C., Grainger, D. W., Lanspa, M. J., Orme, J. F., Clemmer, T. P., Weaver, L. K., Thomas, F. O., Grissom, C. K., et al. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac143 · Cited by 24
Kasun, M., Ryan, K., Paik, J., Lane-McKinley, K., Dunn, L. B., Roberts, L. W., & Kim, J. P. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad238 · Cited by 22
Cary, M. P., Grady, S. D., McMillian-Bohler, J. M., Bessias, S., Silcox, C., Silva, S. G., Guilamo‐Ramos, V., McCall, J., Sperling, J. B., & Goldstein, B. A. (2025) · Nursing Outlook · doi:10.1016/j.outlook.2025.102395 · Cited by 23
Self-citationGarg, P. K., Krishna, M., Kulkarni, P., Horne, D. A., Salgia, R., & Singhal, S. S. (2025) · Cancers · doi:10.3390/cancers17172799 · Cited by 25
Palmowski, L., Nowak, H., Witowski, A., Koos, B., Wolf, A., Weber, M., Kleefisch, D., Unterberg, M., Haberl, H., von Busch, A., et al. (2024) · PLoS ONE · doi:10.1371/journal.pone.0300739 · Cited by 25
Hu, F., Zhu, J., Zhang, S., Wang, C., Zhang, L., Zhou, H. J., & Shi, H. (2023) · European journal of medical research · doi:10.1186/s40001-023-01255-8 · Cited by 21
Kim, J. P., Ryan, K., Kasun, M., Hogg, J., Dunn, L. B., & Roberts, L. W. (2023) · JMIR AI · doi:10.2196/47449 · Cited by 20
Aslan, A. T., Permana, B., Harris, P. N. A., Naidoo, K. D., Pienaar, M. A., & Irwin, A. (2023) · Current Infectious Disease Reports · doi:10.1007/s11908-023-00818-4 · Cited by 20
Predicting Therapeutic Response to Unfractionated Heparin Therapy: Machine Learning Approach
Abdel‐Hafez, A., Scott, I. A., Falconer, N., Canaris, S., Bonilla, O., Marxen, S., Garderen, A. V., & Barras, M. A. (2022) · Interactive Journal of Medical Research · doi:10.2196/34533 · Cited by 20
Guiding Clostridioides difficile Infection Prevention Efforts in a Hospital Setting With AI
Tang, S., Shepard, S., Clark, R., Ötleş, E., Udegbunam, C. J., Tran, J., Seiler, M., Ortwine, J., Waljee, A. K., Nagel, J. L., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.15213 · Cited by 16
Gonem, S., Taylor, A., Figueredo, G. P., Forster, S., Quinlan, P., Garibaldi, J. M., McKeever, T. M., & Shaw, D. E. (2022) · Respiratory Research · doi:10.1186/s12931-022-02130-6 · Cited by 22
Sax, D. R., Sturmer, L. R., Mark, D. G., Rana, J. S., & Reed, M. (2022) · Diagnostics · doi:10.3390/diagnostics12102463 · Cited by 15
Pivotal challenges in artificial intelligence and machine learning applications for neonatal care
Jeong, H., & Kamaleswaran, R. (2022) · Seminars in Fetal and Neonatal Medicine · doi:10.1016/j.siny.2022.101393 · Cited by 21
Physicians’ Perceptions of Clinical Decision Support to Treat Patients With Heart Failure in the ED
Casey, S. D., Reed, M., LeMaster, C., Mark, D. G., Gaskin, J., Norris, R. P., & Sax, D. R. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.44393 · Cited by 16
Verma, A. A., Trbovich, P., Mamdani, M., & Shojania, K. G. (2023) · BMJ Quality & Safety · doi:10.1136/bmjqs-2022-015713 · Cited by 19
Ning, J., Spielvogel, C. P., Haberl, D., Trachtová, K., Stoiber, S., Rasul, S., Bystrý, V., Wasinger, G., Baltzer, P., Gurnhofer, E., et al. (2024) · Theranostics · doi:10.7150/thno.96921 · Cited by 19
End user experience of a widely used artificial intelligence based sepsis system
Owoyemi, A., Okpara, E., Salwei, M. E., & Boyd, A. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooae096 · Cited by 15
Machine learning methods for predicting adverse drug events: A systematic review
Chalabianloo, N., Ahmadi, F., Omrani, M. A., Abdullah, S. S., Rostamzadeh, N., Jafari, A., Izzedin, L., Sedig, K., & Muanda, F. T. (2025) · British Journal of Clinical Pharmacology · doi:10.1002/bcp.70377 · Cited by 17
Oh, S. S., Kuang, I., Jeong, H., Song, J., Ren, B., Moon, J. Y., Park, E., & Kawachi, I. I. (2023) · Journal of Medical Internet Research · doi:10.2196/45041 · Cited by 11
Salybekov, A. A., Wolfien, M., Hahn, W., Hidaka, S., & Kobayashi, S. (2024) · Biomedicines · doi:10.3390/biomedicines12030606 · Cited by 20
"It depends": Configuring AI to Improve Clinical Usefulness Across Contexts
Zając, H. D., Ribeiro, J., Ingala, S., Gentile, S., Wanjohi, R., Gitau, S. N., Carlsen, J. F., Nielsen, M. B., & Andersen, T. O. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3660707 · Cited by 23
Wan, Y.-K. J., Wright, M. C., McFarland, M. M., Dishman, D., Nies, M. A., Rush, A., Madaras-Kelly, K., Jeppesen, A., & Del Fiol, G. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad203 · Cited by 11
Lazzarino, R., Borek, A., Honeyford, K., Welch, J., Brent, A., Kinderlerer, A. R., Cooke, G., Patil, S., Gordon, A. C., Glampson, B., et al. (2024) · JMIR Human Factors · doi:10.2196/56949 · Cited by 17
Cheng, Y., Li, J., Lee, J., & Liu, Y. (2025) · Journal of Risk Research · doi:10.1080/13669877.2025.2579317 · Cited by 20
Strechen, I., Wilson, P. M., Eltalhi, T., Piche, K., Tschida-Reuter, D., Howard, D., Sutor, B., Tiong, I., Herasevich, S., Pickering, B. W., et al. (2024) · Trials · doi:10.1186/s13063-024-08254-y · Cited by 12
Zając, H. D., Andersen, T. O., Kwasa, E., Wanjohi, R., Onyinkwa, M. K., Mwaniki, E. K., Gitau, S. N., Yaseen, S. S., Carlsen, J. F., Fraccaro, M., et al. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3715115 · Cited by 22
Trust in Clinical AI: Expanding the Unit of Analysis
Browne, J., Bakker, S., Yu, B., Lloyd, P. A., & Allouch, S. B. (2022) · Frontiers in artificial intelligence and applications · doi:10.3233/faia220192 · Cited by 15
Clinician Perspectives on Decision Support and AI-based Decision Support in a Pediatric ED
Ramgopal, S., Macy, M. L., Hayes, A. A., Florin, T. A., Carroll, M. S., & Kshetrapal, A. (2024) · Hospital Pediatrics · doi:10.1542/hpeds.2023-007653 · Cited by 11
Silvestri, J. A., Kmiec, T. E., Bishop, N. S., Regli, S. H., & Weissman, G. E. (2022) · JMIR Human Factors · doi:10.2196/36976 · Cited by 12
Sandhu, S., Sendak, M., Ratliff, W., Knechtle, W. S., Fulkerson, W. J., & Balu, S. (2023) · Patterns · doi:10.1016/j.patter.2023.100710 · Cited by 16
Self-citationAllen, M. J., James, C. R., Frost, J., Liabo, K., Pearn, K., Monks, T., Zhelev, Z., Logan, S., Everson, R., James, M. A., et al. (2022) · Health and Social Care Delivery Research · doi:10.3310/gvzl5699 · Cited by 10
Machine learning in oncological pharmacogenomics: advancing personalized chemotherapy
Avcı, Ç. B., Bağca, B. G., Shademan, B., Takanlou, L. S., Takanlou, M. S., & Nourazarian, A. (2024) · Functional & Integrative Genomics · doi:10.1007/s10142-024-01462-4 · Cited by 16
Ratzki‐Leewing, A. A., Ryan, B. L., Zou, G., Webster-Bogaert, S., Black, J. E., Stirling, K., Timcevska, K., Khan, N., Buchenberger, J. D., & Harris, S. B. (2022) · JMIR Research Protocols · doi:10.2196/33726 · Cited by 8
Hogg, H. D. J., Sendak, M., Denniston, A. K., Keane, P. A., & Maniatopoulos, G. (2023) · Journal of Medical Artificial Intelligence · doi:10.21037/jmai-23-28 · Cited by 12
Self-citationLu, G., Zhang, J., Shi, T., Liu, Y., Gao, X., Zeng, Q., Ding, J., Chen, J., Yang, K., Ma, Q., et al. (2023) · Microbiology Spectrum · doi:10.1128/spectrum.03096-23 · Cited by 9
Whitney, C., Preis, H., & Vargas, A. R. (2025) · SSM - Qualitative Research in Health · doi:10.1016/j.ssmqr.2025.100540 · Cited by 11
Noselli, M., van der Vegt, A. H., Cordeil, M., Campbell, V. K., Wang, A. P., Shetty, A., & Scott, I. A. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3572494 · Cited by 6
Wingert, T., Williams, T., Syed, B., Hill, B., Grogan, T., Young, A., Antongiorgi, Z. D., Salari, V., Joosten, A., Hofer, I. S., et al. (2026) · British Journal of Anaesthesia · doi:10.1016/j.bja.2025.11.042 · Cited by 8
A Human-Centered Approach to Interpretable Machine Learning in Clinical Decision Support Systems
Nimbalkar, V., Chawla, L., Adnan, M. M., Bhansali, A., Gupta, M., & Kalra, R. (2025) · International Conference on Artificial Intelligence for Innovations in Healthcare Industries (ICAIIHI) · doi:10.1109/icaiihi67124.2025.11403473 · Cited by 7
PAULRAJ, N. J. (2025) · Journal of Computer Science and Technology Studies · doi:10.32996/jcsts.2025.7.3.56 · Cited by 7
Raszke, P., Giebel, G. D., Abels, C., Wasem, J., Adamzik, M., Nowak, H., Palmowski, L., Heinz, P., Mreyen, S., Timmesfeld, N., et al. (2024) · JMIR Research Protocols · doi:10.2196/62704 · Cited by 7
Wang, S. M., Hogg, H. D. J., Sangvai, D. G., Patel, M. R., Weissler, E. H., Kellogg, K. C., Ratliff, W., Balu, S., & Sendak, M. (2023) · JMIR Formative Research · doi:10.2196/43963 · Cited by 11
Self-citationLin, A. L., Parrish, A. B., Cary, M. P., Silcox, C., Balu, S., Jelovsek, J. E., O’Brien, C. L., Pencina, M. J., Poon, E. G., & Economou-Zavlanos, N. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-094601 · Cited by 10
Self-citationNurses’ Perspectives on AI in Clinical Practice: Implications for Training and Workforce Development
Lansdowne, M., Sperling, J. B., Lytle, K. S., Allen, D., Merrill, T., & Cary, M. P. (2025) · Nursing Administration Quarterly · doi:10.1097/naq.0000000000000710 · Cited by 6
Self-citationPALLM: Evaluating and Enhancing Palliative Care Conversations with Large Language Models
Wang, Z., Yuan, F., LeBaron, V. T., Flickinger, T., & Barnes, L. E. (2025) · ACM Transactions on Computing for Healthcare · doi:10.1145/3712300 · Cited by 15
Wu, X., Hu, F., Zhang, T., Pan, Y., He, J., Zhang, R., Zhou, H., & Shi, H. (2025) · Scientific Reports · doi:10.1038/s41598-025-10119-6 · Cited by 4
Nguyen, N. B.-T., Vo, Q.-H.-Q., Le, H.-M., Huynh, N. T., Phan, Q.-H., & Pham, T.-T.-H. (2024) · Optik · doi:10.1016/j.ijleo.2024.171932 · Cited by 4
Factors Influencing Trust in Artificial Intelligence Interventions in Healthcare: A Rapid Review
Murray, L., & Montague, E. N. H. (2025) · Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care · doi:10.1177/2327857925141010 · Cited by 4
Advancements and Challenges in Artificial Intelligence Applications in Healthcare Delivery SystemsPreprint
Aftab, M., Nadeem, A., Zhang, C., Dong, Z., Jiang, Y., & Liu, K. (2024) · Preprints.org · doi:10.20944/preprints202407.0938.v1 · Cited by 5
Metadata
- Authors
- Sandhu, S., Lin, A. L., Brajer, N., Sperling, J., Ratliff, W., Bedoya, A. D., Balu, S., O'Brien, C., & Sendak, M. P.
- Affiliations
- Duke University (US)
- Corresponding author
- Sandhu, Sahil
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2020-11-19
- Volume
- 22
- Issue
- 11
- Pages
- e22421
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC7714645
- OpenAlex
- W3094937636
- Cited by
- 168
- References
- 50
- Citations by year
- 2026: 37 · 2025: 35 · 2024: 38 · 2023: 25 · 2022: 23 · 2021: 10
- FWCI
- 4.29
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Critical Care and Intensive Care Medicine › Sepsis Diagnosis and Treatment
- Keywords
- machine learning integration, clinical workflow, implementation barriers, clinician trust, perceived usefulness, emergency physicians, rapid response teams, nurse-physician communication, clinical decision support, technology adoption
- MeSH terms
- Machine Learning, Humans, Qualitative Research, Workflow
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
