Real-World Integration of a Sepsis Deep Learning Technology Into Routine Clinical Care: Implementation Study
Open accessPublished version of a preprint
The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Abstract
Background: Successful integrations of machine learning into routine clinical care are exceedingly rare, and barriers to its adoption are poorly characterized in the literature. Objective: This study aims to report a quality improvement effort to integrate a deep learning sepsis detection and management platform, Sepsis Watch, into routine clinical care. Methods: In 2016, a multidisciplinary team consisting of statisticians, data scientists, data engineers, and clinicians was assembled by the leadership of an academic health system to radically improve the detection and treatment of sepsis. This report of the quality improvement effort follows the learning health system framework to describe the problem assessment, design, development, implementation, and evaluation plan of Sepsis Watch. Results: Sepsis Watch was successfully integrated into routine clinical care and reshaped how local machine learning projects are executed. Frontline clinical staff were highly engaged in the design and development of the workflow, machine learning model, and application. Novel machine learning methods were developed to detect sepsis early, and implementation of the model required robust infrastructure. Significant investment was required to align stakeholders, develop trusting relationships, define roles and responsibilities, and to train frontline staff, leading to the establishment of 3 partnerships with internal and external research groups to evaluate Sepsis Watch. Conclusions: Machine learning models are commonly developed to enhance clinical decision making, but successful integrations of machine learning into routine clinical care are rare. Although there is no playbook for integrating deep learning into clinical care, learnings from the Sepsis Watch integration can inform efforts to develop machine learning technologies at other health care delivery systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Conversely, alerts that are poorly integrated into workflow can increase interruptions, contribute to unnecessary antibiotics, and shift attention away from other competing tasks.”
From Artificial Intelligence to Facilitate SEP-1 Measure Compliance and Fluid Management in Sepsis (2026) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Sendak et al. performed a single-center implementation study integrating a deep learning-based sepsis detection platform into the emergency department of an academic health system. They used EHR time-series data from ~39,918 adult ED encounters to develop and deploy a multitask Gaussian processes (MGPs)–recurrent neural network (RNN) model within a real-time clinical workflow.”
From Artificial Intelligence to Facilitate SEP-1 Measure Compliance and Fluid Management in Sepsis (2026) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Studies of sepsis alerts frequently report improvements in process measures such as time-to-antibiotics or completion of key bundle elements, particularly when these alerts are coupled to structured triage, escalation, and rapid clinician response.”
From Artificial Intelligence to Facilitate SEP-1 Measure Compliance and Fluid Management in Sepsis (2026) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Implementation barriers for sepsis AI cluster around trust and interpretability, workflow integration, technical readiness and generalizability, governance and safety, leadership alignment, and the resources required for sustainment (Table 4) [10,33,34,46,48,77,78].”
From Artificial Intelligence to Facilitate SEP-1 Measure Compliance and Fluid Management in Sepsis (2026) · cites this paper as [10]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The system was successfully implemented with high clinician engagement and continuous risk prediction, demonstrating feasibility of real-world integration but requiring substantial infrastructure and organizational coordination [10].”
From Artificial Intelligence to Facilitate SEP-1 Measure Compliance and Fluid Management in Sepsis (2026) · cites this paper as [10]
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The third international consensus definitions for sepsis and septic shock (Sepsis-3).
Singer, M., Deutschman, C. S., Seymour, C. W., Shankar-Hari, M., Annane, D., Bauer, M., Bellomo, R., Bernard, G. R., Chiche, J.-D., Coopersmith, C. M., et al. (2016) · JAMA · doi:10.1001/jama.2016.0287 · Cited by 28,250
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496 · cited 1×
“The next goal to drive adoption was to broadly communicate the change vision and empower action [43,44].”
BackgroundHigh-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450 · cited 1×
“Machine learning, a set of statistical methods optimized for prediction on new observations, is central to this transformation [1].”
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
Surviving Sepsis Campaign: International guidelines for management of sepsis and septic shock: 2016.
Rhodes, A., Evans, L. E., Alhazzani, W., Levy, M. M., Antonelli, M., Ferrer, R., Kumar, A., Sevransky, J. E., Sprung, C. L., Nunnally, M. E., et al. (2017) · Intensive Care Medicine · doi:10.1007/s00134-017-4683-6 · Cited by 6,621
Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor
Lebovits, H. (2019) · Public Integrity · doi:10.1080/10999922.2018.1511671 · Cited by 3,279
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · cited 1×
“All new machine learning implementations have the potential for introducing inequality and bias that is not always clearly visible in numeric data [45-48].”
BackgroundRecurrent Neural Networks for Multivariate Time Series with Missing Values
Che, Z., Purushotham, S., Cho, K., Sontag, D., & Liu, Y. (2018) · Scientific Reports · doi:10.1038/s41598-018-24271-9 · Cited by 2,370 · cited 1×
“RNNs are a type of deep learning configured to ingest time series data and are ideally suited to combine static and dynamic features of hospital encounters that vary in length [39,40].”
BackgroundAbràmoff, M. D., Lavin, P. T., Birch, M., Shah, N., & Folk, J. C. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0040-6 · Cited by 1,638 · cited 1×
“Isolated efforts demonstrating the clinical impact of previously validated technologies show the potential of machine learning in health care [3,4].”
BackgroundPrognosis Research Strategy (PROGRESS) 3: Prognostic Model Research
Steyerberg, E. W., Moons, K. G. M., van der Windt, D. A., Hayden, J. A., Perel, P., Schroter, S., Riley, R. D., Hemingway, H., Altman, D. G., & for the PROGRESS Group (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001381 · Cited by 1,508
Research works printed but not linked: 5
- SculleyDHoltGGolovinDHidden Technical Debt in Machine Learning SystemsNIPS Proceedings20142020-06-16https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf
- SariaSSubbaswamyATutorial: Safe and Reliable Machine LearningACM Conference on Fairness, Accountability, and Transparency201901FAT'19January 29-31, 2019Atlanta, USA
- 10.5555/3305381.3305503 · doi:10.5555/3305381.3305503
- Barocas, S, 2016, Cal L Rev, 104, 671
- Moore, G, 2006, Crossing the Chasm: Marketing and Selling High-Tech Products to Mainstream Customers
Cited by
A Systematic Review of the Barriers to the Implementation of Artificial Intelligence in Healthcare
Ahmed, M. I., Spooner, B., Isherwood, J., Lane, M., Orrock, E., & Dennison, A. (2023) · Cureus · doi:10.7759/cureus.46454 · Cited by 364
Power to the People? Opportunities and Challenges for Participatory AI
Birhane, A., Isaac, W., Prabhakaran, V., Diaz, M., Elish, M. C., Gabriel, I., & Mohamed, S. (2022) · Equity and Access in Algorithms Mechanisms and Optimization · doi:10.1145/3551624.3555290 · Cited by 330
“The recent upswing in participatory tools have varied in approach and point in the machine learning product lifecycle, including: auditing methods for machine learning based interventions [48, 75], public consultation methods such as citizen juries [6, 83] or joint problem formulation [61], information disclosures suc…”
BackgroundSelf-citationThe Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice
Delgado, F., Yang, S., Madaio, M., & Yang, Q. (2023) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3617694.3623261 · Cited by 256
Artificial Intelligence Applications in Health Care Practice: Scoping Review
Sharma, M., Savage, C., Nair, M., Larsson, I., Svedberg, P., & Nygren, J. M. (2022) · Journal of Medical Internet Research · doi:10.2196/40238 · Cited by 198
“However, the study by Sendak et al [46] encouraged the separation of developers and clinicians and made conscious efforts to shift focus away from the technical aspects of AI.”
InfluentialNair, M., Svedberg, P., Larsson, I., & Nygren, J. M. (2024) · PLOS ONE · doi:10.1371/journal.pone.0305949 · Cited by 179
Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study
Sandhu, S., Lin, A. L., Brajer, N., Sperling, J., Ratliff, W., Bedoya, A. D., Balu, S., O'Brien, C., & Sendak, M. P. (2020) · Journal of Medical Internet Research · doi:10.2196/22421 · Cited by 168
“According to our institutional definition for sepsis, over 20% of adults admitted through the DUH ED develop sepsis [31], and nearly 68% of sepsis occurs within the first 24 hours of hospital encounter [32].”
BackgroundSelf-citationA Path for Translation of Machine Learning Products into Healthcare Delivery
(2020) · EMJ Innovations · doi:10.33590/emjinnov/19-00172 · Cited by 172
Self-citationImpact of a deep learning sepsis prediction model on quality of care and survival
Boussina, A., Shashikumar, S. P., Malhotra, A., Owens, R. L., El-Kareh, R., Longhurst, C. A., Quintero, K., Donahue, A., Chan, T. C., Nemati, S., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-023-00986-6 · Cited by 170
“To the best of our knowledge, the only deep-learning model previously tested in an ED setting is the Sepsis Watch by investigators at Duke; however, no patient-centered outcomes have been reported thus far 28 .”
Chomutare, T., Tejedor, M., Svenning, T. O., Marco-Ruiz, L., Tayefi, M., Lind, K., Godtliebsen, F., Moen, A., Ismail, L., Makhlysheva, A., et al. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph192316359 · Cited by 142
“Sendak [40] (2020) USA Sepsis Screening Clinicians Deep Learning”
BackgroundInfluentialSivaraman, V., Bukowski, L. A., Levin, J., Kahn, J. M., & Perer, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581075 · Cited by 141
“to sepsis, early-warning systems such as Sepsis Watch and the Targeted Real-time Early Warning System (TREWS) have been accepted by clinicians at the hospitals where they are deployed [35, 69], despite being initially met with ambivalence [34].”
BackgroundRole of artificial intelligence in revolutionizing drug discovery
Rehman, A. U., Li, M., Wu, B., Ali, Y., Rasheed, S., Shaheen, S., Liu, X., Luo, R., & Zhang, J. (2024) · Fundamental Research · doi:10.1016/j.fmre.2024.04.021 · Cited by 209
“An exemplary case is the clinical decision support system (CDSS) known as “Sepsis Watch ” to detect Sepsis early [ 27 ], a severe medical condition that occurs when an infection in the body triggers a chain reaction, resulting in a life-threatening medical emergency, often originating from…”
Artificial intelligence sepsis prediction algorithm learns to say “I don’t know”
Shashikumar, S. P., Wardi, G., Malhotra, A., & Nemati, S. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00504-6 · Cited by 125
Henry, K. E., Adams, R., Parent, C., Soleimani, H., Sridharan, A., Johnson, L., Hager, D. N., Cosgrove, S. E., Markowski, A., Klein, E. Y., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01895-z · Cited by 144
“In sepsis, based on promising retrospective validation, a growing number of tools have been deployed to clinical settings 21,44–49 .”
Rahimi, A. K., Pienaar, O., Ghadimi, M., Canfell, O. J., Pole, J. D., Shrapnel, S., van der Vegt, A. H., & Sullivan, C. M. (2024) · Journal of Medical Internet Research · doi:10.2196/49655 · Cited by 164
“[39,40,42-49, 51, 54 ], qualitative (4 /”
InfluentialThe lifecycle of algorithmic decision-making systems: Organizational choices and ethical challenges
Marabelli, M., Newell, S., & Handunge, V. R. (2021) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2021.101683 · Cited by 170
“This subverted the typical processes at a hospital where it is doctors alerting nurses that a task needs to be performed – in fact several nurses reported being afraid of doing so (Sendak et al. 2020).”
Implementation frameworks for end-to-end clinical AI: derivation of the SALIENT framework
van der Vegt, A. H., Scott, I. A., Dermawan, K., Schnetler, R. J., Kalke, V. R., & Lane, P. J. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad088 · Cited by 109
Dailah, H. G., Koriri, M. D., Sabei, A., Kriry, T., & Zakri, M. (2024) · Healthcare · doi:10.3390/healthcare12242555 · Cited by 148
“Over the past half-century, AI in healthcare has experienced significant evolution, especially with the introduction of machine learning (ML) and deep learning (DL) technologies [65,66].”
Rajashekar, N. C., Shin, Y. E., Pu, Y., Chung, S., You, K., Giuffre, M., Chan, C. E., Saarinen, T., Hsiao, A., Sekhon, J., et al. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642024 · Cited by 85
“Previous eforts at qualitatively characterizing healthcare provider interaction with AI systems has spanned multiple domains, including alert systems [88, 92], risk estimators [10, 12], and image-retrieval [16].”
BackgroundUniversity of California, S. F., Adler-Milstein, J., Aggarwal, N., University of Wisconsin-Madison, Ahmed, M., National Academy of Medicine, Castner, J., Incorporated, C., Evans, B. J., University of Florida, et al. (2022) · NAM Perspectives · doi:10.31478/202209c · Cited by 82
“To avoid drift in workflow or use of AI tools, project leaders should clearly document algorithms and regularly train staff on appropriate use (Sendak et al., 2020c).”
BackgroundMethodSelf-citationConceptualising fairness: three pillars for medical algorithms and health equity
Sikstrom, L., Maslej, M. M., Hui, K., Findlay, Z., Buchman, D. Z., & Hill, S. L. (2022) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2021-100459 · Cited by 76
Evaluation of sepsis prediction models before onset of treatment
Kamran, F., Tjandra, D., Heiler, A., Virzi, J., Singh, K., King, J. E., Valley, T. S., & Wiens, J. (2024) · NEJM AI · doi:10.1056/aioa2300032 · Cited by 74
Zając, H. D., Li, D., Dai, X., Carlsen, J. F., Kensing, F., & Andersen, T. O. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3582430 · Cited by 74
AI and professional liability assessment in healthcare. a revolution in legal medicine?
Terranova, C., Cestonaro, C., Fava, L., & Cinquetti, A. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2023.1337335 · Cited by 73
Artificial Intelligence for Clinical Decision Support in Sepsis
Wu, M., Du, X., Gu, R., & Wei, J. (2021) · Frontiers in Medicine · doi:10.3389/fmed.2021.665464 · Cited by 70
van der Vegt, A. H., Scott, I. A., Dermawan, K., Schnetler, R. J., Kalke, V. R., & Lane, P. J. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad075 · Cited by 72
“Similarly, whether to develop and implement the MLA in-house or use an external vendor (D10), which involved weighing up the capability to future-proof the organization for future AI solutions(41) versus implementation and maintenance challenges arising from separate ownership of the input data and the AI model.(41) T…”
BackgroundMethodInfluentialSingh, H., Mhasawade, V., & Chunara, R. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000023 · Cited by 102
Botha, N. N., Segbedzi, C. E., Dumahasi, V. K., Maneen, S., Kodom, R. V., Tsedze, I. S., Akoto, L. A., Atsu, F. S., Lasim, O. U., & Ansah, E. W. (2024) · Archives of Public Health · doi:10.1186/s13690-024-01414-1 · Cited by 92
“[2, 76, 77, 119, 122, 133].”
BackgroundPreti, L. M., Ardito, V., Compagni, A., Petracca, F., & Cappellaro, G. (2024) · Journal of Medical Internet Research · doi:10.2196/55897 · Cited by 62
“Regardless, feedback collection was described as an iterative activity [46,48,60,61], which also positively influenced professional buy-in [47].”
InfluentialTaylor, R. A., Sangal, R. B., Smith, M. E., Haimovich, A. D., Rodman, A., Iscoe, M. S., Pavuluri, S. K., Rose, C., Janke, A. T., Wright, D. S., et al. (2025) · Academic Emergency Medicine · doi:10.1111/acem.15066 · Cited by 62
“Most health care systems lack the machine learning operations (MLOps) infrastructure necessary to deploy and maintain AI tools effectively (e.g., SepsisWatch implementation 38 ).”
Communications of the Association for Information SystemsPreprint
Marabelli, M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5250701 · Cited by 60
Beldhuis, I. E., Marapin, R. S., Jiang, Y. Y., Simões de Souza, N. F., Georgiou, A., Kaufmann, T., Castela Forte, J., & van der Horst, I. C. C. (2021) · Journal of Critical Care · doi:10.1016/j.jcrc.2021.04.012 · Cited by 58
The use of AI in mental health services to support decision-making: scoping review
Auf, H., Svedberg, P., Nygren, J., Nair, M., & Lundgren, L. E. (2025) · Journal of Medical Internet Research · doi:10.2196/63548 · Cited by 56
“This step requires the inclusion of multidisciplinary individuals who will engage with the workflow using simulations or prototypes to enable the detection of any barriers or potential risks before putting the integrated AI systems in place [98,99].”
Tulk Jesso, S., Kelliher, A., Sanghavi, H., Martin, T., & Henrickson Parker, S. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.830345 · Cited by 54
“…proposed across multiple medical domains addressing issues such as clinician burnout (Arndt et al., 2017; Ash et al., 2019), medical errors (Cai et al., 2019a; Van Camp et al., 2019), and detecting frequently unrecognized and life-threatening conditions such as sepsis (Sendak et al., 2020).”
BackgroundInfluentialSusanto, A. P., Lyell, D., Widyantoro, B., Berkovsky, S., & Magrabi, F. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad180 · Cited by 52
Abbas, S. R., Seol, H., Abbas, Z., & Lee, S. W. (2025) · Healthcare · doi:10.3390/healthcare13141642 · Cited by 80
Bernstam, E. V., Shireman, P. K., Meric‐Bernstam, F., Zozus, M. N., Jiang, X., Brimhall, B. B., Windham, A. K., Schmidt, S., Visweswaran, S., Ye, Y., et al. (2021) · Clinical and Translational Science · doi:10.1111/cts.13175 · Cited by 67
From understanding diseases to drug design: can artificial intelligence bridge the gap?
Pushkaran, A. C., & Arabi, A. A. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10714-5 · Cited by 72
“The platform was trained with 50,000 patient records involving over 32 million data points, and it was proven to improve sepsis patient care (Sendak et al. 2020).”
Nahar, N., Zhang, H., Lewis, G. A., Zhou, S., & Kästner, C. (2023) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain58948.2023.00034 · Cited by 85
“For challenges of unrealistic requirements, several studies mentioned that practitioners found it useful to conduct training sessions with clients and other team members on AI literacy, before starting ML projects [77], [95], [106], [119].”
Integrating artificial intelligence into healthcare systems: more than just the algorithm
Kwong, J. C. C., Nickel, G., Wang, S., & Kvedar, J. C. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01066-z · Cited by 72
Machine Learning Operations in Health Care: A Scoping Review
Rajagopal, A., Ayanian, S., Ryu, A. J., Qian, R., Legler, S. R., Peeler, E. A., Issa, M., Coons, T. J., & Kawamoto, K. (2024) · Mayo Clinic Proceedings: Digital Health · doi:10.1016/j.mcpdig.2024.06.009 · Cited by 52
“66 Achieving this can be facilitated by priming stakeholders before introduction, 67 offering intuitive interfaces, providing continuous education, implementing in phases (particularly for complex interventions), 68,69 and iterating based on periodic feedback.”
Transforming sepsis management: AI-driven innovations in early detection and tailored therapies
Papareddy, P., Lobo, T. J., Holub, M., Bouma, H., Maca, J., Strodthoff, N., & Herwald, H. (2025) · Critical Care · doi:10.1186/s13054-025-05588-0 · Cited by 49
“This framework highlights the potential of AI to bridge molecular insights and clinical decision-making in sepsis management based on continuously updated data ensures that treat-ment strategies remain relevant as patients’ conditions evolve [113].”
A framework for understanding label leakage in machine learning for health care
Davis, S. E., Matheny, M. E., Balu, S., & Sendak, M. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad178 · Cited by 76
Self-citationTang, K., Xiang, Y., Tian, J., Hou, J., Chen, X., Wang, X., & Zhong, Z. (2023) · International Journal of Mechanical Sciences · doi:10.1016/j.ijmecsci.2023.108956 · Cited by 45
The AI Future of Emergency Medicine
Petrella, R. J. (2024) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2024.01.031 · Cited by 43
Kashyap, S., Morse, K. E., Patel, B., & Shah, N. H. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab154 · Cited by 43
Clinical deployment environments: Five pillars of translational machine learning for health
Harris, S. K., Bonnici, T., Keen, T., Lilaonitkul, W., White, M. J., & Swanepoel, N. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.939292 · Cited by 54
“This efficiency will again depend on the problem space: where the algorithmic target depends on information arising from the EHR rather than an isolated device or image, and where the pathway involves multiple end-users, then successful implementation will be near impossible if done sequentially (development then depl…”
BackgroundChan, S. L., Lee, J. W., Ong, M. E. H., Siddiqui, F. J., Graves, N., Ho, A. F. W., & Liu, N. (2023) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2023.02.001 · Cited by 49
Twala, B. (2025) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2025.1638340 · Cited by 54
A smarter perspective: Learning with and from AI-cases
Arbelaez Ossa, L., Rost, M., Lorenzini, G., Shaw, D. M., & Elger, B. S. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2022.102458 · Cited by 39
van der Meijden, S. L., de Hond, A. A. H., Thoral, P. J., Steyerberg, E. W., Kant, I. M. J., Cinà, G., & Arbous, M. S. (2023) · JMIR Human Factors · doi:10.2196/39114 · Cited by 39
“We want to stress that addressing explainability is not the only factor required for a successful AI-CDS implementation; rather, the incorporation of domain expertise, the sociotechnical context, and physicians’ perspectives should be taken into account during the whole development, design, and implementation process…”
BackgroundMethodGalsgaard, A., Doorschodt, T., Holten, A.-L., Müller, F. C., Ploug Boesen, M., & Maas, M. (2022) · European Journal of Radiology · doi:10.1016/j.ejrad.2022.110231 · Cited by 38
ter Avest, E., Carenzo, L., Lendrum, R. A., Christian, M. D., Lyon, R., Coniglio, C., Rehn, M., Lockey, D., & Perkins, Z. (2022) · Critical Care · doi:10.1186/s13054-022-04052-7 · Cited by 38
Cary, M. P., Bessias, S., McCall, J., Pencina, M. J., Grady, S. D., Lytle, K. S., & Economou-Zavlanos, N. (2024) · Journal of Nursing Scholarship · doi:10.1111/jnu.13007 · Cited by 50
“One example of such clinical tools is Duke Health's Sepsis Watch (Sendak et al., 2020 ), a deep learning application that analyzes over 32 million data points to assess a patient's risk for developing sepsis and guide the hospital's rapid response team through the first 3 h of care administration.”
Macrae, C. (2024) · Risk Analysis · doi:10.1111/risa.14273 · Cited by 57
“An enormous amount of work is currently underway to understand how to safely deploy, integrate, use, and manage AI in different health-care settings (CQC, 2020; Feng et al., 2022; Sendak, 2020; Svedberg et al., 2022; Wiens et al., 2019; Winter & Carusi, 2023).”
Nair, M., Nygren, J., Nilsén, P., Gama, F., Neher, M., Larsson, I., & Svedberg, P. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1550459 · Cited by 49
“…detailing clinicians ’ responsibilities in the context of the AI system (40, 44), (Expert 3), (Case 1, Case 2) (cid:129) Addressing algorithm ’ s explainability, availability, quality and safety (45, 48) (cid:129) Using a thoughtful framing about AI in communication with the stakeholders (13)”
InfluentialChallenges of developing artificial intelligence‐assisted tools for clinical medicine
Shung, D. L., & Sung, J. J. �. (2021) · Journal of Gastroenterology and Hepatology · doi:10.1111/jgh.15378 · Cited by 33
Santus, E., Marino, N., Cirillo, D., Chersoni, E., Montagud, A., Chadha, A. S., Valencia, A., Hughes, K. S., & Lindvall, C. J. (2021) · Journal of Medical Internet Research · doi:10.2196/22453 · Cited by 36
“Within two years from the tool introduction, the number of sepsis-induced patients drastically decreased [25], thanks to three key elements: (1) adaptation of the predictive model to a highly specific context; (2) scalability through integration with hospital workflows; and (3) the adopted user…”
Machine learning for healthcare that matters: Reorienting from technical novelty to equitable impact
Balagopalan, A., Baldini, I., Celi, L. A., Gichoya, J. W., McCoy, L. G., Naumann, T., Shalit, U., van der Schaar, M., & Wagstaff, K. L. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000474 · Cited by 42
Enabling collaborative governance of medical AI
Price, W. N., Sendak, M., Balu, S., & Singh, K. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00699-1 · Cited by 27
Self-citationRecommendations to promote fairness and inclusion in biomedical AI research and clinical use
Griffin, A. C., Wang, K., Leung, T. I., & Facelli, J. C. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104693 · Cited by 29
Scaling up artificial intelligence to curb infectious diseases in Africa
Otaigbe, I. I. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.1030427 · Cited by 42
“Sepsis Watch is a Clinical Decision Support System designed for early identi fi cation of patients at risk for Sepsis (14).”
Background"What is Your Envisioned Future?": Toward Human-AI Enrichment in Data Work of Asthma Care
Su, Z., He, L., Jariwala, S. P., Zheng, K., & Chen, Y. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555157 · Cited by 41
Nair, M., Andersson, J., Nygren, J., & Lundgren, L. E. (2023) · JMIR Formative Research · doi:10.2196/47335 · Cited by 28
“The liability concerns that were brought up by the stakeholders provide a pressing need and one more question to the ongoing liability discussion [24,25]: Who is to be held responsible when the clinician disagrees with the application’s outcomes and takes a different course of action that results in an adverse event?…”
BackgroundValan, B., Prakash, A., Ratliff, W., Gao, M., Muthya, S., Thomas, A., Eaton, J. L., Gardner, M., Nichols, M., Revoir, M., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01664-5 · Cited by 21
“From data curation to quality assurance and continuous monitoring, successful integration requires a multifaceted and deliberate approach 28,29 .”
Self-citationCheng, R. K.-H., Aggarwal, A. K., Chakraborty, A., Harish, V., McGowan, M., Roy, A., Szulewski, A., & Nolan, B. (2024) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2024.05.020 · Cited by 27
Invisible clinical labor driving the successful integration of AI in healthcare
Ulloa, M., Rothrock, B., Ahmad, F. S., & Jacobs, M. L. (2022) · Frontiers in Computer Science · doi:10.3389/fcomp.2022.1045704 · Cited by 25
“Few examples exist outlining this translation, but some practices are highlighting the importance of workflow integration (Sendak et al., 2020).”
BackgroundCary, M. P., Grady, S. D., McMillian-Bohler, J. M., Bessias, S., Silcox, C., Silva, S. G., Guilamo‐Ramos, V., McCall, J., Sperling, J. B., & Goldstein, B. A. (2025) · Nursing Outlook · doi:10.1016/j.outlook.2025.102395 · Cited by 23
Stern, A. D., Goldfarb, A., Minssen, T., & Price, W. N. (2022) · NEJM Catalyst · doi:10.1056/cat.21.0242 · Cited by 40
Morris, A. H., Horvat, C. M., Stagg, B. C., Grainger, D. W., Lanspa, M. J., Orme, J. F., Clemmer, T. P., Weaver, L. K., Thomas, F. O., Grissom, C. K., et al. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac143 · Cited by 24
Chen, J., Baxter, S. L., van den Brandt, A., Lieu, A. C., Camp, A. S., Jiun, L., Welsbie, D. S., Moghimi, S., Christopher, M. A., Weinreb, R. N., et al. (2022) · Journal of Glaucoma · doi:10.1097/ijg.0000000000002163 · Cited by 23
“Previous usability studies for CDS tools utilizing de fi ned UX principles have focused on the management of diseases such as sepsis, 48 diabetes, 49,50 and depression, 51 and have mostly been evaluated in a preclinical context with the exception of the SepsisWatch model.”
MethodWang, X., Xiong, D., Cui, S., Duan, B., Ding, G., Huang, Y., & Wang, Q. (2026) · Frontiers in Immunology · doi:10.3389/fimmu.2026.1732079 · Cited by 29
Lui, T. K.-L., Cheung, K. S., & Leung, W. K. (2022) · Hepatology International · doi:10.1007/s12072-022-10370-3 · Cited by 19
“It has already been shown that ML models could apply in many clinical decision-making processes, such as prediction of cardiovascular risk [7], sepsis [8], papilloedema [9], cancer risk [10] and colonic lesion histology [11], etc.”
MethodThe algorithm journey map: a tangible approach to implementing AI solutions in healthcare
Boag, W., Hasan, A., Kim, J. Y., Revoir, M., Nichols, M., Ratliff, W., Gao, M., Zilberstein, S., Samad, Z., Hoodbhoy, Z., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01061-4 · Cited by 19
“…studies have developed, researchers have begun identifying “ lessons learned ” to help increase the adoption of AI software in health systems, including: how to conduct a silent trial 9 , reliability and fairness audits 10 , clinical work fl ow 11 , and change management and outcome monitoring 12 .”
BackgroundSelf-citationLearning decision thresholds for risk stratification models from aggregate clinician behavior
Patel, B., Steinberg, E., Pfohl, S. R., & Shah, N. H. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab159 · Cited by 20
“If so, the corresponding threshold on the predicted score from the risk stratification model’s receiver-operating characteristic (ROC) curve becomes the operating point, which is the risk score cutoff that triggers a subsequent action, such as generating an alert for the early detection and treatment of sepsis.(2) Med…”
BackgroundA scoping review of silent trials for medical artificial intelligence
Tikhomirov, L., Semmler, C., Prizant, N., Bhasin, S., Kenyon, G., van der Vegt, A., Erdman, L., Kurian, N. C., Thompson, H., Palmer, L. J., et al. (2026) · Nature Health · doi:10.1038/s44360-025-00048-z · Cited by 17
Self-citationRigg, J., Doyle, O., McDonogh, N., Leavitt, N., Ali, R., Son, A., & Kreter, B. (2023) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2022-100651 · Cited by 24
Nghiem, J., Adler, D. A., Estrin, D. L., Livesey, C., & Choudhury, T. (2023) · JMIR Formative Research · doi:10.2196/47380 · Cited by 24
“For example, machine learning has been used for clinical decision support, documentation summarization, and aid in medical image–based diagnosis [46-49].”
MethodAslan, A. T., Permana, B., Harris, P. N. A., Naidoo, K. D., Pienaar, M. A., & Irwin, A. (2023) · Current Infectious Disease Reports · doi:10.1007/s11908-023-00818-4 · Cited by 20
“This affirms the importance of EHR systems with the functionality to support complex AI systems [77].”
Exploring AI Problem Formulation with Children via Teachable Machines
Dwivedi, U., Elsayed-Ali, S., Bonsignore, E., & Kacorri, H. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642692 · Cited by 23
“There is an increasing literature in participatory machine learning that engages adults, those who typically have domain knowledge but may lack AI expertise, in the problem formulation stages e.g. , [12, 25,82,87].”
BackgroundGonem, S., Taylor, A., Figueredo, G. P., Forster, S., Quinlan, P., Garibaldi, J. M., McKeever, T. M., & Shaw, D. E. (2022) · Respiratory Research · doi:10.1186/s12931-022-02130-6 · Cited by 22
Verma, A. A., Trbovich, P., Mamdani, M., & Shojania, K. G. (2023) · BMJ Quality & Safety · doi:10.1136/bmjqs-2022-015713 · Cited by 19
“63 Beginning the clinical deployment on a small number of units initially, to refine clinical workflows through Plan- Do- Study-Act cycles, may be helpful in advance of organisation-wide deployment.”
MethodDutta, S., McEvoy, D. S., Rubins, D. M., Dighe, A. S., Filbin, M. R., & Rhee, C. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac115 · Cited by 20
Botha, N. N., Ansah, E. W., Segbedzi, C. E., Dumahasi, V. K., Maneen, S., Kodom, R. V., Tsedze, I. S., Akoto, L. A., & Atsu, F. S. (2024) · BMC Digital Health · doi:10.1186/s44247-024-00091-y · Cited by 20
“…are increasingly introduced into the workflow, the incidence of medical errors associated with workload and stress will significantly reduce [32, 34, 42–44, 47–49, 52–55, 57, 58, 61–64, 67, 68, 70, 71, 73, 74, 77, 79–83, 86, 88, 91, 93, 98, 100, 101, 104, 105, 109–112, 115, 116, 118–122, 124].”
BackgroundArtificial intelligence assisted acute patient journey
Nazir, T. F., Rehman, M. M. U., Asghar, M. R., & Kalia, J. S. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.962165 · Cited by 16
“Early analysis showed that the median detection time was 5h before the patient’s deterioration, providing ample time for the physician to confirm the diagnosis and intervene (Sendak et al., 2020).”
BackgroundEditorial: Surfacing best practices for AI software development and integration in healthcare
Sendak, M., Vidal, D. E., Trujillo, S., Singh, K., Liu, X., & Balu, S. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2023.1150875 · Cited by 16
Self-citationȚocu, G., Lisă, E. L., Tutunaru, D., Mihailov, R., Șerban, C., Luțenco, V., Dimofte, F., Guliciuc, M., Chiscop, I., Ștefănescu, B. I., et al. (2025) · Diagnostics · doi:10.3390/diagnostics15172169 · Cited by 23
“They should be visible directly within the EMR interface through interactive tools that display risk scores, the evolution of vital signs, and the determining factors of the prediction [76].”
Safety-driven design of machine learning for sepsis treatment
Jia, Y., Lawton, T., Burden, J., McDermid, J. A., & Habli, I. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103762 · Cited by 22
Verma, A. A., Pou-Prom, C., McCoy, L. G., Murray, J., Nestor, B., Bell, S., Mourad, O., Fralick, M., Friedrich, J. O., Ghassemi, M., et al. (2023) · Critical Care Explorations · doi:10.1097/cce.0000000000000897 · Cited by 20
Owoyemi, A., Osuchukwu, J., Salwei, M. E., & Boyd, A. (2025) · JMIRx Med · doi:10.2196/65565 · Cited by 18
"It depends": Configuring AI to Improve Clinical Usefulness Across Contexts
Zając, H. D., Ribeiro, J., Ingala, S., Gentile, S., Wanjohi, R., Gitau, S. N., Carlsen, J. F., Nielsen, M. B., & Andersen, T. O. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3660707 · Cited by 23
“Researchers also investigated AI’s position in a clinical decision-making process [41] and the rationale behind integration opportunities into clinical practice [39, 66, 68, 90].”
BackgroundMLHOps: Machine Learning Health Operations
Khattak, F. K., Subasri, V., Krishnan, A., Pou-Prom, C., Koçak, S. A., Dolatabadi, E., Pandya, D., Seyyed-Kalantari, L., & Rudzicz, F. (2024) · IEEE Access · doi:10.1109/access.2024.3521279 · Cited by 39
“There are various examples of ML being applied to medical data, including prediction of sepsis [265], in-hospital mortality, prolonged length-of-stay, patient deterioration, and unplanned readmission [240].”
The Diagnostic Classification of the Pathological Image Using Computer Vision
Matsuzaka, Y., & Yashiro, R. (2025) · Algorithms · doi:10.3390/a18020096 · Cited by 18
Ghosh, P., Posner, K. L., Hyland, S. L., Cleve, W. V., Bristow, M., Long, D. R., Palla, K., Nair, B. G., Fong, C. T., Pauldine, R. W., et al. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3589953 · Cited by 14
Smith, M. A., Adelaine, S., Bednarz, L., Patterson, B. W., Pothof, J. J., & Liao, F. (2021) · NEJM Catalyst · doi:10.1056/cat.20.0650 · Cited by 10
Low responsiveness of machine learning models to critical or deteriorating health conditions
Pias, T. S., Afrose, S., Tuli, M. D., Trisha, I. H., Deng, X., Nemeroff, C. B., & Yao, D. D. (2025) · Communications Medicine · doi:10.1038/s43856-025-00775-0 · Cited by 10
“Since then, AI machine learning (ML) based predictive technologies are rapidly made available for incorporation into clinical work fl ows 2 , e.g., for early sepsis detection 3 and predicting surgery time 4 .”
Machine Learning in Optimising Nursing Care Delivery Models: An Empirical Analysis of Hospital Wards
Aslan, M., & Toros, E. (2025) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.70001 · Cited by 16
“Meanwhile, the rise of digital literacy in nursing education emphasises the role of new technologies, such as ML and artificial intelligence (AI), in transforming care delivery and improving efficiency [20, 21].”
Zając, H. D., Andersen, T. O., Kwasa, E., Wanjohi, R., Onyinkwa, M. K., Mwaniki, E. K., Gitau, S. N., Yaseen, S. S., Carlsen, J. F., Fraccaro, M., et al. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3715115 · Cited by 22
“New studies emerge, investigating AI’s place in the broader ecosystem of collaborative medical work practices [56, 110, 116, 145], origin of trust [56, 98], explainability [74], bias [11] or collaboration [17].”
Thieme, A., Rajamohan, A. M., Cooper, B., Groombridge, H., Simister, R. J., Wong, B., Woznitza, N., Pinnock, M. A., Wetscherek, M. T., Morrison, C. P., et al. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3716500 · Cited by 20
Leveraging Artificial Intelligence for Expediting Implementation Efforts
Younas, A., & Reynolds, S. S. (2024) · Creative Nursing · doi:10.1177/10784535241239059 · Cited by 15
“Studies have demonstrated the usefulness of predictive analytics in implementation planning (Amarasingham et al., 2014; Moussa et al., 2021; Ng & Tan, 2021; Sendak et al., 2020; Sharma et al., 2022).”
Thieme, A., Rajamohan, A., Cooper, B., Groombridge, H., Simister, R., Wong, B., Woznitza, N., Pinnock, M. A., Wetscherek, M. T., Morrison, C., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.05299 · Cited by 10
Metadata
- Authors
- Sendak, M. P., Ratliff, W., Sarro, D., Alderton, E., Futoma, J., Gao, M., Nichols, M., Revoir, M., Yashar, F., Miller, C., Kester, K., Sandhu, S., Corey, K., Brajer, N., Tan, C., Lin, A., Brown, T., Engelbosch, S., Anstrom, K., Elish, M. C., Heller, K., Donohoe, R., Theiling, J., Poon, E., Balu, S., Bedoya, A., & O'Brien, C.
- Affiliations
- Duke University (US)
- Duke University Hospital (US)
- Harvard University (US)
- Duke Clinical Research Institute (US)
- Data & Society Research Institute (US)
- Google (United States) (US)
- Corresponding author
- Sendak, Mark P
- ORCID
- Sendak, Mark P
- Ratliff, William
- Sarro, Dina
- Alderton, Elizabeth
- Futoma, Joseph
- Gao, Michael
- Nichols, Marshall
- Revoir, Mike
- Yashar, Faraz
- Miller, Corinne
- Kester, Kelly
- Sandhu, Sahil
- Corey, Kristin
- Brajer, Nathan
- Tan, Christelle
- Lin, Anthony
- Brown, Tres
- Engelbosch, Susan
- Anstrom, Kevin
- Elish, Madeleine Clare
- Heller, Katherine
- Donohoe, Rebecca
- Theiling, Jason
- Poon, Eric
- Balu, Suresh
- Bedoya, Armando
- O'Brien, Cara
- Venue
- JMIR Medical Informatics
- Publisher
- JMIR Publications Inc.
- Published
- 2020-07-15
- Volume
- 8
- Issue
- 7
- Pages
- e15182
- ISSN
- 2291-9694 (ISSN-L)
- PMC
- PMC7391165
- OpenAlex
- W2998524378
- Cited by
- 287
- References
- 48
- Citations by year
- 2026: 102 · 2025: 59 · 2024: 36 · 2023: 32 · 2022: 38 · 2021: 18 · 2020: 2
- FWCI
- 11.04
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Informatics (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- John D. and Catherine T. MacArthur Foundation
- Duke Institute for Health Innovation
- Topics
- Health Sciences › Medicine › Critical Care and Intensive Care Medicine › Sepsis Diagnosis and Treatment
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- sepsis, deep learning, clinical implementation, early sepsis detection, learning health systems, quality improvement, clinical decision support, stakeholder engagement, clinical workflow integration, health care delivery systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.


