CoChat
Sign Up
Cited by 19Artificial Intelligence in Healthcare and EducationMachine Learning in HealthcareElectronic Health Records Systems

The algorithm journey map: a tangible approach to implementing AI solutions in healthcare

Open access
Boag, William, Hasan, Alifia, Kim, Jee Young, Revoir, Mike, Nichols, Marshall, Ratliff, William, Gao, Michael, Zilberstein, Shira +7 more·2024·npj Digital Medicine
DOI 10.1038/s41746-024-01061-4PMC PMC11003994
Open this paper in CoChat — free account

Abstract

When integrating AI tools in healthcare settings, complex interactions between technologies and primary users are not always fully understood or visible. This deficient and ambiguous understanding hampers attempts by healthcare organizations to adopt AI/ML, and it also creates new challenges for researchers to identify opportunities for simplifying adoption and developing best practices for the use of AI-based solutions. Our study fills this gap by documenting the process of designing, building, and maintaining an AI solution called SepsisWatch at Duke University Health System. We conducted 20 interviews with the team of engineers and scientists that led the multi-year effort to build the tool, integrate it into practice, and maintain the solution. This “Algorithm Journey Map” enumerates all social and technical activities throughout the AI solution’s procurement, development, integration, and full lifecycle management. In addition to mapping the “who?” and “what?” of the adoption of the AI tool, we also show several ‘lessons learned’ throughout the algorithm journey maps including modeling assumptions, stakeholder inclusion, and organizational structure. In doing so, we identify generalizable insights about how to recognize and navigate barriers to AI/ML adoption in healthcare settings. We expect that this effort will further the development of best practices for operationalizing and sustaining ethical principles—in algorithmic systems.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper documents an “Algorithm Journey Map” of SepsisWatch, an AI sepsis prediction tool at Duke University Health System, based on 20 interviews with six innovation-team members. It maps social and technical activities across problem identification, development, integration, and lifecycle management, and extracts lessons on modeling assumptions, stakeholder inclusion, and organizational structure.

Key findings

  • The map covers four lifecycle stages: problem identification, development (with model build/validation and UI/UX sub-stages), integration (technical and clinical sub-stages), and lifecycle management; the authors report it included 7 components for 4 stages plus a separate table for lifecycle-management activities.
  • Interviewee-identified lessons cluster into modeling assumptions (e.g., early truncation of data at ICU transfer limited later expansion; outcome definition needed modularization; real-time input availability and runtime constraints affected model architecture), stakeholder inclusion (physician leaders were engaged but nursing leadership was initially omitted, complicating clinical integration), and organizational structure (e.g., creating a patient response program to align RRT nurse incentives; statistician recruitment required senior support).
  • Operational details include 6 months of due diligence on Epic’s Cognitive Computing Platform before deciding it could not run the tool, a custom pipeline pulling real-time data every 5 min, recurring updates to refresh data element “groupers” every 6 months, and 2 owners for the tool (the clinical champion and innovation team project manager).

Methods

  • This was a quality improvement project; IRB review and approval was not required according to Duke University policy, and all interviewed staff provided consent.
  • Six participants from the Duke Institute for Health Innovation were recruited—a clinical data scientist, an innovation program manager, a data engineer, two solutions architects, and an innovation program director—and 20 unstructured interviews were conducted with no interview guide.
  • Participants were asked which stakeholders were involved, what work was done and by whom, what decisions were made, what data/information informed decisions, and what decision points were not considered but should have been; early interviews used detailed notes or interactive Microsoft Visio schematics, and the team met with sepsis project leaders multiple times to refine the map. Pre-deployment processes were mapped with swimlane diagrams, while post-deployment efforts were represented as a table because of their concurrent and asynchronous nature.

Limitations

  • The paper states three primary limitations: focus on a single algorithm within a single setting; journey maps may emphasize a specific perspective and amplify existing power structures rather than reflect less-powerful stakeholders, and may not capture all context needed to scale across locations or time periods; and lack of standardization for how much detail to include.
  • The authors also state they scoped out activities prior to problem selection for the sepsis AI tool and did not include new sepsis models developed as separate projects during lifecycle management.
  • The paper notes that patients were not included at any step and that this did not come up during interviews as an opportunity for reflection; it also states the journey-map effort is not scalable and does not recommend algorithm journey maps accompany every health AI adoption.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41746-024-01061-4

Cites

24 of 24 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

6 of 13 cited sources checked · 3 supported · 0 flagged · accuracy not yet measured

  • Findingreasoning

    Up until that point, the project team had been working primarily with physician leaders at both the hospital and department levels, which left out nurse leaders at the health system level and within relevant service lines who needed to deploy resources to support the rollout.

    “Up until this point, the project team had been working primarily with physician leaders at both the hospital and department levels. Unfortunately, this left out nurse leaders at the health system level and within relevant service lines (e.g., emergency department) who needed to deploy resources to support the rollout.”
    Quote in the paperSubstantial claim
  • Findingreasoning

    Because the tool did not make clinical decisions or treatment recommendations and supported independent review by clinicians, regulatory leaders determined that the technology qualified as non-device CDS.

    “Because the tool did not make clinical decisions or treatment recommendations and supported independent review by clinicians, regulatory leaders determined that the technology qualified as non-device CDS.”
    Quote in the paperSubstantial claim
  • Findingnone given

    The proposal to develop a sepsis AI tool was selected by health system leaders for funding.

    “The proposal to develop a sepsis AI tool was selected by health system leaders for funding.”
    Quote in the paperNot a substantial claim
  • Findingreasoning

    These nurses reported to the cardiology service line, which was not primarily responsible for sepsis care quality.

    “These nurses reported to the cardiology service line, which was not primarily responsible for sepsis care quality.”
    Quote in the paperSubstantial claim
  • Findingreasoning

    Post-rollout lifecycle management is not a linear-flow process.

    “Unlike prior stages, post-rollout lifecycle management is not a linear-flow process.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 58
  • May combine two claims. 27

  • Classified as probably not a substantial claim.

    Premisecitation

    The algorithm journey map is organized around four stages based on related work defining algorithm adoption stages.

    “The algorithm journey map is organized around four stages based on related work defining algorithm adoption stages17.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [17] Kim, J. Y. et al. Organizational governance of emerging technologies: AI adoption in healthcare. In 2023 ACM Conference on Fairness, Accountability, Transparency, FAccT ’23, Chicago, IL, USA (ACM, 2023). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Treatment was to be measured based on sepsis bundle compliance as defined by SEP-1 sepsis bundle regulations issued by CMS.

    “Treatment was to be measured based on sepsis bundle compliance as defined by SEP-1 sepsis bundle regulations issued by CMS20.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [20] Gesten, F. & Evans, L. SEP-1—taking the measure of a measure. JAMA Netw. Open 4, e2138823 (2021). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A survey of 95 randomized controlled trials of AI software found mixed results of their clinical impact.

    “a survey of 95 randomized controlled trials (RCTs) of AI software found mixed results of their clinical impact4.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [4] Zhou, Q., Chen, Z.-h, Cao, Y.-h & Peng, S. Clinical impact and quality of randomized controlled trials involving interventions evaluating artificial intelligence prediction tools: a systematic review. NPJ Digit. Med. 4, 154 (2021). source

      The Results (General characteristics; page not available) report 65 RCTs, not 95, and include traditional statistical prediction tools as well as AI tools. The Results (AI tool intervention and its performance; page not available) support the characterization of mixed clinical impact: 61.5% of trials had positive primary outcomes, while 38.5% showed no statistically significant benefit. (LLM, conf=0.90)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation

    Models of innovation are dependent on political, cultural, and institutional factors, requiring a high degree of contextual specificity in each case study.

    “Models of innovation are dependent on political, cultural, and institutional factors, requiring a high degree of contextual specificity in each case study23,24.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [23] Pfotenhauer, S. & Jasanoff, S. Panacea or diagnosis? Imaginaries of innovation and the ‘MIT model’ in three political cultures. Soc. Stud. Sci. 47, 783–810 (2017). source
    • cites [24] Sendak, M. P., Gao, M., Brajer, N. & Balu, S. Presenting machine learning model information to clinical end users with model facts labels. NPJ Digit. Med. 3, 41 (2020). source

      The Related work section says machine-learning product labels should address the context-specific challenges of health care. It does not discuss whether models of innovation depend on political, cultural, or institutional factors, or whether each case study requires contextual specificity; searches for those terms and related phrasing found no such passage. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    The data quality activities conducted for this project in 2016 - 2017 laid the foundation for a data quality assurance framework that was formally validated at a later date.

    “The data quality activities conducted for this project in 2016 - 2017 laid the foundation for a data quality assurance framework that was formally validated at a later date19.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [19] Sendak, M. et al. Development and validation of ML-DQA–a machine learning data quality assurance framework for healthcare. Machine Learning for Healthcare Conference. PMLR, 741–759 (2022). source

      The Conclusion (p. 16) says the study developed and validated the ML-DQA framework, supporting the later-validation portion of the claim. The Project Selection discussion (p. 5) dates the five projects’ data-quality assurance work to 2021–2022; our copy does not say that activities from the citing document’s project in 2016–2017 laid the framework’s foundation. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    There is a lack of standardization for the amount of detail to include in a journey map.

    “The third limitation is a lack of standardization for the amount of detail to include in a journey map.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    While developing the algorithm journey map, the authors aligned the structure with 4 stages that can be adopted by other projects.

    “While developing the algorithm journey map, we did align the structure with 4 stages that can be adopted by other projects.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    An example of a recurring update that occurs every 6 months is a coordinated effort between the innovation team and clinical champions to refresh data element "groupers."

    “An example of a recurring update that occurs every 6 months is a coordinated effort between the innovation team and clinical champions to refresh data element “groupers.””
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    During the initial meeting with the chief nursing officer, it became clear that there were additional stakeholders who needed to be engaged before the AI tool could be launched.

    “During this initial meeting, it became clear that there were additional stakeholders who needed to be engaged before the AI tool could be launched.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given

    In 2016, machine learning expertise within the school of medicine was limited and there was no mechanism for faculty in quantitative sciences departments to directly collaborate on operational health system projects.

    “In 2016, machine learning expertise within the school of medicine was limited and there was no mechanism for faculty in quantitative sciences departments to directly collaborate on operational health system projects.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Early clinical collaborators felt the sepsis AI tool would not be used in the ICU, so data was truncated at the time of ICU transfer.

    “Early on, clinical collaborators at that time felt the sepsis AI tool would not be used in the ICU, so data was thus truncated at the time of ICU transfer.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    The paper shows several 'lessons learned' throughout the algorithm journey maps including modeling assumptions, stakeholder inclusion, and organizational structure.

    “In addition to mapping the “who?” and “what?” of the adoption of the AI tool, we also show several ‘lessons learned’ throughout the algorithm journey maps including modeling assumptions, stakeholder inclusion, and organizational structure.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The study does not address general questions such as administrative roles, measures of AI tool viability, and form and frequency of communication required for AI tools to be successfully integrated.

    “Our study does not address general questions such as the administrative roles, measures of AI tool viability, and form and frequency of communication required for AI tools to be successfully integrated.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    Modeling became easier once the outcome definition was modularized, allowing for easily changing the criteria and retraining the model.

    “Modeling became easier once the outcome definition was modularized, allowing for easily changing the criteria and retraining the model.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    The Algorithm Journey Map enumerates all social and technical activities throughout the AI solution's procurement, development, integration, and full lifecycle management.

    “This “Algorithm Journey Map” enumerates all social and technical activities throughout the AI solution’s procurement, development, integration, and full lifecycle management.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Working closely with nurse stakeholders, several adaptations were made to the workflow, including direct communication between the RRT nurse and bedside nurses in the ED.

    “Working closely with nurse stakeholders, several adaptations were made to the workflow, including direct communication between the RRT nurse and bedside nurses in the ED.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The journey map was scoped to not include any activities prior to problem selection for the sepsis AI tool, even though previous technologies were used to detect sepsis.

    “We scoped the journey map to not include any activities prior to problem selection for the sepsis AI tool, even though previous technologies were used to detect sepsis.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    Monitoring involves both regular dev-ops duties as well as periodic, thorough data science analysis.

    “Monitoring involves both regular dev-ops duties (e.g., is the system still up? Were there large changes in data volume?) as well as periodic, thorough data science analysis (e.g., does the model still perform well? Have clinical outcomes improved?).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Clinical integration involved fine-tuning the workflow and user interface, developing training material, and assembling a governance committee.

    “This involved fine-tuning the workflow and user interface, developing training material, and assembling a governance committee.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    During technical integration, the team realized the initial model approach required matrix inversion and significant computation.

    “The plan had been to integrate this model into production, but during technical integration, the team realized this approach required matrix inversion and significant computation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Over a period of 6 months, the two teams conducted due diligence on the Epic solution and determined it was not able to run the sepsis AI tool.

    “Over a period of 6 months, the two teams conducted due diligence on the Epic solution and determined it was not able to run the sepsis AI tool.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    An example of a task that spun off into a standalone product is the alert notification system that now supports many AI tools.

    “An example of a task that spun off into a standalone product is the alert notification system that now supports many AI tools.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    By design, the traceability artifact is extremely enmeshed with the particular use case and setting.

    “By design, this traceability artifact is extremely enmeshed with our particular use case and setting.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    The teams agreed to develop a custom solution that extracts data from the EHR, pipes it to a server that runs the model, and sends those predictions to a database that displays results on a custom web application.

    “The teams agreed to develop a custom solution that extracts data from the EHR, pipes it to a server that runs the model, and sends those predictions to a database that displays results on a custom web application.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    RRT nurses were historically cardiac critical care nurses who supported care in the cardiac ICU when not responding to urgent events.

    “RRT nurses were historically cardiac critical care nurses who supported care in the cardiac ICU when not responding to urgent events.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    As a result of this oversight, the clinical integration stage involved meeting many levels of nursing leadership as well as directly engaging certified nurse educators to finalize programmatic decisions and develop training material.

    “As a result of this oversight, the clinical integration stage involved meeting many levels of nursing leadership (health system-level, hospital-level, and department-level) as well as directly engaging certified nurse educators to finalize programmatic decisions and develop training material.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The original goal of the tool was just to flag high-risk patients in a dashboard, but the RRT nurses communicated that it would be even better if the tool helped them not just detect but also manage interventions to treat sepsis.

    “For instance, the original goal of the tool was just to flag high-risk patients in a dashboard, but the RRT nurses communicated that it would be even better if the tool helped them not just detect but also manage interventions to treat sepsis.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Updates are performed both as needed and on a regular schedule in order to ensure the robust performance of the sepsis AI tool.

    “Updates are performed both as needed (event-based, semi-regularly) and on a regular schedule (recurring) in order to ensure the robust performance of the sepsis AI tool.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The journey map did not include any new sepsis models developed as separate projects during lifecycle management.

    “We also did not include any new sepsis models developed as separate projects during lifecycle management.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    This involved working with hospital leaders to move RRT nurses out of cardiology and into a new structure that became responsible for sepsis care quality.

    “This involved working with hospital leaders to move RRT nurses out of cardiology and into a new structure that became responsible for sepsis care quality.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingreasoning

    When a user requested to expand the use of the AI tool to inpatient wards, a new "2.0" project had to be initiated.

    “When a user requested to expand the use of the AI tool to inpatient wards, a new “2.0” project had to be initiated.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingreasoning

    They concluded that such a scenario would require additional training for users to understand how to interpret time-based plots, so instead they focused on point-in-time visualizations to more closely match the setting the model was trained on without as large a risk of user misconceptions.

    “They concluded that such a scenario would require additional training for users to understand how to interpret time-based plots, so instead they focused on point-in-time visualizations to more closely match the setting the model was trained on without as large a risk of user misconceptions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Physicians had differing opinions about which outcome to use, and the published literature didn't show consensus.

    “Physicians had differing opinions about which outcome to use, and the published literature didn’t show consensus.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    Initial versions of the model involved a technically complex Multi-task Gaussian Process for data imputation, followed by an LSTM classifier.

    “Initial versions of the model involved a technically complex Multi-task Gaussian Process for data imputation, followed by an LSTM classifier.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingreasoning

    The UI designer and RRT nurse agreed on a workflow with four patient states (Triage, Screened, Monitoring, and Treatment) and the user would move the patients through the process as sepsis is detected and managed.

    “The UI designer and RRT nurse agreed on a workflow with four patient states (Triage, Screened, Monitoring, and Treatment) and the user would move the patients through the process as sepsis is detected and managed.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    After some feedback sessions with users, the UI designers began to worry that the users would use the trajectory/trend as an indicator, itself, and begin to over-rely on it.

    “However, after some feedback sessions with users, the UI designers began to worry that the users would use the trajectory/trend as an indicator, itself, and begin to over-rely on it.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    The algorithm journey map included 7 components for 4 stages and a separate table to capture all lifecycle management activities.

    “In total, the algorithm journey map included 7 components for 4 stages and a separate table to capture all lifecycle management activities.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    The team eventually used fill-forward data imputation for the LSTM.

    “Eventually, the team used fill-forward data imputation for the LSTM”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    After a few months post-clinical integration, the innovation team learned that the patient workflow coordinators were too busy with other duties to be using the sepsis AI tool; the model predictions were not as critical to PWC nurses' immediate priorities.

    “However, after a few months post-clinical integration, the innovation team learned that the patient workflow coordinators were too busy with other duties to be using the sepsis AI tool; the model predictions were not as critical to PWC nurses’ immediate priorities.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    When determining which data elements to include as inputs for the model, the team had not initially considered that any data for the model needed to not just be captured in the EHR but also available in real-time.

    “When determining which data elements to include as inputs for the model, the team had not initially considered that any data for the model needed to not just be captured in the EHR but also available in real-time.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The UI design was an iterative process that began with scoping what the tool can help with based on the status quo workflow for delivery of care and the general capabilities for what the tool can do.

    “This was an iterative process that began with scoping what the tool can help with based on the status quo workflow for delivery of care (i.e., reacting to sepsis once the patient already starts to deteriorate) and the general capabilities for what the tool can do (i.e., forecast who is at risk for deteriorating in the next N hours).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The AI sepsis tool adoption captured by this journey map did not include patients at any step, and that idea did not even come up during the interviews as an opportunity to reflect on where they should have been included.

    “For instance, the AI sepsis tool adoption captured by this journey map did not include patients at any step, and – unlike for the cases of nurses—that idea did not even come up during the interviews as an opportunity to reflect on where they should have been included.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Many projects in this field have struggled with the question around patient inclusion.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    Sharing and documenting knowledge about the full range of roles critical to technological work matches frameworks such as the Data Feminist principle to "Make Labor Visible."

    “Sharing and documenting knowledge about the full range of roles critical to technological work matches frameworks such as the Data Feminist16 principle to “Make Labor Visible.””
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [16] D’Ignazio, C. & Klein, L. F. Data Feminism (MIT Press, 2020). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Once the data pipeline was functioning, the innovation team submitted another study proposal to run a 'silent trial' to enable end-to-end system monitoring and testing.

    “Once the data pipeline was functioning, the innovation team submitted another study proposal to run a ‘silent trial’ to enable end-to-end system monitoring and testing.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Premisecitation

    Workflow diagrams can empower stakeholders and function as communication tools for knowledge legitimation and diffusion.

    “Workflow diagrams can empower stakeholders and function as communication tools for knowledge legitimation and diffusion.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [14] Huising, R. Moving off the map: how knowledge of organizational operations empowers and alienates. Organ. Sci. 30, 1 (2019). source

  • May combine two claims.

    Study design and conductreasoning

    This study was a quality improvement project with minimal risk to the participants.

    “This study was a quality improvement project with minimal risk to the participants.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    This effort was critical to ensure alignment in organizational priorities because otherwise, busy nurses would likely not consistently be able to make time to use the sepsis AI tool.

    “This effort was critical to ensure alignment in organizational priorities because otherwise, busy nurses would likely not consistently be able to make time to use (let alone offer feedback about) the sepsis AI tool.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    A governance committee was established including stakeholders from both physician and nursing leadership to meet monthly during the initial rollout in order to resolve any emergent issues.

    “Finally, a governance committee was established including stakeholders from both physician and nursing leadership to meet monthly during the initial rollout in order to resolve any emergent issues.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    We interviewed every stakeholder on the innovation team who was involved in the project.

    “We interviewed every stakeholder on the innovation team (Duke Institute for Health Innovation) who was involved in the project.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    Even if a different organization doesn't follow the exact same steps, there will be commonalities in the types of stakeholders, challenges, and enablers (e.g., institutional silos, differences in stakeholder priorities, technical barriers).

    “and because even if a different organization doesn’t follow the exact same steps, there will be commonalities in the types of stakeholders, challenges, and enablers (e.g., institutional silos, differences in stakeholder priorities, technical barriers).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    A few months of testing were done to ensure the system could handle the volume of data being extracted, particularly because vitals are collected very frequently.

    “A few months of testing were done to ensure the system could handle the volume of data being extracted, particularly because vitals are collected very frequently.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The project team needed to ensure that organizational incentives enabled RRT nurses to prioritize the use of the tool.

    “the project team needed to ensure that organizational incentives enabled RRT nurses to prioritize the use of the tool.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    Documenting on-the-ground processes creates a stable and tangible basis for knowledge building and legitimation.

    “Furthermore, documenting on-the-ground processes creates a stable and tangible basis for knowledge building and legitimation15.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15] Daston, L. & Galison, P. The image of objectivity. Representations 40, 81–128 (1992). source

  • May combine two claims.

    Premisecitation

    A notable incident at UCSF: alert fatigue, coupled with a confusing user interface featuring different units for adult (mg) and pediatric (mg/kg) patients, resulted in a doctor accidentally ordering a 39x overdose, which was subsequently administered to a pediatric patient, nearly proving fatal.

    “A notable incident at UCSF serves as a stark reminder. Here, alert fatigue, coupled with a confusing user interface featuring different units for adult (mg) and pediatric (mg/kg) patients, resulted in a doctor accidentally ordering a 39x overdose, which was subsequently administered to a pediatric patient, nearly proving fatal13.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [13] Salber, P. Wachter’s digital doctor explores the dark side of health IT. Mhealth 1, 18 (2015).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    The algorithm journey map captures an extremely messy process that is far from ideal, and the effort required to surface the process is not scalable.

    “First, the algorithm journey map captures an extremely messy process that is far from ideal, and the effort required to surface the process is not scalable.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The algorithm journey map presented in this study does not map well to any other setting or use case.

    “Unfortunately, while other organizations may learn from our experience, we do not expect that the algorithm journey map presented in this study maps well to any other setting or use case.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    With the algorithm journey map, researchers can now tailor documentation and transparency artifacts to specific decision points in a process.

    “With the algorithm journey map, researchers can now tailor documentation and transparency artifacts to specific decision points in a process.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: There is limited understanding of the specific user, knowledge base, context-of-use, and decision made using information contained in the documentation artifact.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationnone given

    The specific path visualized in the sepsis AI tool algorithm journey map is highly circuitous, confusing, and not meant to ever be repeated.

    “The specific path visualized in the sepsis AI tool algorithm journey map is highly circuitous, confusing, and not meant to ever be repeated.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    Although the algorithm journey map is highly specific to the Duke SepsisWatch context, the exercise will be very valuable to other researchers because it provides a blueprint for how one can build their own algorithm journey map.

    “Although this algorithm journey map is highly specific to the Duke SepsisWatch context, the exercise will be very valuable to other researchers both because it provides a blueprint for how one can build their own algorithm journey map”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisenone given

    Epic's backend databases involve both a real-time feed of the current day and a historical archive, and access to real-time data requires the IT department to build specific data endpoints for each kind of element.

    “Epic’s backend databases involve both a real-time feed of the current day and a historical archive, and access to real-time data requires the IT department to build specific data endpoints for each kind of element.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Once the iterative feedback was incorporated into the design, the prototype was presented to the clinical champion to ensure that the tool would be aligned with the project's goals.

    “Once the iterative feedback was incorporated into the design, the prototype was presented to the clinical champion to ensure that the tool would be aligned with the project’s goals.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The statistician reviewed the output of the model with the clinical champion, both with summary statistics and chart reviews to assess whether the model was ready to move forward.

    “The statistician then reviewed the output of the model with the clinical champion, both with summary statistics and chart reviews to assess whether the model was ready to move forward.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    A given algorithm journey map might not be able to capture all of the relevant context to scale across locations or time periods.

    “Further, a given algorithm journey map might not be able to capture all of the relevant context to scale across locations or time periods.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    It should have been knowable at the time that the runtime environment would limit expensive model architectures and decisions.

    “but it should have been knowable at the time that the runtime environment would limit expensive model architectures and decisions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The decisions made during the development stage guided the design and development of the sepsis AI tool and had significant downstream implications.

    “As will be described in the following sections, the above decisions guided the design and development of the sepsis AI tool and had significant downstream implications.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationcitation

    Algorithm journey maps might emphasize a specific perspective, amplifying existing power structures rather than allowing less-powerful stakeholders to have their experiences properly reflected.

    “Second, and relatedly, algorithm journey maps might emphasize a specific perspective, amplifying existing power structures rather than allowing less-powerful stakeholders to have their experiences properly reflected22.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [22] Moser, I. & Law, J. Fluids or flows? Information and qualculation in medical practice. Inf. Technol. People 19, (2006). https://doi.org/10.1108/09593840610649961. source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    All staff who were interviewed for the project provided consent prior to interviews.

    “All staff who were interviewed for the project provided consent prior to interviews.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The IRB reviewed the project to approve the development of the algorithm on retrospective data and granted a waiver of consent to use patient data for model development.

    “The IRB reviewed the project to approve the development of the algorithm on retrospective data and granted a waiver of consent to use patient data for model development.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Sharing operational knowledge allows stakeholders to navigate the organization and gain agency as they understand the outcomes and goals of their roles.

    “Sharing operational knowledge, or knowledge of day-to-day operations, allows stakeholders to navigate the organization and gain agency as they understand the outcomes and goals of their roles14.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [14] Huising, R. Moving off the map: how knowledge of organizational operations empowers and alienates. Organ. Sci. 30, 1 (2019). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Once the 'silent trial' was approved, the innovation team conducted user testing to get feedback about the UI and the performance of the model.

    “Once the ‘silent trial’ was approved, the innovation team conducted user testing to get feedback about the UI and the performance of the model.”
    Quote in the paperNot a substantial claim
70 of 70

  • The paper relies on this without support.

    Complex interactions between AI technologies and primary users in healthcare settings are not always fully understood or visible, and this deficient and ambiguous understanding hampers healthcare organizations’ attempts to adopt AI/ML.

    “complex interactions between technologies and primary users are not always fully understood or visible. This deficient and ambiguous understanding hampers attempts by healthcare organizations to adopt AI/ML”

  • The paper relies on this without support.

    There is limited understanding of the specific user, knowledge base, context-of-use, and decision made using information contained in the documentation artifact.

    “There’s limited understanding of the specific user, knowledge base, context-of-use, and decision made using information contained in the documentation artifact.”

  • The paper relies on this without support.

    Many machine learning in healthcare studies seek to develop and validate frameworks across multiple models and settings.

    “While many machine learning in healthcare studies seek to develop and validate frameworks across multiple models and settings, our objective was different.”

  • The paper relies on this without support.

    Many projects in this field have struggled with the question around patient inclusion.

    “Many projects in this field also have struggled with this question around patient inclusion; as more projects are more tangible, the field will be able to learn from what methods of patient engagement (and other instantiations of ethical computing principles) are more/less meaningful.”
4 of 4

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Implementation challenges for AI in clinical use include securing significant investment in funding, building infrastructure, employing skilled personnel, ensuring regulatory compliance, and integration into clinical workflows.

    “It entails overcoming implementation challenges, such as securing significant investment in funding, building infrastructure, employing skilled personnel, ensuring regulatory compliance, and integration into clinical workflows14–16.”

    From Establishing organizational AI governance in healthcare: a case study in Canada (2025) · cites this paper as [14]

    The Results (Algorithm journey map; page unavailable) describe building a real-time data pipeline and server infrastructure, consulting regulatory affairs leadership, adapting clinical workflows, and periodically securing funding and resources. The Results (Interviewee-identified opportunities for improvement; page unavailable) also describe recruiting statistical expertise. These passages support the listed implementation challenges, but do not establish that the required financial investment was significant. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Cites

Flagged first, then by citations

Research works printed but not linked: 6

  1. Park, E. The AI Bill of Rights: a step in the right direction. Orange Cty. Lawyer Mag. 65, 2 (2023).
  2. Boag, W. Evidence-based AI Ethics. Dissertation, Massachusetts Institute of Technology (2022).
  3. Tonekaboni, S. et al. How to validate machine learning models prior to deployment: silent trial protocol for evaluation of real-time models at ICU. In Proc. Conference on Health, Inference, and Learning, PMLR. Vol. 174 (eds Gerardo F., George H C., Tom P., Joyce C H., & Tristan N.) 169–182 (Proceedings of Machine Learning Research (PMLR), 2022).
  4. Haraway, D. Simians, Cyborgs, and Women: The Reinvention of Nature (Routledge, 1990).
  5. Richardson, A. Using customer journey maps to improve customer experience. Harv. Bus. Rev. 15, 2–5 (2010).
  6. Damelio, R. The Basics of Process Mapping (CRC Press, 2011).

Cited by

Sorted by citations

Related

Authors
Boag, W., Hasan, A., Kim, J. Y., Revoir, M., Nichols, M., Ratliff, W., Gao, M., Zilberstein, S., Samad, Z., Hoodbhoy, Z., Ali, M., Khan, N. S., Patel, M., Balu, S., & Sendak, M.
Affiliations
  • Duke University (US)
  • Harvard University (US)
  • Aga Khan University (PK)
Corresponding author
Sendak, Mark
ORCID
Venue
npj Digital Medicine
Publisher
Springer Science and Business Media LLC
Published
2024-04-09
Volume
7
Issue
1
Pages
87-87
ISSN
2398-6352 (ISSN-L)
DOI
10.1038/s41746-024-01061-4
PMC
PMC11003994
OpenAlex
W4394621882
Semantic Scholar
2fd67fcc8b48140a4afc69ac9cc3961504908766
Cited by
19
References
22
Citations by year
2026: 7 · 2025: 9 · 2024: 3
FWCI
1.78
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • eCommons - AKU (Aga Khan University) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Gordon and Betty Moore Foundation
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
Keywords
AI adoption in healthcare, sepsis prediction, machine learning implementation, AI adoption barriers, stakeholder inclusion, organizational structure, AI lifecycle management, lessons learned

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-11-02.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-11-02.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review