Implementing a machine learning screening tool for malnutrition: insights from qualitative research applicable to other machine learning-based clinical decision support systems

Integration with electronic health records was common, particularly in more mature tools.

“Integration with electronic health records (EHRs) was common, particularly in more mature tools [23,24,26–34,36].”

Health professionals positively perceived, and were more likely to use, XAI systems that fit into their existing workflow.

“Health professionals positively perceived, and were more likely to use, XAI systems that fit into their existing workflow [23–26,30,31,33,35].”

Perceptions of the XAI model's data quality impacted participants' trust and use of tools, and trust in outputs was reduced when XAI systems used hospital data that participants knew was unreliable.

“Similarly, perceptions of the XAI model’s data quality impacted participants trust and use of tools and trust in outputs were reduced when XAI system’s used hospital data that participants knew was unreliable [23,25,26,29,32,33,36].”

While several systems had been implemented or were near live, the majority remained at prototype or pre-implementation stages.

“While several systems had been implemented or were near live [26,30,31,33,36], the majority remained at prototype or pre-implementation stages.”

Participants commonly reported that XAI tools support task completion.

“Across these studies participants commonly reported that the XAI tools support task completion [23,26,28,29,31,33].”

The theme perceived usefulness was identified in 13 papers.

“The theme perceived usefulness was identified in 13 papers, where participants reported the how the systems impacted their work and associated tasks [23,24,26,28–33,35–38].”

Eight studies linked perceived usefulness to the tool's impact on workload.

“Eight studies linked perceived usefulness to the tool’s impact on workload [26,29–31,33,35,37,38].”

Impact on workflow arose as a theme in 9 papers.

“Impact in workflow related to how XAI systems aligned with and influenced work processes, and arose as a theme in 9 papers [23–27,30,31,33,35].”

Education and training was a theme that arose in 9 studies.

“Education and training was a theme that arose in 9 studies, where participants described their need for, or experience of, learning how to use the XAI system [23,25,26,28–30,32,35,36].”

Aligning the XAI tool with existing systems such as embedding it into the hospital electronic health record was important.

“Similarly aligning the XAI tool with existing systems such as embedding it into the hospital electronic health record was important [23,26,35].”

In three studies of XAI tools that were already implemented, some participants lacked understanding of the model or features.

“In three studies of XAI tools that were already implemented, some participants lacked understanding of the model or features [26,30,36].”

In Besculides et al.'s study the XAI tool was already implemented and was perceived to increase workload as it identified more high-risk patients needing review.

“Conversely, in Besculides et al.’s [26] study the XAI tool was already implemented and was perceived to increase workload as it identified more high-risk patients needing review.”

Perceptions on the model's performance were reported in 13 studies.

“Perceptions on the model’s performance was reported in 13 studies [23–26,29–37].”

Besculides et al. found depth of understanding differed for those who were trained by the data science team and those who were trained by supervisors or managers.

“Besculides et al. [26] also found depth of understanding differed for those who were trained by data science team and those who were trained by supervisors or managers.”

Participants wanted to exercise their professional judgement and override the XAI recommendations when perceived to be inaccurate.

“Participants wanted to exercise their professional judgement and override the XAI recommendations when perceived to be inaccurate [26], and were concerned “that regulatory agencies might use these systems to standardize care even in scenarios where a clinician disagrees with the system, potentially leading to overtreatment and patient harm” [30].”

Validation of the model output was a major reason for explainability, reported in nine studies.

“Validation of the model output was a major reason for explainability, and was reported in nine studies [25–27,30,32,33,35,36,38].”

Besculides et al. reported dieticians checked factors contributing to the score when they questioned the prediction.

“Besculides et al. [26] reported dieticians checked factors contributing to the score when they questioned the prediction.”

Participants had concerns with performance regarding unique patients believed to be outliers.

“Participants also had concerns with performance regarding unique patients believed to be outliers [25,26,29,36].”

Five studies included perceptions related to health professionals' autonomy.

“Five studies included perceptions related to health professionals’ autonomy [26,29,30,33,35].”

In Besculides et al.'s study of a system to predict patient risk of malnutrition, participants reported they did not mention the prediction score to patients and communication with patients remained the same post-implementation.

“In contrast, in Besculidies et al’s [26] study of a system to predict patient risk of malnutrition, participants reported that they did not mention the prediction score to patients and communication with patients remained the same post-implementation.”

Health professionals' perceptions involving design and usability were found in 14 papers.

“Health professionals’ perceptions involving design and usability were found in 14 papers [23–34,37,38].”