Engaging Multidisciplinary clinical users in the design of an artificial intelligence-powered graphical user interface for intensive care unit instability decision support

Health professionals described XAI systems as useful for supporting clinical decision making.

“Health professionals also described XAI systems as useful for supporting clinical decision making [29,31,33,35–37].”

Participants commonly reported that XAI tools support task completion.

“Across these studies participants commonly reported that the XAI tools support task completion [23,26,28,29,31,33].”

Eight studies linked perceived usefulness to the tool's impact on workload.

“Eight studies linked perceived usefulness to the tool’s impact on workload [26,29–31,33,35,37,38].”

The theme perceived usefulness was identified in 13 papers.

“The theme perceived usefulness was identified in 13 papers, where participants reported the how the systems impacted their work and associated tasks [23,24,26,28–33,35–38].”

Integration with electronic health records was common, particularly in more mature tools.

“Integration with electronic health records (EHRs) was common, particularly in more mature tools [23,24,26–34,36].”

Education and training was a theme that arose in 9 studies.

“Education and training was a theme that arose in 9 studies, where participants described their need for, or experience of, learning how to use the XAI system [23,25,26,28–30,32,35,36].”

Perceptions on the model's performance were reported in 13 studies.

“Perceptions on the model’s performance was reported in 13 studies [23–26,29–37].”

Perceptions regarding efficiency or saving time were reported in six studies.

“Perceptions regarding efficiency or saving time were reported in six studies [23,28–30,32,33].”

Helman et al. reported it took a significant amount of time to explain how to use their XAI tool, which was viewed as a barrier to use in practice.

“Helman et al. [29] reported it took a significant amount of time to explain how to use their XAI tool, which was viewed as a barrier to use in practice.”

Participants had concerns with performance regarding unique patients believed to be outliers.

“Participants also had concerns with performance regarding unique patients believed to be outliers [25,26,29,36].”

Five studies included perceptions related to health professionals' autonomy.

“Five studies included perceptions related to health professionals’ autonomy [26,29,30,33,35].”

Participants in included studies raised concerns that XAI systems could lead to over-reliance and reduce critical thinking skills.

“Participants also raised concerns that the XAI system could lead to over-reliance and reduce critical thinking skills [29,30,33,35].”

Several studies reported trust and acceptance of model outputs were facilitated by providing evidence for the model effectiveness on patients and/or evidence of the output.

“Several studies reported trust and acceptance of model outputs were facilitated by providing evidence for the model effectiveness on patients and/or evidence of the output [23,29,30,33,35,36].”

XAI model predictions were reportedly more likely to be accepted when they aligned with health professionals' knowledge or with the patient context.

“XAI model predictions were reportedly more likely to be accepted when they aligned with health professionals’ knowledge, or with the patient context [23,25,27–30,32–36]”

Perceptions of the XAI model's data quality impacted participants' trust and use of tools, and trust in outputs was reduced when XAI systems used hospital data that participants knew was unreliable.

“Similarly, perceptions of the XAI model’s data quality impacted participants trust and use of tools and trust in outputs were reduced when XAI system’s used hospital data that participants knew was unreliable [23,25,26,29,32,33,36].”

Health professionals' perceptions involving design and usability were found in 14 papers.

“Health professionals’ perceptions involving design and usability were found in 14 papers [23–34,37,38].”