“Nothing works without the doctor:” physicians’ perception of clinical decision-making and artificial intelligence

Health professionals described XAI systems as useful for supporting clinical decision making.

“Health professionals also described XAI systems as useful for supporting clinical decision making [29,31,33,35–37].”

Education and training was a theme that arose in 9 studies.

“Education and training was a theme that arose in 9 studies, where participants described their need for, or experience of, learning how to use the XAI system [23,25,26,28–30,32,35,36].”

Aligning the XAI tool with existing systems such as embedding it into the hospital electronic health record was important.

“Similarly aligning the XAI tool with existing systems such as embedding it into the hospital electronic health record was important [23,26,35].”

Health professionals positively perceived, and were more likely to use, XAI systems that fit into their existing workflow.

“Health professionals positively perceived, and were more likely to use, XAI systems that fit into their existing workflow [23–26,30,31,33,35].”

Participants in included studies raised concerns that XAI systems could lead to over-reliance and reduce critical thinking skills.

“Participants also raised concerns that the XAI system could lead to over-reliance and reduce critical thinking skills [29,30,33,35].”

In two studies, participants reported they were not interested in understanding the model itself, but rather the output.

“In two studies participants reported they were not interested in understanding the model itself, but rather the output [27,35].”

XAI model predictions were reportedly more likely to be accepted when they aligned with health professionals' knowledge or with the patient context.

“XAI model predictions were reportedly more likely to be accepted when they aligned with health professionals’ knowledge, or with the patient context [23,25,27–30,32–36]”

Health professionals in three studies reported XAI explanations could help communicate risks or escalations in care to patients.

“Health professionals in three studies reported XAI explanations could help communicate risks or escalations in care to patients [23,35,37].”

The theme perceived usefulness was identified in 13 papers.

“The theme perceived usefulness was identified in 13 papers, where participants reported the how the systems impacted their work and associated tasks [23,24,26,28–33,35–38].”

Eight studies linked perceived usefulness to the tool's impact on workload.

“Eight studies linked perceived usefulness to the tool’s impact on workload [26,29–31,33,35,37,38].”

Impact on workflow arose as a theme in 9 papers.

“Impact in workflow related to how XAI systems aligned with and influenced work processes, and arose as a theme in 9 papers [23–27,30,31,33,35].”

Participants wanted the XAI tool to be incorporated into existing decision making processes and to require action at the relevant time so they do not create additional work.

“For example, participants wanted the XAI tool to be incorporated into existing decision making processes, [35] and require action at the relevant time so they do not create additional work [25].”

Samhammer et al. found that when the rationale for the recommendation could not be understood, physicians would revert to their own assessment and not rely on the model.

“Samhammer et al. [35] found that when the rationale for the recommendation could not be understood, physicians would revert to their own assessment and not rely on the model.”

Validation of the model output was a major reason for explainability, reported in nine studies.

“Validation of the model output was a major reason for explainability, and was reported in nine studies [25–27,30,32,33,35,36,38].”

Five studies included perceptions related to health professionals' autonomy.

“Five studies included perceptions related to health professionals’ autonomy [26,29,30,33,35].”

Several studies reported trust and acceptance of model outputs were facilitated by providing evidence for the model effectiveness on patients and/or evidence of the output.

“Several studies reported trust and acceptance of model outputs were facilitated by providing evidence for the model effectiveness on patients and/or evidence of the output [23,29,30,33,35,36].”