CarePre: An Intelligent Clinical Decision Assistance System

Integration with electronic health records was common, particularly in more mature tools.

“Integration with electronic health records (EHRs) was common, particularly in more mature tools [23,24,26–34,36].”

Four studies used more than one explanation technique in their model.

“Four studies used more than one explanation technique in their model [24,32,34,38].”

Several studies lacked detail on the recruitment of participants.

“Several studies lacked detail on the recruitment of participants [32,34,37,38].”

Five studies did not provide sufficient detail on their qualitative data analysis process, making it difficult to determine the validity of the findings.

“Five studies did not provide sufficient detail on their qualitative data analysis process, and this made it difficult to determine the validity of the findings [31,32,34,37,38].”

Perceived usefulness of XAI systems was reported to be user or context dependent.

“Perceived usefulness was reported to be user or context dependent [24,31–33,36,38].”

The theme perceived usefulness was identified in 13 papers.

“The theme perceived usefulness was identified in 13 papers, where participants reported the how the systems impacted their work and associated tasks [23,24,26,28–33,35–38].”

Education and training was a theme that arose in 9 studies.

“Education and training was a theme that arose in 9 studies, where participants described their need for, or experience of, learning how to use the XAI system [23,25,26,28–30,32,35,36].”

Perceptions regarding efficiency or saving time were reported in six studies.

“Perceptions regarding efficiency or saving time were reported in six studies [23,28–30,32,33].”

In two studies of XAI prototypes, participants perceived that further training or practice using the tool would have increased their understanding.

“In two studies of XAI prototypes, participants perceived if they received further training or practice using the tool it would have increased their understanding [23,32].”

Perceptions on the model's performance were reported in 13 studies.

“Perceptions on the model’s performance was reported in 13 studies [23–26,29–37].”

Participants in several studies described the role of the model's performance in influencing trust.

“Participants in several studies described the role of the model’s performance in influencing trust [23,25,31–33,36].”

Some health professionals had limited time to interpret model explanations and make a decision.

“Considering context and users was also reported, with some health professionals having limited time to interpret model explanations and make a decision [23,25,27,32].”

Validation of the model output was a major reason for explainability, reported in nine studies.

“Validation of the model output was a major reason for explainability, and was reported in nine studies [25–27,30,32,33,35,36,38].”

A number of studies reported trust could be earned over time as model performance was proven.

“A number of studies also reported trust could be earned over time as model performance was proven [23,25,31–33,36], for example, an electrophysiologist said “I just think I should see that it confirms our decisions in enough cases - then I would feel comfortable about colleagues leaning on it” [33].”

XAI model predictions were reportedly more likely to be accepted when they aligned with health professionals' knowledge or with the patient context.

“XAI model predictions were reportedly more likely to be accepted when they aligned with health professionals’ knowledge, or with the patient context [23,25,27–30,32–36]”

Perceptions of the XAI model's data quality impacted participants' trust and use of tools, and trust in outputs was reduced when XAI systems used hospital data that participants knew was unreliable.

“Similarly, perceptions of the XAI model’s data quality impacted participants trust and use of tools and trust in outputs were reduced when XAI system’s used hospital data that participants knew was unreliable [23,25,26,29,32,33,36].”

Health professionals' perceptions involving design and usability were found in 14 papers.

“Health professionals’ perceptions involving design and usability were found in 14 papers [23–34,37,38].”

Example-based explanations were used in two studies.

“Example-based explanations were used in two studies [24,32].”

Of the four studies that implemented multiple explanation techniques, only one directly examined participants' perceptions of the different methods.

“Of the four studies that implemented multiple explanation techniques within their models [24,32,34,38], only one directly examined participants’ perceptions of the different methods [34].”

In Jin et al.'s study, two participants reported highlighting similar patients and their disease progression was very informative, and helped to validate predictions.

“In Jin et al.’s [32] study two participants reported highlighting similar patients and their disease progression was very informative, and helped to validate predictions.”

Eight studies reported participants desired actionable information from XAI systems.

“Eight studies in this review reported participants desired actionable information from the XAI systems [23–25,28,30,32,36,37].”

In three studies, participants reported wanting to see variables that could be altered through interventions in feature-based explanations.

“With regard to feature based explanations, participants in three studies reported wanting to see the variables that could be altered through interventions [23,25,28,32].”

Health professionals valued the presentation of contextual information such as patient results alongside predictions and explanations.

“Health professionals valued the presentation of contextual information such as patient results alongside predictions and explanations”