Examining explainable clinical decision support systems with think aloud protocols

Integration with electronic health records was common, particularly in more mature tools.

“Integration with electronic health records (EHRs) was common, particularly in more mature tools [23,24,26–34,36].”

Four studies used more than one explanation technique in their model.

“Four studies used more than one explanation technique in their model [24,32,34,38].”

Health professionals positively perceived, and were more likely to use, XAI systems that fit into their existing workflow.

“Health professionals positively perceived, and were more likely to use, XAI systems that fit into their existing workflow [23–26,30,31,33,35].”

Of the four studies that implemented multiple explanation techniques, only one directly examined participants' perceptions of the different methods.

“Of the four studies that implemented multiple explanation techniques within their models [24,32,34,38], only one directly examined participants’ perceptions of the different methods [34].”

Simple, interpretable visualizations, such as bar charts and line graphs, were preferred for understanding predictions at a glance.

“Simple, interpretable visualizations, such as bar charts and line graphs, were preferred for understanding predictions at a glance.”

Healthcare professionals preferred grouping features and ordering them by their contribution to predicted risk to allow easy and quick interpretation.

“For presentation of features, healthcare professionals preferred grouping features and ordering them by their contribution to predicted risk to allow easy and quick interpretation.”

In Anjara et al.'s study, the majority of participants found the examples confusing and felt the system was simply retrieving similar patient records without revealing how the model actually analysed the data.

“In Anjara et. al.’s [24] study the majority of participants found the examples confusing and felt the system was simply retrieving similar patient records without revealing how the model actually analysed the data.”

The theme perceived usefulness was identified in 13 papers.

“The theme perceived usefulness was identified in 13 papers, where participants reported the how the systems impacted their work and associated tasks [23,24,26,28–33,35–38].”

Perceived usefulness of XAI systems was reported to be user or context dependent.

“Perceived usefulness was reported to be user or context dependent [24,31–33,36,38].”

Impact on workflow arose as a theme in 9 papers.

“Impact in workflow related to how XAI systems aligned with and influenced work processes, and arose as a theme in 9 papers [23–27,30,31,33,35].”

Perceptions on the model's performance were reported in 13 studies.

“Perceptions on the model’s performance was reported in 13 studies [23–26,29–37].”

Eight studies reported participants desired actionable information from XAI systems.

“Eight studies in this review reported participants desired actionable information from the XAI systems [23–25,28,30,32,36,37].”

Example-based explanations were used in two studies.

“Example-based explanations were used in two studies [24,32].”

Health professionals' perceptions involving design and usability were found in 14 papers.

“Health professionals’ perceptions involving design and usability were found in 14 papers [23–34,37,38].”

Explanations were viewed as useful when conducting research.

“Explanations were also viewed as useful when conducting research [24].”

One study did not include any of the terms from our qualitative category in its title, abstract, or keywords.

“For example, one study did not include any of the terms from our qualitative category in its title, abstract, or keywords [24]”