A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare

Health professionals positively perceived, and were more likely to use, XAI systems that fit into their existing workflow.

“Health professionals positively perceived, and were more likely to use, XAI systems that fit into their existing workflow [23–26,30,31,33,35].”

Perceptions of the XAI model's data quality impacted participants' trust and use of tools, and trust in outputs was reduced when XAI systems used hospital data that participants knew was unreliable.

“Similarly, perceptions of the XAI model’s data quality impacted participants trust and use of tools and trust in outputs were reduced when XAI system’s used hospital data that participants knew was unreliable [23,25,26,29,32,33,36].”

A number of studies reported trust could be earned over time as model performance was proven.

“A number of studies also reported trust could be earned over time as model performance was proven [23,25,31–33,36], for example, an electrophysiologist said “I just think I should see that it confirms our decisions in enough cases - then I would feel comfortable about colleagues leaning on it” [33].”

Healthcare professionals preferred grouping features and ordering them by their contribution to predicted risk to allow easy and quick interpretation.

“For presentation of features, healthcare professionals preferred grouping features and ordering them by their contribution to predicted risk to allow easy and quick interpretation.”

In three studies, participants reported wanting to see variables that could be altered through interventions in feature-based explanations.

“With regard to feature based explanations, participants in three studies reported wanting to see the variables that could be altered through interventions [23,25,28,32].”

Simple, interpretable visualizations, such as bar charts and line graphs, were preferred for understanding predictions at a glance.

“Simple, interpretable visualizations, such as bar charts and line graphs, were preferred for understanding predictions at a glance.”

Some health professionals had limited time to interpret model explanations and make a decision.

“Considering context and users was also reported, with some health professionals having limited time to interpret model explanations and make a decision [23,25,27,32].”

Participants had concerns with performance regarding unique patients believed to be outliers.

“Participants also had concerns with performance regarding unique patients believed to be outliers [25,26,29,36].”

Impact on workflow arose as a theme in 9 papers.

“Impact in workflow related to how XAI systems aligned with and influenced work processes, and arose as a theme in 9 papers [23–27,30,31,33,35].”

Education and training was a theme that arose in 9 studies.

“Education and training was a theme that arose in 9 studies, where participants described their need for, or experience of, learning how to use the XAI system [23,25,26,28–30,32,35,36].”

Perceptions on the model's performance were reported in 13 studies.

“Perceptions on the model’s performance was reported in 13 studies [23–26,29–37].”

Participants in several studies described the role of the model's performance in influencing trust.

“Participants in several studies described the role of the model’s performance in influencing trust [23,25,31–33,36].”

Participants wanted the XAI tool to be incorporated into existing decision making processes and to require action at the relevant time so they do not create additional work.

“For example, participants wanted the XAI tool to be incorporated into existing decision making processes, [35] and require action at the relevant time so they do not create additional work [25].”

Validation of the model output was a major reason for explainability, reported in nine studies.

“Validation of the model output was a major reason for explainability, and was reported in nine studies [25–27,30,32,33,35,36,38].”

In two studies, some participants reported wanting to understand the model at a global level.

“Conversely, in two studies some participants reported wanting to also understand the model at a global level [25,36]”

XAI model predictions were reportedly more likely to be accepted when they aligned with health professionals' knowledge or with the patient context.

“XAI model predictions were reportedly more likely to be accepted when they aligned with health professionals’ knowledge, or with the patient context [23,25,27–30,32–36]”

Five studies specifically described using SHAP.

“Five studies [23,25,27,28,34] specifically described using SHapley Additive exPlanations (SHAP)”

Explanations were wanted by users for educational purposes, for example, so health professionals could learn from the model.

“Additionally, explanations were wanted by users for educational purposes, for example, so health professionals could learn from the model [25].”

Health professionals' perceptions involving design and usability were found in 14 papers.

“Health professionals’ perceptions involving design and usability were found in 14 papers [23–34,37,38].”

Eight studies reported participants desired actionable information from XAI systems.

“Eight studies in this review reported participants desired actionable information from the XAI systems [23–25,28,30,32,36,37].”

Flexibility in presentation formats was valued, enabling alignment with individual user preferences.

“Flexibility in presentation formats was also valued, enabling alignment with individual user preferences.”

Health professionals valued the presentation of contextual information such as patient results alongside predictions and explanations.

“Health professionals valued the presentation of contextual information such as patient results alongside predictions and explanations”

Similar to others, we found that extraction of directly interpretable patient data and actionable needs was a high priority.

“Similar to others,73–75 we found that extraction of directly interpretable patient data and actionable needs was a high priority (Subtheme 4.d).”