Clinician preimplementation perspectives of a decision-support tool for the prediction of cardiac arrhythmia based on machine learning: near-live feasibility and qualitative study

Perceptions of the XAI model's data quality impacted participants' trust and use of tools, and trust in outputs was reduced when XAI systems used hospital data that participants knew was unreliable.

“Similarly, perceptions of the XAI model’s data quality impacted participants trust and use of tools and trust in outputs were reduced when XAI system’s used hospital data that participants knew was unreliable [23,25,26,29,32,33,36].”

Simple, interpretable visualizations, such as bar charts and line graphs, were preferred for understanding predictions at a glance.

“Simple, interpretable visualizations, such as bar charts and line graphs, were preferred for understanding predictions at a glance.”

Decimals, negative values, and outputs that did not sum to 100% were viewed as confusing and potentially misleading.

“while decimals, negative values, and outputs that did not sum to 100% were viewed as confusing and potentially misleading.”

Health professionals preferred risk predictions expressed as whole-number percentages.

“For numeric information, health professionals reported preferring risk predictions expressed as whole-number percentages”

While several systems had been implemented or were near live, the majority remained at prototype or pre-implementation stages.

“While several systems had been implemented or were near live [26,30,31,33,36], the majority remained at prototype or pre-implementation stages.”

Eight studies linked perceived usefulness to the tool's impact on workload.

“Eight studies linked perceived usefulness to the tool’s impact on workload [26,29–31,33,35,37,38].”

Perceived usefulness of XAI systems was reported to be user or context dependent.

“Perceived usefulness was reported to be user or context dependent [24,31–33,36,38].”

Participants commonly reported that XAI tools support task completion.

“Across these studies participants commonly reported that the XAI tools support task completion [23,26,28,29,31,33].”

Health professionals described XAI systems as useful for supporting clinical decision making.

“Health professionals also described XAI systems as useful for supporting clinical decision making [29,31,33,35–37].”

The theme perceived usefulness was identified in 13 papers.

“The theme perceived usefulness was identified in 13 papers, where participants reported the how the systems impacted their work and associated tasks [23,24,26,28–33,35–38].”

Health professionals positively perceived, and were more likely to use, XAI systems that fit into their existing workflow.

“Health professionals positively perceived, and were more likely to use, XAI systems that fit into their existing workflow [23–26,30,31,33,35].”

Impact on workflow arose as a theme in 9 papers.

“Impact in workflow related to how XAI systems aligned with and influenced work processes, and arose as a theme in 9 papers [23–27,30,31,33,35].”

Perceptions regarding efficiency or saving time were reported in six studies.

“Perceptions regarding efficiency or saving time were reported in six studies [23,28–30,32,33].”

Perceptions on the model's performance were reported in 13 studies.

“Perceptions on the model’s performance was reported in 13 studies [23–26,29–37].”

Participants in several studies described the role of the model's performance in influencing trust.

“Participants in several studies described the role of the model’s performance in influencing trust [23,25,31–33,36].”

Five studies included perceptions related to health professionals' autonomy.

“Five studies included perceptions related to health professionals’ autonomy [26,29,30,33,35].”

XAI model predictions were reportedly more likely to be accepted when they aligned with health professionals' knowledge or with the patient context.

“XAI model predictions were reportedly more likely to be accepted when they aligned with health professionals’ knowledge, or with the patient context [23,25,27–30,32–36]”

A number of studies reported trust could be earned over time as model performance was proven.

“A number of studies also reported trust could be earned over time as model performance was proven [23,25,31–33,36], for example, an electrophysiologist said “I just think I should see that it confirms our decisions in enough cases - then I would feel comfortable about colleagues leaning on it” [33].”

Several studies reported trust and acceptance of model outputs were facilitated by providing evidence for the model effectiveness on patients and/or evidence of the output.

“Several studies reported trust and acceptance of model outputs were facilitated by providing evidence for the model effectiveness on patients and/or evidence of the output [23,29,30,33,35,36].”

Validation of the model output was a major reason for explainability, reported in nine studies.

“Validation of the model output was a major reason for explainability, and was reported in nine studies [25–27,30,32,33,35,36,38].”

LIME were used by models in three studies.

“Local Interpretable Model-Agnostic Explanations (LIME) were also used by models in three studies [33,34,37]”

Integration with electronic health records was common, particularly in more mature tools.

“Integration with electronic health records (EHRs) was common, particularly in more mature tools [23,24,26–34,36].”

Participants in included studies raised concerns that XAI systems could lead to over-reliance and reduce critical thinking skills.

“Participants also raised concerns that the XAI system could lead to over-reliance and reduce critical thinking skills [29,30,33,35].”