CoChat

Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making

Open access
Lee, Min Hun, Chew, Chong Jun · 2023 · Proceedings of the ACM on Human-Computer Interaction

The DOI matches its Crossref record, with a small difference in title or year; 80 of the 80 references it cites were found in a registry or an index; 53 of the 76 sources its claims cite were checked for support.

Abstract

Artificial intelligence (AI) is increasingly being considered to assist human decision-making in high-stake domains (e.g. health). However, researchers have discussed an issue that humans can over-rely on wrong suggestions of the AI model instead of achieving human AI complementary performance. In this work, we utilized salient feature explanations along with what-if, counterfactual explanations to make humans review AI suggestions more analytically to reduce overreliance on AI and explored the effect of these explanations on trust and reliance on AI during clinical decision-making. We conducted an experiment with seven therapists and ten laypersons on the task of assessing post-stroke survivors' quality of motion, and analyzed their performance, agreement level on the task, and reliance on AI without and with two types of AI explanations. Our results showed that the AI model with both salient features and counterfactual explanations assisted therapists and laypersons to improve their performance and agreement level on the task when 'right' AI outputs are presented. While both therapists and laypersons over-relied on 'wrong' AI outputs, counterfactual explanations assisted both therapists and laypersons to reduce their over-reliance on 'wrong' AI outputs by 21% compared to salient feature explanations. Specifically, laypersons had higher performance degrades by 18.0 f1-score with salient feature explanations and 14.0 f1-score with counterfactual explanations than therapists with performance degrades of 8.6 and 2.8 f1-scores respectively. Our work discusses the potential of counterfactual explanations to better estimate the accuracy of an AI model and reduce over-reliance on 'wrong' AI outputs and implications for improving human-AI collaborative decision-making.

Overview

CoChat Summary

## TL;DR

This within-subject study tested whether counterfactual explanations change trust and reliance on AI in a stroke rehabilitation assessment task. Counterfactual explanations reduced overreliance on wrong AI outputs compared with salient feature explanations, but they also increased decision time and were rated less favorably on several usability measures.

## Key findings

  • With right AI outputs, both explanation types improved human+AI performance and agreement, but with wrong AI outputs both harmed performance; counterfactual explanations led to less degraded performance than salient features.
  • Compared with salient feature analysis, counterfactual explanations produced more “right” decisions and fewer “wrong” decisions overall by 21% (29/136), with the same pattern reported for therapists and laypersons separately.
  • For wrong AI outputs, therapists’ performance degradation was 8.6 F1 with salient features vs 2.8 F1 with counterfactual explanations; laypersons’ degradation was 18.0 F1 vs 14.0 F1.
  • Self-reported trust was higher for salient features than counterfactuals: 73.76/100 vs 45.20/100; participants also spent more time per task with counterfactuals (75 s overall vs 57 s with salient features).

## Methods

  • Within-subject online experiment comparing two interfaces: AI with salient feature analysis and AI with salient features + counterfactual explanations.
  • Participants/data: 7 therapists and 10 laypersons assessed the quality of motion of 15 post-stroke survivors using videos from a dataset of 300 videos.
  • Analysis: Participants made initial judgments without AI and final judgments after seeing AI outputs/explanations; outcomes were evaluated using performance (F1-score), agreement level, counts of right/wrong decisions including overreliance, decision time, and usability questionnaires, with Wilcoxon tests reported.

## Limitations

  • The paper states that it did not consider explanation fidelity in the experimental design, which it says should be studied further.
  • The paper states a generalizability limitation because of the small sample size, the single clinical task, and the focus on particular explanation visualizations and a feed-forward neural network.
  • Evident from the text: the study used a fixed set of 3 right and 5 wrong AI outputs per condition, so the findings are tied to that specific task setup and output mix.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3610218

Cites

80 of 80 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 The AI-based decision support system that presents (a) the video of post-stroke survivor's exercises and (b) the predicted assessment score by AI along with salient feature-based explanations that compare a poststroke survivor's unaffected and affected side using the top three most important features and counterfactual, what-if explanations that describe how input features need to be changed to flip an AI output (e.g. 'correct' to 'incorrect' ROM).
Fig. 1 The AI-based decision support system that presents (a) the video of post-stroke survivor's exercises and (b) the predicted assessment score by AI along with salient feature-based explanations that compare a poststroke survivor's unaffected and affected side using the top three most important features and counterfactual, what-if explanations that describe how input features need to be changed to flip an AI output (e.g. 'correct' to 'incorrect' ROM).From the PDF
(a) E1-Unaffected
(a) E1-UnaffectedFrom the PDF
(b) E1-Affected
(b) E1-AffectedFrom the PDF
Fig. 3 The overall procedure of the user study: a participant completed rehabilitation assessment tasks using the AI with salient features (Condition A) and the AI with salient features and counterfactual explanations (Condition B). In each condition, the participant first completed the initial assessment after reviewing only a video and then provided the final assessment after reviewing the AI outputs and explanations. When the participant completed the tasks on each condition, the participant completed the usability questionnaires on each condition. At the end of the study, the participant completed the overall, post-survey about their preferences.
Fig. 3 The overall procedure of the user study: a participant completed rehabilitation assessment tasks using the AI with salient features (Condition A) and the AI with salient features and counterfactual explanations (Condition B). In each condition, the participant first completed the initial assessment after reviewing only a video and then provided the final assessment after reviewing the AI outputs and explanations. When the participant completed the tasks on each condition, the participant completed the usability questionnaires on each condition. At the end of the study, the participant completed the overall, post-survey about their preferences.From the PDF
Fig. 4 Performance of rehabilitation assessment tasks by therapists (TPs) and laypersons (LPs) using AI with Features and Counterfactuals for (1) all cases, (2) cases with 'right' AI outputs, and (3) cases with 'wrong' AI outputs. Although participants' performances improved after reviewing 'right' AI outputs with both salient feature analysis and counterfactual explanations, their performance reduced after reviewing 'wrong' AI outputs. Counterfactual explanations assisted participants to have lower degraded performance than salient feature analysis. *, **, and *** indicate 90%, 95%, and 99% statistical significance levels.
Fig. 4 Performance of rehabilitation assessment tasks by therapists (TPs) and laypersons (LPs) using AI with Features and Counterfactuals for (1) all cases, (2) cases with 'right' AI outputs, and (3) cases with 'wrong' AI outputs. Although participants' performances improved after reviewing 'right' AI outputs with both salient feature analysis and counterfactual explanations, their performance reduced after reviewing 'wrong' AI outputs. Counterfactual explanations assisted participants to have lower degraded performance than salient feature analysis. *, **, and *** indicate 90%, 95%, and 99% statistical significance levels.From the PDF
Fig. 5 Agreement level of rehabilitation assessment tasks by therapists (TPs) and laypersons (LPs) using AI with Features and Counterfactuals for (1) all cases, (2) cases with 'right' AI outputs, and (3) cases with 'wrong' AI outputs. After reviewing 'right' AI outputs with salient feature analysis or counterfactual explanations, both TPs and LPs increased their agreement levels. After reviewing 'wrong' AI outputs, both TPs and LPs using salient features increased their agreement levels while TPs and LPs using counterfactual explanations decreased their agreement levels. *, **, and *** indicate 90%, 95%, and 99% statistical significance levels.
Fig. 5 Agreement level of rehabilitation assessment tasks by therapists (TPs) and laypersons (LPs) using AI with Features and Counterfactuals for (1) all cases, (2) cases with 'right' AI outputs, and (3) cases with 'wrong' AI outputs. After reviewing 'right' AI outputs with salient feature analysis or counterfactual explanations, both TPs and LPs increased their agreement levels. After reviewing 'wrong' AI outputs, both TPs and LPs using salient features increased their agreement levels while TPs and LPs using counterfactual explanations decreased their agreement levels. *, **, and *** indicate 90%, 95%, and 99% statistical significance levels.From the PDF
Fig. 6 The counts of 'right' and 'wrong' decisions by all participants (All), therapists (TPs), and laypersons (LPs). Counterfactual explanations assisted the participants to increase their 'right' decisions and reduce their 'wrong' decisions compared to the salient feature analysis. * and *** indicates 90% and 99% statistical significance levels.
Fig. 6 The counts of 'right' and 'wrong' decisions by all participants (All), therapists (TPs), and laypersons (LPs). Counterfactual explanations assisted the participants to increase their 'right' decisions and reduce their 'wrong' decisions compared to the salient feature analysis. * and *** indicates 90% and 99% statistical significance levels.From the PDF
Fig. 7 Detailed analysis of (a) 'right' and (b) 'wrong' human decisions by all participants (All), therapists (TPs), and laypersons (LPs): Counterfactual explanations assisted the participants to increase the number of 'right' decisions on rejecting 'wrong' AI outputs and reduce the number of 'wrong' decisions on agreeing 'wrong' AI outputs.
Fig. 7 Detailed analysis of (a) 'right' and (b) 'wrong' human decisions by all participants (All), therapists (TPs), and laypersons (LPs): Counterfactual explanations assisted the participants to increase the number of 'right' decisions on rejecting 'wrong' AI outputs and reduce the number of 'wrong' decisions on agreeing 'wrong' AI outputs.From the PDF
Fig. 8 Participants' usability responses on the system with (1) salient feature analysis (Features) and (2) counterfactual explanations (Counterfacts). Overall, participants expressed higher usage intent and potential with the system with salient feature analysis. They considered that the system with salient feature analysis is considered to be more useful, provides more insights without significance, requires less effort on assessment, is more reliable, transparent without significance, more trustful, and less frustrating. *** indicates 99% statistical significance level.
Fig. 8 Participants' usability responses on the system with (1) salient feature analysis (Features) and (2) counterfactual explanations (Counterfacts). Overall, participants expressed higher usage intent and potential with the system with salient feature analysis. They considered that the system with salient feature analysis is considered to be more useful, provides more insights without significance, requires less effort on assessment, is more reliable, transparent without significance, more trustful, and less frustrating. *** indicates 99% statistical significance level.From the PDF

Verification

Claims the paper makes

28 Valid · 3 Not valid · 5 Not checked · 76 Own results and interpretations

47 more claims · 4 with their cited sources checked

Create a free account to see all 112 claims and to verify the 1 reference still open.

See all claims — free account
  • Probably not a substantial claim. 35
  • It may state two separate claims as one. 30
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Citations

Cites

Show all 80 references

Cited by

Metadata

Authors
Lee, M. H., & Chew, C. J.
Affiliations
Singapore Management University (SG)
Venue
Proceedings of the ACM on Human-Computer Interaction
Published
2023-09-28
Volume
7
Issue
CSCW2
Pages
1-22
ISSN
2573-0142 (ISSN-L)
OpenAlex
W4385895991
Cited by
78
References
77
Citations by year
2026: 28 · 2025: 41 · 2024: 9
FWCI
7.34
Type
journal article
Language
en
Open access
Locations
  • Proceedings of the ACM on Human-Computer Interaction (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Funders
The Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) Tier 1 grant
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
Keywords
counterfactual explanations, trust in AI, overreliance on AI, human-AI collaboration, clinical decision-making, laypeople, therapists, explainable AI, AI-assisted diagnosis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.