The DOI matches its Crossref record, with a small difference in title or year; 79 of the 79 references it cites were found in a registry or an index; 53 of the 76 sources its claims cite were checked for support.
Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making
Open accessAbstract
Artificial intelligence (AI) is increasingly being considered to assist human decision-making in high-stake domains (e.g. health). However, researchers have discussed an issue that humans can over-rely on wrong suggestions of the AI model instead of achieving human AI complementary performance. In this work, we utilized salient feature explanations along with what-if, counterfactual explanations to make humans review AI suggestions more analytically to reduce overreliance on AI and explored the effect of these explanations on trust and reliance on AI during clinical decision-making. We conducted an experiment with seven therapists and ten laypersons on the task of assessing post-stroke survivors' quality of motion, and analyzed their performance, agreement level on the task, and reliance on AI without and with two types of AI explanations. Our results showed that the AI model with both salient features and counterfactual explanations assisted therapists and laypersons to improve their performance and agreement level on the task when 'right' AI outputs are presented. While both therapists and laypersons over-relied on 'wrong' AI outputs, counterfactual explanations assisted both therapists and laypersons to reduce their over-reliance on 'wrong' AI outputs by 21% compared to salient feature explanations. Specifically, laypersons had higher performance degrades by 18.0 f1-score with salient feature explanations and 14.0 f1-score with counterfactual explanations than therapists with performance degrades of 8.6 and 2.8 f1-scores respectively. Our work discusses the potential of counterfactual explanations to better estimate the accuracy of an AI model and reduce over-reliance on 'wrong' AI outputs and implications for improving human-AI collaborative decision-making.
Overview
CoChat Summary
## TL;DR
This within-subject study tested whether counterfactual explanations change trust and reliance on AI in a stroke rehabilitation assessment task. Counterfactual explanations reduced overreliance on wrong AI outputs compared with salient feature explanations, but they also increased decision time and were rated less favorably on several usability measures.
## Key findings
- With right AI outputs, both explanation types improved human+AI performance and agreement, but with wrong AI outputs both harmed performance; counterfactual explanations led to less degraded performance than salient features.
- Compared with salient feature analysis, counterfactual explanations produced more “right” decisions and fewer “wrong” decisions overall by 21% (29/136), with the same pattern reported for therapists and laypersons separately.
- For wrong AI outputs, therapists’ performance degradation was 8.6 F1 with salient features vs 2.8 F1 with counterfactual explanations; laypersons’ degradation was 18.0 F1 vs 14.0 F1.
- Self-reported trust was higher for salient features than counterfactuals: 73.76/100 vs 45.20/100; participants also spent more time per task with counterfactuals (75 s overall vs 57 s with salient features).
## Methods
- Within-subject online experiment comparing two interfaces: AI with salient feature analysis and AI with salient features + counterfactual explanations.
- Participants/data: 7 therapists and 10 laypersons assessed the quality of motion of 15 post-stroke survivors using videos from a dataset of 300 videos.
- Analysis: Participants made initial judgments without AI and final judgments after seeing AI outputs/explanations; outcomes were evaluated using performance (F1-score), agreement level, counts of right/wrong decisions including overreliance, decision time, and usability questionnaires, with Wilcoxon tests reported.
## Limitations
- The paper states that it did not consider explanation fidelity in the experimental design, which it says should be studied further.
- The paper states a generalizability limitation because of the small sample size, the single clinical task, and the focus on particular explanation visualizations and a feed-forward neural network.
- Evident from the text: the study used a fixed set of 3 right and 5 wrong AI outputs per condition, so the findings are tied to that specific task setup and output mix.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures









Verification
Claims the paper makes
28 Valid · 3 Not valid · 5 Not checked · 76 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, the previous research discusses the issues of these feature-based explanations on overtrust in AI [29, 37, 64].”
What the sources say
- Not checked — no full text of the sourcecites [29] [29] Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, and Jennifer Wortman Vaughan. 2020. Interpreting interpretability: understanding data scientists' use of interpretability tools for machine learning. In Proceedings of the 2020 CHI conference on human factors in computing systems . 1-14. - source
- Partly supportedcites [37] [37] Vivian Lai, Chacha Chen, Q Vera Liao, Alison Smith-Renner, and Chenhao Tan. 2021. Towards a science of human-ai decision making: a survey of empirical studies. arXiv preprint arXiv:2112.11471 (2021). - sourceExperimental — accuracy not yet measured
Section 4.2 (p. 15) discusses feature-importance explanations, and Section 5.2 (p. 25) discusses over-reliance on AI as a trust-related measure. The passages do not establish that feature-based explanations cause or contribute to overtrust; that connection is unsupported by the passages found. (LLM, conf=0.78)
- Not checked — no full text of the sourcecites [64] [64] Xinru Wang and Ming Yin. 2021. Are explanations helpful? a comparative study of the effects of explanations in ai-assisted decision-making. In 26th International Conference on Intelligent User Interfaces . 318-328. - source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Other works that utilize AI/ML models focused on tasks that do not require domain experts, such as judges or clinicians [5, 38, 64].”
What the sources say
- Supportedcites [5] [5] Gagan Bansal, Besmira Nushi, Ece Kamar, Walter S Lasecki, Daniel S Weld, and Eric Horvitz. 2019. Beyond accuracy: The role of mental models in human-AI team performance. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , Vol. 7. 2-11. - source
Section 4 (Experiments Setup) describes a game-like assembly-line task in which participants decide whether objects are defective using AI recommendations. It explicitly says the task abstracts away human expertise in problem solving, supporting the characterization of this study as using a task that does not require domain experts. The judges and clinicians mentioned in the Background are examples of real-world applications, not participants in these experiments. (LLM, conf=0.91)
- Supportedcites [38] [38] Vivian Lai and Chenhao Tan. 2019. On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of the conference on fairness, accountability, and transparency . 29-38. - source
The Experimental Setup (p. 3) says the study recruited Amazon Mechanical Turk participants to judge whether hotel reviews were genuine or deceptive. The Concluding Discussion (p. 9) treats bail decisions and medical diagnosis as tasks for future study, rather than tasks tested here. (LLM, conf=0.92)
- Not checked — no full text of the sourcecites [64] [64] Xinru Wang and Ming Yin. 2021. Are explanations helpful? a comparative study of the effects of explanations in ai-assisted decision-making. In 26th International Conference on Intelligent User Interfaces . 318-328. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, clinicians can be biased in their decision making and expert disagreement is prevalent in medical decision-making tasks [7, 33, 34].”
What the sources say
- Partly supportedcites [7] [7] Michael L Barnett, Dhruv Boddupalli, Shantanu Nundy, and David W Bates. 2019. Comparative accuracy of diagnosis by collective intelligence of multiple physicians vs individual physicians. JAMA network open 2, 3 (2019). - sourceExperimental — accuracy not yet measured
The Discussion reports that final diagnoses differed from referral diagnoses in 21% of subspecialist referrals, supporting disagreement in a specific diagnostic setting. It does not establish that expert disagreement is prevalent across medical decision-making tasks generally; the paper also notes that most referral diagnoses were unchanged or only refined. (LLM, conf=0.82)
- Partly supportedcites [33] [33] Peter Knapp and Jenny Hewison. 1999. Disagreement in patient and carer assessment of functional abilities after stroke. Stroke 30, 5 (1999), 934-938. - sourceExperimental — accuracy not yet measured
The introduction (p. 1) reports that doctors and nurses have given different ratings of patients’ functional abilities, supporting an example of clinical disagreement. The study itself examines stroke patients and informal carers, not experts, and neither passage establishes that expert disagreement is prevalent across medical decision-making tasks. (LLM, conf=0.82)
- Partly supportedcites [34] [34] Jonathan Krause, Varun Gulshan, Ehsan Rahimy, Peter Karth, Kasumi Widner, Greg S Corrado, Lily Peng, and Dale R Webster. 2018. Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Ophthalmology 125, 8 (2018), 1264-1272. - sourceExperimental — accuracy not yet measured
The Discussion (p. 7) describes frequent disagreement among retinal specialists grading diabetic retinopathy images, including subjective differences at grade boundaries. That supports expert disagreement in this medical task, but the paper does not establish that disagreement is prevalent across medical decision-making tasks generally. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Similar to the previous research [11], our study also shows a positive correlation between user's trust and reliance: the more a user trusts the system, the more the user is likely to over-rely on outputs of the system even when 'wrong' AI outputs are presented.”
What the sources say
- Supportedcites [11] [11] Adrian Bussone, Simone Stumpf, and Dympna O'Sullivan. 2015. The role of explanations on trust and reliance in clinical decision support systems. In 2015 international conference on healthcare informatics . IEEE, 160-169. - source
The Discussion (p. 11) reports a strong relationship between trust and reliance: users who highly trust the CDSS are likely to over-rely on its suggestions. The System Reliability and Use section (p. 4) defines over-reliance as agreeing with incorrect suggestions. The citing authors' statement about their own study is not attributed to this source. (LLM, conf=0.92)
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, clinicians can be biased in their decision making and expert disagreement is prevalent in medical decision-making tasks [7, 33, 34].”
What the sources say
- Could not verifycites [7] [7] Michael L Barnett, Dhruv Boddupalli, Shantanu Nundy, and David W Bates. 2019. Comparative accuracy of diagnosis by collective intelligence of multiple physicians vs individual physicians. JAMA network open 2, 3 (2019). - sourceExperimental — accuracy not yet measured
The Introduction discusses misdiagnosis, but does not attribute it to bias in clinicians’ decision-making. The Discussion > Limitations mentions possible bias in the study’s assessment of diagnostic solutions, not bias in clinicians’ decisions. Searches for cognitive bias and biased physician judgments found no passage addressing the attributed claim. (LLM, conf=0.75)
- Partly supportedcites [33] [33] Peter Knapp and Jenny Hewison. 1999. Disagreement in patient and carer assessment of functional abilities after stroke. Stroke 30, 5 (1999), 934-938. - sourceExperimental — accuracy not yet measured
The introduction (p. 1) discusses possible bias in assessments used clinically and reports that doctors and nurses gave different ratings of patients’ abilities. The study itself examines disagreement between stroke patients and carers, not bias in clinicians’ decision-making, so it supports only the narrower point about variability in assessments. (LLM, conf=0.78)
- Partly supportedcites [34] [34] Jonathan Krause, Varun Gulshan, Ehsan Rahimy, Peter Karth, Kasumi Widner, Greg S Corrado, Lily Peng, and Dale R Webster. 2018. Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Ophthalmology 125, 8 (2018), 1264-1272. - sourceExperimental — accuracy not yet measured
The Discussion (p. 7) reports subjective differences among retina specialists in how they applied diabetic-retinopathy grading boundaries, and Grading and Adjudication (p. 5) documents ophthalmologists’ grading errors. These passages support the narrower point that clinical judgments can vary, but they do not establish bias in clinicians’ decision-making generally. (LLM, conf=0.78)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
47 more claims · 4 with their cited sources checked
Create a free account to see all 112 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 35
- It may state two separate claims as one. 30
- The quote only partly states this claim. 2
Citations
Cites
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
Gauriat, C.-M., Pencolé, Y., Ribot, P., & Brouillet, G. (2024) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.4230/oasics.dx.2024.27 · Cited by 12,586
Parkinson's disease: clinical features and diagnosis
Jankovic, J. (2008) · Journal of Neurology, Neurosurgery & Psychiatry · doi:10.1136/jnnp.2007.131045 · Cited by 5,720
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
The Mythos of Model Interpretability
Lipton, Z. C. (2016) · arXiv · doi:10.48550/arxiv.1606.03490 · Cited by 5,422
Humans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554 · cited 1×
“In addition to the contradictory perspectives on the effect of explanations on trust, our research community still requires additional studies to understand the effect of AI explanations on overreliance [54].”
BackgroundRajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-021-01614-0 · Cited by 3,212
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524
Cited by
Large language models in healthcare and medical domain: A review
Nazi, Z. A., & Peng, W. (2024) · Informatics · doi:10.3390/informatics11030057 · Cited by 415
The Typing Cure: Experiences with Large Language Model Chatbots for Mental Health Support
Song, I., Pendse, S. R., Kumar, N., & De Choudhury, M. (2024) · arXiv · doi:10.48550/arxiv.2401.14362 · Cited by 64
The Typing Cure: Experiences with Large Language Model Chatbots for Mental Health Support
Song, I., Pendse, S. R., Kumar, N., & Choudhury, M. D. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757430 · Cited by 193
From human-centered to symbiotic artificial intelligence: a focus on medical applications
Desolda, G., Esposito, A., Lanzilotti, R., Piccinno, A., & Costabile, M. F. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-20414-5 · Cited by 69
Emotionally adaptive support: a narrative review of affective computing for mental health
Schlicher, M., Li, Y., Murthy, S. M. K., Sun, Q., & Schuller, B. W. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1657031 · Cited by 60
“Existing studies have explored various techniques such as visualizing attention in medical imaging in order to understand the central parts of the AI’s decision (147), as well as salient features and counterfactual explanations to prevent clinicians from over-relying on incorrect AI outputs (148).”
Metadata
- Authors
- Lee, M. H., & Chew, C. J.
- Affiliations
- Singapore Management University (SG)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2023-09-28
- Volume
- 7
- Issue
- CSCW2
- Pages
- 1-22
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W4385895991
- Semantic Scholar
- afe05c9afdeb48e111712bb6fe5f8b68632145fb
- Cited by
- 78
- References
- 77
- Citations by year
- 2026: 28 · 2025: 41 · 2024: 9
- FWCI
- 7.34
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- The Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) Tier 1 grant
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- counterfactual explanations, trust in AI, overreliance on AI, human-AI collaboration, clinical decision-making, laypeople, therapists, explainable AI, AI-assisted diagnosis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making. https://app.cochat.ai/papers/10.1145/3610218
- The paper
- Lee, M. H., & Chew, C. J. (2023). Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making. Proceedings of the ACM on Human-Computer Interaction. https://doi.org/10.1145/3610218
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.