Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making
Open accessThe DOI matches its Crossref record, with a small difference in title or year; 80 of the 80 references it cites were found in a registry or an index; 53 of the 76 sources its claims cite were checked for support.
Abstract
Artificial intelligence (AI) is increasingly being considered to assist human decision-making in high-stake domains (e.g. health). However, researchers have discussed an issue that humans can over-rely on wrong suggestions of the AI model instead of achieving human AI complementary performance. In this work, we utilized salient feature explanations along with what-if, counterfactual explanations to make humans review AI suggestions more analytically to reduce overreliance on AI and explored the effect of these explanations on trust and reliance on AI during clinical decision-making. We conducted an experiment with seven therapists and ten laypersons on the task of assessing post-stroke survivors' quality of motion, and analyzed their performance, agreement level on the task, and reliance on AI without and with two types of AI explanations. Our results showed that the AI model with both salient features and counterfactual explanations assisted therapists and laypersons to improve their performance and agreement level on the task when 'right' AI outputs are presented. While both therapists and laypersons over-relied on 'wrong' AI outputs, counterfactual explanations assisted both therapists and laypersons to reduce their over-reliance on 'wrong' AI outputs by 21% compared to salient feature explanations. Specifically, laypersons had higher performance degrades by 18.0 f1-score with salient feature explanations and 14.0 f1-score with counterfactual explanations than therapists with performance degrades of 8.6 and 2.8 f1-scores respectively. Our work discusses the potential of counterfactual explanations to better estimate the accuracy of an AI model and reduce over-reliance on 'wrong' AI outputs and implications for improving human-AI collaborative decision-making.
Overview
CoChat Summary
## TL;DR
This within-subject study tested whether counterfactual explanations change trust and reliance on AI in a stroke rehabilitation assessment task. Counterfactual explanations reduced overreliance on wrong AI outputs compared with salient feature explanations, but they also increased decision time and were rated less favorably on several usability measures.
## Key findings
- With right AI outputs, both explanation types improved human+AI performance and agreement, but with wrong AI outputs both harmed performance; counterfactual explanations led to less degraded performance than salient features.
- Compared with salient feature analysis, counterfactual explanations produced more “right” decisions and fewer “wrong” decisions overall by 21% (29/136), with the same pattern reported for therapists and laypersons separately.
- For wrong AI outputs, therapists’ performance degradation was 8.6 F1 with salient features vs 2.8 F1 with counterfactual explanations; laypersons’ degradation was 18.0 F1 vs 14.0 F1.
- Self-reported trust was higher for salient features than counterfactuals: 73.76/100 vs 45.20/100; participants also spent more time per task with counterfactuals (75 s overall vs 57 s with salient features).
## Methods
- Within-subject online experiment comparing two interfaces: AI with salient feature analysis and AI with salient features + counterfactual explanations.
- Participants/data: 7 therapists and 10 laypersons assessed the quality of motion of 15 post-stroke survivors using videos from a dataset of 300 videos.
- Analysis: Participants made initial judgments without AI and final judgments after seeing AI outputs/explanations; outcomes were evaluated using performance (F1-score), agreement level, counts of right/wrong decisions including overreliance, decision time, and usability questionnaires, with Wilcoxon tests reported.
## Limitations
- The paper states that it did not consider explanation fidelity in the experimental design, which it says should be studied further.
- The paper states a generalizability limitation because of the small sample size, the single clinical task, and the focus on particular explanation visualizations and a feed-forward neural network.
- Evident from the text: the study used a fixed set of 3 right and 5 wrong AI outputs per condition, so the findings are tied to that specific task setup and output mix.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures









Verification
Claims the paper makes
28 Valid · 3 Not valid · 5 Not checked · 76 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, the previous research discusses the issues of these feature-based explanations on overtrust in AI [29, 37, 64].”
What the sources say
- Not checked — no full text of the sourcecites [29] [29] Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, and Jennifer Wortman Vaughan. 2020. Interpreting interpretability: understanding data scientists' use of interpretability tools for machine learning. In Proceedings of the 2020 CHI conference on human factors in computing systems . 1-14. - source
- Partly supportedcites [37] [37] Vivian Lai, Chacha Chen, Q Vera Liao, Alison Smith-Renner, and Chenhao Tan. 2021. Towards a science of human-ai decision making: a survey of empirical studies. arXiv preprint arXiv:2112.11471 (2021). - sourceExperimental — accuracy not yet measured
Section 4.2 (p. 15) discusses feature-importance explanations, and Section 5.2 (p. 25) discusses over-reliance on AI as a trust-related measure. The passages do not establish that feature-based explanations cause or contribute to overtrust; that connection is unsupported by the passages found. (LLM, conf=0.78)
- Not checked — no full text of the sourcecites [64] [64] Xinru Wang and Ming Yin. 2021. Are explanations helpful? a comparative study of the effects of explanations in ai-assisted decision-making. In 26th International Conference on Intelligent User Interfaces . 318-328. - source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Other works that utilize AI/ML models focused on tasks that do not require domain experts, such as judges or clinicians [5, 38, 64].”
What the sources say
- Supportedcites [5] [5] Gagan Bansal, Besmira Nushi, Ece Kamar, Walter S Lasecki, Daniel S Weld, and Eric Horvitz. 2019. Beyond accuracy: The role of mental models in human-AI team performance. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , Vol. 7. 2-11. - source
Section 4 (Experiments Setup) describes a game-like assembly-line task in which participants decide whether objects are defective using AI recommendations. It explicitly says the task abstracts away human expertise in problem solving, supporting the characterization of this study as using a task that does not require domain experts. The judges and clinicians mentioned in the Background are examples of real-world applications, not participants in these experiments. (LLM, conf=0.91)
- Supportedcites [38] [38] Vivian Lai and Chenhao Tan. 2019. On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of the conference on fairness, accountability, and transparency . 29-38. - source
The Experimental Setup (p. 3) says the study recruited Amazon Mechanical Turk participants to judge whether hotel reviews were genuine or deceptive. The Concluding Discussion (p. 9) treats bail decisions and medical diagnosis as tasks for future study, rather than tasks tested here. (LLM, conf=0.92)
- Not checked — no full text of the sourcecites [64] [64] Xinru Wang and Ming Yin. 2021. Are explanations helpful? a comparative study of the effects of explanations in ai-assisted decision-making. In 26th International Conference on Intelligent User Interfaces . 318-328. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, clinicians can be biased in their decision making and expert disagreement is prevalent in medical decision-making tasks [7, 33, 34].”
What the sources say
- Partly supportedcites [7] [7] Michael L Barnett, Dhruv Boddupalli, Shantanu Nundy, and David W Bates. 2019. Comparative accuracy of diagnosis by collective intelligence of multiple physicians vs individual physicians. JAMA network open 2, 3 (2019). - sourceExperimental — accuracy not yet measured
The Discussion reports that final diagnoses differed from referral diagnoses in 21% of subspecialist referrals, supporting disagreement in a specific diagnostic setting. It does not establish that expert disagreement is prevalent across medical decision-making tasks generally; the paper also notes that most referral diagnoses were unchanged or only refined. (LLM, conf=0.82)
- Partly supportedcites [33] [33] Peter Knapp and Jenny Hewison. 1999. Disagreement in patient and carer assessment of functional abilities after stroke. Stroke 30, 5 (1999), 934-938. - sourceExperimental — accuracy not yet measured
The introduction (p. 1) reports that doctors and nurses have given different ratings of patients’ functional abilities, supporting an example of clinical disagreement. The study itself examines stroke patients and informal carers, not experts, and neither passage establishes that expert disagreement is prevalent across medical decision-making tasks. (LLM, conf=0.82)
- Partly supportedcites [34] [34] Jonathan Krause, Varun Gulshan, Ehsan Rahimy, Peter Karth, Kasumi Widner, Greg S Corrado, Lily Peng, and Dale R Webster. 2018. Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Ophthalmology 125, 8 (2018), 1264-1272. - sourceExperimental — accuracy not yet measured
The Discussion (p. 7) describes frequent disagreement among retinal specialists grading diabetic retinopathy images, including subjective differences at grade boundaries. That supports expert disagreement in this medical task, but the paper does not establish that disagreement is prevalent across medical decision-making tasks generally. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Similar to the previous research [11], our study also shows a positive correlation between user's trust and reliance: the more a user trusts the system, the more the user is likely to over-rely on outputs of the system even when 'wrong' AI outputs are presented.”
What the sources say
- Supportedcites [11] [11] Adrian Bussone, Simone Stumpf, and Dympna O'Sullivan. 2015. The role of explanations on trust and reliance in clinical decision support systems. In 2015 international conference on healthcare informatics . IEEE, 160-169. - source
The Discussion (p. 11) reports a strong relationship between trust and reliance: users who highly trust the CDSS are likely to over-rely on its suggestions. The System Reliability and Use section (p. 4) defines over-reliance as agreeing with incorrect suggestions. The citing authors' statement about their own study is not attributed to this source. (LLM, conf=0.92)
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, clinicians can be biased in their decision making and expert disagreement is prevalent in medical decision-making tasks [7, 33, 34].”
What the sources say
- Could not verifycites [7] [7] Michael L Barnett, Dhruv Boddupalli, Shantanu Nundy, and David W Bates. 2019. Comparative accuracy of diagnosis by collective intelligence of multiple physicians vs individual physicians. JAMA network open 2, 3 (2019). - sourceExperimental — accuracy not yet measured
The Introduction discusses misdiagnosis, but does not attribute it to bias in clinicians’ decision-making. The Discussion > Limitations mentions possible bias in the study’s assessment of diagnostic solutions, not bias in clinicians’ decisions. Searches for cognitive bias and biased physician judgments found no passage addressing the attributed claim. (LLM, conf=0.75)
- Partly supportedcites [33] [33] Peter Knapp and Jenny Hewison. 1999. Disagreement in patient and carer assessment of functional abilities after stroke. Stroke 30, 5 (1999), 934-938. - sourceExperimental — accuracy not yet measured
The introduction (p. 1) discusses possible bias in assessments used clinically and reports that doctors and nurses gave different ratings of patients’ abilities. The study itself examines disagreement between stroke patients and carers, not bias in clinicians’ decision-making, so it supports only the narrower point about variability in assessments. (LLM, conf=0.78)
- Partly supportedcites [34] [34] Jonathan Krause, Varun Gulshan, Ehsan Rahimy, Peter Karth, Kasumi Widner, Greg S Corrado, Lily Peng, and Dale R Webster. 2018. Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Ophthalmology 125, 8 (2018), 1264-1272. - sourceExperimental — accuracy not yet measured
The Discussion (p. 7) reports subjective differences among retina specialists in how they applied diabetic-retinopathy grading boundaries, and Grading and Adjudication (p. 5) documents ophthalmologists’ grading errors. These passages support the narrower point that clinical judgments can vary, but they do not establish bias in clinicians’ decision-making generally. (LLM, conf=0.78)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
47 more claims · 4 with their cited sources checked
Create a free account to see all 112 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 35
- It may state two separate claims as one. 30
- The quote only partly states this claim. 2
Citations
Cites
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
Gauriat, C.-M., Pencolé, Y., Ribot, P., & Brouillet, G. (2024) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.4230/oasics.dx.2024.27 · Cited by 12,586
Parkinson's disease: clinical features and diagnosis
Jankovic, J. (2008) · Journal of Neurology, Neurosurgery & Psychiatry · doi:10.1136/jnnp.2007.131045 · Cited by 5,720
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
The Mythos of Model Interpretability
Lipton, Z. C. (2016) · arXiv · doi:10.48550/arxiv.1606.03490 · Cited by 5,422
Humans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554 · cited 1×
“In addition to the contradictory perspectives on the effect of explanations on trust, our research community still requires additional studies to understand the effect of AI explanations on overreliance [54].”
BackgroundRajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-021-01614-0 · Cited by 3,212
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524
Cited by
Large language models in healthcare and medical domain: A review
Nazi, Z. A., & Peng, W. (2024) · Informatics · doi:10.3390/informatics11030057 · Cited by 415
The Typing Cure: Experiences with Large Language Model Chatbots for Mental Health Support
Song, I., Pendse, S. R., Kumar, N., & De Choudhury, M. (2024) · arXiv · doi:10.48550/arxiv.2401.14362 · Cited by 64
The Typing Cure: Experiences with Large Language Model Chatbots for Mental Health Support
Song, I., Pendse, S. R., Kumar, N., & Choudhury, M. D. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757430 · Cited by 196
From human-centered to symbiotic artificial intelligence: a focus on medical applications
Desolda, G., Esposito, A., Lanzilotti, R., Piccinno, A., & Costabile, M. F. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-20414-5 · Cited by 69
Emotionally adaptive support: a narrative review of affective computing for mental health
Schlicher, M., Li, Y., Murthy, S. M. K., Sun, Q., & Schuller, B. W. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1657031 · Cited by 61
“Existing studies have explored various techniques such as visualizing attention in medical imaging in order to understand the central parts of the AI’s decision (147), as well as salient features and counterfactual explanations to prevent clinicians from over-relying on incorrect AI outputs (148).”
Göndöcs, D., Horváth, S., & Dörfler, V. (2025) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2025.103622 · Cited by 52
Nicora, G., Catalano, M., Bortolotto, C., Achilli, M., Messana, G., Tito, A. L., Consonni, A., Cutti, S., Comotto, F., Stella, G., et al. (2024) · Journal of Imaging · doi:10.3390/jimaging10050117 · Cited by 15
“…may support AI-based clinical decisions by facilitating conditions, i.e., by increasing the individual trust that technical infrastructure is in fact supporting the use of the AI-based system, which has been highlighted as an important factor in predicting the intention to use AI [43,44].”
BackgroundWang, L., Smriti, D., Yuan, H., & Huh, J. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650846 · Cited by 18
From Text to Trust: Empowering AI-assisted Decision Making with Adaptive LLM-powered Analysis
Li, Z., Zhu, H., Lu, Z., Xiao, Z., & Yin, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713133 · Cited by 37
Yun, S., & Lim, Y.-K. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713347 · Cited by 33
Eckhardt, S., Kühl, N., Dolata, M., & Schwabe, G. (2025) · ACM Computing Surveys · doi:10.1145/3776528 · Cited by 44
"It depends": Configuring AI to Improve Clinical Usefulness Across Contexts
Zając, H. D., Ribeiro, J., Ingala, S., Gentile, S., Wanjohi, R., Gitau, S. N., Carlsen, J. F., Nielsen, M. B., & Andersen, T. O. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3660707 · Cited by 23
The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates
Russo, G., Horta Ribeiro, M., Davidson, T. R., Veselovsky, V., & West, R. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757667 · Cited by 12
Wolf, S., Grundgeiger, T., Zähringer, R., Shishkova, L., Maas, F., Dilling, C., & Happel, O. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713372 · Cited by 12
Xu, Y., Shao, Y., Dong, J., Shi, S., Jiang, C., & Li, Q. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713772 · Cited by 16
Zając, H. D., Andersen, T. O., Kwasa, E., Wanjohi, R., Onyinkwa, M. K., Mwaniki, E. K., Gitau, S. N., Yaseen, S. S., Carlsen, J. F., Fraccaro, M., et al. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3715115 · Cited by 22
Breckner, K., Neumayr, T., Mara, M., Streit, M., & Augstein, M. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2482742 · Cited by 16
Selective Trust: Understanding Human-AI Partnerships in Personal Health Decision-Making Process
van Arum, S., Genç, H. U., Reidsma, D., & Karahanoğlu, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713462 · Cited by 30
“While considerable progress has been made in recent years in understanding the perspectives of healthcare providers in making decisions through human-AI collaboration [e.g., 32, 41, 60], this knowledge has not yet been transferred to human-AI collaboration in personal health decision-making.”
Counterfactual Explanations May Not Be the Best Algorithmic Recourse Approach
Upadhyay, S., Lakkaraju, H., & Gajos, K. Z. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712095 · Cited by 20
POTDAI: A Tool to Evaluate the Perceived Operational Trust Degree in Artificial Intelligence Systems
Martín-Moncunill, D., García-Laredo, E., & Nieves, J. C. (2024) · IEEE Access · doi:10.1109/access.2024.3454061 · Cited by 10
“…on the detection of serious pathologies such as cancer [29], one on trust in the use of AI in surgery [31], and another on clinical decision making [32]), two studies on the extent to which a user believes that his or her initial trust can be improved by explanations provided by the system itself…”
BackgroundMethodMastrianni, A., Kim, M. S., Sullivan, T. M., Sippel, G. J., Burd, R. S., Gajos, K. Z., & Sarcevic, A. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757512 · Cited by 6
Raees, M., Khan, V.-J., Lykourentzou, I., & Papangelis, K. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791467 · Cited by 5
Artificial intelligence impacts in education and pediatric mental health
Liberatore, G., Kim, A., Brenner, J., & Milanaik, R. L. (2025) · Current Opinion in Pediatrics · doi:10.1097/mop.0000000000001453 · Cited by 7
Füßl, A., & Nissen, V. (2025) · Requirements Engineering · doi:10.1007/s00766-025-00437-6 · Cited by 8
Kuhl, U., & Bush, A. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-08333-3_12 · Cited by 12
“This finding challenges assumptions regarding the universally positive effects of CEs on human-AI collaborative decision-making [23], suggesting a more complex relationship than previously theorized.”
InfluentialInfo-CELS: Informative Saliency Map-Guided Counterfactual Explanation for Time Series Classification
Li, P., Bahri, O., Hosseinzadeh, P., Boubrahimi, S. F., & Hamdi, S. M. (2025) · Electronics · doi:10.3390/electronics14071311 · Cited by 10
Fact or Fiction? Exploring Explanations to Identify Factual Confabulations in RAG-Based LLM Systems
Reinhard, P., Li, M. M., Fina, M., & Leimeister, J. M. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3720249 · Cited by 13
AI-Powered Collective Decision-Making Systems and the Future Trends
Khan, Z., & Ambadekar, S. P. (2024) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt61001.2024.10725853 · Cited by 7
“Min Hun Lee and Chong Jun Chew's[8] study investigates the impact of counterfactual explanations on AI trust and reliance in clinical decision-making.”
BackgroundReimagining Multidisciplinary Teams: Challenges and Opportunities for LLMs in Cancer MDTs
Paulo, S. F., Neto, I., Figueiredo, N., Lopes, D. S., & Nicolau, H. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711055 · Cited by 4
Towards Uncertainty Aware Task Delegation and Human-AI Collaborative Decision-Making
Lee, M. H., & Tok, M. Z. Y. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732155 · Cited by 14
Self-citationMultidisciplinary perspectives on human‑AI team trust
Tielman, M. L., Bailey, M. E., Frattolillo, F., Jorge, C. C., Ulfert, A., & Meyer-Vitali, A. (2025) · Interaction Studies Social Behaviour and Communication in Biological and Artificial Systems · doi:10.1075/is.24048.tie · Cited by 5
Mehrotra, S., Gadiraju, U., Bittner, E. A. C., van Delden, F., Jonker, C. M., & Tielman, M. L. (2025) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3699682.3728343 · Cited by 8
Sun, Y., Yue, W., Jin, X., Ma, S., Ma, X., & Wang, X. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757705 · Cited by 6
Cóias, A. R., Lee, M. H., Bernardino, A. J. M., Smailagic, A., Mateus, M. P. D. N., Fernandes, D., & Trapola, S. (2025) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2025.3602548 · Cited by 5
“The saliency map provides us with insights about VLC decision [53].”
Self-citationChen, Y., Verbert, K., Gerling, K., Abeele, M.-E. V., & Abeele, V. V. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712139 · Cited by 7
Ghaffar, F., Islam, Y. Z., Furtado, N. M., & Burns, C. M. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.111132 · Cited by 4
FashionCook: A Visual Analytics System for Human–AI Collaboration in Fashion E-Commerce Design
Shao, Y., Liu, S., Chen, G., Ma, R., Wang, X., & Li, Q. (2025) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2025.3597849 · Cited by 2
User Decision Guidance with Selective Explanation Presentation from Explainable-AI
Fukuchi, Y., & Yamada, S. (2024) · IEEE RO-MAN · doi:10.1109/ro-man60168.2024.10731188 · Cited by 6
“Previous studies have integrated such XAI methods into IDSSs and shown their effectiveness in presenting explanations along with AI predictions in diverse applications [3], [4].”
MethodImproving Tactical Decision-Making Through Multiobjective Contrastive Explanations
Blom, M. L., Singh, R., Miller, T., Sonenberg, L., Trentelman, K., Saulwick, A., & Wark, S. (2025) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2025.3591163 · Cited by 2
“Recent research has investigated XAI for a range of tasks, such as document matching [27], financial advice [28], question answering [29], medical diagnosis and treatment planning [30], [31], [32], and even mushroom picking [22].”
Mariyono, D., Jalil, A. H. A., Mustafida, F., Muslim, M., & Mazhabi, Z. (2025) · New Ideas in Psychology · doi:10.1016/j.newideapsych.2025.101187 · Cited by 7
Data-Driven Driver Training via Counterfactual and Language-Based Guidance in Racing Scenarios
Bae, J., Nam, H., Ryu, K., Lee, J., Kim, J., Chun, H., Han, J., & Choi, J. (2025) · IEEE Access · doi:10.1109/access.2025.3615128 · Cited by 2
“To address this, various studies have explored counterfactual approaches using XAI results [14], [15], [40], [41].”
Raees, M., & Papagelis, K. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3798834 · Cited by 1
Discrimination Exposed? On the Reliability of Explanations for Discrimination Detection
Skirzyński, J., Danks, D. J., & Ustun, B. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732167 · Cited by 7
More Isn't Always Better: Balancing Decision Accuracy and Conformity Pressures in Multi-AI Advice
Tsuchiya, Y., & Baba, Y. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791648 · Cited by 4
Ruijs, N., Pluyter, J. R., Chen, L.-L., & Funk, M. (2026) · ACM Transactions on Computing for Healthcare · doi:10.1145/3803858 · Cited by 1
Uncertainties and Emerging Uses of Human-Ai Medical Diagnosis in Collaborative Clinical Practice
Correia, A., Fonseca, B., Schneider, D. M., Chaves, R., & Kärkkäinen, T. (2025) · International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT) · doi:10.1109/ismsit67332.2025.11267990 · Cited by 1
“AI-enabled practices across collaborative clinical settings were also studied by Lee and Chew [10], who demonstrated the role of counterfactual explanations in reducing the risk of overtrust in decision-making.”
iRULER: Intelligible Rubric-Based User-Defined LLM Evaluation for Revision
Bai, J., Cheong, W. S., Müller, P., & Lim, B. Y. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790539 · Cited by 10
Visual-action AI agents for medical diagnosis and treatment: advances and future outlook
Yao, Y., Luxue, Y., Chen, X., Shen, C., Yang, H., Lee, S., Chen, T., Ge, H., & Shen, J. (2025) · The Visual Computer · doi:10.1007/s00371-025-04153-8 · Cited by 2
“Lee and Chew [68] also showed in their study that counterfactual explanations can greatly improve clinicians’ trust calibration in AI suggestions.”
Yoon, H., Jeon, G., Nam, C. S., & Lee, S. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2630099 · Cited by 1
Reinhard, P., Li, M. M., Peters, C., & Leimeister, J. M. (2026) · Information Systems Frontiers · doi:10.1007/s10796-026-10716-4 · Cited by 11
Cabitza, F., Gallazzi, E., & Papale, A. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3816965
Data-Informed, Not Data-Driven: Leadership Practices for Responsible AI Integration in K-12 Schools
Ragab, K. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-4908-4.ch011
Core Techniques for Explaining LLMs in Healthcare
Farhadi, A., & Siasifar, R. (2026) · Springer eBooks · doi:10.1007/978-3-032-15088-2_4 · Cited by 1
Hou, Y., Yang, J., Wong, R. Y., & Howell, N. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3816945
Wang, Y., Qin, M., Tao, D., & Liu, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-30047-8_8
Counterfactual AI in Healthcare: Enhancing Decision-Making and Outcome Prediction
Sharma, H., Pradhan, R., Mogha, H., Singh, S., & Kumar, R. (2025) · International Conference on Computational Intelligence and Communication Technologies (CCICT) · doi:10.1109/ccict65753.2025.00097
“Judea Pearl’s seminal work in 2000 introduced a formal framework for understanding causality through Structural Causal Models (SCMs) [9], [10].”
Fernández, R. C., & Solórzano, M. (2026) · Applied Artificial Intelligence · doi:10.1080/08839514.2026.2726917
Quilter, E. E. V., Nyangacha, R. M., Kinyeru, E., Fuster‐Casanovas, A., Brigden, A., O'Hara, K. P., Atieno, D., Ruiz-Postigo, J. A., & Carrión, C. (2025) · doi:10.2196/preprints.81829
Enhancing Interview Preparation for Autistic Job Seekers through Human-AI Collaboration
Garrison, E., MacNeil, S., Hong, S.-S., Ara, Z., Hantula, D. A., & Vučetić, S. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3816928 · Cited by 1
EvidenceTrail: Auditable Evidence Graphs for Process-Excursion Decision SupportPreprint
hu, Q. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7292207
Promise and Caution: Mapping Opportunities for AI Decision Support in Emergency Medical Services
Zhang, Z., Agbugba, V. F., Bai, E., Billings, S., Luo, X., & Adelgais, K. (2026) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi69079.2026.00086
“Clinicians may hesitate to act on recommendations produced by “black-box” models with unclear rationale [5], [22], and concerns about algorithmic bias, uncertain reliability, and limited explainability can further erode confidence and appropriate reliance [8], [23], [24], [25].”
Warren, G., Sun, J., Shklovski, I., & Augenstein, I. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812358 · Cited by 4
Ziegfeld, L., van Drie, R. A. N., Adhikari, A., & van der Waa, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-35387-0_34
Parwez, K. (2026) · International Conference on Intelligent Systems and Digital Applications (ISDA) · doi:10.1109/isda70544.2026.11606120
“Although the explainability elements increase transparency, healthcare providers might excessively depend on AI-generated predictions, especially when the justifications seem medically reasonable [32].”
Dhanorkar, S., Passi, S., & Vorvoreanu, M. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812402 · Cited by 32
Na, Y., Chen, Y., Liu, Y., & Lai, K. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-0880-8_33
Faster but Not Smarter? Temporal Constraints and User Compliance with AI Explanations
Shahu, A., Greinecker, N., & Wintersberger, P. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3798697 · Cited by 1
How Information Detail and Confidence in AI Responses Shape User Beliefs
Wu, Z., Jobanputra, M., Demberg, V., Hullman, J., & Feit, A. M. (2026) · International Conference on Human-AI Interaction and Experience Design (HAXD) · doi:10.1109/haxd70072.2026.11620844
Quilter, E. E. V., Nyangacha, R. M., Kinyeru, E., Fuster‐Casanovas, A., Brigden, A., O’Hara, K., Atieno, D., Ruiz-Postigo, J. A., & Carrión, C. (2026) · JMIR mhealth and uhealth · doi:10.2196/81829
“This risk is echoed in broader literature, which highlights that AI tools may contribute to clinician deskilling, diminished critical reasoning, and reduced capacity to navigate complex or atypical cases through overreliance [48-53].”
Between Privacy and Transparency: Exploring the Ways of Communicating Credibility Assessment
Tang, H., Rohles, B., Chuai, Y., Lenzini, G., & Sergeeva, A. (2025) · European Symposium on Usable Security (EuroUSEC) · doi:10.1109/eurousec69254.2025.00018 · Cited by 1
Metadata
- Authors
- Lee, M. H., & Chew, C. J.
- Affiliations
- Singapore Management University (SG)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2023-09-28
- Volume
- 7
- Issue
- CSCW2
- Pages
- 1-22
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W4385895991
- Semantic Scholar
- afe05c9afdeb48e111712bb6fe5f8b68632145fb
- Cited by
- 78
- References
- 77
- Citations by year
- 2026: 28 · 2025: 41 · 2024: 9
- FWCI
- 7.34
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- The Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) Tier 1 grant
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- counterfactual explanations, trust in AI, overreliance on AI, human-AI collaboration, clinical decision-making, laypeople, therapists, explainable AI, AI-assisted diagnosis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.