Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making
Open accessThe DOI matches its Crossref record, with a small difference in title or year; 80 of the 80 references it cites were found in a registry or an index; 53 of the 76 sources its claims cite were checked for support.
Abstract
Artificial intelligence (AI) is increasingly being considered to assist human decision-making in high-stake domains (e.g. health). However, researchers have discussed an issue that humans can over-rely on wrong suggestions of the AI model instead of achieving human AI complementary performance. In this work, we utilized salient feature explanations along with what-if, counterfactual explanations to make humans review AI suggestions more analytically to reduce overreliance on AI and explored the effect of these explanations on trust and reliance on AI during clinical decision-making. We conducted an experiment with seven therapists and ten laypersons on the task of assessing post-stroke survivors' quality of motion, and analyzed their performance, agreement level on the task, and reliance on AI without and with two types of AI explanations. Our results showed that the AI model with both salient features and counterfactual explanations assisted therapists and laypersons to improve their performance and agreement level on the task when 'right' AI outputs are presented. While both therapists and laypersons over-relied on 'wrong' AI outputs, counterfactual explanations assisted both therapists and laypersons to reduce their over-reliance on 'wrong' AI outputs by 21% compared to salient feature explanations. Specifically, laypersons had higher performance degrades by 18.0 f1-score with salient feature explanations and 14.0 f1-score with counterfactual explanations than therapists with performance degrades of 8.6 and 2.8 f1-scores respectively. Our work discusses the potential of counterfactual explanations to better estimate the accuracy of an AI model and reduce over-reliance on 'wrong' AI outputs and implications for improving human-AI collaborative decision-making.
Overview
CoChat Summary
## TL;DR
This within-subject study tested whether counterfactual explanations change trust and reliance on AI in a stroke rehabilitation assessment task. Counterfactual explanations reduced overreliance on wrong AI outputs compared with salient feature explanations, but they also increased decision time and were rated less favorably on several usability measures.
## Key findings
- With right AI outputs, both explanation types improved human+AI performance and agreement, but with wrong AI outputs both harmed performance; counterfactual explanations led to less degraded performance than salient features.
- Compared with salient feature analysis, counterfactual explanations produced more “right” decisions and fewer “wrong” decisions overall by 21% (29/136), with the same pattern reported for therapists and laypersons separately.
- For wrong AI outputs, therapists’ performance degradation was 8.6 F1 with salient features vs 2.8 F1 with counterfactual explanations; laypersons’ degradation was 18.0 F1 vs 14.0 F1.
- Self-reported trust was higher for salient features than counterfactuals: 73.76/100 vs 45.20/100; participants also spent more time per task with counterfactuals (75 s overall vs 57 s with salient features).
## Methods
- Within-subject online experiment comparing two interfaces: AI with salient feature analysis and AI with salient features + counterfactual explanations.
- Participants/data: 7 therapists and 10 laypersons assessed the quality of motion of 15 post-stroke survivors using videos from a dataset of 300 videos.
- Analysis: Participants made initial judgments without AI and final judgments after seeing AI outputs/explanations; outcomes were evaluated using performance (F1-score), agreement level, counts of right/wrong decisions including overreliance, decision time, and usability questionnaires, with Wilcoxon tests reported.
## Limitations
- The paper states that it did not consider explanation fidelity in the experimental design, which it says should be studied further.
- The paper states a generalizability limitation because of the small sample size, the single clinical task, and the focus on particular explanation visualizations and a feed-forward neural network.
- Evident from the text: the study used a fixed set of 3 right and 5 wrong AI outputs per condition, so the findings are tied to that specific task setup and output mix.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures









Verification
Claims the paper makes
28 Valid · 3 Not valid · 5 Not checked · 76 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, the previous research discusses the issues of these feature-based explanations on overtrust in AI [29, 37, 64].”
What the sources say
- Not checked — no full text of the sourcecites [29] [29] Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, and Jennifer Wortman Vaughan. 2020. Interpreting interpretability: understanding data scientists' use of interpretability tools for machine learning. In Proceedings of the 2020 CHI conference on human factors in computing systems . 1-14. - source
- Partly supportedcites [37] [37] Vivian Lai, Chacha Chen, Q Vera Liao, Alison Smith-Renner, and Chenhao Tan. 2021. Towards a science of human-ai decision making: a survey of empirical studies. arXiv preprint arXiv:2112.11471 (2021). - sourceExperimental — accuracy not yet measured
Section 4.2 (p. 15) discusses feature-importance explanations, and Section 5.2 (p. 25) discusses over-reliance on AI as a trust-related measure. The passages do not establish that feature-based explanations cause or contribute to overtrust; that connection is unsupported by the passages found. (LLM, conf=0.78)
- Not checked — no full text of the sourcecites [64] [64] Xinru Wang and Ming Yin. 2021. Are explanations helpful? a comparative study of the effects of explanations in ai-assisted decision-making. In 26th International Conference on Intelligent User Interfaces . 318-328. - source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Other works that utilize AI/ML models focused on tasks that do not require domain experts, such as judges or clinicians [5, 38, 64].”
What the sources say
- Supportedcites [5] [5] Gagan Bansal, Besmira Nushi, Ece Kamar, Walter S Lasecki, Daniel S Weld, and Eric Horvitz. 2019. Beyond accuracy: The role of mental models in human-AI team performance. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , Vol. 7. 2-11. - source
Section 4 (Experiments Setup) describes a game-like assembly-line task in which participants decide whether objects are defective using AI recommendations. It explicitly says the task abstracts away human expertise in problem solving, supporting the characterization of this study as using a task that does not require domain experts. The judges and clinicians mentioned in the Background are examples of real-world applications, not participants in these experiments. (LLM, conf=0.91)
- Supportedcites [38] [38] Vivian Lai and Chenhao Tan. 2019. On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of the conference on fairness, accountability, and transparency . 29-38. - source
The Experimental Setup (p. 3) says the study recruited Amazon Mechanical Turk participants to judge whether hotel reviews were genuine or deceptive. The Concluding Discussion (p. 9) treats bail decisions and medical diagnosis as tasks for future study, rather than tasks tested here. (LLM, conf=0.92)
- Not checked — no full text of the sourcecites [64] [64] Xinru Wang and Ming Yin. 2021. Are explanations helpful? a comparative study of the effects of explanations in ai-assisted decision-making. In 26th International Conference on Intelligent User Interfaces . 318-328. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, clinicians can be biased in their decision making and expert disagreement is prevalent in medical decision-making tasks [7, 33, 34].”
What the sources say
- Partly supportedcites [7] [7] Michael L Barnett, Dhruv Boddupalli, Shantanu Nundy, and David W Bates. 2019. Comparative accuracy of diagnosis by collective intelligence of multiple physicians vs individual physicians. JAMA network open 2, 3 (2019). - sourceExperimental — accuracy not yet measured
The Discussion reports that final diagnoses differed from referral diagnoses in 21% of subspecialist referrals, supporting disagreement in a specific diagnostic setting. It does not establish that expert disagreement is prevalent across medical decision-making tasks generally; the paper also notes that most referral diagnoses were unchanged or only refined. (LLM, conf=0.82)
- Partly supportedcites [33] [33] Peter Knapp and Jenny Hewison. 1999. Disagreement in patient and carer assessment of functional abilities after stroke. Stroke 30, 5 (1999), 934-938. - sourceExperimental — accuracy not yet measured
The introduction (p. 1) reports that doctors and nurses have given different ratings of patients’ functional abilities, supporting an example of clinical disagreement. The study itself examines stroke patients and informal carers, not experts, and neither passage establishes that expert disagreement is prevalent across medical decision-making tasks. (LLM, conf=0.82)
- Partly supportedcites [34] [34] Jonathan Krause, Varun Gulshan, Ehsan Rahimy, Peter Karth, Kasumi Widner, Greg S Corrado, Lily Peng, and Dale R Webster. 2018. Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Ophthalmology 125, 8 (2018), 1264-1272. - sourceExperimental — accuracy not yet measured
The Discussion (p. 7) describes frequent disagreement among retinal specialists grading diabetic retinopathy images, including subjective differences at grade boundaries. That supports expert disagreement in this medical task, but the paper does not establish that disagreement is prevalent across medical decision-making tasks generally. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Similar to the previous research [11], our study also shows a positive correlation between user's trust and reliance: the more a user trusts the system, the more the user is likely to over-rely on outputs of the system even when 'wrong' AI outputs are presented.”
What the sources say
- Supportedcites [11] [11] Adrian Bussone, Simone Stumpf, and Dympna O'Sullivan. 2015. The role of explanations on trust and reliance in clinical decision support systems. In 2015 international conference on healthcare informatics . IEEE, 160-169. - source
The Discussion (p. 11) reports a strong relationship between trust and reliance: users who highly trust the CDSS are likely to over-rely on its suggestions. The System Reliability and Use section (p. 4) defines over-reliance as agreeing with incorrect suggestions. The citing authors' statement about their own study is not attributed to this source. (LLM, conf=0.92)
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, clinicians can be biased in their decision making and expert disagreement is prevalent in medical decision-making tasks [7, 33, 34].”
What the sources say
- Could not verifycites [7] [7] Michael L Barnett, Dhruv Boddupalli, Shantanu Nundy, and David W Bates. 2019. Comparative accuracy of diagnosis by collective intelligence of multiple physicians vs individual physicians. JAMA network open 2, 3 (2019). - sourceExperimental — accuracy not yet measured
The Introduction discusses misdiagnosis, but does not attribute it to bias in clinicians’ decision-making. The Discussion > Limitations mentions possible bias in the study’s assessment of diagnostic solutions, not bias in clinicians’ decisions. Searches for cognitive bias and biased physician judgments found no passage addressing the attributed claim. (LLM, conf=0.75)
- Partly supportedcites [33] [33] Peter Knapp and Jenny Hewison. 1999. Disagreement in patient and carer assessment of functional abilities after stroke. Stroke 30, 5 (1999), 934-938. - sourceExperimental — accuracy not yet measured
The introduction (p. 1) discusses possible bias in assessments used clinically and reports that doctors and nurses gave different ratings of patients’ abilities. The study itself examines disagreement between stroke patients and carers, not bias in clinicians’ decision-making, so it supports only the narrower point about variability in assessments. (LLM, conf=0.78)
- Partly supportedcites [34] [34] Jonathan Krause, Varun Gulshan, Ehsan Rahimy, Peter Karth, Kasumi Widner, Greg S Corrado, Lily Peng, and Dale R Webster. 2018. Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Ophthalmology 125, 8 (2018), 1264-1272. - sourceExperimental — accuracy not yet measured
The Discussion (p. 7) reports subjective differences among retina specialists in how they applied diabetic-retinopathy grading boundaries, and Grading and Adjudication (p. 5) documents ophthalmologists’ grading errors. These passages support the narrower point that clinical judgments can vary, but they do not establish bias in clinicians’ decision-making generally. (LLM, conf=0.78)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
47 more claims · 4 with their cited sources checked
Create a free account to see all 112 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 35
- It may state two separate claims as one. 30
- The quote only partly states this claim. 2
Citations
Cites
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
Gauriat, C.-M., Pencolé, Y., Ribot, P., & Brouillet, G. (2024) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.4230/oasics.dx.2024.27 · Cited by 12,586
Parkinson's disease: clinical features and diagnosis
Jankovic, J. (2008) · Journal of Neurology, Neurosurgery & Psychiatry · doi:10.1136/jnnp.2007.131045 · Cited by 5,720
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
The Mythos of Model Interpretability
Lipton, Z. C. (2016) · arXiv · doi:10.48550/arxiv.1606.03490 · Cited by 5,422
Humans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554 · cited 1×
“In addition to the contradictory perspectives on the effect of explanations on trust, our research community still requires additional studies to understand the effect of AI explanations on overreliance [54].”
BackgroundRajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-021-01614-0 · Cited by 3,212
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 1×
“For the presentation of these salient features, we utilized a radar chart to effectively show the comparison of identified features on post-stroke survivors’ unaffected and affected sides to follow the therapist’s practices [3, 41].”
MethodFeigin, V. L., Norrving, B., & Mensah, G. A. (2017) · Circulation Research · doi:10.1161/circresaha.116.308413 · Cited by 2,184
Ardila, D., Kiraly, A. P., Bharadwaj, S., Choi, B., Reicher, J. J., Peng, L., Tse, D., Etemadi, M., Ye, W., Corrado, G., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0447-x · Cited by 2,167 · cited 1×
“…(AI) and machine learning (ML) models have achieved equivalent results or outperformed humans at decision-making tasks (e.g. screening lung cancer [4]), these AI and ML models are increasingly being considered to increase efficiency and reduce the cost of performing decision-making tasks from…”
BackgroundCzaja, S. J., Charness, N., Fisk, A. D., Hertzog, C., Nair, S. N., Rogers, W. A., & Sharit, J. (2006) · Psychology and Aging · doi:10.1037/0882-7974.21.2.333 · Cited by 2,113 · cited 1×
“To understand the participant’s background in technology, we asked them to respond to a set of technical experience questions, which were based on survey questions designed by the Center for Research and Education on Aging and Technology Enhancement (CREATE) [17].”
MethodExplaining machine learning classifiers through diverse counterfactual explanations
Mothilal, R. K., Sharma, A., & Tan, C. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372850 · Cited by 1,233 · cited 3×
“Previous research [10] discusses the potential of cognitive forcing functions (e.g. not showing AI suggestions by default or waiting before showing AI suggestions) to increase analytical thinking and reduce overreliance.”
BackgroundMethodBuçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
Proceedings of the ACM on Human-Computer Interaction
Karahalios, K. G., Monroy‐Hernández, A., Lampinen, A., & Fitzpatrick, G. (2018) · Cited by 1,074
Unintended consequences of machine learning in medicine
Cabitza, F., Rasoini, R., & Gensini, G. F. (2017) · JAMA · doi:10.1001/jama.2017.7797 · Cited by 1,066
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018) · The Quarterly Journal of Economics · doi:10.1093/qje/qjx032 · Cited by 1,040
Designing Theory-Driven User-Centric Explainable AI
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300831 · Cited by 906
Letham, B., Rudin, C., McCormick, T. H., & Madigan, D. (2015) · The Annals of Applied Statistics · doi:10.1214/15-aoas848 · Cited by 766 · cited 2×
“Our work advances ongoing discussions around the implications for improving human-AI collaborative decision-making in high-stake domains (e.g. health) [15, 42, 44].”
BackgroundDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757 · cited 1×
“In this section, we discussed the potential benefits and limitations of two types of AI explanations (i.e. salient feature and counterfactual), their effects on domain experts and laypersons’ decision-making, and suggestions for more effect human-AI collaborative, clinical decision-making.”
BackgroundDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728 · cited 1×
“…been a growing number of studies that have evaluated the effect of AI explanations in diverse decision-making tasks (e.g. house price prediction [55], image classification [2], student admission [16], deception detection [38], stroke rehabilitation assessment [42]) and aspects, such as whether…”
BackgroundManipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1802.07810 · Cited by 711
Beede, E., Baylor, E., Hersch, F., Iurchenko, A., Wilcox, L., Ruamviboonsuk, P., & Vardoulakis, L. M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376718 · Cited by 614 · cited 3×
“For instance, a deep learning-based system has been used in clinics to bring new data-driven insights to assist the diagnosis of cancer [15], the detection of diabetic eye disease [8], or the assessment of physical stroke rehabilitation assessment [42].”
BackgroundMethodCai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594
Principles of Explanatory Debugging to Personalize Interactive Machine Learning
Kulesza, T., Burnett, M., Wong, W.-K., & Stumpf, S. (2015) · Proceedings of the 20th International Conference on Intelligent User Interfaces · doi:10.1145/2678025.2701399 · Cited by 576
Gradio: Hassle-Free Sharing and Testing of ML Models in the WildPreprint
Abid, A., Abdalla, A., Ali, A., Khan, D., Alfozan, A., & Zou, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.02569 · Cited by 572
Krause, J., Gulshan, V., Rahimy, E., Karth, P., Widner, K., Corrado, G. S., Peng, L., & Webster, D. R. (2018) · Ophthalmology · doi:10.1016/j.ophtha.2018.01.034 · Cited by 564
Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376219 · Cited by 564
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2017) · doi:10.3386/w23180 · Cited by 545
Treatment Techniques and Clinical Guidelines for Photocoagulation of Diabetic Macular Edema
(1987) · Ophthalmology · doi:10.1016/s0161-6420(87)33527-4 · Cited by 544
Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance
Bansal, G., Nushi, B., Kamar, E., Lasecki, W. S., Weld, D. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5285 · Cited by 466 · cited 9×
“To this end, we conduct a within-subject experiment with seven therapists and ten laypersons to compare the effect of counterfactual explanations with one of the widely used AI explanations, salient feature explanations [5, 16, 38, 42].”
BackgroundMethodInfluentialSullivan, K. J., Tilson, J. K., Cen, S. Y., Rose, D. K., Hershberg, J., Correa, A., Gallichio, J., McLeod, M., Moore, C., Wu, S. S., et al. (2011) · Stroke · doi:10.1161/strokeaha.110.592766 · Cited by 445
The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems
Bussone, A., Stumpf, S., & O'Sullivan, D. (2015) · 2015 International Conference on Healthcare Informatics · doi:10.1109/ichi.2015.26 · Cited by 441
Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making
Cai, C. J., Reif, E., Hegde, N., Hipp, J., Kim, B., Smilkov, D., Wattenberg, M., Viegas, F., Corrado, G. S., Stumpe, M. C., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300234 · Cited by 405
A review of instance selection methods
Olvera-López, J. A., Carrasco-Ochoa, J. A., Martínez-Trinidad, J. F., & Kittler, J. (2010) · Artificial Intelligence Review · doi:10.1007/s10462-010-9165-y · Cited by 394
The Principles and Limits of Algorithm-in-the-Loop Decision Making
Green, B., & Chen, Y. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359152 · Cited by 392 · cited 3×
“In addition, an additional study is required to investigate how people can effectively evaluate the performance and trustworthiness of an ML model and calibrate their trust and reliance to improve human-AI/algorithm interaction [22].”
BackgroundLai, V., & Tan, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287590 · Cited by 377 · cited 9×
“In contrast to prior research that describes the improved performance of humans with AI explanations [38, 41], our results demonstrated that the human + AI team with both salient feature and counterfactual explanations performed worse than the human alone (Figure 4).”
BackgroundMethodResultInfluentialFactual and Counterfactual Explanations for Black Box Decision Making
Guidotti, R., Monreale, A., Giannotti, F., Pedreschi, D., Ruggieri, S., & Turini, F. (2019) · IEEE Intelligent Systems · doi:10.1109/mis.2019.2957223 · Cited by 349 · cited 2×
“The counterfactual explanations describe what changes in feature values lead to updating an AI output in a certain way [12, 24, 47].”
BackgroundMethodReasons For Physicians Not Adopting Clinical Decision Support Systems: Critical Analysis
Khairat, S., Marc, D., Crosby, W., & Al Sanousi, A. (2018) · JMIR Medical Informatics · doi:10.2196/medinform.8912 · Cited by 335
Counterfactual Explanations for Machine Learning: A Review.Preprint
Verma, S., Dickerson, J. P., & Hines, K. (2020) · arXiv (Cornell University) · Cited by 332
Wang, X., & Yin, M. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450650 · Cited by 320 · cited 9×
“Building upon growing works on the usage of explainable AI methods for improving AI-assisted decision-making [10, 11, 64], we hypothesize that counterfactual explanations [47, 62], a type of AI explanations that describe how the inputs can be modified to achieve an AI output in a certain way, will…”
BackgroundInfluentialExplaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders
Cheng, H.-F., Wang, R., Zhang, Z., O'Connell, F., Gray, T., Harper, F. M., & Zhu, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300789 · Cited by 292 · cited 5×
“To this end, we conduct a within-subject experiment with seven therapists and ten laypersons to compare the effect of counterfactual explanations with one of the widely used AI explanations, salient feature explanations [5, 16, 38, 42].”
BackgroundMethodInfluentialCounterfactuals in Explainable Artificial Intelligence (XAI): Evidence from Human Reasoning
Byrne, R. M. J. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/876 · Cited by 285 · cited 2×
“The counterfactual explanations describe what changes in feature values lead to updating an AI output in a certain way [12, 24, 47].”
BackgroundCounterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review
Verma, S., Boonsanong, V., Hoang, M., Hines, K. E., Dickerson, J. P., & Shah, C. (2020) · arXiv · doi:10.48550/arxiv.2010.10596 · Cited by 267 · cited 3×
“In this work, we assume that reviewing counterfactual explanations [47, 62, 64] will allow a user to critically think of how to change an AI output and more rigorously review an AI output than widely used AI explanations (e.”
BackgroundMethodYang, Q., Steinfeld, A., & Zimmerman, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300468 · Cited by 247
Mueller, S. T., Hoffman, R. R., Clancey, W., Emrey, A., & Klein, G. (2019) · arXiv · doi:10.48550/arxiv.1902.01876 · Cited by 215
Panigutti, C., Beretta, A., Giannotti, F., & Pedreschi, D. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502104 · Cited by 215
Brown, A., Chouldechova, A., Putnam-Hornstein, E., Tobin, A., & Vaithianathan, R. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300271 · Cited by 207
"The human body is a black box": supporting clinical decision-making with deep learning
Sendak, M., Elish, M. C., Gao, M., Futoma, J., Ratliff, W., Nichols, M., Bedoya, A., Balu, S., & O'Brien, C. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372827 · Cited by 196
Naiseh, M., Al-Thani, D., Jiang, N., & Ali, R. (2023) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102941 · Cited by 170 · cited 2×
“…uncontrolled AI model outputs and explanations implemented by the dataset of 15 post-stroke survivors in contrast to existing previous research that utilizes simulated and controlled AI outputs and explanations [10, 11, 49] to understand the effect of AI explanations and the issue of overreliance.”
BackgroundResultEvaluating saliency map explanations for convolutional neural networks: a user study
Alqaraawi, A., Schuessler, M., Weiß, P., Costanza, E., & Berthouze, N. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377519 · Cited by 163
Let Me Explain: Impact of Personal and Impersonal Explanations on Trust in Recommender Systems
Kunkel, J., Donkers, T., Michael, L., Barbu, C.-M., & Ziegler, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300717 · Cited by 156
Barnett, M. L., Boddupalli, D., Nundy, S., & Bates, D. W. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.0096 · Cited by 154 · cited 1×
“However, clinicians can be biased in their decision making and expert disagreement is prevalent in medical decision-making tasks [7, 33, 34].”
BackgroundArtificial Intelligence and Life in 2030: The One Hundred Year Study on Artificial Intelligence
Stone, P., Brooks, R. A., Brynjolfsson, E., Calo, R., Etzioni, O., Hager, G., Hirschberg, J., Kalyanakrishnan, S., Kamar, E. S., Kraus, S., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.06318 · Cited by 153
Kawakami, A., Sivaraman, V., Cheng, H.-F., Stapleton, L., Cheng, Y., Qing, D., Perer, A., Wu, Z. S., Zhu, H., & Holstein, K. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517439 · Cited by 143 · cited 1×
“…fully autonomous AI systems, researchers have explored the feasibility of human-AI collaborative decision-making, in which an AI model provides humans new data-driven insights on a task for achieving complementary performance, outperforming neither of the AI or the human alone [5, 30, 42, 43].”
BackgroundA Human-AI Collaborative Approach for Clinical Decision Making on Rehabilitation Assessment
Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S. B. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445472 · Cited by 137
Self-citationNourani, M., King, J., & Ragan, E. (2020) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v8i1.7469 · Cited by 116
What can AI do for me?: evaluating machine learning interpretations in cooperative play
Feng, S., & Boyd-Graber, J. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302265 · Cited by 92
ViCE: visual counterfactual explanations for machine learning models
Gomez, O., Holter, S., Yuan, J., & Bertini, E. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377536 · Cited by 91
Yang, Q., Steinfeld, A. M., & Zimmerman, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.09612 · Cited by 80
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies
Lai, V., Chen, C., Liao, Q. V., Smith-Renner, A., & Tan, C. (2021) · arXiv · doi:10.48550/arxiv.2112.11471 · Cited by 75 · cited 7×
“However, the previous research describes the issues of these explanations on overtrust in AI [29, 37, 64].”
BackgroundMethodInfluentialLearning to assess the quality of stroke rehabilitation exercises
Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Badia, S. B. I. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302273 · Cited by 71 · cited 6×
“We utilized the dataset of a ‘Bring a cup to the mouth’ upper-limb exercise from 15 post-stroke survivors [40].”
BackgroundMethodInfluentialSelf-citationArdila, D., Kiraly, A. P., Bharadwaj, S., Choi, B., Reicher, J. J., Peng, L. H., Tse, D., Etemadi, M., Ye, W., Corrado, G. S., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0536-x · Cited by 69
Disagreement in Patient and Carer Assessment of Functional Abilities After Stroke
Knapp, P., & Hewison, J. (1999) · Stroke · doi:10.1161/01.str.30.5.934 · Cited by 59
How model accuracy and explanation fidelity influence user trust
Papenmeier, A., Englebienne, G., & Seifert, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.12652 · Cited by 58
Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415227 · Cited by 57
Self-citationEvaluating Saliency Map Explanations for Convolutional Neural Networks: A User StudyPreprint
Alqaraawi, A., Schuessler, M., Weiß, P., Costanza, E., & Bianchi‐Berthouze, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.00772 · Cited by 41
What can AI do for me: Evaluating Machine Learning Interpretations in Cooperative PlayPreprint
Feng, S., & Boyd-Graber, J. L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.09648 · Cited by 40
"The Human Body is a Black Box": Supporting Clinical Decision-Making with Deep LearningPreprint
Sendak, M., Elish, M. C., Gao, M. C., Futoma, J., Ratliff, W., Nichols, M., Bedoya, A., Balu, S., & O’Brien, C. L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.08089 · Cited by 35
Improving Visualization Interpretation Using Counterfactuals
Kaul, S., Borland, D., Cao, N., & Gotz, D. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114779 · Cited by 33
ViCE: Visual Counterfactual Explanations for Machine Learning ModelsPreprint
Gómez, Ó., Holter, S., Yuan, J., & Bertini, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.02428 · Cited by 20
Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511112 · Cited by 19 · cited 2×
“Specifically, researchers have discussed the potential of human and AI/ML teaming to achieve better results than either humans or AI/ML models alone [5, 42, 43].”
BackgroundSelf-citationAlgorithm-in-the-Loop Decision Making
Green, B., & Chen, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i09.7115 · Cited by 15
Lee, M. H., & Choy, Y. J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.05525 · Cited by 8
Self-citationHow Different Explanations Impact Trust Calibration: The Case of Clinical Decision Support SystemsPreprint
Naiseh, M., Al‐Thani, D., Jiang, N., & Ali, R. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4098528 · Cited by 2
(2017) · doi:10.9728/dcs.2024.25.7.1847 · Cited by 2
Cited by
Large language models in healthcare and medical domain: A review
Nazi, Z. A., & Peng, W. (2024) · Informatics · doi:10.3390/informatics11030057 · Cited by 415
The Typing Cure: Experiences with Large Language Model Chatbots for Mental Health Support
Song, I., Pendse, S. R., Kumar, N., & De Choudhury, M. (2024) · arXiv · doi:10.48550/arxiv.2401.14362 · Cited by 64
The Typing Cure: Experiences with Large Language Model Chatbots for Mental Health Support
Song, I., Pendse, S. R., Kumar, N., & Choudhury, M. D. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757430 · Cited by 196
From human-centered to symbiotic artificial intelligence: a focus on medical applications
Desolda, G., Esposito, A., Lanzilotti, R., Piccinno, A., & Costabile, M. F. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-20414-5 · Cited by 69
Emotionally adaptive support: a narrative review of affective computing for mental health
Schlicher, M., Li, Y., Murthy, S. M. K., Sun, Q., & Schuller, B. W. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1657031 · Cited by 61
“Existing studies have explored various techniques such as visualizing attention in medical imaging in order to understand the central parts of the AI’s decision (147), as well as salient features and counterfactual explanations to prevent clinicians from over-relying on incorrect AI outputs (148).”
Metadata
- Authors
- Lee, M. H., & Chew, C. J.
- Affiliations
- Singapore Management University (SG)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2023-09-28
- Volume
- 7
- Issue
- CSCW2
- Pages
- 1-22
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W4385895991
- Semantic Scholar
- afe05c9afdeb48e111712bb6fe5f8b68632145fb
- Cited by
- 78
- References
- 77
- Citations by year
- 2026: 28 · 2025: 41 · 2024: 9
- FWCI
- 7.34
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- The Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) Tier 1 grant
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- counterfactual explanations, trust in AI, overreliance on AI, human-AI collaboration, clinical decision-making, laypeople, therapists, explainable AI, AI-assisted diagnosis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.