Are Explanations Helpful? A Comparative Study of the Effects of Explanations in AI-Assisted Decision-Making
Open accessThe DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.
Abstract
This paper contributes to the growing literature in empirical evaluation of explainable AI (XAI) methods by presenting a comparison on the effects of a set of established XAI methods in AI-assisted decision making. Specifically, based on our review of previous literature, we highlight three desirable properties that ideal AI explanations should satisfy—improve people’s understanding of the AI model, help people recognize the model uncertainty, and support people’s calibrated trust in the model. Through randomized controlled experiments, we evaluate whether four types of common model-agnostic explainable AI methods satisfy these properties on two types of decision making tasks where people perceive themselves as having different levels of domain expertise in (i.e., recidivism prediction and forest cover prediction). Our results show that the effects of AI explanations are largely different on decision making tasks where people have varying levels of domain expertise in, and many AI explanations do not satisfy any of the desirable properties for tasks that people have little domain expertise in. Further, for decision making tasks that people are more knowledgeable, feature contribution explanation is shown to satisfy more desiderata of AI explanations, while the explanation that is considered to resemble how human explain decisions (i.e., counterfactual explanation) does not seem to improve calibrated trust. We conclude by discussing the implications of our study for improving the design of XAI methods to better support human decision making.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, the previous research discusses the issues of these feature-based explanations on overtrust in AI [29, 37, 64].”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [64]
In the citing paper
“Overall, our findings indicate user's overreliance on 'wrong' AI outputs, which follows the previous studies that describe salient feature explanations increases user's overreliance on the AI model [6, 64].”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [64]
In the citing paper
“Other works that utilize AI/ML models focused on tasks that do not require domain experts, such as judges or clinicians [5, 38, 64].”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [64]
In the citing paper
“Wang and Yin [64] conducted the randomized experiment using four types of common model-agnostic explainable AI methods and discussed the effect of AI explanations could be largely different where people have varying levels of domain expertise.”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [64]
In the citing paper
“However, across many domains, empirical studies of explanations have produced mixed results [4,11,49,53,58,68,82,87], with some suggesting that explanations increase decisionmakers' tendency to rely on AI predictions even when the AI system is wrong [4,68,82,87]-a phenomenon often referred to as overreliance [10,78].”
From Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations (2023) · cites this paper as [82]
16 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
UCI Machine Learning Repository
Asuncion, A. (2007) · Medical Entomology and Zoology · Cited by 25,370
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 4×
“For example, sophisticated techniques such as LIME [60] and SHAP [48] have been used to illustrate how much each feature contributes to a model’s final prediction.”
MethodInfluential"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · cited 5×
“For example, sophisticated techniques such as LIME [60] and SHAP [48] have been used to illustrate how much each feature contributes to a model’s nal prediction.”
MethodInfluential(2011) · Cited by 14,317
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“the training dataset or counterfactual examples to explain model predictions [36, 39, 65], and conducting model distillation [10, 32].”
BackgroundGauriat, C.-M., Pencolé, Y., Ribot, P., & Brouillet, G. (2024) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.4230/oasics.dx.2024.27 · Cited by 12,586
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 1×
“…interpretable models such as generalized additive models and decision sets [15, 34, 43, 66], as well as visualizing what deep neural network has learned in its intermediate layers and how its predictions are affected by different part of the inputs (e.g., through saliency map) [37, 61, 68].”
BackgroundPeeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · cited 1×
“havior of the entire AI model is categorized as global explanations, while methods that provide reasons for specic model predictions are categorized as local explanations [2, 24, 26].”
MethodLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 1×
“Typical methods include ask people to rank the input data features based on their inuence to overall predictions [18, 33], to indicate the direction of change in the model’s prediction when a feature’s value is altered [16, 18, 29], to simulate the model’s predictions [16, 24, 40, 47, 59], to answer “what-if” question…”
MethodDeep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 1×
“…interpretable models such as generalized additive models and decision sets [15, 34, 43, 66], as well as visualizing what deep neural network has learned in its intermediate layers and how its predictions are affected by different part of the inputs (e.g., through saliency map) [37, 61, 68].”
Background
Research works printed but not linked: 6
- Dennis Collaris Leo M Vink and Jarke J van Wijk. 2018. Instance-level explanations for fraud detection: A case study. arXiv preprint arXiv:1806.07129(2018). Dennis Collaris Leo M Vink and Jarke J van Wijk. 2018. Instance-level explanations for fraud detection: A case study. arXiv preprint arXiv:1806.07129(2018). · arXiv
- 10.2139/ssrn.3465622 · DOI · Paper page
- Isaac Lage Emily Chen Jeffrey He Menaka Narayanan Been Kim Sam Gershman and Finale Doshi-Velez. 2019. An evaluation of the human-interpretability of explanation. arXiv preprint arXiv:1902.00006(2019). Isaac Lage Emily Chen Jeffrey He Menaka Narayanan Been Kim Sam Gershman and Finale Doshi-Velez. 2019. An evaluation of the human-interpretability of explanation. arXiv preprint arXiv:1902.00006(2019). · arXiv
- Jeff Larson , Surya Mattu , Lauren Kirchner , and Julia Angwin . 2016. How we analyzed the COMPAS recidivism algorithm. ProPublica (5 2016) 9, 1 ( 2016 ). Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin. 2016. How we analyzed the COMPAS recidivism algorithm. ProPublica (5 2016) 9, 1 (2016).
- Maranda McBride and Shona Morgan . 2010. Trust calibration for automated decision aids . Institute for Homeland Security Solutions.[Online] . Available: https://www. ihssnc. org/portals/0/Documents/VIMSDocuments/McBride_Research_Brief. pdf(2010). Maranda McBride and Shona Morgan. 2010. Trust calibration for automated decision aids. Institute for Homeland Security Solutions.[Online]. Available: https://www. ihssnc. org/portals/0/Documents/VIMSDocuments/McBride_Research_Brief. pdf(2010).
- Jennifer Wortman Vaughan and Hanna Wallach . 2020. A Human-Centered Agenda for Intelligible Machine Learning. Machines We Trust: Getting Along with Artificial Intelligence ( 2020 ). Jennifer Wortman Vaughan and Hanna Wallach. 2020. A Human-Centered Agenda for Intelligible Machine Learning. Machines We Trust: Getting Along with Artificial Intelligence (2020).
Cited by
Explanations Can Reduce Overreliance on AI Systems During Decision-Making
Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M. S., & Krishna, R. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579605 · Cited by 402
“Future work should explore our theory on high-stakes tasks such as recidivism prediction [92], where the consequences of a wrong decision are much higher.”
BackgroundChen, H., Gomez, C., Huang, C.-M., & Unberath, M. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00699-2 · Cited by 300
“…images, plots) [20], channel ( e. g. graphical interface) [21], and interactivity ( e. g. whether users can provide feedback or refine model outputs) can drastically affect users’ experience and performance [22–24], and therefore, must be aligned with the goal of transparent system development.”
BackgroundTowards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3331846 · Cited by 278
“The agreement rate of users with the model decisions is commonly used [9, 11, 12, 25] as a measure of observed trust.”
BackgroundInfluentialGomez, C., Cho, S. M., Ke, S., Huang, C.-M., & Unberath, M. (2025) · Frontiers in Computer Science · doi:10.3389/fcomp.2024.1521066 · Cited by 209
“…et al., 2022; Schemmer et al., 2023; Mackeprang et al., 2019); image labeling (Cau et al., 2023) 8 Law Recidivism prediction (Grgic-Hlaca et al., 2019; Wang and Yin, 2021; Alufaisan et al., 2021; Liu et al., 2021); criminal referral decision (Zytek et al., 2022); penal sentence prediction (Kahr…”
Lai, V., Chen, C., Smith-Renner, A., Liao, Q. V., & Tan, C. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594087 · Cited by 204
“The most commonly used metric is forward simulation [1, 2, 15, 21, 24, 36, 48, 80, 81, 83, 87, 105, 111], by asking participants to simulate a model’s predictions on unseen instances. and what design impacts the perception is an active research area.”
BackgroundMethodInfluentialLeichtmann, B., Humer, C., Hinterreiter, A., Streit, M., & Mara, M. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107539 · Cited by 200
Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations
Schemmer, M., Kuehl, N., Benz, C., Bartos, A., & Satzger, G. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584066 · Cited by 208
“In this work, we focus on AR with respect to performance ( 𝑃 ) following Talone [69], Wang and Yin [74].”
BackgroundMethodResultInfluentialFairness and Explanation in AI-Informed Decision Making
Angerschmid, A., Zhou, J., Theuermann, K., Chen, F., & Holzinger, A. (2022) · Machine Learning and Knowledge Extraction · doi:10.3390/make4020026 · Cited by 161
“Wang et al. [54] compared different explanation types such as feature importance, feature contribution, nearest neighbour and counterfactual explanation from three perspectives of improving people’s understanding of the AI model, helping people recognize the model uncertainty, and supporting…”
BackgroundDesigning for Responsible Trust in AI Systems: A Communication Perspective
Liao, Q. V., & Sundar, S. S. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533182 · Cited by 165
“It provides an alternative explanation to the observations that adding transparency features often increase people’s trust even if the model should not be relied upon [3, 59, 66, 70]: they may have enhanced people’s intention and process based trust rather than ability based trust.”
BackgroundUnderstanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Chen, V., Liao, Q. V., Wortman Vaughan, J., & Bansal, G. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610219 · Cited by 220
The role of explainable AI in the context of the AI Act
Panigutti, C., Hamon, R., Hupont, I., Fernandez Llorca, D., Fano Yela, D., Junklewitz, H., Scalzo, S., Mazzini, G., Sanchez, I., Soler Garrido, J., et al. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594069 · Cited by 137
“While most of the studies that have performed a human evaluation of XAI methods uses laypeople recruited on crowdsourcing platforms [18, 117, 121] or domain experts on prototype-level AI applications [90], there is a lack of field experiments conducted in real-world settings, i.”
BackgroundMa, S., Lei, Y., Wang, X., Zheng, C., Shi, C., Yin, M., & Ma, X. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581058 · Cited by 143
“Third, AI’s confdence scores in these samples should be well-calibrated to refect its actual CL [100, 106].”
BackgroundInfluentialSelf-citationEnhancing AI-Assisted Group Decision Making through LLM-Powered Devil's Advocate
Chiang, C.-W., Lu, Z., Li, Z., & Yin, M. (2024) · Proceedings of the 29th International Conference on Intelligent User Interfaces · doi:10.1145/3640543.3645199 · Cited by 142
“For example, many studies specifically examine whether people exhibit any over-reliance or under-reliance on AI recommendations [11, 15, 23, 33, 37, 45, 53, 56, 72, 76, 84, 98, 99, 105, 109, 110].”
BackgroundSelf-citationSivaraman, V., Bukowski, L. A., Levin, J., Kahn, J. M., & Perer, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581075 · Cited by 141
“A mixed-methods analysis of the think-aloud transcripts and structured decision responses showed that explanations improved clinicians’ perceptions of the AI’s usefulness and made them more confdent in their own decisions, a fnding that is consistent with prior literature [1, 80, 85].”
BackgroundMethodResultInfluentialKim, S. S. Y., Liao, Q. V., Vorvoreanu, M., Ballard, S., & Vaughan, J. W. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658941 · Cited by 147
Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation
Lai, V., Carton, S., Bhatnagar, R., Liao, Q. V., Zhang, Y., & Tan, C. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501999 · Cited by 145
Panigutti, C., Beretta, A., Giannotti, F., & Pedreschi, D. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502104 · Cited by 126
“However, several works have shown that users’ domain expertise is relevant to the trust calibration process [32, 62, 82, 86], e.”
BackgroundHuman-LLM Collaborative Annotation Through Effective Verification of LLM Labels
Wang, X., Kim, H., Rahman, S., Mitra, K., & Miao, Z. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641960 · Cited by 128
Self-citationHuman-centered evaluation of explainable AI applications: a systematic review
Kim, J., Maathuis, H., & Sent, D. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1456486 · Cited by 130
Game changers: A generative AI prompt protocol to enhance human-AI knowledge co-construction
Robertson, J., Ferreira, C. C., Botha, E., & Oosthuizen, K. (2024) · Business Horizons · doi:10.1016/j.bushor.2024.04.008 · Cited by 196
Co-design of human-centered, explainable AI for clinical decision support
Panigutti, C., Beretta, A., Fadda, D., Giannotti, F., Pedreschi, D., Perotti, A., & Rinzivillo, S. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3587271 · Cited by 127
Inkpen, K., Chappidi, S., Mallari, K., Nushi, B., Ramesh, D., Michelucci, P., Mandava, V., Vepřek, L. H., & Quinn, G. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3534561 · Cited by 132
“This inding is further supported by a series of work at the intersection of human-computer interaction and machine learning interpretability, which highlights concerns around the phenomenon of increased, but not appropriate, reliance on automation [6, 9, 34, 46, 58].”
BackgroundRePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions
Wang, Y., Shen, S., & Lim, B. Y. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581402 · Cited by 120
“complexity of AI systems, and many studies have shown that XAI can increase user trust and understanding [4, 81].”
BackgroundHuman-Centered Explainable AI (XAI): From Algorithms to User Experiences
Liao, Q. V., & Varshney, K. R. (2021) · arXiv · doi:10.48550/arxiv.2110.10790 · Cited by 120
“Such tests to assess people’s understanding without a specific end goal can fail to predict the success of using an XAI technique in real tasks that people seek explanations for, such as debugging models [54] or improving decision-making [16, 100].”
BackgroundDikmen, M., & Burns, C. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102792 · Cited by 116
Liu, H., Lai, V., & Tan, C. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479552 · Cited by 111
“AI assistance improves human performance compared to without any assistance; however, the performance of human-AI teams seldom surpasses AI alone in challenging prediction tasks [3, 5, 7, 20, 21, 32, 33, 37, 43, 54, 65, 66, 73].”
BackgroundKnowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems
He, G., Kuiper, L., & Gadiraju, U. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581025 · Cited by 107
“At the same time, explanations have been shown to improve the human understanding and interpretation of AI advice [3, 52, 79], which can also potentially contribute to appropriate reliance.”
BackgroundMethodInfluentialZhang, S., Yu, J., Xu, X., Yin, C., Lu, Y., Yao, B., Tory, M., Padilla, L. M., Caterino, J., Zhang, P., et al. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642343 · Cited by 99
“…we begin our project with an open-ended semi-structured interview [50] with six domain experts as a formative study to gather information on 1) the clinicians’ daily practice of sepsis decision making, and 2) the user experience and user needs of AI-based sepsis decision support systems.”
BackgroundResultRechkemmer, A., & Yin, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501967 · Cited by 97
Self-citationHemmer, P., Westphal, M., Schemmer, M., Vetter, S., Vössing, M., & Satzger, G. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584052 · Cited by 93
“and can, therefore, either accept or override the recommendation [61, 71].”
BackgroundAI-enhanced collective intelligence
Cui, H., & Yasseri, T. (2024) · Patterns · doi:10.1016/j.patter.2024.101074 · Cited by 143
“163 A previous study also proposes a framework highlighting trust relationships within human-AI teams.”
Fairness Perceptions of Artificial Intelligence: A Review and Path Forward
Narayanan, D., Nagpal, M., McGuire, J., Schweitzer, S., & De Cremer, D. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2023.2210890 · Cited by 80
Fok, R., & Weld, D. S. (2024) · AI Magazine · doi:10.1002/aaai.12182 · Cited by 81
Effects of Explanations in AI-Assisted Decision Making: Principles and Comparisons
Wang, X., & Yin, M. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3519266 · Cited by 74
“We note that this work is an extended version of Wang and Yin [78].”
MethodSelf-citationLee, M. H., & Chew, C. J. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610218 · Cited by 78
“Building upon growing works on the usage of explainable AI methods for improving AI-assisted decision-making [10, 11, 64], we hypothesize that counterfactual explanations [47, 62], a type of AI explanations that describe how the inputs can be modified to achieve an AI output in a certain way, will…”
BackgroundInfluentialCultural Bias in Explainable AI Research: A Systematic Analysis
Peters, U., & Carman, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14888 · Cited by 113
“However, the ability to understand the explanations that XAI systems produce may differ between individuals (Wang & Yin, 2021), and it has been noted that XAI designers frequently adopt a one-PETERS More importantly, while several recent studies report that WEIRD sampling may severely limit the…”
BackgroundPerformance and Metacognition Disconnect when Reasoning in Human-AI Interaction
Fernandes, D., Villa, S., Nicholls, S., Haavisto, O., Buschek, D., Schmidt, A., Kosch, T., Shen, C., & Welsch, R. (2024) · arXiv · doi:10.48550/arxiv.2409.16708 · Cited by 71
“…interaction [57]. e t Furthermore, explanations from AI systems are often uninformative, ignored, or lead to cognitive biases themselves [9, 30, 88, 93]. d r b For instance, Zamfirescu-Pereira et al. [97] found that users have difficulty crafting effective prompts, while Dang et al. [23] noted…”
Better Together? An Evaluation of AI-Supported Code Translation
Weisz, J. D., Muller, M., Ross, S. I., Martinez, F., Houde, S., Agarwal, M., Talamadupula, K., & Richards, J. T. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511157 · Cited by 69
“Wang and Yin [95] examined the role of explanations in AI-supported decision making and found that various XAI methods were ineffective in supporting human decision makers on tasks for which they had limited domain expertise.”
Bove, C., Aigrain, J., Lesot, M.-J., Tijus, C., & Detyniecki, M. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511139 · Cited by 67
“Yet, a more widespread adoption of ML models is for now limited by their difficulty to provide explanations about the rationale behind their predictions [5, 36].”
BackgroundAI makes you smarter but none the wiser: The disconnect between performance and metacognition
Fernandes, D., Villa, S., Nicholls, S., Haavisto, O., Buschek, D., Schmidt, A., Kosch, T., Shen, C., & Welsch, R. (2025) · Computers in Human Behavior · doi:10.1016/j.chb.2025.108779 · Cited by 188
“…interaction [57]. e t Furthermore, explanations from AI systems are often uninformative, ignored, or lead to cognitive biases themselves [9, 30, 88, 93]. d r b For instance, Zamfirescu-Pereira et al. [97] found that users have difficulty crafting effective prompts, while Dang et al. [23] noted…”
Towards relatable explainable AI with the perceptual process
Zhang, W., & Lim, B. Y. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501826 · Cited by 67
“[107] that attribution explanations were more useful than counterfactuals, possibly due to the difference of interpreting structured or unstructured data.”
BackgroundThe effects of explanations on automation bias
Vered, M., Livni, T., Howe, P. D. L., Miller, T., & Sonenberg, L. (2023) · Artificial Intelligence · doi:10.1016/j.artint.2023.103952 · Cited by 117
Exploring the Design of Generative AI in Supporting Music-based Reminiscence for Older Adults
Jin, Y., Cai, W., Chen, L., Zhang, Y., Doherty, G. J., & Jiang, T. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642800 · Cited by 90
Liao, Q. V., Zhang, Y., Luss, R., Doshi-Velez, F., & Dhurandhar, A. (2022) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v10i1.21995 · Cited by 69
Subramanian, H. V., Canfield, C., & Shank, D. B. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102780 · Cited by 84
Chiang, C.-W., Lu, Z., Li, Z., & Yin, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581015 · Cited by 56
Self-citationKim, S. S. Y., Vaughan, J. W., Liao, Q. V., Lombrozo, T., & Russakovsky, O. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714020 · Cited by 58
“However, prior work suggests that in many settings, the very presence of an explanation can increase trust and reliance, whether or not it is warranted [6, 33, 93, 97, 117, 132].”
InfluentialVisser, R., Peters, T. M., Scharlau, I., & Hammer, B. (2025) · Cognitive Systems Research · doi:10.1016/j.cogsys.2025.101357 · Cited by 58
“For example, Wang and Yin (2021, 2022) assess behavioral trust by distinguishing between appropriate trust, undertrust, and overtrust, depending on the participants’ usage of the model and the correctness of the model.”
InfluentialMa, S., Wang, X., Lei, Y., Shi, C., Yin, M., & Ma, X. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642671 · Cited by 54
“However, recent research has unveiled a potential drawback of providing AI explanations: the risk of increased over-reliance on AI systems when AI provides incorrect suggestions [2, 64, 83, 89].”
InfluentialSelf-citationMa, S., Wang, X., Lei, Y., Shi, C., Yin, M., & Ma, X. (2024) · arXiv · doi:10.48550/arxiv.2403.09552 · Cited by 56
Self-citationBashkirova, A., & Krpan, D. (2024) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2024.100066 · Cited by 55
Knowledge graph-based rich and confidentiality preserving Explainable Artificial Intelligence (XAI)
Rožanec, J. M., Fortuna, B., & Mladenić, D. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.11.015 · Cited by 55
Lu, Z., Li, P., Wang, W., & Yin, M. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555562 · Cited by 54
“In recent years, a rich set of empirical research has been conducted in understanding whether and how the presence of recommendations from an AI model can influence people’s decision-making ability [9, 14, 27, 29, 41, 53, 56, 57].”
BackgroundMethodSelf-citationHuang, W., Suominen, H., Liu, T., Rice, G. E., Salomón, C., & Barnard, A. S. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104365 · Cited by 68
Selective Explanations: Leveraging Human Input to Align Explainable AI
Lai, V., Zhang, Y., Chen, C., Liao, Q. V., & Tan, C. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610206 · Cited by 53
“Future work can also explore utilizing beliefs about user preferences to augment the selection of examples in example-based explanations [42, 67, 98].”
BackgroundFok, R., & Weld, D. S. (2023) · arXiv · doi:10.48550/arxiv.2305.07722 · Cited by 54
“Beyond simply improving the performance of a human-AI team, we are interested in complementary performance , where the human-AI system performs better (for example, its accuracy is higher or decisions are made more cheaply) than either the human or AI separately.”
BackgroundYou’d Better Stop! Understanding Human Reliance on Machine Learning Models under Covariate Shift
Chiang, C.-W., & Yin, M. (2021) · 13th ACM Web Science Conference 2021 · doi:10.1145/3447535.3462487 · Cited by 53
“In addition, people’s first impression and mental model of an ML model [2, 26], the model’s confidence and interpretability [22, 29, 32], and the consistency between the model and humans in both their decisions and rationales [15, 33] are all shown to impact people’s reliance on the model.”
BackgroundSelf-citationTowards Explainability for AI Fairness
Zhou, J., Chen, F., & Holzinger, A. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-04083-2_18 · Cited by 43
Mitigating knowledge imbalance in AI-advised decision-making through collaborative user involvement
Gomez, C., Unberath, M., & Huang, C.-M. (2023) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102977 · Cited by 50
Will You Accept the AI Recommendation? Predicting Human Behavior in AI-Assisted Decision Making
Wang, X., Lu, Z., & Yin, M. (2022) · Proceedings of the ACM Web Conference 2022 · doi:10.1145/3485447.3512240 · Cited by 49
“…indicators of the AI model [20, 35, 45, 47], the recent surge of interests in increasing the interpretability of AI (e.g., [16, 18, 36]) has led to an increasing number of evaluations on understanding whether and how AI explanations impact people’s trust in the AI model [23, 31, 41, 47].”
BackgroundSelf-citationAl-Ansari, N., Al‐Thani, D., & Al-Mansoori, R. S. (2024) · Human Behavior and Emerging Technologies · doi:10.1155/2024/4628855 · Cited by 71
Kahr, P. K., Rooks, G., Willemsen, M. C., & Snijders, C. C. P. (2024) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3686164 · Cited by 49
Ma, S., Chen, Q., Wang, X., Zheng, C., Peng, Z., Yin, M., & Ma, X. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713423 · Cited by 48
Self-citationHow Time Pressure in Different Phases of Decision-Making Influences Human-AI Collaboration
Cao, S., Gomez, C., & Huang, C.-M. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610068 · Cited by 48
“People’s domain expertise and knowledge of AI technology [45, 71], as well as their math and logic skills [66], have also been studied to understand the complex interplay between human cognition and data-driven AI models.”
BackgroundAn objective metric for Explainable AI: How and why to estimate the degree of explainability
Sovrano, F., & Vitali, F. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110866 · Cited by 79
“Instead, the metrics proposed in [30, 32, 22] rely on user studies, as many other works [64, 48, 74, 70, 15, 52], based on classical usability metrics (i.e., effectiveness, efficiency, satisfaction).”
MethodEnhancing Household Energy Consumption Predictions Through Explainable AI Frameworks
Bhandary, A., Dobariya, V., Yenduri, G., Jhaveri, R. H., Gochhait, S., & Benedetto, F. (2024) · IEEE Access · doi:10.1109/access.2024.3373552 · Cited by 58
“In response to the pressing demand for understanding the complex nature of machine learning models, we dig into the area of XAI, a game-changing concept that covers a variety of tools and methodologies aimed at providing AI models with interpretability and transparency [10].”
MethodIs Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant
He, G., Aishwarya, N., & Gadiraju, U. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712133 · Cited by 115
“This is consistent with the findings of previous empirical studies of human-AI collaboration [62, 119, 130].”
InfluentialDu, Y., Antoniadi, A. M., McNestry, C., McAuliffe, F. M., & Mooney, C. (2022) · Applied Sciences · doi:10.3390/app122010323 · Cited by 47
Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty
Zhou, K., Hwang, J. D., Ren, X., & Sap, M. (2024) · arXiv · doi:10.48550/arxiv.2401.06730 · Cited by 44
A Missing Piece in the Puzzle: Considering the Role of Task Complexity in Human-AI Decision Making
Salimzadeh, S., He, G., & Gadiraju, U. (2023) · Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization · doi:10.1145/3565472.3592959 · Cited by 42
“Medical Diagnosis (7,4,2)[79],(6,1,0)[9], (6,240,0)[32] Nutrition Prediction (5,26,156)[16],(3,24,0)[15], (4,48,24)[33] Recidivism Prediction (10,64,32)[97],(11,200,50)[40], (8,12,6)[29],(132,10,0)[95], (11,50,0)[69] Monitoring and Administration (15,130,390)[81] Job Application Approval (5,24,0)[82]”
BackgroundInfluentialChiang, C.-W., & Yin, M. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511121 · Cited by 42
“One common approach used for promoting more appropriate reliance on ML models is to provide explanations of model predictions [2, 33, 47, 61, 63], though these attempts show mixed success.”
MethodSelf-citationSalimzadeh, S., He, G., & Gadiraju, U. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641905 · Cited by 41
“For example, in high-stake scenarios such as in the medical [39, 61, 67, 97], legal [6, 81, 86, 131], and financial [27, 36, 43, 45, 46] domains, individuals tend to exhibit a preference for human decision-makers over AI systems.”
BackgroundLeichtmann, B., Hinterreiter, A., Humer, C., Streit, M., & Mara, M. (2023) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2023.2221605 · Cited by 59
“…1999), many empirical user studies on XAI have been conducted only recently (e.g., Buc¸inca et al., 2020; Cai et al., 2019; Kenny et al., 2021; Leichtmann et al., 2023; Nguyen et al., 2021; van der Waa et al., 2021; Wang & Yin, 2021; Yang et al., 2020; Zhang et al., 2020) and vary in results.”
Tran, M., Balasooriya, C. D., Semmler, C., & Rhee, J. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01823-8 · Cited by 72
Rezaeian, O., Bayrak, A. E., & Asan, O. (2026) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2025.2539458 · Cited by 39
“Wang et al. [57] also evaluated XAI methods in recidivism and forest cover prediction tasks, comparing global approaches like feature importance with local methods such as counter-factual explanations and nearest-neighbor examples.”
Does More Advice Help? The Effects of Second Opinions in AI-Assisted Decision Making
Lu, Z., Wang, D., & Yin, M. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3653708 · Cited by 39
“…sometimes the provision of certain kinds of explanations may even result in significant over-reliance on AI models for some people [5, 10, 16, 80, 102, 106]—Schaffer et al. [80] showed that showing explanations to people who reported higher familiarity with the decision making tasks led to…”
Self-citationRezaeian, O., Bayrak, A. E., & Asan, O. (2025) · arXiv · doi:10.48550/arxiv.2501.16693 · Cited by 39
“Wang et al. [57] also evaluated XAI methods in recidivism and forest cover prediction tasks, comparing global approaches like feature importance with local methods such as counter-factual explanations and nearest-neighbor examples.”
Evaluating the impact of human explanation strategies on human-AI visual decision-making
Morrison, K., Shin, D., Holstein, K., & Perer, A. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579481 · Cited by 37
“Wang and Yin [65] compare the impacts of four different explanation techniques including counterfactual explanations, feature importance, feature contribution, and nearest neighbors on two different tasks.”
BackgroundResultWang, X., & Yin, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581366 · Cited by 38
“These studies look into dierent aspects of eects of AI explanations, including how they aect people’s understandings of the AI model [19, 67, 87, 88], awareness of AI uncertainty [87, 88, 95], trust in the AI model [14, 65, 95], de-”
BackgroundInfluentialSelf-citationTo Rely or Not to Rely? Evaluating Interventions for Appropriate Reliance on Large Language Models
Bo, J. Y., Wan, S., & Anderson, A. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714097 · Cited by 36
Cabitza, F., Fregosi, C., Campagner, A., & Natali, C. (2024) · Communications in Computer and Information Science · doi:10.1007/978-3-031-63803-9_14 · Cited by 35
“However, issues arise with the acknowledgment of the very real possibility that not all explanations sufficiently clarify the advice to the user in a manner that is understandable [30] and, more critically, that inadequate explanations are not always necessarily indicative of incorrect advice.”
BackgroundUnderstanding the Role of Explanation Modality in AI-assisted Decision-making
Robbemond, V., Inel, O., & Gadiraju, U. (2022) · Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization · doi:10.1145/3503252.3531311 · Cited by 34
“In the context of decision-support systems such as recommender systems or classification models, the most common explanation modalities are textual and visual (graphical) [8, 31, 54, 59, 63, 64, 72, 73].”
BackgroundResultImpact of Model Interpretability and Outcome Feedback on Trust in AI
Ahn, D., Almaatouq, A., Gulabani, M., & Hosanagar, K. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642780 · Cited by 69
Buçinca, Z., Swaroop, S., Paluch, A. E., Doshi‐Velez, F., & Gajos, K. Z. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713229 · Cited by 76
“Early optimism that AI decision-support tools will inevitably enhance human decision quality [67] has dwindled in light of accruing empirical evidence that paints a more complex picture [5, 11, 35, 82, 98].”
Should I Follow AI-based Advice? Measuring Appropriate Reliance in Human-AI Decision-Making
Schemmer, M., Hemmer, P., Kühl, N., Benz, C., & Satzger, G. (2022) · arXiv · doi:10.48550/arxiv.2204.06916 · Cited by 33
“In this context, additional work uses the term “appropriate trust” with a similar interpretation as the behavior to follow “the fraction of tasks where participants used the model’s prediction when the model was correct and did not use the model’s prediction when the model was wrong” [34, p. 323].”
MethodTransparency for AI systems: a value-based approach
Buijsman, S. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09770-w · Cited by 52
“…sense that the reasons for individual outputs are understood by users), and in fact the academic literature has so far found only marginal improvements in objective measures such as the ability to predict outputs or to make correct decisions (Chromik et al., 2021; Wang & Yin, 2021) based on SHAP.”
Li, Y., Deng, X., Hu, X., & Jing, L. (2024) · Journal of theoretical and applied electronic commerce research · doi:10.3390/jtaer19040126 · Cited by 64
Explainable AI decision support improves accuracy during telehealth strep throat screening
Gomez, C., Smith, B.-L., Zayas, A., Unberath, M., & Canares, T. (2024) · Communications Medicine · doi:10.1038/s43856-024-00568-x · Cited by 30
“Participants ’ task knowledge can also affecttheirwillingnesstoadoptAI ’ sadvice 65,66 .”
BackgroundSupporting High-Uncertainty Decisions through AI and Logic-Style Explanations
Cau, F. M., Hauptmann, H., Spano, L. D., & Tintarev, N. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584080 · Cited by 30
“Nevertheless, the technique is widely used in the XAI literature [14, 55, 56, 58].”
MethodGround(less) Truth: A Causal Framework for Proxy Labels in Human-Algorithm Decision-Making
Guerdan, L., Coston, A., Wu, Z. S., & Holstein, K. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3594036 · Cited by 28
“, predicting current forest cover from satellite imagery [100]; a task with”
BackgroundA Decision Theoretic Framework for Measuring AI Reliance
Guo, Z., Wu, Y., Hartline, J. D., & Hullman, J. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658901 · Cited by 27
“Future work could extend our definition to continuous recommendation spaces.”
BackgroundExplainable predictive process monitoring: a user evaluation
Rizzi, W., Comuzzi, M., Francescomarino, C. D., Ghidini, C., Lee, S., Maggi, F. M., & Nolte, A. (2024) · Process Science · doi:10.1007/s44311-024-00003-3 · Cited by 45
“For instance in Wang and Yin (2021), users were asked to carry out forward-simulation tasks.”
Large Language Models Help Humans Verify Truthfulness – Except When They Are Convincingly Wrong
Si, C., Goyal, N., Wu, T., Zhao, C., Feng, S., Daumé Iii, H., & Boyd-Graber, J. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.81 · Cited by 26
Trusting Autonomous Teammates in Human-AI Teams - A Literature Review
Duan, W., Flathmann, C., McNeese, N. J., Scalia, M. J., Zhang, R., Gorman, J. C., Freeman, G., Zhou, S., Hauptman, A. I., & Yin, X. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713527 · Cited by 50
Reicherts, L., Zhang, Z. T., von Oswald, E., Liu, Y., Rogers, Y., & Hassib, M. E. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713295 · Cited by 63
“While there are instances where explanations lead to more appropriate reliance [58, 60, 62], in many cases, explanations have had little effect on users’ decision accuracy [15] or even increased overre-liance [1, 3, 12, 14, 21, 33, 53, 54].”
Explanations Can Reduce Overreliance on AI Systems During Decision-Making
Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M., & Krishna, R. (2022) · arXiv · Cited by 403
“Future work should explore our theory on high-stakes tasks such as recidivism prediction [92], where the consequences of a wrong decision are much higher.”
BackgroundSchoeffer, J., & Kuehl, N. (2021) · Companion Publication of the 2021 Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/3462204.3481742 · Cited by 24
“Very recently, Wang and Yin [35], after thoroughly reviewing and summarizing existing research, put forward that explanations should a) improve people’s understanding of an ADS; b) help peoplerecognize the uncertainty underlying a prediction and nudge people to rely on the ADS more on high…”
Exploring explainable AI in the tax domain
Górski, Ł., Kuźniacki, B., Almada, M., Tyliński, K., Calvo, M., Asnaghi, P. M., Almada, L., Iñiguez, H. I., Rubianes, F., Pera, O., et al. (2024) · Artificial Intelligence and Law · doi:10.1007/s10506-024-09395-w · Cited by 43
“Social sciences maintain that they offer a similar conceptual framework to that employed when humans explain their decision to each other; empirical evidence suggest that for such explanation to be useful, ultimately a domain expertise is nevertheless needed (Wang and Yin 2021).”
He, G., Demartini, G., & Gadiraju, U. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713218 · Cited by 23
“For example, increasing the transparency of AI systems by providing confidence scores [112], explanations [104], trustworthiness cues [58], and uncertainty communication [50].”
Bove, C., Lesot, M., Tijus, C., & Detyniecki, M. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581641.3584082 · Cited by 43
“Several forms have been proposed to present counterfactual explanations, such as textual [46], visual [11, 12, 48] or vocal [42].”
BackgroundMethodInfluential
Metadata
- Authors
- Wang, X., & Yin, M.
- Affiliations
- Purdue University West Lafayette (US)
- ORCID
- Venue
- 26th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2021-04-14
- Pages
- 318-328
- OpenAlex
- W3156106752
- Semantic Scholar
- c79ba865574cb9635dbca135cec99ee56125778c
- Cited by
- 320
- References
- 66
- Citations by year
- 2026: 63 · 2025: 76 · 2024: 82 · 2023: 56 · 2022: 34 · 2021: 9
- FWCI
- 20.90
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion, cc-by)
- Funders
- National Science Foundation (1850335)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- explainable AI, AI-assisted decision-making, model-agnostic explanation, counterfactual explanations, model uncertainty, domain knowledge, recidivism prediction, human-AI interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.