Are Explanations Helpful? A Comparative Study of the Effects of Explanations in AI-Assisted Decision-Making
Open accessThe DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index.
Abstract
This paper contributes to the growing literature in empirical evaluation of explainable AI (XAI) methods by presenting a comparison on the effects of a set of established XAI methods in AI-assisted decision making. Specifically, based on our review of previous literature, we highlight three desirable properties that ideal AI explanations should satisfy—improve people’s understanding of the AI model, help people recognize the model uncertainty, and support people’s calibrated trust in the model. Through randomized controlled experiments, we evaluate whether four types of common model-agnostic explainable AI methods satisfy these properties on two types of decision making tasks where people perceive themselves as having different levels of domain expertise in (i.e., recidivism prediction and forest cover prediction). Our results show that the effects of AI explanations are largely different on decision making tasks where people have varying levels of domain expertise in, and many AI explanations do not satisfy any of the desirable properties for tasks that people have little domain expertise in. Further, for decision making tasks that people are more knowledgeable, feature contribution explanation is shown to satisfy more desiderata of AI explanations, while the explanation that is considered to resemble how human explain decisions (i.e., counterfactual explanation) does not seem to improve calibrated trust. We conclude by discussing the implications of our study for improving the design of XAI methods to better support human decision making.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, the previous research discusses the issues of these feature-based explanations on overtrust in AI [29, 37, 64].”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [64]
In the citing paper
“Overall, our findings indicate user's overreliance on 'wrong' AI outputs, which follows the previous studies that describe salient feature explanations increases user's overreliance on the AI model [6, 64].”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [64]
In the citing paper
“Other works that utilize AI/ML models focused on tasks that do not require domain experts, such as judges or clinicians [5, 38, 64].”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [64]
In the citing paper
“Wang and Yin [64] conducted the randomized experiment using four types of common model-agnostic explainable AI methods and discussed the effect of AI explanations could be largely different where people have varying levels of domain expertise.”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [64]
In the citing paper
“However, across many domains, empirical studies of explanations have produced mixed results [4,11,49,53,58,68,82,87], with some suggesting that explanations increase decisionmakers' tendency to rely on AI predictions even when the AI system is wrong [4,68,82,87]-a phenomenon often referred to as overreliance [10,78].”
From Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations (2023) · cites this paper as [82]
16 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
UCI Machine Learning Repository
Asuncion, A. (2007) · Medical Entomology and Zoology · Cited by 25,410
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 4×
“For example, sophisticated techniques such as LIME [60] and SHAP [48] have been used to illustrate how much each feature contributes to a model’s final prediction.”
MethodInfluential"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · cited 5×
“For example, sophisticated techniques such as LIME [60] and SHAP [48] have been used to illustrate how much each feature contributes to a model’s nal prediction.”
MethodInfluential(2011) · Cited by 14,317
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Gauriat, C.-M., Pencolé, Y., Ribot, P., & Brouillet, G. (2024) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.4230/oasics.dx.2024.27 · Cited by 12,586
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 1×
“…interpretable models such as generalized additive models and decision sets [15, 34, 43, 66], as well as visualizing what deep neural network has learned in its intermediate layers and how its predictions are affected by different part of the inputs (e.g., through saliency map) [37, 61, 68].”
BackgroundPeeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · cited 1×
“havior of the entire AI model is categorized as global explanations, while methods that provide reasons for specic model predictions are categorized as local explanations [2, 24, 26].”
MethodLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 1×
“Typical methods include ask people to rank the input data features based on their inuence to overall predictions [18, 33], to indicate the direction of change in the model’s prediction when a feature’s value is altered [16, 18, 29], to simulate the model’s predictions [16, 24, 40, 47, 59], to answer “what-if” question…”
MethodDeep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 1×
“…interpretable models such as generalized additive models and decision sets [15, 34, 43, 66], as well as visualizing what deep neural network has learned in its intermediate layers and how its predictions are affected by different part of the inputs (e.g., through saliency map) [37, 61, 68].”
Background
Research works printed but not linked: 6
- Dennis Collaris Leo M Vink and Jarke J van Wijk. 2018. Instance-level explanations for fraud detection: A case study. arXiv preprint arXiv:1806.07129(2018). Dennis Collaris Leo M Vink and Jarke J van Wijk. 2018. Instance-level explanations for fraud detection: A case study. arXiv preprint arXiv:1806.07129(2018). · arXiv
- 10.2139/ssrn.3465622 · DOI · Paper page
- Isaac Lage Emily Chen Jeffrey He Menaka Narayanan Been Kim Sam Gershman and Finale Doshi-Velez. 2019. An evaluation of the human-interpretability of explanation. arXiv preprint arXiv:1902.00006(2019). Isaac Lage Emily Chen Jeffrey He Menaka Narayanan Been Kim Sam Gershman and Finale Doshi-Velez. 2019. An evaluation of the human-interpretability of explanation. arXiv preprint arXiv:1902.00006(2019). · arXiv
- Jeff Larson , Surya Mattu , Lauren Kirchner , and Julia Angwin . 2016. How we analyzed the COMPAS recidivism algorithm. ProPublica (5 2016) 9, 1 ( 2016 ). Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin. 2016. How we analyzed the COMPAS recidivism algorithm. ProPublica (5 2016) 9, 1 (2016).
- Maranda McBride and Shona Morgan . 2010. Trust calibration for automated decision aids . Institute for Homeland Security Solutions.[Online] . Available: https://www. ihssnc. org/portals/0/Documents/VIMSDocuments/McBride_Research_Brief. pdf(2010). Maranda McBride and Shona Morgan. 2010. Trust calibration for automated decision aids. Institute for Homeland Security Solutions.[Online]. Available: https://www. ihssnc. org/portals/0/Documents/VIMSDocuments/McBride_Research_Brief. pdf(2010).
- Jennifer Wortman Vaughan and Hanna Wallach . 2020. A Human-Centered Agenda for Intelligible Machine Learning. Machines We Trust: Getting Along with Artificial Intelligence ( 2020 ). Jennifer Wortman Vaughan and Hanna Wallach. 2020. A Human-Centered Agenda for Intelligible Machine Learning. Machines We Trust: Getting Along with Artificial Intelligence (2020).
Cited by
Items 301–332 of 332
Explaining at the Speed of Sight: Attention-Aware XAI for OODA-Inspired AI Design
Wright, D., & Menon, V. (2025) · IEEE International Conference on Cognitive Machine Intelligence (CogMI) · doi:10.1109/cogmi67134.2025.00032
Degachi, S., Niforatos, E., & Kortuem, G. (2026) · Behaviour and Information Technology · doi:10.1080/0144929x.2026.2638907 · Cited by 1
“Recent work shows that even the mere presence of many explanation types can increase blind, inappropriate, trust in, and overreliance on, AI tools (Bansal et al. 2021; Pafla, Larson, and Hancock 2024; Si et al. 2024; X. Wang and Yin 2021; Zhang, Vera Liao, and Bellamy 2020).”
Vereschak, O. (2022) · theses.fr (ABES) · doi:10.70675/7ecd53bfza7ccz4edfz8f1azdb8052ba1233 · Cited by 2
Interpretable Music Recommender Systems
Afchar, D. (2023) · theses.fr (ABES) · doi:10.70675/acfb6676z8613z4e08zad14z98d4a9c2ee74 · Cited by 3
Understanding the Affordances of Control in AI Reasoning for Human-AI Decision-Making
Moon, J., Huang, C., & Xiao, Z. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3798555 · Cited by 1
Holland, T., Sparrow, L. A., Smith, W., & Webber, S. E. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791146 · Cited by 1
More Isn't Always Better: Balancing Decision Accuracy and Conformity Pressures in Multi-AI Advice
Tsuchiya, Y., & Baba, Y. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791648 · Cited by 4
Guided Reflection in AI-Assisted Decision-Making: Effects on AI Overreliance and Decision Accuracy
Li, S., Li, J., Li, H., Zhu, H., & Li, X. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790632 · Cited by 8
Wright, D., & Menon, V. (2026) · IEEE Conference on Cognitive and Computational Aspects of Situation Management (CogSIMA) · doi:10.1109/cogsima68896.2026.11481231
Aligning explanations with human values: context-sensitive trust calibration in AI decision systems
Singh, S., & Siemon, D. (2026) · Behaviour and Information Technology · doi:10.1080/0144929x.2026.2662402
A Socioinformatic Approach to xAI
Wilhelm, A., & Zweig, K. A. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-25308-8_27
Aligning AI Explanations with User Needs: A Qualitative Study of XAI Methods
Donoso-Guzmán, I., Parra, D., & Verbert, K. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-19105-2_10
Analyzing Empirical Findings on User Reliance Behaviors in XAI-Assisted Decision-Making
Filho, J. C. D. S., Belloum, R., & de Oliveira, K. M. (2026) · Lecture notes in business information processing · doi:10.1007/978-3-032-26836-5_3
From Curiosity to Caution: How Expertise Shapes the Use of Interpretable Machine Learning
Naqvi, S., Balaji, G., Laju, A., & Kaur, H. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812376
“Prior work has studied user reliance on interpretability tools for tasks such as exploratory data science (e.g., [22, 79, 92]), AI-assisted decision-making (e.g., [143, 180, 189]), and cognitive reasoning (e.g., [175]).”
Reliance-Aware Design of Interactive Explainable User Interfaces
Filho, J. C. D. S., Belloum, R., Monteiro, I. T., & de Oliveira, K. M. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812231 · Cited by 2
“XAI systems have been widely used to support decision-making processes in several fields, such as medical diagnosis [50, 52], loan approval [55], criminal recidivism [42, 74], education [10], and entertainment [39, 69].”
When Explanations Deceive: Understanding Unintentional and Intentional Deception in XAI
Tekkesinoglu, S. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812351
Yang, M., Kagawa, R., & Baba, Y. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812288
“In this collaborative workflow, Explainable AI (XAI) plays an important role [28, 63, 64].”
Explanation Depth and Trust Calibration in AI-Driven Financial Advice: A Behavioural ExperimentPreprint
Kashyap, S., & Srivastava, N. (2026) · Research Square · doi:10.21203/rs.3.rs-9888939/v1
Human–AI teams for decision-making in banking and finance: A systematic scoping review
Aksu, F., Morandini, S., & Pietrantoni, L. (2026) · Human Systems Management · doi:10.1177/01672533261461402
Human-AI Sensemaking in DH Archives: Comparing Expert and LLM Interpretations of Topic Word Lists
Huang, Q., Karaosmanoglu, S., Kruse, L., Klomfaß, V., Nantke, J., & Steinicke, F. (2026) · Creativity and Cognition · doi:10.1145/3803784.3807528
Linguistic Uncertainty Markers for Trust Calibration in AI-Assisted Decision-Making
Spillner, L., Rockstroh, J., Wenig, N., Zargham, N., Kiefner, L., Porzel, R., & Malaka, R. (2026) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3816046.3816231 · Cited by 1
Costa, G., & Ortale, R. (2026) · Applied Intelligence · doi:10.1007/s10489-026-07399-4
“The above challenges highlight the need for IDSS that emphasize transparency [2], interpretability [3], and explain-ability [4, 5], while maintaining computational efficiency.”
Not All Trust Is the Same: Effects of Decision Workflow and Explanations in Human-AI Decision Making
Spillner, L., Ringe, R., Porzel, R., & Malaka, R. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-26717-7_15 · Cited by 5
How do AI feedback types affect students’ continuance usage intention in AI hallucinations?
Guo, Y., Huang, T., Lu, H., Zong, L., Huang, L., & Chen, M. (2026) · Interactive Learning Environments · doi:10.1080/10494820.2026.2711950
Artificial intelligence-driven decision-making and its impact on board accountability
dalaien, B. O. A. A., Al-Hroub, M., Syouf, R. A., Hazaea, S. A., & Aldalaeen, S. O. A. (2026) · Corporate Board role duties and composition · doi:10.22495/cbv22i3art9
Wójcik, S., Rulkiewicz, A., & Domienik‐Karłowicz, J. (2026) · doi:10.2196/preprints.110249
Fiebrink, R., Gillies, M. F. P., & Ramos, G. (2026) · Cambridge University Press eBooks · doi:10.1017/9781108872225.010
Schiedermair, I., Weitemeyer, R., Kopp, T., Beyer, J., Sen, S., & Kinkel, S. (2026) · HMD Praxis der Wirtschaftsinformatik · doi:10.1365/s40702-026-01324-6
Laoh, E., Ahmad, O. B., & Liu, T. (2026) · Smart Health · doi:10.1016/j.smhl.2026.100724 · Cited by 1
Chen, H., Lin, C., & Pu, Y. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2739361
Zipperling, D., Deck, L., Lanzl, J., & Kühl, N. (2026) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3858913
Performance and Metacognition Disconnect when Reasoning in Human-AI Interaction
Fernandes, D., Villa, S., Nicholls, S., Haavisto, O., Buschek, D., Schmidt, A., Kosch, T., Shen, C., & Welsch, R. (2024) · arXiv · doi:10.48550/arxiv.2409.16708 · Cited by 71
“…interaction [57]. e t Furthermore, explanations from AI systems are often uninformative, ignored, or lead to cognitive biases themselves [9, 30, 88, 93]. d r b For instance, Zamfirescu-Pereira et al. [97] found that users have difficulty crafting effective prompts, while Dang et al. [23] noted…”
Metadata
- Authors
- Wang, X., & Yin, M.
- Affiliations
- Purdue University West Lafayette (US)
- ORCID
- Venue
- 26th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2021-04-14
- Pages
- 318-328
- OpenAlex
- W3156106752
- Semantic Scholar
- c79ba865574cb9635dbca135cec99ee56125778c
- Cited by
- 320
- References
- 66
- Citations by year
- 2026: 63 · 2025: 76 · 2024: 82 · 2023: 56 · 2022: 34 · 2021: 9
- FWCI
- 20.90
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion, cc-by)
- Funders
- National Science Foundation (1850335)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- explainable AI, AI-assisted decision-making, model-agnostic explanation, counterfactual explanations, model uncertainty, domain knowledge, recidivism prediction, human-AI interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.