Explanations Considered Harmful: The Impact of Misleading Explanations on Accuracy in Hybrid Human-AI Decision Making
Open accessThe DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
. EXplainable AI (XAI) has the potential to enhance decision-making in human-AI collaborations, yet existing research indicates that explanations can also lead to undue reliance on AI recommendations, a dilemma often referred to as the ’white box paradox.’ This paradox illustrates how persuasive explanations for incorrect advice might foster inappropriate trust in AI systems. Our study extends beyond the traditional scope of the white box paradox by proposing a framework for examining explanation inadequacy. We specifically investigate how accurate AI advice, when paired with misleading explanations, affects decision-making in logic puzzle tasks. Our findings introduce the concept of the ‘XAI halo effect,’ where participants were influenced by the misleading explanations to the extent that they did not verify the correctness of the advice, despite its accuracy. This effect reveals a nuanced challenge in XAI, where even correct advice can lead to misjudgment if the accompanying explanations are not coherent and contextually relevant. The study highlights the critical need for explanations to be both accurate and relevant, especially in contexts where decision accuracy is paramount. This calls into question the use of explanations in situations where their potential to mislead outweighs their transparency or educational value.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
The false hope of current approaches to explainable artificial intelligence in health care
Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00208-9 · Cited by 1,724 · cited 2×
“Explainable Artificial Intelligence (XAI), a term introduced by Van Lent et al. [28], represents the hope to address the limitations of current AI systems, intended as “black boxes”, by enhancing trust and transparency through the provision of explanations [17, 22].”
BackgroundShin, D. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2020.102551 · Cited by 1,548
Explainable AI (XAI): Core Ideas, Techniques, and Solutions
Dwivedi, R., Dave, D., Naik, H., Singhal, S., Omer, R., Patel, P., Qian, B., Wen, Z., Shah, T., Morgan, G., et al. (2023) · ACM Computing Surveys · doi:10.1145/3561048 · Cited by 1,386
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757 · cited 3×
“Evidence Evidence strongly confirms and supports the utility of XAI in decision-making processes, highlighting its contribution to improved accuracy [10, 2, 3].”
BackgroundResultDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems
Bussone, A., Stumpf, S., & O'Sullivan, D. (2015) · 2015 International Conference on Healthcare Informatics · doi:10.1109/ichi.2015.26 · Cited by 441 · cited 1×
“…can increase the user’s confidence in the final decision, leading to greater satisfaction with the interaction [27]; foster appropriate reliance [2, 5] — enabling the user to trust the system when appropriate and be skeptical when the system might mislead — and thereby calibrate their trust [20,…”
Background
Cited by
Brankovic, A., Cook, D., Rahman, J., Delaforce, A., Li, J., Magrabi, F., Cabitza, F., Coiera, E., & Bradford, D. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01764-2 · Cited by 37
“Item 7 .”
Self-citationAI-overdependence and human cognitive decline: Hazards, evidence, and mitigation strategies
Noorbehbahani, F., & Oyibo, K. (2026) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2026.101102 · Cited by 12
Don’t Be Fooled: The Misinformation Effect of Explanations in Human–AI Collaboration
Spitzer, P., Holstein, J., Morrison, K., Holstein, K., Satzger, G., & Kühl, N. (2026) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2025.2574511 · Cited by 12
“Correct AI advice [3, 39, 42, 55, 68, 76, 77, 94, 102, 104] [17, 61] Incorrect AI advice [4, 15, 16, 18, 20, 30, 53, 54, 72, 75] [56, 61, 64]”
InfluentialCabitza, F., Famiglini, L., Fregosi, C., Pe, S., Parimbelli, E., La Maida, G. A., & Gallazzi, E. (2025) · Proceedings of the 30th International Conference on Intelligent User Interfaces · doi:10.1145/3708359.3712157 · Cited by 11
“Even if one explanation appears more per-suasive (yet remains incorrect [12]), the presence of alternative perspectives serves as a safeguard, fostering critical thinking and reducing—but not entirely eliminating—automation bias.”
Self-citationRaees, M., Khan, V.-J., Lykourentzou, I., & Papangelis, K. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791467 · Cited by 5
Metadata
- Authors
- Cabitza, F., Fregosi, C., Campagner, A., & Natali, C.
- Venue
- Communications in Computer and Information Science
- Publisher
- Springer Nature Switzerland
- Published
- 2024
- Pages
- 255-269
- ISSN
- 1865-0929 (ISSN-L)
- 1865-0937
- OpenAlex
- W4400439181
- Cited by
- 35
- References
- 17
- FWCI
- 10.65
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Explanations Considered Harmful: The Impact of Misleading Explanations on Accuracy in Hybrid Human-AI Decision Making. https://app.cochat.ai/papers/10.1007/978-3-031-63803-9_14
- The paper
- Cabitza, F., Fregosi, C., Campagner, A., & Natali, C. (2024). Explanations Considered Harmful: The Impact of Misleading Explanations on Accuracy in Hybrid Human-AI Decision Making. Communications in Computer and Information Science. https://doi.org/10.1007/978-3-031-63803-9_14
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.