The DOI matches its Crossref record; 55 of the 57 references it cites were found in a registry or an index.
I Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI
Abstract
Unintended consequences of deployed AI systems fueled the call for more interpretability in AI systems. Often explainable AI (XAI) systems provide users with simplifying local explanations for individual predictions but leave it up to them to construct a global understanding of the model behavior. In this work, we examine if non-technical users of XAI fall for an illusion of explanatory depth when interpreting additive local explanations. We applied a mixed methods approach consisting of a moderated study with 40 participants and an unmoderated study with 107 crowd workers using a spreadsheet-like explanation interface based on the SHAP framework. We observed what non-technical users do to form their mental models of global AI model behavior from local explanations and how their perception of understanding decreases when it is examined.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Tasks that involve inherent uncertainty are often those where humans tend to rely on AI systems for advice, such as loan approval [27, 34, 124], recidivism prediction [31, 47, 86], house price estimation [2, 13, 25], and student admission [13, 24].”
From Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making (2024) · cites this paper as [27]
In the citing paper
“However, recent research shows that people do not always engage with explainability tools enough to help improve decision making.”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [2]
In the citing paper
“Participants who do seem to believe that they understand explanatory information, leading to over-confidence [2], even when presented with explanations that contain no useful information [5].”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [2]
In the citing paper
“Some experimental studies [2,5,42] indicate that most participants do not cognitively engage with explainability tools.”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [2]
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
(2011) · Cited by 14,317
Shapley, L. (1988) · Cambridge University Press eBooks · doi:10.1017/cbo9780511528446.003 · Cited by 10,698
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
From local explanations to global understanding with explainable AI for trees
Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., & Lee, S.-I. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-019-0138-9 · Cited by 10,551
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Shortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224
Definitions, methods, and applications in interpretable machine learning
Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., & Yu, B. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1900654116 · Cited by 2,167 · cited 2×
“For an explanation to be effective, it does not only need to have a sufficient level of fidelity but must "provide insight for a particular audience into a chosen domain problem" [36].”
Background
Flagged first, then by citations
2 of the 57 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- Or Biran and Courtenay Cotton . 2017 . Explanation and justification in machine learning: A survey . In IJCAI-17 workshop on explainable AI (XAI), Vol. 8. 1. Or Biran and Courtenay Cotton. 2017. Explanation and justification in machine learning: A survey. In IJCAI-17 workshop on explainable AI (XAI), Vol. 8. 1.
- Adrian Bussone , S. Stumpf , and D. O’Sullivan . 2015 . The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems. 2015 International Conference on Healthcare Informatics ( 2015 ), 160–169. Adrian Bussone, S. Stumpf, and D. O’Sullivan. 2015. The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems. 2015 International Conference on Healthcare Informatics (2015), 160–169.
- Finale Doshi-Velez and Been Kim. 2017. Towards A Rigorous Science of Interpretability. CoRR abs/1702.08608(2017). arxiv:1702.08608http://arxiv.org/abs/1702.08608 Finale Doshi-Velez and Been Kim. 2017. Towards A Rigorous Science of Interpretability. CoRR abs/1702.08608(2017). arxiv:1702.08608http://arxiv.org/abs/1702.08608 · arXiv
- Tae-Nyun Kim and Hayeon Song . 2020 . The Effect of Message Framing and Timing on the Acceptance of Artificial Intelligence’s Suggestion. Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (2020). Tae-Nyun Kim and Hayeon Song. 2020. The Effect of Message Framing and Timing on the Acceptance of Artificial Intelligence’s Suggestion. Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (2020).
- Gregory L. Murphy and Douglas L. Medin . 1985. The role of theories in conceptual coherence.Psychological review 92 3 ( 1985 ), 289–316. Gregory L. Murphy and Douglas L. Medin. 1985. The role of theories in conceptual coherence.Psychological review 92 3 (1985), 289–316.
Cited by
Sorted by citations
Items 101–146 of 146
Rahman, J. S., Delaforce, A., Bradford, D., Li, J., Magrabi, F., Cook, D., & Brankovic, A. (2026) · JMIR Human Factors · doi:10.2196/81460 · Cited by 2
“Moreover, recent research has shown that such explanations can inflate users’ trust in the model even when that trust is not warranted [26,33,39,40].”
Empowering Individuals in Automated Decision-Making: Explainability, Contestability and Beyond
Pi, Y. (2024) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3678884.3682043 · Cited by 3
“In previous XAI literature, some explanations are less likely to be presented using tables; for example, feature-based explanations are often presented graphically or color-coded (i.e., SHAP Bar Plot) [5, 6, 22].”
Leveraging Complementary AI Explanations to Mitigate Misunderstanding in XAI
Xuan, Y., Sokol, K., Sanderson, M., & Chan, J. (2025) · IEEE Swiss Conference on Data Science (SDS) · doi:10.1109/sds66131.2025.00034 · Cited by 2
“A key cognitive bias that contributes to such a misunder-standing is the illusion of explanatory depth , where users believe that they understand a system in greater detail than they actually do [1], [2].”
AI Literacy for Legal AI Systems: a Practical Approach
Gültekin-Várkonyi, G. (2025) · Iustum Aequum Salutare · doi:10.69695/ias.2025.4.01 · Cited by 6
Human-Centered Explainable AI (XAI)
Liao, Q. V., & Varshney, K. R. (2026) · Cambridge University Press eBooks · doi:10.1017/9781108872225.011 · Cited by 448
Towards Understanding Human-AI Reliance Patterns Through Explanation Styles
Casolin, E. R., & Salim, F. D. (2024) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/3675094.3678996 · Cited by 2
“Prior work [3, 13, 15, 16, 22] has suggested that this lack of appropriate reliance is due to the mere presence of explanations invoking cognitive biases and heuristics that inevitably lead to blind trust in, and deference to, AI systems.”
Leveraging Generative AI to Improve Comprehensibility in Social Recommender Systems
Ashaduzzaman, M., & Tsai, C.-H. (2025) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3708319.3733696 · Cited by 3
“Subjective understanding often relies on post-task questionnaires, using standardized or adapted methods to gauge users’ perceived clarity of explanations [15].”
Improving Tactical Decision-Making Through Multiobjective Contrastive Explanations
Blom, M. L., Singh, R., Miller, T., Sonenberg, L., Trentelman, K., Saulwick, A., & Wark, S. (2025) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2025.3591163 · Cited by 1
“As noted, studies [49], [50]; show participants often fail to engage with explainability tools and overestimate their understanding.”
Anik, A. I., & Bunt, A. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789087 · Cited by 2
“Recognizing the limitations of a solely algorithm-centered focus [34, 45, 101], researchers have called for a shift to broader contexts [88, 89].”
Cybersophy: A framework for human cognition, identity, and values in the age of artificial agents
Ning, H., & Ding, J. (2026) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2026.100385 · Cited by 1
Seamful Design Considerations for Human-in-the-Loop Digital Phenotyping of Mental Health
Swain, V. D., Tongtawee, T., Wang, O., Hsu, J., Jacobson, N. C., & Mishra, V. (2026) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3832021 · Cited by 2
“Even if we aggregate enough representative data to train such models, explaining these models rarely provides actionable insight to non-technical end-users [20].”
INFEATURE: An Interactive Feature-Based-Explanation Framework for Non-technical Users
Pi, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-35891-3_16 · Cited by 1
“DecisionTime”: A Configurable Framework for Reproducible Human-AI Decision-Making Studies
Salimzadeh, S., & Gadiraju, U. (2024) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3631700.3664885 · Cited by 1
“AI systems have been increasingly adapted in various domains to assist individuals in making decisions that directly impact their lives, such as allocating resources [4], recommending products [10], and even predicting medical conditions [12].”
BackgroundKrämer, N. C., Wischnewski, M., & Müller, E. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/h32ze · Cited by 2
Pranata, Y., Mirwanti, R., Bahtiar, Y. A., & Purwanti, H. (2026) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s621469 · Cited by 1
“13,15,16 This distinction is especially consequential in the ICU, where the accountability demanded of clinical decisions calls for a deeper understanding than a mere summary of model patterns or trends.”
Sonenberg, L. (2023) · Logic Journal of IGPL · doi:10.1093/jigpal/jzad006
Optimal Neighborhood Contexts in Explainable AI: An Explanandum-Based Evaluation
Pawar, U., O’Shea, D., O’Reilly, R., Costello, M., & Beder, C. (2024) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2024.3389781 · Cited by 1
Explainable AI in Ad Enforcement: Striking the Balance Between Transparency and Safety
Shah, B. M. (2025) · Journal of Information Systems Engineering & Management · doi:10.52783/jisem.v10i41s.7584
“● Progressive disclosure: Use expandable sections to allow advertisers to access additional detail as needed [104].”
Rahman, J. S., Delaforce, A., Bradford, D., Li, J., Magrabi, F., Cook, D., & Brankovic, A. (2025) · doi:10.2196/preprints.81460
Explainability of Language Models (XAI and LLM)
Cecco, A. D., & Gianfagna, L. (2025) · Springer eBooks · doi:10.1007/978-3-031-92229-9_8
MAINLE: A Multi-Agent, Interactive, Natural Language Local Explainer of Classification Tasks
Serafim, P. B. S., Filho, R. F. F., Freitas, S., Gezi̇ci, G., Giannotti, F., Raimondi, F., & dos Santos, A. H. R. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-06078-5_9
Beyond Satisfaction: From Placebic to Actionable Explanations for Enhanced Understandability
Shymanski, J., Brue, J., & Sen, S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-01399-6_10
“Chromik et al. found that users receiving explanations, especially non-domain expert users, experienced an illusion of understanding when presented with insufficient local explanations [4].”
Vromman, F. V., Kerckhove, C. V., Gagnon, J., Pelletier, C., Dufresne, Y., & Coulombe, S. (2025) · doi:10.2196/preprints.88750
EXPLAINABLE ARTIFICIAL INTELLIGENCE AS A SERVICE: ALGORITHMS FOR ASSESSMENT OF CHARACTERISTICS
Kharchenko, V. V., Yakovlev, S., Veprytska, O., & Fesenko, H. V. (2026) · Kibernetyka ta Systemnyi Analiz · doi:10.34229/kca2522-9664.26.1.1
Explainable Artificial Intelligence as a Service: Algorithms for Evaluation of Characteristics
Kharchenko, V. S., Yakovlev, S., Veprytska, O., & Fesenko, H. V. (2026) · Cybernetics and Systems Analysis · doi:10.1007/s10559-026-00839-6
Accuracy is not all you need! The Reasons to Require AI Explainability
Buijsman, S. (2026) · Minds and Machines · doi:10.1007/s11023-026-09768-x · Cited by 1
“At the same time, these studies also find that people perform no better on the common metrics for how well they understand the AI system, such as whether they are able to predict the output of the system for new inputs Chromik et al. (2021); Van Der Waa et al. (2021).”
Explainable AI for Well-Being Prediction From Lifestyle Data: 2-Study Design
Vromman, F. V., Kerckhove, C. V., Gagnon, J., Pelletier, C., Dufresne, Y., & Coulombe, S. (2026) · JMIR Mental Health · doi:10.2196/88750
Shank, D. B., Canfield, C., & Subramanian, H. V. (2024) · OSF Preprints (OSF Preprints)
AI CHAOS! 1st Workshop on the Challenges for Human Oversight of AI Systems
Schrills, T., Kahr, P. K., Langer, M., Kaur, H., & Gadiraju, U. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742414.3794953
“Poorly implemented oversight mechanisms can create a false sense of safety, leading stakeholders to believe that systems are properly monitored when they are not [2–4], and can obscure accountability and delay the identification of system problems [7, 8].”
MIRAGE: Misleading Impacts Resulting from AI Generated Explanations (Workshop)
Daly, E., Ehsan, U., Quercia, D., & Stumpf, S. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742414.3794955
“For example, providing explanations has been often linked with over-or under-reliance on AI predictions [5, 6, 15], it might lead to inflated confidence and trust [2, 7], or increase bias of AI systems[18, 21], while explanations can also give an illusion of control and accountability [16, 19].”
Explainability for machine learning models : from data adaptability to user perception
Delaunay, J. (2023) · theses.fr (ABES) · doi:10.70675/933366c4z576ez464cz9fcfz24627f86dd2f
Interpretable Music Recommender Systems
Afchar, D. (2023) · theses.fr (ABES) · doi:10.70675/acfb6676z8613z4e08zad14z98d4a9c2ee74 · Cited by 1
Bertrand, A. (2024) · theses.fr (ABES) · doi:10.70675/216d3647z78bcz4c68z9ba9z2d0a029cf1b4 · Cited by 1
AI CHAOS! 2nd Workshop on the Challenges for Human Oversight of AI Systems
Khadar, M., Cecil, J., Neut, L. V. D., Banović, N., Baum, K. L., Chancellor, S., Costanza, E., Eslami, M., Feit, A. M., Gaube, S., et al. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3778736
“Poorly designed oversight mechanisms can, for example, create a false sense of safety, leading stakeholders to believe that systems are properly monitored when they are not [3, 6, 10].”
Does Sycophancy Change Decisions? Effect of LLM Sycophancy on AI-Assisted Decision-Making
Li, Z., Pan, J., Liu, Q., Xi, Y., Zhou, Y., Jin, Y., Mao, R., & Chen, P. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790934 · Cited by 5
“In high-risk tasks, users both rely more on AI and demand greater transparency [21, 63], though poor trust calibration can lead to over-reliance or rejection [9, 27, 45].”
Algorithmic Authority Bias in AI-Assisted Learning: An Integrative Cognitive Framework
Sony, S. D., & Mus, K. (2026) · International Educational Review · doi:10.58693/ier.415
“Moreover, later experiments confirmed that AI-generated explanations can strengthen this illusion (Chromik et al., 2021).”
Connelly, L. C., Pedersen, T. M., Øvlisen, K., Bilstrup, K.-E. K., Petersen, M. G., & Nyrup, R. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812233
“Recent work has raised and begun to engage with these issues of scope [4, 18], evaluation [32, 34], and context agnostic explanations [12, 51].”
Visible Reasoning: User-Facing Decision Transparency for Generative AI Systems
Anbalagan, S. K., Nie, X., Kommalapati, A., Kanamarlapudi, V. K., Radhakrishnan, S., Zhao, X., & Mohan, U. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-30849-8_1
“Chromik et al. [7] showed that users can overestimate their understanding from AI explanations alone, motivating more evidence-grounded transparency mechanisms.”
Kauttonen, J., Aunimo, L., Alamäki, A. V., & Gálvez-Ruiz, D. (2026) · Behaviour and Information Technology · doi:10.1080/0144929x.2026.2711030
Rosbach, E., Ammeling, J., Bertram, C., Riener, A., & Aubreville, M. (2026) · Veterinary Pathology · doi:10.1177/03009858261472493
Fiebrink, R., Gillies, M. F. P., & Ramos, G. (2026) · Cambridge University Press eBooks · doi:10.1017/9781108872225.010
Schiedermair, I., Weitemeyer, R., Kopp, T., Beyer, J., Sen, S., & Kinkel, S. (2026) · HMD Praxis der Wirtschaftsinformatik · doi:10.1365/s40702-026-01324-6
Explanation Needs Emerge Before Explicit Requests in Human-Robot Interaction
Kontogiorgos, D., Gustafson, J., & Shah, J. (2026) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3776574.3831198
“Continuous decision-making settings, such as finance and autonomous driving, predominantly use automatic explanations [7, 12, 14, 15, 63, 67], while human resources and e-commerce often combine automatic and on-demand delivery [10, 17, 22, 60, 81].”
Hosseinian, F., & Sheikhansari, I. (2026) · AI and Ethics · doi:10.1007/s43681-026-01437-0
Alon, L. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6172315
Human-Centered Explainable AI (XAI): From Algorithms to User Experiences
Liao, Q. V., & Varshney, K. R. (2021) · arXiv · doi:10.48550/arxiv.2110.10790 · Cited by 120
“In addition, there is the concern of illusory understanding, with which one subjectively over-estimates the understanding they gain from XAI [22].”
Background
Metadata
- Authors
- Chromik, M., Eiband, M., Buchner, F., Krüger, A., & Butz, A.
- Affiliations
- Ludwig-Maximilians-Universität München (DE)
- Venue
- 26th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2021-04-14
- Pages
- 307-317
- OpenAlex
- W3156666346
- Cited by
- 144
- References
- 57
- Citations by year
- 2026: 26 · 2025: 31 · 2024: 38 · 2023: 30 · 2022: 16 · 2021: 3
- FWCI
- 9.96
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- explainable AI, illusion of explanatory depth, local explanations, SHAP, non-technical users, mental models, interpretability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
