The DOI matches its Crossref record; 55 of the 57 references it cites were found in a registry or an index.
I Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI
Abstract
Unintended consequences of deployed AI systems fueled the call for more interpretability in AI systems. Often explainable AI (XAI) systems provide users with simplifying local explanations for individual predictions but leave it up to them to construct a global understanding of the model behavior. In this work, we examine if non-technical users of XAI fall for an illusion of explanatory depth when interpreting additive local explanations. We applied a mixed methods approach consisting of a moderated study with 40 participants and an unmoderated study with 107 crowd workers using a spreadsheet-like explanation interface based on the SHAP framework. We observed what non-technical users do to form their mental models of global AI model behavior from local explanations and how their perception of understanding decreases when it is examined.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Tasks that involve inherent uncertainty are often those where humans tend to rely on AI systems for advice, such as loan approval [27, 34, 124], recidivism prediction [31, 47, 86], house price estimation [2, 13, 25], and student admission [13, 24].”
From Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making (2024) · cites this paper as [27]
In the citing paper
“However, recent research shows that people do not always engage with explainability tools enough to help improve decision making.”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [2]
In the citing paper
“Participants who do seem to believe that they understand explanatory information, leading to over-confidence [2], even when presented with explanations that contain no useful information [5].”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [2]
In the citing paper
“Some experimental studies [2,5,42] indicate that most participants do not cognitively engage with explainability tools.”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [2]
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
(2011) · Cited by 14,317
Shapley, L. (1988) · Cambridge University Press eBooks · doi:10.1017/cbo9780511528446.003 · Cited by 10,698
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
From local explanations to global understanding with explainable AI for trees
Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., & Lee, S.-I. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-019-0138-9 · Cited by 10,551
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Shortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224
Definitions, methods, and applications in interpretable machine learning
Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., & Yu, B. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1900654116 · Cited by 2,167 · cited 2×
“For an explanation to be effective, it does not only need to have a sufficient level of fidelity but must "provide insight for a particular audience into a chosen domain problem" [36].”
Background(2013) · Cited by 1,951
The misunderstood limits of folk science: An illusion of explanatory depth
Rozenblit, L., & Keil, F. (2002) · Cognitive Science · doi:10.1207/s15516709cog2605_1 · Cited by 1,032
Designing Theory-Driven User-Centric Explainable AI
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300831 · Cited by 906
Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods
Slack, D., Hilgard, S., Jia, E., Singh, S., & Lakkaraju, H. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375830 · Cited by 900
Abdul, A., Vermeulen, J., Wang, D., Lim, B. Y., & Kankanhalli, M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174156 · Cited by 743
Explainable machine learning in deployment
Bhatt, U., Xiang, A., Sharma, S., Weller, A., Taly, A., Jia, Y., Ghosh, J., Puri, R., Moura, J. M. F., & Eckersley, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3375624 · Cited by 633
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629
Cai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594
Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376219 · Cited by 564
Metrics for explainable AI: challenges and prospects
Hoffman, R. R., Mueller, S. T., Klein, G., & Litman, J. (2018) · arXiv · doi:10.48550/arxiv.1812.04608 · Cited by 518
A systematic review and taxonomy of explanations in decision support and recommender systems
Nunes, I., & Jannach, D. (2017) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-017-9195-0 · Cited by 416
The Search for Simplicity: A Fundamental Cognitive Principle?
Chater, N. (1999) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/713755819 · Cited by 336
Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems
Buçinca, Z., Lin, P., Gajos, K. Z., & Glassman, E. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377498 · Cited by 307
Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders
Cheng, H.-F., Wang, R., Zhang, Z., O'Connell, F., Gray, T., Harper, F. M., & Zhu, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300789 · Cited by 292 · cited 4×
“Prior work reported that providing users with interactive explanation facilities improved their subjective and objective model understanding [12].”
BackgroundMethodInfluentialSokol, K., & Flach, P. A. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3372870 · Cited by 278
The challenge of crafting intelligible intelligence
Weld, D. S., & Bansal, G. (2019) · Communications of the ACM · doi:10.1145/3282486 · Cited by 253
Mueller, S. T., Hoffman, R. R., Clancey, W., Emrey, A., & Klein, G. (2019) · arXiv · doi:10.48550/arxiv.1902.01876 · Cited by 215
McGuirl, J. M., & Sarter, N. B. (2006) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872006779166334 · Cited by 212
Automated rationale generation: a technique for explainable AI and its effects on human perceptions
Ehsan, U., Tambwekar, P., Chan, L., Harrison, B., & Riedl, M. O. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302316 · Cited by 210
Mills, C. M., & Keil, F. C. (2003) · Journal of Experimental Child Psychology · doi:10.1016/j.jecp.2003.09.003 · Cited by 187
Evaluating saliency map explanations for convolutional neural networks: a user study
Alqaraawi, A., Schuessler, M., Weiß, P., Costanza, E., & Berthouze, N. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377519 · Cited by 163
Missing the trees for the forest: A construal level account of the illusion of explanatory depth.
Alter, A. L., Oppenheimer, D. M., & Zemla, J. C. (2010) · Journal of Personality and Social Psychology · doi:10.1037/a0020218 · Cited by 158
Narayanan, M., Chen, E., He, J., Kim, B., Gershman, S., & Doshi-Velez, F. (2018) · arXiv · doi:10.48550/arxiv.1802.00682 · Cited by 153
Scharrer, L., Rupieper, Y., Stadtler, M., & Bromme, R. (2016) · Public Understanding of Science · doi:10.1177/0963662516680311 · Cited by 142
Can we do better explanations? A proposal of user-centered explainable AI
Ribera, M., & Lapedriza, À. (2019) · IUI Workshops · Cited by 134
Automation Bias and Errors: Are Crews Better Than Individuals?
Skitka, L. J., Mosier, K. L., Burdick, M., & Rosenblatt, B. (2000) · The International Journal of Aviation Psychology · doi:10.1207/s15327108ijap1001_5 · Cited by 127
Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems
Tomsett, R., Braines, D., Harborne, D., Preece, A. D., & Chakraborty, S. (2018) · ORCA Online Research @Cardiff (Cardiff University) · doi:10.48550/arxiv.1806.07552 · Cited by 127
I can do better than your AI: expertise and explanations
Schaffer, J., O'Donovan, J., Michaelis, J., Raglin, A., & Höllerer, T. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302308 · Cited by 119
Countering the anchoring and adjustment bias with decision support systems
George, J. F., Duffy, K., & Ahuja, M. K. (2000) · Decision Support Systems · doi:10.1016/s0167-9236(00)00074-9 · Cited by 117
The science of cycology: Failures to understand how everyday objects work
Lawson, R. (2006) · Memory & Cognition · doi:10.3758/bf03195929 · Cited by 85
The Pragmatic Turn in Explainable Artificial Intelligence (XAI)
Páez, A. (2019) · PhilPapers (PhilPapers Foundation) · Cited by 80
Generating Counterfactual and Contrastive Explanations using SHAPPreprint
Rathi, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.09293 · Cited by 76
A Survey of Evaluation Methods and Measures for Interpretable Machine LearningPreprint
Mohseni, S., Zarei, N., & Ragan, E. D. (2018) · arXiv (Cornell University) · Cited by 69
Understanding the role of computer-mediated counter-argument in countering confirmation bias
Huang, H.-H., Hsu, J. S.-C., & Ku, C.-Y. (2012) · Decision Support Systems · doi:10.1016/j.dss.2012.03.009 · Cited by 58
Lau, A. Y. S., & Coiera, E. W. (2008) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2557 · Cited by 53
Leveraging rationales to improve human task performance
Das, D., & Chernova, S. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377512 · Cited by 52
The effects of mixing machine learning and human judgment
Vaccaro, M., & Waldo, J. (2019) · Communications of the ACM · doi:10.1145/3359338 · Cited by 49
Investigating the Importance of First Impressions and Explainable AI with Interactive Video Analysis
Nourani, M., Honeycutt, D. R., Block, J. E., Roy, C., Rahman, T., Ragan, E. D., & Gogate, V. (2020) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3334480.3382967 · Cited by 36
Do Not Trust Additive Explanations
Gosiewska, A., & Biecek, P. (2019) · arXiv · doi:10.48550/arxiv.1903.11420 · Cited by 34
Customization bias in decision support systems
Solomon, J. (2014) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557211 · Cited by 20
Considerations on explainable AI and users’ mental models
Rutjes, H., Willemsen, M. C., & IJsselsteijn, W. A. (2019) · TU/e Research Portal · Cited by 18
The Illusion of Explanatory Depth in a Misunderstood Field: The IOED in Mental Disorders.
Zeveney, A. S., & Marsh, J. K. (2016) · eScholarship (California Digital Library) · Cited by 17
Instance-Level Explanations for Fraud Detection: A Case StudyPreprint
Collaris, D., Vink, L. M., & van Wijk, J. J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07129 · Cited by 17
reSHAPe: A Framework for Interactive Explanations in XAI Based on SHAP
Chromik, M. (2020) · Gesellschaft für Informatik (GI) · doi:10.18420/ecscw2020_p06 · Cited by 9
Self-citation
Research works printed but not linked: 5
- Or Biran and Courtenay Cotton . 2017 . Explanation and justification in machine learning: A survey . In IJCAI-17 workshop on explainable AI (XAI), Vol. 8. 1. Or Biran and Courtenay Cotton. 2017. Explanation and justification in machine learning: A survey. In IJCAI-17 workshop on explainable AI (XAI), Vol. 8. 1.
- Adrian Bussone , S. Stumpf , and D. O’Sullivan . 2015 . The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems. 2015 International Conference on Healthcare Informatics ( 2015 ), 160–169. Adrian Bussone, S. Stumpf, and D. O’Sullivan. 2015. The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems. 2015 International Conference on Healthcare Informatics (2015), 160–169.
- Finale Doshi-Velez and Been Kim. 2017. Towards A Rigorous Science of Interpretability. CoRR abs/1702.08608(2017). arxiv:1702.08608http://arxiv.org/abs/1702.08608 Finale Doshi-Velez and Been Kim. 2017. Towards A Rigorous Science of Interpretability. CoRR abs/1702.08608(2017). arxiv:1702.08608http://arxiv.org/abs/1702.08608 · arXiv
- Tae-Nyun Kim and Hayeon Song . 2020 . The Effect of Message Framing and Timing on the Acceptance of Artificial Intelligence’s Suggestion. Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (2020). Tae-Nyun Kim and Hayeon Song. 2020. The Effect of Message Framing and Timing on the Acceptance of Artificial Intelligence’s Suggestion. Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (2020).
- Gregory L. Murphy and Douglas L. Medin . 1985. The role of theories in conceptual coherence.Psychological review 92 3 ( 1985 ), 289–316. Gregory L. Murphy and Douglas L. Medin. 1985. The role of theories in conceptual coherence.Psychological review 92 3 (1985), 289–316.
Cited by
Sorted by citations
Artificial intelligence and illusions of understanding in scientific research
Messeri, L., & Crockett, M. J. (2024) · Nature · doi:10.1038/s41586-024-07146-0 · Cited by 707
“Recent work suggests that this illusion inflates user understandings of AI systems 109 .”
Towards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3331846 · Cited by 278
“Similar findings have been made with respect to objective and subjective understanding [12, 35, 40].”
BackgroundMethodResultInfluentialHaque, A. B., Islam, A. K. M. N., & Mikalef, P. (2022) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2022.122120 · Cited by 314
Survey on Explainable AI: From Approaches, Limitations and Applications Aspects
Yang, W., Wei, Y.-C., Wei, H., Chen, Y., Huang, G., Li, X., Li, R., Yao, N., Wang, X., Gu, X., et al. (2023) · Human-Centric Intelligent Systems · doi:10.1007/s44230-023-00038-y · Cited by 303
Lai, V., Chen, C., Smith-Renner, A., Liao, Q. V., & Tan, C. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594087 · Cited by 204
“The most commonly used metric is forward simulation [1, 2, 15, 21, 24, 36, 48, 80, 81, 83, 87, 105, 111], by asking participants to simulate a model’s predictions on unseen instances. and what design impacts the perception is an active research area.”
BackgroundInfluential
Metadata
- Authors
- Chromik, M., Eiband, M., Buchner, F., Krüger, A., & Butz, A.
- Affiliations
- Ludwig-Maximilians-Universität München (DE)
- Venue
- 26th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2021-04-14
- Pages
- 307-317
- OpenAlex
- W3156666346
- Cited by
- 144
- References
- 57
- Citations by year
- 2026: 26 · 2025: 31 · 2024: 38 · 2023: 30 · 2022: 16 · 2021: 3
- FWCI
- 9.96
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- explainable AI, illusion of explanatory depth, local explanations, SHAP, non-technical users, mental models, interpretability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
