The DOI matches its Crossref record; 55 of the 57 references it cites were found in a registry or an index.
I Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI
Abstract
Unintended consequences of deployed AI systems fueled the call for more interpretability in AI systems. Often explainable AI (XAI) systems provide users with simplifying local explanations for individual predictions but leave it up to them to construct a global understanding of the model behavior. In this work, we examine if non-technical users of XAI fall for an illusion of explanatory depth when interpreting additive local explanations. We applied a mixed methods approach consisting of a moderated study with 40 participants and an unmoderated study with 107 crowd workers using a spreadsheet-like explanation interface based on the SHAP framework. We observed what non-technical users do to form their mental models of global AI model behavior from local explanations and how their perception of understanding decreases when it is examined.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Tasks that involve inherent uncertainty are often those where humans tend to rely on AI systems for advice, such as loan approval [27, 34, 124], recidivism prediction [31, 47, 86], house price estimation [2, 13, 25], and student admission [13, 24].”
From Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making (2024) · cites this paper as [27]
In the citing paper
“However, recent research shows that people do not always engage with explainability tools enough to help improve decision making.”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [2]
In the citing paper
“Participants who do seem to believe that they understand explanatory information, leading to over-confidence [2], even when presented with explanations that contain no useful information [5].”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [2]
In the citing paper
“Some experimental studies [2,5,42] indicate that most participants do not cognitively engage with explainability tools.”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [2]
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
(2011) · Cited by 14,317
Shapley, L. (1988) · Cambridge University Press eBooks · doi:10.1017/cbo9780511528446.003 · Cited by 10,698
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
From local explanations to global understanding with explainable AI for trees
Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., & Lee, S.-I. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-019-0138-9 · Cited by 10,551
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Shortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224
Definitions, methods, and applications in interpretable machine learning
Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., & Yu, B. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1900654116 · Cited by 2,167 · cited 2×
“For an explanation to be effective, it does not only need to have a sufficient level of fidelity but must "provide insight for a particular audience into a chosen domain problem" [36].”
Background
Research works printed but not linked: 5
- Or Biran and Courtenay Cotton . 2017 . Explanation and justification in machine learning: A survey . In IJCAI-17 workshop on explainable AI (XAI), Vol. 8. 1. Or Biran and Courtenay Cotton. 2017. Explanation and justification in machine learning: A survey. In IJCAI-17 workshop on explainable AI (XAI), Vol. 8. 1.
- Adrian Bussone , S. Stumpf , and D. O’Sullivan . 2015 . The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems. 2015 International Conference on Healthcare Informatics ( 2015 ), 160–169. Adrian Bussone, S. Stumpf, and D. O’Sullivan. 2015. The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems. 2015 International Conference on Healthcare Informatics (2015), 160–169.
- Finale Doshi-Velez and Been Kim. 2017. Towards A Rigorous Science of Interpretability. CoRR abs/1702.08608(2017). arxiv:1702.08608http://arxiv.org/abs/1702.08608 Finale Doshi-Velez and Been Kim. 2017. Towards A Rigorous Science of Interpretability. CoRR abs/1702.08608(2017). arxiv:1702.08608http://arxiv.org/abs/1702.08608 · arXiv
- Tae-Nyun Kim and Hayeon Song . 2020 . The Effect of Message Framing and Timing on the Acceptance of Artificial Intelligence’s Suggestion. Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (2020). Tae-Nyun Kim and Hayeon Song. 2020. The Effect of Message Framing and Timing on the Acceptance of Artificial Intelligence’s Suggestion. Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (2020).
- Gregory L. Murphy and Douglas L. Medin . 1985. The role of theories in conceptual coherence.Psychological review 92 3 ( 1985 ), 289–316. Gregory L. Murphy and Douglas L. Medin. 1985. The role of theories in conceptual coherence.Psychological review 92 3 (1985), 289–316.
Cited by
Sorted by citations
Artificial intelligence and illusions of understanding in scientific research
Messeri, L., & Crockett, M. J. (2024) · Nature · doi:10.1038/s41586-024-07146-0 · Cited by 707
“Recent work suggests that this illusion inflates user understandings of AI systems 109 .”
Towards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3331846 · Cited by 278
“Similar findings have been made with respect to objective and subjective understanding [12, 35, 40].”
BackgroundMethodResultInfluentialHaque, A. B., Islam, A. K. M. N., & Mikalef, P. (2022) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2022.122120 · Cited by 314
Survey on Explainable AI: From Approaches, Limitations and Applications Aspects
Yang, W., Wei, Y.-C., Wei, H., Chen, Y., Huang, G., Li, X., Li, R., Yao, N., Wang, X., Gu, X., et al. (2023) · Human-Centric Intelligent Systems · doi:10.1007/s44230-023-00038-y · Cited by 303
Lai, V., Chen, C., Smith-Renner, A., Liao, Q. V., & Tan, C. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594087 · Cited by 204
“The most commonly used metric is forward simulation [1, 2, 15, 21, 24, 36, 48, 80, 81, 83, 87, 105, 111], by asking participants to simulate a model’s predictions on unseen instances. and what design impacts the perception is an active research area.”
BackgroundInfluentialMiller, T. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594001 · Cited by 172
“Some experimental studies [Chromik et al., 2021, Eiband et al., 2019, Ghai et al., 2021, Vera Liao and Varshney, 2021] indicate that most participants do not cognitively engage with explainability tools.”
BackgroundInfluentialGeneral Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models
Molnar, C., König, G., Herbinger, J., Freiesleben, T., Dandl, S., Scholbeck, C. A., Casalicchio, G., Grosse-Wentrup, M., & Bischl, B. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-04083-2_4 · Cited by 211
Mathew, D. E., Ebem, D. U., Ikegwu, A. C., Ukeoma, P. E., & Dibiaezue, N. F. (2025) · Neural Processing Letters · doi:10.1007/s11063-025-11732-2 · Cited by 168
“Some works have provided explanatory depth on XAI; Chromik et al. [30] stated that explanatory depth shows how deeply an AI system can explain its decisions, going beyond seeming or shallow explanations and the concept of explanatory depth is vital because not all AI explanations are equally…”
How Cognitive Biases Affect XAI-assisted Decision-making: A Systematic Review
Bertrand, A., Belloum, R., Eagan, J. R., & Maxwell, W. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534164 · Cited by 129
“[10] using local feature importance (SHAP [1]) explanations.”
BackgroundMethodSivaraman, V., Bukowski, L. A., Levin, J., Kahn, J. M., & Perer, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581075 · Cited by 141
“[12] to more recent work [20, 85, 90] has shown that these explanations tend to increase trust in the model even when it is unwarranted.”
BackgroundThe Who in XAI: How AI Background Shapes Perceptions of AI Explanations
Ehsan, U., Passi, S., Liao, Q. V., Chan, L., Lee, I.-H., Muller, M., & Riedl, M. O. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642474 · Cited by 123
“Negative consequences of cognitive biases such as over-trust in XAI could be attributed to a System 1 heuristic of associating explanations with AI competence [29, 124].”
BackgroundPlant Genotype to Phenotype Prediction Using Machine Learning
Danilevicz, M. F., Gill, M., Anderson, R., Batley, J., Bennamoun, M., Bayer, P. E., & Edwards, D. (2022) · Frontiers in Genetics · doi:10.3389/fgene.2022.822173 · Cited by 133
“…to the original criticism of describing the extraction of meaning post-hoc from the black box as a practice with potential bias [57], the ability to purposely engineer explanations (Slack et al., 2020) and the likelihood of false conclusions being made by inexperienced users (Chromik et al., 2021).”
BackgroundDikmen, M., & Burns, C. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102792 · Cited by 116
Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems
He, G., Kuiper, L., & Gadiraju, U. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581025 · Cited by 107
“[11] compared perceived understanding of XAI methods with their actual understanding to reveal users’ illusion of explanatory depth.”
BackgroundFok, R., & Weld, D. S. (2024) · AI Magazine · doi:10.1002/aaai.12182 · Cited by 81
Effects of Explanations in AI-Assisted Decision Making: Principles and Comparisons
Wang, X., & Yin, M. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3519266 · Cited by 74
Nannini, L., Balayn, A., & Smith, A. L. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594074 · Cited by 101
Toward involving end-users in interactive human-in-the-loop AI fairness
Nakao, Y., Stumpf, S., Ahmed, S., Naseer, A., & Strappelli, L. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3514258 · Cited by 69
“Some work has warned that explanations can be misleading and can influence users to over-rely on AI decisions [15, 20].”
Bove, C., Aigrain, J., Lesot, M., Tijus, C., & Detyniecki, M. (2022) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3490099.3511139 · Cited by 68
Medical artificial intelligence for clinicians: the lost cognitive perspective
Tikhomirov, L., Semmler, C., McCradden, M. D., Searston, R. A., Ghassemi, M., & Oakden‐Rayner, L. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00095-5 · Cited by 69
Verma, H., Mlynář, J., Schaer, R., Reichenbach, J., Jreige, M., Prior, J. O., Évéquoz, F., & Depeursinge, A. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581506 · Cited by 70
“, [15, 19, 52, 59, 88]), and arguments have been made to operationalize holistic explainability of AI [74] and foster interactive attainment of explanations [65], which are aligned with the context of use, involved stakeholders, organizational values, and industrial standards.”
BackgroundSubramanian, H. V., Canfield, C., & Shank, D. B. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102780 · Cited by 73
Bertrand, A., Viard, T., Belloum, R., Eagan, J. R., & Maxwell, W. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581314 · Cited by 71
Fok, R., & Weld, D. S. (2023) · arXiv · doi:10.48550/arxiv.2305.07722 · Cited by 54
“One hypothesis for these negative results suggests decision makers do not actually cognitively engage with AI explanations [14, 22, 25, 28], resulting in diminished utility of an explanation regardless of its type or quality.”
BackgroundDefining Explanation and Explanatory Depth in XAI
Buijsman, S. (2022) · Minds and Machines · doi:10.1007/s11023-022-09607-9 · Cited by 58
“Furthermore, Chromik et al. (2021) found that users generalize from a collection of (contrasting, so including counterfactuals) local (Shapley) explanations, and typically do this incorrectly.”
BackgroundAl-Ansari, N., Al‐Thani, D., & Al-Mansoori, R. S. (2024) · Human Behavior and Emerging Technologies · doi:10.1155/2024/4628855 · Cited by 58
Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant
He, G., Aishwarya, N., & Gadiraju, U. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712133 · Cited by 87
“However, the increase in trust and reliance may potentially stem from an illusion of their understanding of explanatory depth [20, 96].”
Eisbach, S., Langer, M., & Hertel, G. (2023) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2023.100015 · Cited by 46
Understanding plant phenotypes in crop breeding through explainable AI
Danilevicz, M. F., Upadhyaya, S. R., Batley, J., Bennamoun, M., Bayer, P. E., & Edwards, D. (2025) · Plant Biotechnology Journal · doi:10.1111/pbi.70208 · Cited by 51
“…machine learning prediction to avoid misleading insights, as studies show that non-technical users and even data scientists over-trust model predictions and XAI explanations, missing the opportunity to use XAI to effectively uncover data or model issues (Chromik et al ., 2021; Kaur et al ., 2020).”
A Missing Piece in the Puzzle: Considering the Role of Task Complexity in Human-AI Decision Making
Salimzadeh, S., He, G., & Gadiraju, U. (2023) · Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization · doi:10.1145/3565472.3592959 · Cited by 42
“For example, recidivism prediction algorithms have been used to help judges determine whether defendants are likely to re-offend [26, 31, 65, 69], medical diseases are being diagnosed with AI systems [19, 57, 62], and loan risk prediction algorithms are employed to approve or reject loan applications [11, 25, 38, 95].”
MethodSalimzadeh, S., He, G., & Gadiraju, U. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641905 · Cited by 41
“Tasks that involve inherent uncertainty are often those where humans tend to rely on AI systems for advice, such as loan approval [27, 34, 124], recidivism prediction [31, 47, 86], house price estimation [2, 13, 25], and student admission [13, 24].”
He, G., Buijsman, S., & Gadiraju, U. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610067 · Cited by 56
C-XAI: A conceptual framework for designing XAI tools that support trust calibration
Naiseh, M., Simkute, A., Zieni, B., Jiang, N., & Ali, R. (2024) · Journal of Responsible Technology · doi:10.1016/j.jrt.2024.100076 · Cited by 46
“To assist this process, C-XAI proposes four behavioural metrics proposed in the literature to measure trust calibration risks during human-AI task e.g., (Buçinca, Malaya & Gajos, 2021; Chromik, Eiband, Buchner, Krüger & Butz, 2021), summarised in Table 4.”
MethodTransparency for AI systems: a value-based approach
Buijsman, S. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09770-w · Cited by 44
“…sense that the reasons for individual outputs are understood by users), and in fact the academic literature has so far found only marginal improvements in objective measures such as the ability to predict outputs or to make correct decisions (Chromik et al., 2021; Wang & Yin, 2021) based on SHAP.”
Intelligent systems in healthcare: A systematic survey of explainable user interfaces
Cálem, J., Moreira, C., & Jorge, J. A. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108908 · Cited by 32
Why should I trust you? Influence of explanation design on consumer behavior in AI-based services
Nizette, F., Hammedi, W., van Riel, A. C. R., & Steils, N. (2024) · Journal of service management · doi:10.1108/josm-05-2024-0223 · Cited by 36
Supporting High-Uncertainty Decisions through AI and Logic-Style Explanations
Cau, F. M., Hauptmann, H., Spano, L. D., & Tintarev, N. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581641.3584080 · Cited by 35
“The tabular representation of the SHAP explanations was inspired by SHAPTable [16] (see Fig.”
BackgroundMethodBoudanga, Z., Benhadou, S., & Medromi, H. (2023) · F1000Research · doi:10.12688/f1000research.138867.1 · Cited by 41
“49,50 53 For digital assistants, there are applications of XAI-based interactive agents that are trustworthy and more user-centered.”
Toward Human-centered XAI in Practice: A survey
Kong, X., Liu, S., & Zhu, L. (2024) · Machine Intelligence Research · doi:10.1007/s11633-022-1407-3 · Cited by 35
Villareale, J., Harteveld, C., & Zhu, J. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3549482 · Cited by 30
“Studies have primarily focused on how to influence mental model accuracy [7, 37, 38].”
BackgroundHow can Explainability Methods be Used to Support Bug Identification in Computer Vision Models?
Balayn, A., Rikalo, N., Lofi, C., Yang, J., & Bozzon, A. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517474 · Cited by 26
“Researchers have conducted user-studies on the use of certain explanations for certain stakeholders and data types [2, 12, 13, 23, 25, 52].”
BackgroundHe, G., Demartini, G., & Gadiraju, U. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713218 · Cited by 23
“Another example is that users can develop an illusion of explanatory depth brought by explainable AI techniques [10], which leads to uncalibrated trust in the AI system.”
Bove, C., Lesot, M., Tijus, C., & Detyniecki, M. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581641.3584082 · Cited by 23
“We conduct the user study in a lab setting at INSEAD-Sorbonne University Behavioural Lab, as it has been shown that the presence of a moderator increases participants’ focus [7].”
MethodOkolo, C. T., Agarwal, D., Dell, N. L., & Vashistha, A. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637348 · Cited by 28
“Such methods could help prevent users from forming false beliefs about explained ML predictions [24] and provide these users with the agency to actively challenge these systems.”
How Do HCI Researchers Study Cognitive Biases? A Scoping Review
Boonprakong, N., Tag, B., Gonçalves, J., & Dingler, T. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713450 · Cited by 37
“Meanwhile, a limited number of papers study the effects of cognitive biases by using qualitative (2 articles) [61, 173] or mixed methods (6 articles) [37, 43, 50, 68, 134, 144].”
A scoping review of silent trials for medical artificial intelligence
Tikhomirov, L., Semmler, C., Prizant, N., Bhasin, S., Kenyon, G., van der Vegt, A., Erdman, L., Kurian, N. C., Thompson, H., Palmer, L. J., et al. (2026) · Nature Health · doi:10.1038/s44360-025-00048-z · Cited by 17
Responsible AI-Based Business Process Management and Improvement
Pisoni, G., & Moloney, M. (2024) · Digital Society · doi:10.1007/s44206-024-00105-2 · Cited by 18
“…users, that is developers of AI solutions and business experts, the need to also teach business users regarding explainability and how to “consume” explanations generated, risks involved, and common ways in which the explanations will be used (Chromik et al., 2021; Felländer et al., 2022 ).”
Do Expressions Change Decisions? Exploring the Impact of AI's Explanation Tone on Decision-Making
Okoso, A., Yang, M., & Baba, Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713744 · Cited by 26
“Prediction-related information includes uncertainty of the predictions [25, 39], local feature importance [1, 14], rule-based explanations [38, 40], and example-based explanations [8, 11].”
Deshpande, N. M., Gite, S., Pradhan, B., & Assiri, M. E. (2022) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2022.021225 · Cited by 22
(Explainable) Artificial Intelligence in Aerospace Safety-Critical Systems
Sutthithatip, S., Perinpanayagam, S., & Aslam, S. (2022) · IEEE Aerospace Conference (AERO) · doi:10.1109/aero53065.2022.9843612 · Cited by 18
Jung, J., Kang, S., Choi, J., El-Kareh, R. E., Lee, H., & Kim, H. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.105943 · Cited by 19
The Who in Explainable AI: How AI Background Shapes Perceptions of AI ExplanationsPreprint
Ehsan, U., Passi, S., Liao, Q. V., Chan, L., Lee, I.-H., Muller, M. J., & Riedl, M. O. (2021) · arXiv (Cornell University) · Cited by 17
Piscitelli, M. S., Razzano, G., Buscemi, G., & Capozzoli, A. (2024) · Energy and Buildings · doi:10.1016/j.enbuild.2024.115115 · Cited by 16
“In the medical domain [38,39], education [40–42], transportation [43,44], and finance [45,46], XAI applications demonstrate their broad applicability.”
Belaid, M. K., Bornemann, R., Rabus, M., Krestel, R., & Hüllermeier, E. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-44067-0_5 · Cited by 16
Valdez, A. S. C., Heine, M., Franke, T., Jochems, N., Jetter, H.-C., & Schrills, T. (2024) · i-com · doi:10.1515/icom-2024-0014 · Cited by 20
Framework for human–XAI symbiosis: extended self from the dual-process theory perspective
Litvinova, Y., Mikalef, P., & Luo, X. R. (2024) · Journal of Business Analytics · doi:10.1080/2573234x.2024.2396366 · Cited by 16
“However, the effect of the XAI explanation provided can differ depending on explanation type (Buçinca et al., 2020, March; Lai & Tan, 2019).”
InfluentialInherently Interpretable Machine Learning: A Contrasting Paradigm to Post-hoc Explainable AI
Zschech, P., Weinzierl, S., & Kraus, M. (2025) · Business & Information Systems Engineering · doi:10.1007/s12599-025-00964-0 · Cited by 24
Information that matters: Exploring information needs of people affected by algorithmic decisions
Schmude, T., Koesten, L. M., Möller, T. B., & Tschiatschek, S. (2024) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2024.103380 · Cited by 17
“In consequence, self-reports of understanding can be unreliable and prone to the "illusion of explanatory depth" [23, 80]: an overestimation of one’s understanding.”
BackgroundMethodKnowing (Not) to Know: Explainable Artificial Intelligence and Human Metacognition
von Zahn, M., Liebich, L., Jussupow, E., Hinz, O. F., & Bauer, K. (2025) · Information Systems Research · doi:10.1287/isre.2024.1431 · Cited by 21
The SAGE Framework for Explaining Context in Explainable Artificial Intelligence
Mill, E., Garn, W., Ryman-Tubb, N. F., & Turner, C. J. (2024) · Applied Artificial Intelligence · doi:10.1080/08839514.2024.2318670 · Cited by 14
“However, a recent study of non-technical XAI users provides an interesting perspective on how an audience’s mental model can change (Chromik et al. 2021).”
Eisbach, S., Mai, O., & Hertel, G. (2024) · New Ideas in Psychology · doi:10.1016/j.newideapsych.2024.101077 · Cited by 12
Transparency and Explainability for Public Policy
Vredenburgh, K. (2024) · LSE Public Policy Review · doi:10.31389/lseppr.111 · Cited by 12
Debating with More Persuasive LLMs Leads to More Truthful Answers
Khan, A., Hughes, J., Valentine, D., Ruis, L., Sachan, K., Radhakrishnan, A., Grefenstette, E., Bowman, S. R., Rocktäschel, T., & Perez, E. (2024) · arXiv · doi:10.48550/arxiv.2402.06782 · Cited by 13
Lodovico, C. D., Torrielli, F., Di, L., & Rapp, A. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2491009 · Cited by 13
“In response to this challenge, a growing body of research in the field of explainable AI (XAI) focuses on the development of strategies and techniques that enhance the transparency and comprehensibility of AI systems’ behavior (Chromik et al., 2021; Rozenblit & Keil, 2002).”
InfluentialMaking SHAP Rap: Bridging Local and Global Insights Through Interaction and Narratives
Chromik, M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-85616-8_37 · Cited by 12
“Then, they rated their level of understanding on a 7-point scale(4) [8].”
BackgroundSelf-citationSalimzadeh, S., & Gadiraju, U. (2024) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3627043.3659567 · Cited by 14
“…increasing capabilities of AI systems to perform tasks with high accuracy have led to increasing interest in incorporating these systems into human decision-making processes across various fields, such as finance [24, 32, 42, 44], healthcare [37, 62, 65, 84], and the legal domain [6, 71, 76, 106].”
BackgroundExplaining AI Decisions: Towards Achieving Human-Centered Explainability in Smart Home Environments
Shajalal, M., Boden, A., Stevens, G., Du, D. K., & Kern, D.-R. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-63803-9_23 · Cited by 15
“Chromik et al. [50] investigated the capability of non-expert users to understand and form a mental model of global explanations to comprehend the behavior of a model.”
BackgroundXuan, Y., Small, E. A., Sokol, K., Hettiachchi, D., & Sanderson, M. (2024) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2024.103376 · Cited by 17
“A similar finding was observed by Chromik et al. [13] who indicated that the limitations of explanations is a more challenging task than just attempting to understand explicated information, in which case user confidence is more likely to be poorly calibrated.”
BackgroundResultInfluentialBrachman, M., Pan, Q., Do, H. J., Dugan, C., Chaudhary, A., Johnson, J. M., Rai, P., Chakraborti, T., Gschwind, T., Laredo, J. A., et al. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584088 · Cited by 11
“Explanations for a task assignment process improved users’ mental models [105], but non-technical users may overestimate the correctness of their mental models [28].”
BackgroundWalter, Y. (2023) · AI and Ethics · doi:10.1007/s43681-023-00276-7 · Cited by 14
“This is at least part of the reason why many players have refused to give the public access to these applications at the beginning (cf. [21, 84]).”
The Centers and Margins of Modeling Humans in Well-being Technologies
Zhu, J., Sanches, P., Tsaknaki, V., van der Maden, W., & Kaklopoulou, I. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713940 · Cited by 13
“For example, constructs such as mental models [16, 31, 76, 136], usability and learnability [11, 48, 78, 122], and human factors [14, 140] have all found renewed interest in recent years.”
Balayn, A., Rikalo, N., Yang, J., & Bozzon, A. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581555 · Cited by 13
“Researchers have conducted user-studies around explanations for certain stakeholders and data types [2, 24, 27, 51, 54, 127], but none involves computer vision failures.”
BackgroundFinding AI’s Faults with AAR/AI: An Empirical Study
Khanna, R., Dodge, J. E., Anderson, A. W., Dikkala, R., Irvine, J., Shureih, Z., Lam, K.-H., Matthews, C., Lin, Z., Kahng, M., et al. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3487065 · Cited by 10
On the Need for Collaborative Intelligence in Cybersecurity
Martin, T. (2022) · Electronics · doi:10.3390/electronics11132067 · Cited by 9
“Although this makes research easier (and more reproducible) it is potentially misleading [31] as there is only intuitive support for a link between interpretability and tree compactness / explanation length / etc.”
BackgroundCo-Designing with Users the Explanations for a Proactive Auto-Response Messaging Agent
Jain, P., Farzan, R., & Lee, A. J. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3604248 · Cited by 8
Classification Metrics for Image Explanations: Towards Building Reliable XAI-Evaluations
Frész, B., Lörcher, L., & Huber, M. F. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658537 · Cited by 9
“However, this is still an open problem despite various approaches to assess the properties of the saliency method, mainly due to the subjective nature of evaluations [28], the fallibility of user studies [13], and the different concepts used to evaluate such metrics [39].”
BackgroundResultDigital Psychology: Introducing a Conceptual Impact Model and the Future of Work
Walter, Y. (2024) · Trends in Psychology · doi:10.1007/s43076-024-00408-w · Cited by 16
SIDEs: Separating Idealization from Deceptive 'Explanations' in xAI
Sullivan, E. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658999 · Cited by 12
Explainable AI in Disease Diagnosis
Bedi, P., Thukral, A., & Dhiman, S. (2024) · Computational intelligence methods and applications · doi:10.1007/978-981-97-3705-5_5 · Cited by 7
Rosenbacke, R. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4773350 · Cited by 7
Delaunay, J., Galárraga, L. A., Largouët, C., & van Berkel, N. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711011 · Cited by 6
“These findings are in line with existing research [15, 30, 64, 77].”
Schäfer, U., Sipos, L., & Müller-Birn, C. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757451 · Cited by 10
Tooling for Developing Data-Driven Applications: Overview and Outlook
Weber, T., & Hußmann, H. (2022) · Mensch und Computer (MuC) · doi:10.1145/3543758.3543779 · Cited by 6
“the current point in time, the preferred target group seems to be end-users [13, 14, 20].”
BackgroundPredicting the need for XAI from high-granularity interaction data
de Santana, V. F., Fucs, A., Segura, V., de Moraes, D. B., & Cerqueira, R. (2023) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2023.103029 · Cited by 6
Schrills, T., Gruner, M., Peuscher, H., & Franke, T. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-35748-0_23 · Cited by 7
Exploring the impact of explainable AI and cognitive capabilities on users’ decisions
Cau, F. M., & Spano, L. D. (2025) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-025-09438-0 · Cited by 16
“This aligns with the findings of Chromik et al. (2021), suggesting that users might overestimate their understanding of local feature explanations due to the illusion of explanatory depth.”
InfluentialResilience Through Appropriation: Pilots’ View on Complex Decision Support
Zhang, Z. T., Storath, C., Liu, Y., & Butz, A. M. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581641.3584056 · Cited by 11
Self-citationUnderstanding User Preferences in Explainable Artificial Intelligence: A Mapping Function Proposal
Hashemi, M., Darejeh, A., & Cruz, F. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3733837 · Cited by 11
“Chromik et al. [29] conducted a study to understand how non-technical users form a global understanding of a model when provided with local explanations.”
Szymczyk, N., Szczuka, J. M., Krämer, N. C., & Wischnewski, M. (2025) · Mensch und Computer (MuC) · doi:10.1145/3743049.3743072 · Cited by 4
Bhattacharya, A., & Verbert, K. (2024) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3631700.3664911 · Cited by 4
Ferrell, B., Raskin, S. E., & Zimmerman, E. B. (2023) · JMIR Formative Research · doi:10.2196/41516 · Cited by 5
“This coincides with the aims of the study by Chromik et al [10], who considered how people without technical expertise attempt to understand a model’s behavior by creating mental models that reflect their belief about how a system works.”
BackgroundXAI4RE – Using Explainable AI for Responsible and Ethical AI
Tal, A. S., & Sheidin, J. (2025) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3708319.3733707 · Cited by 3
“…predetermined conclusions [9, 20, 23, 37, 38], non-technical users may develop an illusion of understanding when interpreting XAI explanations [7, 42]; XAI explanations may not accurately predict real-world performance [4]; the explanations can become manipulation tools in persuasive…”
Mehrotra, S., Gadiraju, U., Bittner, E. A. C., van Delden, F., Jonker, C. M., & Tielman, M. L. (2025) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3699682.3728343 · Cited by 6
“This lack of effect can be understood through the lens of the illusion of explanatory depth [14], a cognitive bias where individualsoverestimatetheirunderstanding ofcomplexsystems.”
Understanding the Effects of Explaining Predictive but Unintuitive Features in Human-XAI Interaction
Qu, J., Arguello, J., & Wang, Y. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732021 · Cited by 4
Erlei, A., Cau, F. M., Georgiev, R., Kumar, S. C., Bizer, K., & Gadiraju, U. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791420 · Cited by 7
“For tabular data, we selected the loan prediction problem dataset 10 as a testbed, given its utility in several previous works in AI-assisted decisions [19, 39, 48, 57, 59, 61, 63, 141].”
Understanding the Effects of AI-Assisted Critical Thinking on Human-AI Decision Making
Tian, H. Y., Amin, H., & Yin, M. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790785 · Cited by 10
Qu, J., Arguello, J., & Wang, Y. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658547 · Cited by 3
Pisoni, G., & Moloney, M. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4822711 · Cited by 3
Dataism, skepticism, and intuition for interpretable machine learning
Naser, M. Z. (2025) · AI and Ethics · doi:10.1007/s43681-025-00831-4 · Cited by 2
“For instance, if the underlying model remains opaque, the real question might be whether stakeholders can meaningfully contest or revise the model’s decisions or will be content with the illusions of transparency 2 [83].”
An Empirical Examination of the Evaluative AI Framework
Kornowicz, J. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2581260 · Cited by 2
“This has led to a situation where users often engage superficially with explanations and develop an overreliance on AI (Bansal et al., 2021; Bu¸cinca et al., 2021; Chen et al., 2023; Chromik et al., 2021), shifting from the original problem of un-derreliance.”
Metadata
- Authors
- Chromik, M., Eiband, M., Buchner, F., Krüger, A., & Butz, A.
- Affiliations
- Ludwig-Maximilians-Universität München (DE)
- Venue
- 26th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2021-04-14
- Pages
- 307-317
- OpenAlex
- W3156666346
- Cited by
- 144
- References
- 57
- Citations by year
- 2026: 26 · 2025: 31 · 2024: 38 · 2023: 30 · 2022: 16 · 2021: 3
- FWCI
- 9.96
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- explainable AI, illusion of explanatory depth, local explanations, SHAP, non-technical users, mental models, interpretability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
