I Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI
Abstract
Unintended consequences of deployed AI systems fueled the call for more interpretability in AI systems. Often explainable AI (XAI) systems provide users with simplifying local explanations for individual predictions but leave it up to them to construct a global understanding of the model behavior. In this work, we examine if non-technical users of XAI fall for an illusion of explanatory depth when interpreting additive local explanations. We applied a mixed methods approach consisting of a moderated study with 40 participants and an unmoderated study with 107 crowd workers using a spreadsheet-like explanation interface based on the SHAP framework. We observed what non-technical users do to form their mental models of global AI model behavior from local explanations and how their perception of understanding decreases when it is examined.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Tasks that involve inherent uncertainty are often those where humans tend to rely on AI systems for advice, such as loan approval, recidivism prediction, house price estimation, and student admission.
“Tasks that involve inherent uncertainty are often those where humans tend to rely on AI systems for advice, such as loan approval [27, 34, 124], recidivism prediction [31, 47, 86], house price estimation [2, 13, 25], and student admission [13, 24].”
From Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making (2024) · cites this paper as [27]
Recent research shows that people do not always engage with explainability tools enough to improve decision making.
“However, recent research shows that people do not always engage with explainability tools enough to help improve decision making.”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [2]
Participants who do engage with explainability tools seem to believe they understand explanatory information, leading to over-confidence, even when explanations contain no useful information.
“Participants who do seem to believe that they understand explanatory information, leading to over-confidence [2], even when presented with explanations that contain no useful information [5].”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [2]
Some experimental studies indicate that most participants do not cognitively engage with explainability tools.
“Some experimental studies [2,5,42] indicate that most participants do not cognitively engage with explainability tools.”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [2]
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
(2011) · Cited by 14,317 · see citing works
Shapley, L. (1988) · Cambridge University Press eBooks · doi:10.1017/cbo9780511528446.003 · Cited by 10,698 · see citing works
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · see citing works
From local explanations to global understanding with explainable AI for trees
Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., & Lee, S.-I. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-019-0138-9 · Cited by 10,551 · see citing works
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · see citing works
The mythos of model interpretability
Lipton, Z. C. (2018) · Communications of the ACM · doi:10.1145/3233231 · Cited by 5,427 · see citing works
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302 · see citing works
Shortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224 · see citing works
Definitions, methods, and applications in interpretable machine learning
Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., & Yu, B. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1900654116 · Cited by 2,167 · cited 2× · see citing works
“For an explanation to be effective, it does not only need to have a sufficient level of fidelity but must "provide insight for a particular audience into a chosen domain problem" [36].”
Background(2013) · Cited by 1,951 · see citing works
The misunderstood limits of folk science: An illusion of explanatory depth
Rozenblit, L., & Keil, F. (2002) · Cognitive Science · doi:10.1207/s15516709cog2605_1 · Cited by 1,032 · see citing works
Designing Theory-Driven User-Centric Explainable AI
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300831 · Cited by 906 · see citing works
Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods
Slack, D., Hilgard, S., Jia, E., Singh, S., & Lakkaraju, H. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375830 · Cited by 900 · see citing works
Abdul, A., Vermeulen, J., Wang, D., Lim, B. Y., & Kankanhalli, M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174156 · Cited by 743 · see citing works
Explainable machine learning in deployment
Bhatt, U., Xiang, A., Sharma, S., Weller, A., Taly, A., Jia, Y., Ghosh, J., Puri, R., Moura, J. M. F., & Eckersley, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3375624 · Cited by 633 · see citing works
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · see citing works
Cai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594 · see citing works
Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376219 · Cited by 564 · see citing works
Metrics for explainable AI: challenges and prospects
Hoffman, R. R., Mueller, S. T., Klein, G., & Litman, J. (2018) · arXiv · doi:10.48550/arxiv.1812.04608 · Cited by 518 · see citing works
A systematic review and taxonomy of explanations in decision support and recommender systems
Nunes, I., & Jannach, D. (2017) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-017-9195-0 · Cited by 416 · see citing works
The Search for Simplicity: A Fundamental Cognitive Principle?
Chater, N. (1999) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/713755819 · Cited by 327 · see citing works
Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems
Buçinca, Z., Lin, P., Gajos, K. Z., & Glassman, E. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377498 · Cited by 307 · see citing works
Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders
Cheng, H.-F., Wang, R., Zhang, Z., O'Connell, F., Gray, T., Harper, F. M., & Zhu, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300789 · Cited by 292 · cited 4× · see citing works
“Prior work reported that providing users with interactive explanation facilities improved their subjective and objective model understanding [12].”
BackgroundMethodInfluentialSokol, K., & Flach, P. A. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3372870 · Cited by 278 · see citing works
The challenge of crafting intelligible intelligence
Weld, D. S., & Bansal, G. (2019) · Communications of the ACM · doi:10.1145/3282486 · Cited by 253 · see citing works
Mueller, S. T., Hoffman, R. R., Clancey, W., Emrey, A., & Klein, G. (2019) · arXiv · doi:10.48550/arxiv.1902.01876 · Cited by 215 · see citing works
McGuirl, J. M., & Sarter, N. B. (2006) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872006779166334 · Cited by 212 · see citing works
Automated rationale generation: a technique for explainable AI and its effects on human perceptions
Ehsan, U., Tambwekar, P., Chan, L., Harrison, B., & Riedl, M. O. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302316 · Cited by 210 · see citing works
Mills, C. M., & Keil, F. C. (2003) · Journal of Experimental Child Psychology · doi:10.1016/j.jecp.2003.09.003 · Cited by 187 · see citing works
Evaluating saliency map explanations for convolutional neural networks: a user study
Alqaraawi, A., Schuessler, M., Weiß, P., Costanza, E., & Berthouze, N. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377519 · Cited by 163 · see citing works
Missing the trees for the forest: A construal level account of the illusion of explanatory depth.
Alter, A. L., Oppenheimer, D. M., & Zemla, J. C. (2010) · Journal of Personality and Social Psychology · doi:10.1037/a0020218 · Cited by 158 · see citing works
Narayanan, M., Chen, E., He, J., Kim, B., Gershman, S., & Doshi-Velez, F. (2018) · arXiv · doi:10.48550/arxiv.1802.00682 · Cited by 153 · see citing works
Scharrer, L., Rupieper, Y., Stadtler, M., & Bromme, R. (2016) · Public Understanding of Science · doi:10.1177/0963662516680311 · Cited by 142 · see citing works
Can we do better explanations? A proposal of user-centered explainable AI
Ribera, M., & Lapedriza, À. (2019) · IUI Workshops · Cited by 134 · see citing works
Automation Bias and Errors: Are Crews Better Than Individuals?
Skitka, L. J., Mosier, K. L., Burdick, M., & Rosenblatt, B. (2000) · The International Journal of Aviation Psychology · doi:10.1207/s15327108ijap1001_5 · Cited by 127 · see citing works
Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems
Tomsett, R., Braines, D., Harborne, D., Preece, A. D., & Chakraborty, S. (2018) · ORCA Online Research @Cardiff (Cardiff University) · doi:10.48550/arxiv.1806.07552 · Cited by 127 · see citing works
I can do better than your AI: expertise and explanations
Schaffer, J., O'Donovan, J., Michaelis, J., Raglin, A., & Höllerer, T. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302308 · Cited by 119 · see citing works
Countering the anchoring and adjustment bias with decision support systems
George, J. F., Duffy, K., & Ahuja, M. K. (2000) · Decision Support Systems · doi:10.1016/s0167-9236(00)00074-9 · Cited by 117 · see citing works
The Pragmatic Turn in Explainable Artificial Intelligence (XAI)
Páez, A. (2019) · PhilPapers (PhilPapers Foundation) · Cited by 80 · see citing works
Generating Counterfactual and Contrastive Explanations using SHAPPreprint
Rathi, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.09293 · Cited by 76 · see citing works
The science of cycology: Failures to understand how everyday objects work
Lawson, R. (2006) · Memory & Cognition · doi:10.3758/bf03195929 · Cited by 75 · see citing works
A Survey of Evaluation Methods and Measures for Interpretable Machine LearningPreprint
Mohseni, S., Zarei, N., & Ragan, E. D. (2018) · arXiv (Cornell University) · Cited by 69 · see citing works
Understanding the role of computer-mediated counter-argument in countering confirmation bias
Huang, H.-H., Hsu, J. S.-C., & Ku, C.-Y. (2012) · Decision Support Systems · doi:10.1016/j.dss.2012.03.009 · Cited by 58 · see citing works
Lau, A. Y. S., & Coiera, E. W. (2008) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2557 · Cited by 53 · see citing works
Leveraging rationales to improve human task performance
Das, D., & Chernova, S. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377512 · Cited by 52 · see citing works
The effects of mixing machine learning and human judgment
Vaccaro, M., & Waldo, J. (2019) · Communications of the ACM · doi:10.1145/3359338 · Cited by 49 · see citing works
Investigating the Importance of First Impressions and Explainable AI with Interactive Video Analysis
Nourani, M., Honeycutt, D. R., Block, J. E., Roy, C., Rahman, T., Ragan, E. D., & Gogate, V. (2020) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3334480.3382967 · Cited by 36 · see citing works
Do Not Trust Additive Explanations
Gosiewska, A., & Biecek, P. (2019) · arXiv · doi:10.48550/arxiv.1903.11420 · Cited by 34 · see citing works
Customization bias in decision support systems
Solomon, J. (2014) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557211 · Cited by 20 · see citing works
Considerations on explainable AI and users’ mental models
Rutjes, H., Willemsen, M. C., & IJsselsteijn, W. A. (2019) · TU/e Research Portal · Cited by 19 · see citing works
The Illusion of Explanatory Depth in a Misunderstood Field: The IOED in Mental Disorders.
Zeveney, A. S., & Marsh, J. K. (2016) · eScholarship (California Digital Library) · Cited by 17 · see citing works
Instance-Level Explanations for Fraud Detection: A Case StudyPreprint
Collaris, D., Vink, L. M., & van Wijk, J. J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07129 · Cited by 17 · see citing works
reSHAPe: A Framework for Interactive Explanations in XAI Based on SHAP
Chromik, M. (2020) · Gesellschaft für Informatik (GI) · doi:10.18420/ecscw2020_p06 · Cited by 9 · see citing works
Self-citation
Flagged first, then by citations
2 of the 57 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Artificial intelligence and illusions of understanding in scientific research
Messeri, L., & Crockett, M. J. (2024) · Nature · doi:10.1038/s41586-024-07146-0 · Cited by 707 · see citing works
Towards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3331846 · Cited by 278 · see citing works
Haque, A. B., Islam, A. K. M. N., & Mikalef, P. (2022) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2022.122120 · Cited by 280 · see citing works
Survey on Explainable AI: From Approaches, Limitations and Applications Aspects
Yang, W., Wei, Y.-C., Wei, H., Chen, Y., Huang, G., Li, X., Li, R., Yao, N., Wang, X., Gu, X., et al. (2023) · Human-Centric Intelligent Systems · doi:10.1007/s44230-023-00038-y · Cited by 302 · see citing works
Lai, V., Chen, C., Smith-Renner, A., Liao, Q. V., & Tan, C. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594087 · Cited by 204 · see citing works
Miller, T. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594001 · Cited by 172 · see citing works
General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models
Molnar, C., König, G., Herbinger, J., Freiesleben, T., Dandl, S., Scholbeck, C. A., Casalicchio, G., Grosse-Wentrup, M., & Bischl, B. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-04083-2_4 · Cited by 211 · see citing works
Mathew, D. E., Ebem, D. U., Ikegwu, A. C., Ukeoma, P. E., & Dibiaezue, N. F. (2025) · Neural Processing Letters · doi:10.1007/s11063-025-11732-2 · Cited by 168 · see citing works
How Cognitive Biases Affect XAI-assisted Decision-making: A Systematic Review
Bertrand, A., Belloum, R., Eagan, J. R., & Maxwell, W. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534164 · Cited by 129 · see citing works
Sivaraman, V., Bukowski, L. A., Levin, J., Kahn, J. M., & Perer, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581075 · Cited by 141 · see citing works
The Who in XAI: How AI Background Shapes Perceptions of AI Explanations
Ehsan, U., Passi, S., Liao, Q. V., Chan, L., Lee, I.-H., Muller, M., & Riedl, M. O. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642474 · Cited by 123 · see citing works
Plant Genotype to Phenotype Prediction Using Machine Learning
Danilevicz, M. F., Gill, M., Anderson, R., Batley, J., Bennamoun, M., Bayer, P. E., & Edwards, D. (2022) · Frontiers in Genetics · doi:10.3389/fgene.2022.822173 · Cited by 127 · see citing works
Dikmen, M., & Burns, C. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102792 · Cited by 116 · see citing works
Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems
He, G., Kuiper, L., & Gadiraju, U. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581025 · Cited by 107 · see citing works
“[11] compared perceived understanding of XAI methods with their actual understanding to reveal users’ illusion of explanatory depth.”
BackgroundEffects of Explanations in AI-Assisted Decision Making: Principles and Comparisons
Wang, X., & Yin, M. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3519266 · Cited by 82 · see citing works
Nannini, L., Balayn, A., & Smith, A. L. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594074 · Cited by 87 · see citing works
Toward involving end-users in interactive human-in-the-loop AI fairness
Nakao, Y., Stumpf, S., Ahmed, S., Naseer, A., & Strappelli, L. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3514258 · Cited by 69 · see citing works
Bove, C., Aigrain, J., Lesot, M., Tijus, C., & Detyniecki, M. (2022) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3490099.3511139 · Cited by 68 · see citing works
Medical artificial intelligence for clinicians: the lost cognitive perspective
Tikhomirov, L., Semmler, C., McCradden, M. D., Searston, R. A., Ghassemi, M., & Oakden‐Rayner, L. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00095-5 · Cited by 68 · see citing works
Verma, H., Mlynář, J., Schaer, R., Reichenbach, J., Jreige, M., Prior, J. O., Évéquoz, F., & Depeursinge, A. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581506 · Cited by 63 · see citing works
Subramanian, H. V., Canfield, C., & Shank, D. B. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102780 · Cited by 67 · see citing works
Bertrand, A., Viard, T., Belloum, R., Eagan, J. R., & Maxwell, W. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581314 · Cited by 59 · see citing works
Fok, R., & Weld, D. S. (2024) · AI Magazine · doi:10.1002/aaai.12182 · Cited by 81 · see citing works
Fok, R., & Weld, D. S. (2023) · arXiv · doi:10.48550/arxiv.2305.07722 · Cited by 54 · see citing works
Defining Explanation and Explanatory Depth in XAI
Buijsman, S. (2022) · Minds and Machines · doi:10.1007/s11023-022-09607-9 · Cited by 53 · see citing works
Al-Ansari, N., Al‐Thani, D., & Al-Mansoori, R. S. (2024) · Human Behavior and Emerging Technologies · doi:10.1155/2024/4628855 · Cited by 52 · see citing works
Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant
He, G., Aishwarya, N., & Gadiraju, U. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712133 · Cited by 53 · see citing works
Eisbach, S., Langer, M., & Hertel, G. (2023) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2023.100015 · Cited by 46 · see citing works
Understanding plant phenotypes in crop breeding through explainable AI
Danilevicz, M. F., Upadhyaya, S. R., Batley, J., Bennamoun, M., Bayer, P. E., & Edwards, D. (2025) · Plant Biotechnology Journal · doi:10.1111/pbi.70208 · Cited by 43 · see citing works
A Missing Piece in the Puzzle: Considering the Role of Task Complexity in Human-AI Decision Making
Salimzadeh, S., He, G., & Gadiraju, U. (2023) · Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization · doi:10.1145/3565472.3592959 · Cited by 42 · see citing works
Salimzadeh, S., He, G., & Gadiraju, U. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641905 · Cited by 41 · see citing works
“Tasks that involve inherent uncertainty are often those where humans tend to rely on AI systems for advice, such as loan approval [27, 34, 124], recidivism prediction [31, 47, 86], house price estimation [2, 13, 25], and student admission [13, 24].”
He, G., Buijsman, S., & Gadiraju, U. (2023) · Proc. ACM Hum. Comput. Interact. · doi:10.1145/3610067 · Cited by 59 · see citing works
C-XAI: A conceptual framework for designing XAI tools that support trust calibration
Naiseh, M., Simkute, A., Zieni, B., Jiang, N., & Ali, R. (2024) · Journal of Responsible Technology · doi:10.1016/j.jrt.2024.100076 · Cited by 42 · see citing works
Transparency for AI systems: a value-based approach
Buijsman, S. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09770-w · Cited by 38 · see citing works
Intelligent systems in healthcare: A systematic survey of explainable user interfaces
Cálem, J., Moreira, C., & Jorge, J. A. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108908 · Cited by 32 · see citing works
Why should I trust you? Influence of explanation design on consumer behavior in AI-based services
Nizette, F., Hammedi, W., van Riel, A. C. R., & Steils, N. (2024) · Journal of service management · doi:10.1108/josm-05-2024-0223 · Cited by 36 · see citing works
Supporting High-Uncertainty Decisions through AI and Logic-Style Explanations
Cau, F. M., Hauptmann, H., Spano, L. D., & Tintarev, N. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581641.3584080 · Cited by 31 · see citing works
Boudanga, Z., Benhadou, S., & Medromi, H. (2023) · F1000Research · doi:10.12688/f1000research.138867.1 · Cited by 29 · see citing works
Toward Human-centered XAI in Practice: A survey
Kong, X., Liu, S., & Zhu, L. (2024) · Machine Intelligence Research · doi:10.1007/s11633-022-1407-3 · Cited by 31 · see citing works
Villareale, J., Harteveld, C., & Zhu, J. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3549482 · Cited by 26 · see citing works
How can Explainability Methods be Used to Support Bug Identification in Computer Vision Models?
Balayn, A., Rikalo, N., Lofi, C., Yang, J., & Bozzon, A. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517474 · Cited by 24 · see citing works
He, G., Demartini, G., & Gadiraju, U. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713218 · Cited by 23 · see citing works
Bove, C., Lesot, M., Tijus, C., & Detyniecki, M. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581641.3584082 · Cited by 23 · see citing works
Okolo, C. T., Agarwal, D., Dell, N. L., & Vashistha, A. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637348 · Cited by 21 · see citing works
How Do HCI Researchers Study Cognitive Biases? A Scoping Review
Boonprakong, N., Tag, B., Gonçalves, J., & Dingler, T. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713450 · Cited by 22 · see citing works
A scoping review of silent trials for medical artificial intelligence
Tikhomirov, L., Semmler, C., Prizant, N., Bhasin, S., Kenyon, G., van der Vegt, A., Erdman, L., Kurian, N. C., Thompson, H., Palmer, L. J., et al. (2026) · Nature Health · doi:10.1038/s44360-025-00048-z · Cited by 17 · see citing works
Responsible AI-Based Business Process Management and Improvement
Pisoni, G., & Moloney, M. (2024) · Digital Society · doi:10.1007/s44206-024-00105-2 · Cited by 18 · see citing works
Do Expressions Change Decisions? Exploring the Impact of AI's Explanation Tone on Decision-Making
Okoso, A., Yang, M., & Baba, Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713744 · Cited by 19 · see citing works
Deshpande, N. M., Gite, S., Pradhan, B., & Assiri, M. E. (2022) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2022.021225 · Cited by 20 · see citing works
(Explainable) Artificial Intelligence in Aerospace Safety-Critical Systems
Sutthithatip, S., Perinpanayagam, S., & Aslam, S. (2022) · IEEE Aerospace Conference (AERO) · doi:10.1109/aero53065.2022.9843612 · Cited by 18 · see citing works
Jung, J., Kang, S., Choi, J., El-Kareh, R. E., Lee, H., & Kim, H. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.105943 · Cited by 19 · see citing works
The Who in Explainable AI: How AI Background Shapes Perceptions of AI ExplanationsPreprint
Ehsan, U., Passi, S., Liao, Q. V., Chan, L., Lee, I.-H., Muller, M. J., & Riedl, M. O. (2021) · arXiv (Cornell University) · Cited by 17 · see citing works
Piscitelli, M. S., Razzano, G., Buscemi, G., & Capozzoli, A. (2024) · Energy and Buildings · doi:10.1016/j.enbuild.2024.115115 · Cited by 16 · see citing works
Belaid, M. K., Bornemann, R., Rabus, M., Krestel, R., & Hüllermeier, E. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-44067-0_5 · Cited by 15 · see citing works
Valdez, A. S. C., Heine, M., Franke, T., Jochems, N., Jetter, H.-C., & Schrills, T. (2024) · i-com · doi:10.1515/icom-2024-0014 · Cited by 18 · see citing works
Framework for human–XAI symbiosis: extended self from the dual-process theory perspective
Litvinova, Y., Mikalef, P., & Luo, X. R. (2024) · Journal of Business Analytics · doi:10.1080/2573234x.2024.2396366 · Cited by 15 · see citing works
Inherently Interpretable Machine Learning: A Contrasting Paradigm to Post-hoc Explainable AI
Zschech, P., Weinzierl, S., & Kraus, M. (2025) · Business & Information Systems Engineering · doi:10.1007/s12599-025-00964-0 · Cited by 13 · see citing works
Information that matters: Exploring information needs of people affected by algorithmic decisions
Schmude, T., Koesten, L. M., Möller, T. B., & Tschiatschek, S. (2024) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2024.103380 · Cited by 13 · see citing works
Knowing (Not) to Know: Explainable Artificial Intelligence and Human Metacognition
von Zahn, M., Liebich, L., Jussupow, E., Hinz, O. F., & Bauer, K. (2025) · Information Systems Research · doi:10.1287/isre.2024.1431 · Cited by 14 · see citing works
The SAGE Framework for Explaining Context in Explainable Artificial Intelligence
Mill, E., Garn, W., Ryman-Tubb, N. F., & Turner, C. J. (2024) · Applied Artificial Intelligence · doi:10.1080/08839514.2024.2318670 · Cited by 13 · see citing works
Eisbach, S., Mai, O., & Hertel, G. (2024) · New Ideas in Psychology · doi:10.1016/j.newideapsych.2024.101077 · Cited by 12 · see citing works
Transparency and Explainability for Public Policy
Vredenburgh, K. (2024) · LSE Public Policy Review · doi:10.31389/lseppr.111 · Cited by 12 · see citing works
Debating with More Persuasive LLMs Leads to More Truthful Answers
Khan, A., Hughes, J., Valentine, D., Ruis, L., Sachan, K., Radhakrishnan, A., Grefenstette, E., Bowman, S. R., Rocktäschel, T., & Perez, E. (2024) · arXiv · doi:10.48550/arxiv.2402.06782 · Cited by 13 · see citing works
Lodovico, C. D., Torrielli, F., Di, L., & Rapp, A. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2491009 · Cited by 12 · see citing works
Making SHAP Rap: Bridging Local and Global Insights Through Interaction and Narratives
Chromik, M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-85616-8_37 · Cited by 12 · see citing works
Self-citationSalimzadeh, S., & Gadiraju, U. (2024) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3627043.3659567 · Cited by 11 · see citing works
Explaining AI Decisions: Towards Achieving Human-Centered Explainability in Smart Home Environments
Shajalal, M., Boden, A., Stevens, G., Du, D. K., & Kern, D.-R. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-63803-9_23 · Cited by 13 · see citing works
Xuan, Y., Small, E. A., Sokol, K., Hettiachchi, D., & Sanderson, M. (2024) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2024.103376 · Cited by 12 · see citing works
Brachman, M., Pan, Q., Do, H. J., Dugan, C., Chaudhary, A., Johnson, J. M., Rai, P., Chakraborti, T., Gschwind, T., Laredo, J. A., et al. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584088 · Cited by 11 · see citing works
Walter, Y. (2023) · AI and Ethics · doi:10.1007/s43681-023-00276-7 · Cited by 11 · see citing works
The Centers and Margins of Modeling Humans in Well-being Technologies
Zhu, J., Sanches, P., Tsaknaki, V., van der Maden, W., & Kaklopoulou, I. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713940 · Cited by 10 · see citing works
Balayn, A., Rikalo, N., Yang, J., & Bozzon, A. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581555 · Cited by 10 · see citing works
Boudanga, Z., Benhadou, S., & Medromi, H. (2023) · F1000Research · doi:10.12688/f1000research.138867.2 · Cited by 9 · see citing works
Finding AI’s Faults with AAR/AI: An Empirical Study
Khanna, R., Dodge, J. E., Anderson, A. W., Dikkala, R., Irvine, J., Shureih, Z., Lam, K.-H., Matthews, C., Lin, Z., Kahng, M., et al. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3487065 · Cited by 10 · see citing works
On the Need for Collaborative Intelligence in Cybersecurity
Martin, T. (2022) · Electronics · doi:10.3390/electronics11132067 · Cited by 9 · see citing works
Co-Designing with Users the Explanations for a Proactive Auto-Response Messaging Agent
Jain, P., Farzan, R., & Lee, A. J. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3604248 · Cited by 8 · see citing works
Classification Metrics for Image Explanations: Towards Building Reliable XAI-Evaluations
Frész, B., Lörcher, L., & Huber, M. F. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658537 · Cited by 8 · see citing works
Digital Psychology: Introducing a Conceptual Impact Model and the Future of Work
Walter, Y. (2024) · Trends in Psychology · doi:10.1007/s43076-024-00408-w · Cited by 16 · see citing works
SIDEs: Separating Idealization from Deceptive 'Explanations' in xAI
Sullivan, E. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658999 · Cited by 8 · see citing works
Explainable AI in Disease Diagnosis
Bedi, P., Thukral, A., & Dhiman, S. (2024) · Computational intelligence methods and applications · doi:10.1007/978-981-97-3705-5_5 · Cited by 7 · see citing works
Rosenbacke, R. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4773350 · Cited by 7 · see citing works
Delaunay, J., Galárraga, L. A., Largouët, C., & van Berkel, N. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711011 · Cited by 6 · see citing works
Schäfer, U., Sipos, L., & Müller-Birn, C. (2025) · Proc. ACM Hum. Comput. Interact. · doi:10.1145/3757451 · Cited by 11 · see citing works
Tooling for Developing Data-Driven Applications: Overview and Outlook
Weber, T., & Hußmann, H. (2022) · Mensch und Computer (MuC) · doi:10.1145/3543758.3543779 · Cited by 6 · see citing works
Predicting the need for XAI from high-granularity interaction data
de Santana, V. F., Fucs, A., Segura, V., de Moraes, D. B., & Cerqueira, R. (2023) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2023.103029 · Cited by 6 · see citing works
Schrills, T., Gruner, M., Peuscher, H., & Franke, T. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-35748-0_23 · Cited by 6 · see citing works
Exploring the impact of explainable AI and cognitive capabilities on users’ decisions
Cau, F. M., & Spano, L. D. (2025) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-025-09438-0 · Cited by 9 · see citing works
Resilience Through Appropriation: Pilots’ View on Complex Decision Support
Zhang, Z. T., Storath, C., Liu, Y., & Butz, A. M. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581641.3584056 · Cited by 5 · see citing works
Self-citationUnderstanding User Preferences in Explainable Artificial Intelligence: A Mapping Function Proposal
Hashemi, M., Darejeh, A., & Cruz, F. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3733837 · Cited by 4 · see citing works
Szymczyk, N., Szczuka, J. M., Krämer, N. C., & Wischnewski, M. (2025) · Mensch und Computer (MuC) · doi:10.1145/3743049.3743072 · Cited by 4 · see citing works
Bhattacharya, A., & Verbert, K. (2024) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3631700.3664911 · Cited by 4 · see citing works
Ferrell, B. J., Raskin, S. E., & Zimmerman, E. B. (2023) · JMIR Formative Research · doi:10.2196/41516 · Cited by 4 · see citing works
XAI4RE – Using Explainable AI for Responsible and Ethical AI
Tal, A. S., & Sheidin, J. (2025) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3708319.3733707 · Cited by 3 · see citing works
Mehrotra, S., Gadiraju, U., Bittner, E. A. C., van Delden, F., Jonker, C. M., & Tielman, M. L. (2025) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3699682.3728343 · Cited by 3 · see citing works
Understanding the Effects of Explaining Predictive but Unintuitive Features in Human-XAI Interaction
Qu, J., Arguello, J., & Wang, Y. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732021 · Cited by 4 · see citing works
Erlei, A., Cau, F. M., Georgiev, R., Kumar, S. C., Bizer, K., & Gadiraju, U. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791420 · Cited by 4 · see citing works
Understanding the Effects of AI-Assisted Critical Thinking on Human-AI Decision Making
Tian, H. Y., Amin, H., & Yin, M. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790785 · Cited by 5 · see citing works
Qu, J., Arguello, J., & Wang, Y. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658547 · Cited by 3 · see citing works
Pisoni, G., & Moloney, M. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4822711 · Cited by 3 · see citing works
Dataism, skepticism, and intuition for interpretable machine learning
Naser, M. Z. (2025) · AI and Ethics · doi:10.1007/s43681-025-00831-4 · Cited by 2 · see citing works
- Authors
- Chromik, M., Eiband, M., Buchner, F., Krüger, A., & Butz, A.
- Affiliations
- Ludwig-Maximilians-Universität München (DE)
- Venue
- 26th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2021-04-14
- Pages
- 307-317
- OpenAlex
- W3156666346
- Cited by
- 144
- References
- 57
- Citations by year
- 2026: 26 · 2025: 31 · 2024: 38 · 2023: 30 · 2022: 16 · 2021: 3
- FWCI
- 9.96
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- explainable AI, illusion of explanatory depth, local explanations, SHAP, non-technical users, mental models, interpretability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
