Evaluating saliency map explanations for convolutional neural networks: a user study
The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
Abstract
Convolutional neural networks (CNNs) offer great machine learning performance over a range of applications, but their operation is hard to interpret, even for experts. Various explanation algorithms have been proposed to address this issue, yet limited research effort has been reported concerning their user evaluation. In this paper, we report on an online between-group user study designed to evaluate the performance of "saliency maps" - a popular explanation algorithm for image classification applications of CNNs. Our results indicate that saliency maps produced by the LRP algorithm helped participants to learn about some specific image features the system is sensitive to. However, the maps seem to provide very limited help for participants to anticipate the network's output for new images. Drawing on our findings, we highlight implications for design and further research on explainable AL In particular, we argue the HCI and AI communities should look beyond instance-level explanations.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Alqaraawi et al. [2] conducted a user study on image classification tasks to evaluate the performance of the saliency map, an XAI technique that highlights input pixels in the original images that contribute to a model prediction and discussed the limited usefulness of saliency map to assist participants to anticipate a model output.”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [2]
In the citing paper
“On the other hand, when individuals are faced with tasks that have lower uncertainty, such as annotation and classifcation task [4, 77, 117], they tend to rely less on the AI advice and rely more on their expertise and judgment.”
From Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making (2024) · cites this paper as [4]
In the citing paper
“Unfortunately, the human–XAI interaction techniques face many challenges that need massive work to be solved [103,104].”
From Survey of Explainable AI Techniques in Healthcare (2023) · cites this paper as [104]
In the citing paper
“Unfortunately, the human–XAI interaction techniques face many challenges that need massive work to be solved [103,104].”
From Survey of Explainable AI Techniques in Healthcare (2023) · cites this paper as [104]
In the citing paper
“Such explanations often do not match the end-users' mental models [56], making the explanations ineective [2,40,65] and at times even harmful [13,28,30,47,51].”
From Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their Trust (2023) · cites this paper as [2]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity
White, H. (1980) · Econometrica · doi:10.2307/1912934 · Cited by 25,861
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Pattern Recognition and Machine Learning
(2006) · doi:10.1007/978-0-387-45528-0 · Cited by 9,824
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 2×
“Similarly, the overestimation of a system’s intelligence or capabilities has been shown to impact user interaction negatively [3, 29]. may lead to over-reliance on a system [37], less vigilance towards system failures [61] and unrealistic expectations [61].”
BackgroundPeeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Research works printed but not linked: 6
- Alex Campolo , Madelyn Sanfilippo , Meredith Whittaker , and Kate Crawford . 2018. AI Now 2017 Report. Microsoft Research (Feb . 2018 ). Alex Campolo, Madelyn Sanfilippo, Meredith Whittaker, and Kate Crawford. 2018. AI Now 2017 Report. Microsoft Research (Feb. 2018).
- 10.1167/7-1.10 · doi:10.1167/7-1.10
- 10.1007/978-3-319-46493--0_1 · doi:10.1007/978-3-319-46493--0_1
- Been Kim , Cynthia Rudin , and Julie A Shah . 2014. The Bayesian Case Model: A Generative Approach for Case-Based Reasoning and Prototype Classification . In Advances in Neural Information Processing Systems 27 , Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (Eds.). Curran Associates, Inc. , 1952 --1960. http://papers.nips.cc/paper/5313-the-bayesian-case-model-a-generative-approach-for-case-based-reasoning-and-prototype-classification.pdf Been Kim, Cynthia Rudin, and Julie A Shah. 2014. The Bayesian Case Model: A Generative Approach for Case-Based Reasoning and Prototype Classification. In Advances in Neural Information Processing Systems 27, Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 1952--1960. http://papers.nips.cc/paper/5313-the-bayesian-case-model-a-generative-approach-for-case-based-reasoning-and-prototype-classification.pdf
- Neville Moray . 1999. Mental Models in Theory and Practice. Attention and performance XVII: Cognitive regulation of performance: Interaction of theory and application ( 1999 ), 223--258. Neville Moray. 1999. Mental Models in Theory and Practice. Attention and performance XVII: Cognitive regulation of performance: Interaction of theory and application (1999), 223--258.
- Donald Norman . 2014. On the Relationship between Conceptual and Mental Models . In Gentner et al (e.d) Mental Models . Psychology Press . Donald Norman. 2014. On the Relationship between Conceptual and Mental Models. In Gentner et al (e.d) Mental Models. Psychology Press.
Cited by
Explainable Artificial Intelligence in education
Khosravi, H., Shum, S. B., Chen, G., Conati, C., Tsai, Y.-S., Kay, J., Knight, S., Martinez-Maldonado, R., Sadiq, S., & Gašević, D. (2022) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100074 · Cited by 863
Survey of Explainable AI Techniques in Healthcare
Chaddad, A., Peng, J., Xu, J., & Bouridane, A. (2023) · Sensors · doi:10.3390/s23020634 · Cited by 591
Expanding Explainability: Towards Social Transparency in AI systems
Ehsan, U., Liao, Q. V., Muller, M., Riedl, M. O., & Weisz, J. D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445188 · Cited by 538
Exploring students' perspectives on generative AI-assisted academic writing
Kim, J., Yu, S., Detrick, R., & Li, N. (2025) · Education and Information Technologies · doi:10.1007/s10639-024-12878-7 · Cited by 438
“…consider such socio-technical perspectives and demonstrate a range of explainable AI techniques and methodologies to better assist in sensemaking (Alqaraawi et al., 2020), enhancing user trust (Yang et al., 2020), or enabling shared decision making between student and AI system (Kim et al.,…”
Wang, X., & Yin, M. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450650 · Cited by 320
“[3] image classication saliency map mixed results N/A N/A Chu et al.”
BackgroundAhmad, K., Maabreh, M., Ghaly, M., Khan, K., Qadir, J., & Al-Fuqaha, A. (2022) · Computer Science Review · doi:10.1016/j.cosrev.2021.100452 · Cited by 291
Towards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3331846 · Cited by 278
Lai, V., Chen, C., Smith-Renner, A., Liao, Q. V., & Tan, C. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594087 · Cited by 204
Explainable Deep Reinforcement Learning: State of the Art and Challenges
Vouros, G. A. (2023) · ACM Computing Surveys · doi:10.1145/3527448 · Cited by 160
Explainable artificial intelligence in Alzheimer’s disease classification: a systematic review
Viswan, V., Shaffi, N., Mahmud, M., Subramanian, K., & Hajamohideen, F. (2024) · Cognitive Computation · doi:10.1007/s12559-023-10192-x · Cited by 195
Code-free deep learning for multi-modality medical image classification
Korot, E., Guan, Z., Ferraz, D., Wagner, S. K., Zhang, G., Liu, X., Faes, L., Pontikos, N., Finlayson, S. G., Khalid, H., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00305-2 · Cited by 171
Explainable artificial intelligence for cybersecurity: a literature survey
Charmet, F., Tanuwidjaja, H. C., Ayoubi, S., Gimenez, P.-F., Han, Y., Jmila, H., Blanc, G., Takahashi, T., & Zhang, Z. (2022) · Annals of Telecommunications · doi:10.1007/s12243-022-00926-7 · Cited by 167
Human-XAI Interaction: A Review and Design Principles for Explanation User Interfaces
Chromik, M., & Butz, A. M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-85616-8_36 · Cited by 210
I Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI
Chromik, M., Eiband, M., Buchner, F., Krüger, A., & Butz, A. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450644 · Cited by 144
“Let me explain!”: exploring the potential of virtual agents in explainable AI interaction design
Weitz, K., Schiller, D., Schlagowski, R., Huber, T., & André, E. (2021) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-020-00332-0 · Cited by 119
A Review of Recent Deep Learning Approaches in Human-Centered Machine Learning
Kaluarachchi, T., Reis, A., & Nanayakkara, S. (2021) · Sensors · doi:10.3390/s21072514 · Cited by 124
Visual, textual or hybrid: the effect of user expertise on different explanations
Szymanski, M., Millecamp, M., & Verbert, K. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450662 · Cited by 131
The Who in XAI: How AI Background Shapes Perceptions of AI Explanations
Ehsan, U., Passi, S., Liao, Q. V., Chan, L., Lee, I.-H., Muller, M., & Riedl, M. O. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642474 · Cited by 123
“In contrast, explanation generation methods–used in this paper– aim to explain models that are not directly human-understandable (e.g., deep neural networks).”
MethodOpening the Black Box: A systematic review on explainable artificial intelligence in remote sensing
Höhl, A., Obadić, I., Fernández‐Torres, M., Najjar, H., Oliveira, D. A. B., Akata, Z., Dengel, A. R., & Zhu, X. X. (2024) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2024.3467001 · Cited by 142
Mertes, S., Huber, T., Weitz, K., Heimerl, A., & André, E. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.825565 · Cited by 119
“Furthermore, inspired by Alqaraawi et al. (2020), we additionally used the AI model’s confidence to ensure diversity in the images.”
MethodCharting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAI
Ehsan, U., Saha, K., De Choudhury, M., & Riedl, M. O. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579467 · Cited by 85
Yoon, C., Park, E., Misra, S., Kim, J. Y., Baik, J. W., Kim, K. G., Jung, C. K., & Kim, C. (2024) · Light Science & Applications · doi:10.1038/s41377-024-01554-7 · Cited by 116
Bao, Y., Gong, W., & Yang, K. (2023) · Systems · doi:10.3390/systems11090442 · Cited by 80
Anchoring Bias Affects Mental Model Formation and User Reliance in Explainable AI Systems
Nourani, M., Roy, C., Block, J. E., Honeycutt, D. R., Rahman, T., Ragan, E., & Gogate, V. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450639 · Cited by 107
Nannini, L., Balayn, A., & Smith, A. L. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594074 · Cited by 138
Explainability Pitfalls: Beyond Dark Patterns in Explainable AI
Ehsan, U., & Riedl, M. O. (2021) · arXiv · doi:10.48550/arxiv.2109.12480 · Cited by 99
Riveiro, M., & Thill, S. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103507 · Cited by 79
Petersen, E., Potdevin, Y., Mohammadi, E., Zidowitz, S., Breyer, S., Nowotka, D., Henn, S., Pechmann, L., Leucker, M., Rostalski, P., et al. (2022) · IEEE Access · doi:10.1109/access.2022.3178382 · Cited by 113
Towards falsifiable interpretability researchPreprint
Leavitt, M. L., & Morcos, A. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12016 · Cited by 111
Debugging Tests for Model Explanations
Adebayo, J., Muelly, M., Liccardi, I., & Kim, B. (2020) · arXiv · doi:10.48550/arxiv.2011.05429 · Cited by 88
Automatic diagnosis of COVID-19 from CT images using CycleGAN and transfer learning
Ghassemi, N., Shoeibi, A., Khodatars, M., Heras, J., Rahimi, A., Zare, A., Zhang, Y.-D., Pachori, R. B., & Gorriz, J. M. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110511 · Cited by 78
A Human-Grounded Evaluation Benchmark for Local Explanations of Machine Learning
Mohseni, S., Block, J. E., & Ragan, E. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.05075 · Cited by 45
Extending the Nested Model for User-Centric XAI: A Design Study on GNN-based Drug Repurposing
Wang, Q., Huang, K., Chandak, P., Zitnik, M., & Gehlenborg, N. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209435 · Cited by 82
Fok, R., & Weld, D. S. (2024) · AI Magazine · doi:10.1002/aaai.12182 · Cited by 81
Explainable goal-driven agents and robots-a comprehensive review
Sado, F., Loo, C. K., Liew, W. S., Kerzel, M., & Wermter, S. (2023) · ACM Computing Surveys · doi:10.1145/3564240 · Cited by 79
“However, saliency maps have several disadvantages [3].”
Lee, M. H., & Chew, C. J. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610218 · Cited by 78
Subramanian, H. V., Canfield, C., & Shank, D. B. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102780 · Cited by 84
When Explanations Lie: Why Many Modified BP Attributions FailPreprint
Sixt, L., Granz, M., & Landgraf, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.09818 · Cited by 113
Challenges for machine learning in clinical translation of big data imaging studies
Dinsdale, N. K., Bluemke, E., Sundaresan, V., Jenkinson, M., Smith, S. M., & Namburete, A. I. L. (2022) · Neuron · doi:10.1016/j.neuron.2022.09.012 · Cited by 90
Effects of Explanations in AI-Assisted Decision Making: Principles and Comparisons
Wang, X., & Yin, M. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3519266 · Cited by 74
Explainability pitfalls: beyond dark patterns in explainable AI
Ehsan, U., & Riedl, M. O. (2024) · Patterns · doi:10.1016/j.patter.2024.100971 · Cited by 72
Metrics for Saliency Map Evaluation of Deep Learning Explanation Methods
Gomez, T., Fréour, T., & Mouchère, H. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-09037-0_8 · Cited by 95
Freiesleben, T., König, G., Molnar, C., & Tejero-Cantero, Á. (2024) · Minds and Machines · doi:10.1007/s11023-024-09691-z · Cited by 82
Secure and robust machine learning for healthcare: A survey
Qayyum, A., Qadir, J., Bilal, M., & Al-Fuqaha, A. (2021) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2020.3013489 · Cited by 59
Al-Ansari, N., Al‐Thani, D., & Al-Mansoori, R. S. (2024) · Human Behavior and Emerging Technologies · doi:10.1155/2024/4628855 · Cited by 71
Secure and Robust Machine Learning for Healthcare: A Survey
Qayyum, A., Qadir, J., Bilal, M., & Al-Fuqaha, A. (2020) · arXiv · doi:10.48550/arxiv.2001.08103 · Cited by 59
Defining Explanation and Explanatory Depth in XAI
Buijsman, S. (2022) · Minds and Machines · doi:10.1007/s11023-022-09607-9 · Cited by 53
Quantitative Evaluation of Machine Learning Explanations: A Human-Grounded Benchmark
Mohseni, S., Block, J. E., & Ragan, E. D. (2021) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3397481.3450689 · Cited by 75
Banovic, N., Yang, Z., Ramesh, A., & Liu, A. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579460 · Cited by 55
Fok, R., & Weld, D. S. (2023) · arXiv · doi:10.48550/arxiv.2305.07722 · Cited by 54
Local and global explanations of agent behavior: Integrating strategy summaries with saliency maps
Huber, T., Weitz, K., André, E., & Amir, O. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103571 · Cited by 53
“[4] who evaluated local explanations and came to the conclusion that sole instance-level explanations are not sufficient and should be augmented with global information.”
BackgroundResultInfluentialPredicting the translation efficiency of messenger RNA in mammalian cells
Zheng, D., Persyn, L., Wang, J., Liu, Y., Ulloa-Montoya, F., Cenik, C., & Agarwal, V. (2026) · Nature Biotechnology · doi:10.1038/s41587-025-02712-x · Cited by 52
“Saliency maps are commonly used to explain deep learning model predictions by highlighting the input variables that contribute most toward the predicted label 65,66 .”
Solanke, A. A. (2022) · Forensic Science International: Digital Investigation · doi:10.1016/j.fsidi.2022.301403 · Cited by 52
Towards a multi-stakeholder value-based assessment framework for algorithmic systems
Yurrita, M., Murray-Rust, D., Balayn, A., & Bozzon, A. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533118 · Cited by 39
Müller, R. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2381929 · Cited by 53
Amorim, J. P., Abreu, P. H., Santos, J. A. M., Cortes, M., & Vila, V. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103225 · Cited by 57
Learning Patch-Channel Correspondence for Interpretable Face Forgery Detection
Hua, Y., Shi, R., Wang, P., & Ge, S. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3246793 · Cited by 57
Roadmap of Designing Cognitive Metrics for Explainable Artificial Intelligence (XAI)
Hsiao, J. H.-W., Ngai, H. H. T., Qiu, L., Yang, Y., & Cao, C. C. (2021) · arXiv · doi:10.48550/arxiv.2108.01737 · Cited by 45
Effects of AI and Logic-Style Explanations on Users’ Decisions Under Different Levels of Uncertainty
Cau, F. M., Hauptmann, H., Spano, L. D., & Tintarev, N. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3588320 · Cited by 41
The effect of machine learning explanations on user trust for automated diagnosis of COVID-19
Goel, K., Sindhgatta, R., Kalra, S., Goel, R., & Mutreja, P. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105587 · Cited by 54
A Missing Piece in the Puzzle: Considering the Role of Task Complexity in Human-AI Decision Making
Salimzadeh, S., He, G., & Gadiraju, U. (2023) · Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization · doi:10.1145/3565472.3592959 · Cited by 42
“Sketch Recognition (4,6,0)[18],(6,84,42)[105] Movie Recommendation (2,4,0)[52],(28,1,0)[104], (4,15,0)[63], (10,4,0)[10] Place Recommendation (12,2,0)[96], (51,12,0)[46] Food Recommendation (2,51,0)[37], (9,3,0)[72] Image Classification (3,12,0)[3],(5,216,0)[101], (4,90,0)[47], (3,40,0)[75] Sentiment Analysis (4,3,0)[…”
BackgroundExplainable predictive process monitoring: a user evaluation
Rizzi, W., Comuzzi, M., Francescomarino, C. D., Ghidini, C., Lee, S., Maggi, F. M., & Nolte, A. (2024) · Process Science · doi:10.1007/s44311-024-00003-3 · Cited by 45
Salimzadeh, S., He, G., & Gadiraju, U. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641905 · Cited by 41
“We also considered traffic conditions in both diagnostic and prognostic scenarios, although, in the real world, traffic conditions can change over time and at the time of decision-making, making them predominantly prognostic.”
BackgroundNourani, M., Roy, C., Block, J. E., Honeycutt, D. R., Rahman, T., Ragan, E. D., & Gogate, V. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3531066 · Cited by 48
Human-Centered Explainable AI (HCXAI): Coming of Age
Ehsan, U., Wintersberger, P., Watkins, E. A., Manger, C., Ramos, G., Weisz, J. D., Daumé, H., Riener, A., & Riedl, M. O. (2023) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3573832 · Cited by 43
Exemplary Natural Images Explain CNN Activations Better than State-of-the-Art Feature VisualizationPreprint
Borowski, J., Zimmermann, R. S., Schepers, J., Geirhos, R., Wallis, T. S. A., Bethge, M., & Brendel, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12606 · Cited by 40
Designing Interactive Transfer Learning Tools for ML Non-Experts
Mishra, S., & Rzeszotarski, J. M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445096 · Cited by 32
Nguyen, G., Kim, D., & Nguyen, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.14944 · Cited by 103
Mumuni, F., & Mumuni, A. (2023) · Cognitive Systems Research · doi:10.1016/j.cogsys.2023.101188 · Cited by 41
State-of-the-Art in Responsible, Explainable, and Fair AI for Medical Image Analysis
Amirian, S., Gao, F., Littlefield, N. G., Hill, J. H., Yates, A. J., Plate, J. F., Pantanowitz, L., Rashidi, H. H., & Tafti, A. P. (2025) · IEEE Access · doi:10.1109/access.2025.3555543 · Cited by 37
How to Support Users in Understanding Intelligent Systems? Structuring the Discussion
Eiband, M., Buschek, D., & Hussmann, H. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450694 · Cited by 29
Model-agnostic explainable artificial intelligence for object detection in image data
Moradi, M., Yan, K., Colwell, D., Samwald, M., & Asgari, R. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109183 · Cited by 38
Measure Utility, Gain Trust: Practical Advice for XAI Researchers
Davis, B., Glenski, M., Sealy, W. I. N., & Arendt, D. (2020) · IEEE Workshop on TRust and EXpertise in Visual Analytics (TREX) · doi:10.1109/trex51495.2020.00005 · Cited by 43
Reassuring, Misleading, Debunking: Comparing Effects of XAI Methods on Human Decisions
Humer, C., Hinterreiter, A., Leichtmann, B., Mara, M., & Streit, M. (2024) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3665647 · Cited by 31
Corti, L., Oltmans, R., Jung, J., Balayn, A., Wijsenbeek, M. S., & Yang, J. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642551 · Cited by 43
Speith, T., Crook, B., Mann, S., Schomäcker, A., & Langer, M. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09769-3 · Cited by 34
Okolo, C. T., Agarwal, D., Dell, N. L., & Vashistha, A. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637348 · Cited by 36
Developing an AI-Assisted Low-Resource Spoken Language Learning App for Children
Getman, Y., Phan, N., Al-Ghezi, R., Voskoboinik, E., Singh, M., Grósz, T., Kurimo, M., Salvi, G., Svendsen, T., Strömbergsson, S., et al. (2023) · IEEE Access · doi:10.1109/access.2023.3304274 · Cited by 27
Do Expressions Change Decisions? Exploring the Impact of AI's Explanation Tone on Decision-Making
Okoso, A., Yang, M., & Baba, Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713744 · Cited by 32
How can Explainability Methods be Used to Support Bug Identification in Computer Vision Models?
Balayn, A., Rikalo, N., Lofi, C., Yang, J., & Bozzon, A. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517474 · Cited by 42
A quantitative benchmark of neural network feature selection methods for detecting nonlinear signals
Passemiers, A., Folco, P., Raimondi, D., Birolo, G., Moreau, Y., & Fariselli, P. (2024) · Scientific Reports · doi:10.1038/s41598-024-82583-5 · Cited by 28
Performance of GAN-based augmentation for deep learning COVID-19 image classification
Fedoruk, O., Klimaszewski, K., Ogonowski, A., & Możdżonek, R. (2024) · AIP conference proceedings · doi:10.1063/5.0203379 · Cited by 29
Explainable AI for Interpretation of Ovarian Tumor Classification Using Enhanced ResNet50
Guha, S., Kodipalli, A., Fernandes, S. L., & Dasar, S. K. (2024) · Diagnostics · doi:10.3390/diagnostics14141567 · Cited by 24
Crowdsourcing and Evaluating Concept-driven Explanations of Machine Learning Models
Mishra, S., & Rzeszotarski, J. M. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449213 · Cited by 21
Debiased-CAM to mitigate image perturbations with faithful visual explanations of machine learning
Zhang, W., Dimiccoli, M., & Lim, B. Y. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517522 · Cited by 30
IRIS: Interpretable Rubric-Informed Segmentation for Action Quality Assessment
Matsuyama, H., Kawaguchi, N., & Lim, B. Y. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581641.3584048 · Cited by 25
A User Interface for Explaining Machine Learning Model Explanations
Kadir, M. A., Selim, A. M., Barz, M., & Sonntag, D. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581754.3584131 · Cited by 21
Hierarchical goals contextualize local reward decomposition explanations
Rietz, F., Magg, S., Heintz, F., Stoyanov, T., Wermter, S., & Stork, J. A. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07280-8 · Cited by 19
Navigating the landscape of concept-supported XAI: Challenges, innovations, and future directions
Khoozani, Z. S., Sabri, A. Q. M., Seng, W. C., Seera, M., & Eg, K. Y. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17666-y · Cited by 22
Pafla, M., Larson, K. S., & Hancock, M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642934 · Cited by 24
The AI-DEC: A Card-based Design Method for User-centered AI Explanations
Lee, C. P., Lee, M. K., & Mutlu, B. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3661576 · Cited by 22
Hagberg, E., Hagerman, D., Johansson, R., Hosseini, N., Liu, J., Björnsson, E., Alvén, J., & Hjelmgren, O. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105282 · Cited by 19
Malik, N., & Bzdok, D. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.06.014 · Cited by 21
How to Evaluate Explainability? - A Case for Three Criteria
Speith, T. (2022) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew56159.2022.00024 · Cited by 21
Wait, But Why?: Assessing Behavior Explanation Strategies for Real-Time Strategy Games
Robertson, J., Kokkinakis, A. V., Hook, J., Kirman, B., Block, F., Ursu, M. F., Patra, S., Demediuk, S., Drachen, A., & Olarewaju, O. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450699 · Cited by 18
“In general, many XAI studies test for user trust in the model to make correct decisions [18, 40] and user confidence in the decision-making process [1, 17].”
BackgroundQuestion-Driven Design Process for Explainable AI User Experiences
Liao, Q. V., Pribić, M., Han, J., Miller, S., & Sow, D. (2021) · arXiv · doi:10.48550/arxiv.2104.03483 · Cited by 18
“Another active research area focuses on rigorously evaluating the effectiveness of XAI techniques [9, 18, 19, 23, 57].”
Zhang, W., Shan, X., Fu, B.-Y., Zou, X., & Fu, L. (2022) · Journal of Asian Earth Sciences · doi:10.1016/j.jseaes.2022.105437 · Cited by 18
Oral epithelial cell segmentation from fluorescent multichannel cytology images using deep learning
Sunny, S. P., Khan, A. I., Rangarajan, M., Hariharan, A., Birur, P. N., Pandya, H. J., Shah, N., Kuriakose, M. A., & Suresh, A. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.107205 · Cited by 18
Cabitza, F., Campagner, A., Famiglini, L., Gallazzi, E., & Maida, G. A. L. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-14463-9_3 · Cited by 23
Be careful what you explain: Benefits and costs of explainable AI in a simulated medical task
Rieger, T., Manzey, D., Meussling, B., Onnasch, L., & Roesler, E. (2023) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2023.100021 · Cited by 20
Metadata
- Authors
- Alqaraawi, A., Schuessler, M., Weiß, P., Costanza, E., & Berthouze, N.
- Affiliations
- University College London (GB)
- Technische Universität Berlin (DE)
- UCL Interaction Centre, London, United Kingdom
- Venue
- Proceedings of the 25th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2020-03-17
- Pages
- 275-285
- OpenAlex
- W3010300896
- Cited by
- 163
- References
- 53
- Citations by year
- 2026: 9 · 2025: 24 · 2024: 35 · 2023: 34 · 2022: 26 · 2021: 25 · 2020: 10
- FWCI
- 9.06
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
- Funders
- Bundesministerium für Bildung und Forschung (16DII113)
- King Abdulaziz City for Science and Technology
- Engineering and Physical Sciences Research Council (EP/N014243/1)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- saliency maps, explainable AI, convolutional neural networks, LRP, model interpretability, image classification, instance-level explanation, feature sensitivity, online user study
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.