The DOI matches its Crossref record; 73 of the 76 references it cites were found in a registry or an index.
A Survey of Post-Hoc XAI Methods From a Visualization Perspective: Challenges and Opportunities
Open accessAbstract
XAI (eXplainable AI) has become a pivotal area of research with the advancement of deep learning (DL) technologies and applications. Post-hoc explanation methods interpret deep learning predictions by uncovering the significance of input features, while visualization tools can contribute to a deep understanding of AI model reasoning based on these methods. In this paper, we survey a broad spectrum of post-hoc explanation methods and the visual analytics work based on them. First, we categorize the computational methods into four main types: perturbation-based, gradient-based, decomposition-based, and concept-based. While the first three focus on attributing the model’s output to specific regions of the input image, concept-based methods provide global explanations by mapping human-understandable concepts to high-level features. Then, we examine the methodologies, features, strengths, and limitations of each approach. Moreover, we review existing visualization-focused work based on these computational methods. Finally, we discuss further research challenges and opportunities for XAI visualization with post-hoc explanation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“By contrast, feature- or concept-based explanations appeared only in systems incorporating conventional predictive components amenable to attribution, such as saliency maps, concept bottleneck representations, or SHAP analyses over structured variables [31,32,33].”
From Explainable Agentic Artificial Intelligence in Healthcare: A Scoping Review (2026) · cites this paper as [33]
In the citing paper
“For instance, while models like deep neural networks often demonstrate high detection accuracy, their interpretability is limited, requiring post-hoc XAI tools that can sometimes be computationally expensive (Bhati et al., 2025).”
From Explainable AI: enhancing decision-making in the detection of cyber threats (2026) · cites this paper as Bhati et al., 2025
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 8×
“This scalable tool aggregates neuron activations, enabling users to visually explore how different features influence model predictions, particularly for large datasets like ImageNet [1].”
InfluentialGrad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 2×
“SHAP assigns each feature an importance score by measuring its contribution to the difference between the actual prediction and a baseline prediction, ensuring consistent and theoretically grounded model interpretation by evaluating all possible feature combinations [45].”
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 4×
“A significant draw-back of Occlusion is the computational power required, as inference is needed for each occluded image, which increases with higher resolution images and more detailed heatmaps [43].”
InfluentialLearning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428 · cited 2×
“Illustration of the class activation map (CAM) generation process in [64].”
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · cited 2×
“Focusing specifically on surveys related to post-hoc explanation, Arrieta et al. [7] and Adadi and Berrada [8] comprehensively examine XAI assessment methods and challenges relevant to image classification tasks.”
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 4×
“By weighting the gradient with the input features, the model is locally linearized, providing insight into feature importance [46].”
InfluentialOn Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524 · cited 3×
“Specifically, PatternAttribution employs a class ‘signal’ estimator, trained on the first half of the ImageNet training dataset [1], to estimate the root point for DTD [49].”
Axiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087 · cited 2×
“Integrated Gradients measure the contribution of each input feature to the model’s prediction by computing the integral of gradients along the path from a baseline input to the actual input, enhancing the understanding of model predictions across different input spaces and domains [63].”
Grad-CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks
Chattopadhay, A., Sarkar, A., Howlader, P., & Balasubramanian, V. N. (2018) · 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2018.00097 · Cited by 3,639 · cited 1×
“[65].”
Guidance on Conducting a Systematic Literature Review
Xiao, Y., & Watson, M. (2019) · Journal of Planning Education and Research · doi:10.1177/0739456x17723971 · Cited by 3,467 · cited 1×
“This survey employed a Systematic Literature Review (SLR) approach, known for its rigorous and transparent process in impartially identifying, analyzing, and interpreting evidence [33].”
Striving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447 · cited 5×
“The differences between the three methods, Standard Backpropagation [46], DeconvNet [67], and Guided Back-propagation [52] can be summarized as follows: Standard Backpropagation [46] computes gradients using the chain rule and applies the ReLU mask from the forward pass.”
InfluentialSemantic Image Synthesis With Spatially-Adaptive Normalization
Park, T., Liu, M.-Y., Wang, T.-C., & Zhu, J.-Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00244 · Cited by 2,974 · cited 1×
“Further research has expanded DeconvNets’ applications to image generation [52], unsupervised learning and natural language processing [53], small object detection [54], and semantic image synthesis [55].”
Explainable AI: a review of machine learning interpretability methods
Linardatos, P., Papastefanopoulos, V., & Kotsiantis, S. (2020) · Entropy · doi:10.3390/e23010018 · Cited by 2,922 · cited 1×
“Linardatos et al. [6] contributed by categorizing XAI techniques into ‘‘interpreting closed box models,’’ ‘‘con-structing glass box models,’’ ‘‘promoting fairness in AI,’’ and ‘‘analyzing sensitivity.’’”
Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles
Noroozi, M., & Favaro, P. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46466-4_5 · Cited by 2,775 · cited 1×
“Further research has expanded DeconvNets’ applications to image generation [52], unsupervised learning and natural language processing [53], small object detection [54], and semantic image synthesis [55].”
Learning Important Features Through Propagating Activation Differences
Shrikumar, A., Greenside, P., & Kundaje, A. (2017) · arXiv · doi:10.48550/arxiv.1704.02685 · Cited by 2,761 · cited 1×
“DeepLIFT’s use of a reference point adds sophistication to the interpretation of model predictions, offering deeper insights into how individual features contribute to overall model behavior [73].”
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · cited 1×
“Gilpin et al. [5] examined different perspectives on interpretability, categorizing explanation frameworks and analyzing their limitations in terms of faithfulness and usability.”
Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3060483 · Cited by 1,477 · cited 1×
“Samek et al. [25], [26] have made significant contributions to understanding XAI in image classification, particularly through their work on saliency maps and the development of post-hoc interpretation methods.”
SmoothGrad: removing noise by adding noise
Smilkov, D., Thorat, N., Kim, B., Viégas, F., & Wattenberg, M. (2017) · arXiv · doi:10.48550/arxiv.1706.03825 · Cited by 1,467 · cited 2×
“By averaging gradients across multiple noisy samples of the input data, SmoothGrad produces smoother and more stable heatmaps, enhancing the interpretability of model predictions [48].”
Explainable AI: Interpreting, Explaining and Visualizing Deep Learning
(2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28954-6 · Cited by 1,377 · cited 1×
“Samek et al. [25], [26] have made significant contributions to understanding XAI in image classification, particularly through their work on saliency maps and the development of post-hoc interpretation methods.”
Explainable artificial intelligence (XAI) in deep learning-based medical image analysis
van der Velden, B. H. M., Kuijf, H. J., Gilhuijs, K. G. A., & Viergever, M. A. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102470 · Cited by 1,328 · cited 2×
“Research has also explored XAI’s role in clinical decision support systems [14], medical image analysis [15], [16], trustworthy AI techniques in biomedical imaging [17], [18], [19], AI interpretability in radiology [20], and AI applications in response to COVID-19 [21], [22].”
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., & Sayres, R. (2017) · arXiv · doi:10.48550/arxiv.1711.11279 · Cited by 1,240
Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities
Saeed, W., & Omlin, C. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110273 · Cited by 783
Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
Zeiler, M. D., & Fergus, R. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.3557 · Cited by 755 · cited 2×
“Only neurons with positive forward activations contribute to the gradient: DeconvNet [67] modifies the ReLU during the back-ward pass by allowing only positive gradients to propagate, regardless of the forward activation: Guided Backpropagation [52] combines both constraints,it only permits…”
Explainable Deep Learning Models in Medical Image Analysis
Singh, A., Sengupta, S., & Lakshminarayanan, V. (2020) · Journal of Imaging · doi:10.3390/jimaging6060052 · Cited by 718 · cited 2×
“Research has also explored XAI’s role in clinical decision support systems [14], medical image analysis [15], [16], trustworthy AI techniques in biomedical imaging [17], [18], [19], AI interpretability in radiology [20], and AI applications in response to COVID-19 [21], [22].”
Captum: A unified and generic model interpretability library for PyTorch
Kokhlikyan, N., Miglani, V., Martin, M., Wang, E., Alsallakh, B., Reynolds, J., Melnikov, A., Kliushkina, N., Araya, C., Yan, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.07896 · Cited by 658
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · cited 1×
“Guidotti et al. [3] provided an early survey on methods for explaining closed box models, outlining various interpretability techniques across rule-based, model-agnostic, and perturbation-based approaches.”
Antoniadi, A. M., Du, Y., Guendouz, Y., Wei, L., Mazo, C., Becker, B. A., & Mooney, C. (2021) · Applied Sciences · doi:10.3390/app11115088 · Cited by 629 · cited 1×
“Research has also explored XAI’s role in clinical decision support systems [14], medical image analysis [15], [16], trustworthy AI techniques in biomedical imaging [17], [18], [19], AI interpretability in radiology [20], and AI applications in response to COVID-19 [21], [22].”
Survey of Explainable AI Techniques in Healthcare
Chaddad, A., Peng, J., Xu, J., & Bouridane, A. (2023) · Sensors · doi:10.3390/s23020634 · Cited by 591 · cited 2×
“Other surveys, such as those by Chaddad et al. [12] and Salahuddin et al. [13], have focused on domain-specific applications, emphasizing the importance of transparency and multimodal approaches in AI explanations.”
Inceptionism: Going Deeper into Neural Networks
Mordvintsev, A., Olah, C., & Tyka, M. (2015) · Cited by 574
Huang, X., Kroening, D., Ruan, W., Sharp, J., Sun, Y., Thamo, E., Wu, M., & Yi, X. (2020) · Computer Science Review · doi:10.1016/j.cosrev.2020.100270 · Cited by 532 · cited 1×
“Research has also explored XAI’s role in clinical decision support systems [14], medical image analysis [15], [16], trustworthy AI techniques in biomedical imaging [17], [18], [19], AI interpretability in radiology [20], and AI applications in response to COVID-19 [21], [22].”
Salahuddin, Z., Woodruff, H. C., Chatterjee, A., & Lambin, P. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.105111 · Cited by 508
Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
Das, A., & Rad, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11371 · Cited by 504 · cited 2×
“Adadi and Berrada [8] reviewed XAI assessment meth-ods and challenges, while Das and Rad [9] introduced a taxonomy based on explanation scope and evaluation criteria.”
On the interpretability of artificial intelligence in radiology: challenges and opportunities
Reyes, M., Meier, R., Pereira, S., Silva, C. A., Dahlweid, F.-M., Tengg-Kobligk, H. V., Summers, R. M., & Wiest, R. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020190043 · Cited by 484
The Shattered Gradients Problem: If resnets are the answer, then what is the question?Preprint
Balduzzi, D., Frean, M. R., Leary, L., Lewis, J. P., Ma, W.-D. K., & McWilliams, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.08591 · Cited by 472
Explainable Deep Learning: A Field Guide for the Uninitiated
Ras, G., Xie, N., Van Gerven, M., & Doran, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13200 · Cited by 446
A., S., & R., S. (2023) · Decision Analytics Journal · doi:10.1016/j.dajour.2023.100230 · Cited by 432 · cited 1×
“[11] systematically reviewed Explainable AI models and applications, highlighting advancements in intrinsic and post-hoc methods.”
Layer-Wise Relevance Propagation for Neural Networks with Local Renormalization Layers
Binder, A., Montavon, G., Lapuschkin, S., Müller, K.-R., & Samek, W. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-44781-0_8 · Cited by 418
Omeiza, D., Speakman, S., Cintas, C., & Weldermariam, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.01224 · Cited by 398
On Completeness-aware Concept-Based Explanations in Deep Neural NetworksPreprint
Yeh, C., Kim, B., Arık, S. Ö., Li, C., Pfister, T., & Ravikumar, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.07969 · Cited by 397 · cited 1×
“ConceptSHAP, proposed by Yeh et al. [79], attributes importance to concepts by calculating Shapley values, which determine the contribution of each concept to the model’s output.”
Towards Automatic Concept-based Explanations
Ghorbani, A., Wexler, J., Zou, J., & Kim, B. (2019) · arXiv · doi:10.48550/arxiv.1902.03129 · Cited by 378
Towards better understanding of gradient-based attribution methods for Deep Neural Networks
Ancona, M., Ceolini, E., Öztireli, C., & Groß, M. H. (2017) · Zurich Open Repository and Archive (University of Zurich) · doi:10.3929/ethz-b-000249929 · Cited by 368 · cited 1×
“Deep Taylor Decomposition (DTD) extends LRP by using a first-order Taylor expansion around a chosen root point to decompose a neuron’s activation into contributions from its inputs [70].”
Schwalbe, G., & Finzel, B. (2024) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00867-8 · Cited by 328 · cited 1×
“Schwalbe and Finzel [10] provided a meta-survey, consolidating terminologies and methodologies from over 50 prominent XAI surveys into a unified taxonomy, aiding researchers in selecting appropriate explanation techniques.”
Weber, P., Carl, K. V., & Hinz, O. (2024) · Management Review Quarterly · doi:10.1007/s11301-023-00320-0 · Cited by 292 · cited 2×
“Other works apply post-hoc methods to domains such as medical imaging [15], [16], [20], pandemic response [21], [22], and financial modeling [23], [24].”
Nazir, S., Dickson, D. M., & Akram, M. U. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106668 · Cited by 283 · cited 1×
“Research has also explored XAI’s role in clinical decision support systems [14], medical image analysis [15], [16], trustworthy AI techniques in biomedical imaging [17], [18], [19], AI interpretability in radiology [20], and AI applications in response to COVID-19 [21], [22].”
Visual Analytics for Explainable Deep Learning
Choo, J., & Liu, S. (2018) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2018.042731661 · Cited by 270
A survey of visual analytics techniques for machine learning
Yuan, J., Chen, C., Yang, W., Liu, M., Xia, J., & Liu, S. (2021) · Computational Visual Media · doi:10.1007/s41095-020-0191-7 · Cited by 258 · cited 1×
“Yuan et al. [29] surveyed papers before 2020 using three categories: techniques before model building, techniques during modeling building, and techniques after model building.”
Explaining Classifiers with Causal Concept Effect (CaCE)Preprint
Goyal, Y., Feder, A., Shalit, U., & Kim, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.07165 · Cited by 251 · cited 1×
“The Causal Concept Effect (CaCE) method, introduced by Goyal et al. [78], computes the causal effect of high-level concepts on a model’s predictions.”
Explainable AI for Trees: From Local Explanations to Global Understanding
Lundberg, S., Erion, G., Chen, H., DeGrave, A. J., Prutkin, J. M., Nair, B. G., Katz, R., Himmelfarb, J., Bansal, N., & Lee, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.04610 · Cited by 249 · cited 1×
“Subsequent research has further explored SHAP’s theoretical foundations, applications to models like decision trees and probabilistic models, and its Bayesian interpretation for model interpretability [74].”
A survey of visual analytics for Explainable Artificial Intelligence methods
Alicioglu, G., & Sun, B. (2022) · Computers & Graphics · doi:10.1016/j.cag.2021.09.002 · Cited by 221 · cited 1×
“Alicioglu and Sun [30] reviewed visualization literature based on two different aspects, model usage and visual approaches.”
Hohman, F., Park, H., Robinson, C., & Polo Chau, D. H. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934659 · Cited by 213 · cited 1×
“SUMMIT [83] expands gradient-based visual analytics by introducing an attribution graph that visualizes feature interactions across neuron clusters.”
Explainable Artificial Intelligence (XAI) in Insurance
Owens, E., Sheehan, B., Mullins, M., Cunneen, M., Ressel, J., & Castignani, G. (2022) · Risks · doi:10.3390/risks10120230 · Cited by 182 · cited 2×
“Other works apply post-hoc methods to domains such as medical imaging [15], [16], [20], pandemic response [21], [22], and financial modeling [23], [24].”
From attribution maps to human-understandable explanations through Concept Relevance Propagation
Achtibat, R., Dreyer, M., Eisenbraun, I., Bosse, S., Wiegand, T., Samek, W., & Lapuschkin, S. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00711-8 · Cited by 181 · cited 2×
“Decomposition-based methods , like Layer-wise Relevance Propagation (LRP) [49] and Concept Relevance Propagation (CRP) [50] , offer detailed heatmaps that highlight specific features, focusing on key areas like the cat’s face and body. predictions or decisions.”
Natural and Realistic Single Image Super-Resolution With Explicit Natural Manifold Discrimination
Soh, J. W., Park, G. Y., Jo, J., & Cho, N. I. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00831 · Cited by 157 · cited 1×
“Recent advancements include feature interpolation techniques that improve interpretability in computer vision tasks [56].”
Explaining Convolutional Neural Networks using Softmax Gradient Layer-wise Relevance Propagation
Iwana, B. K., Kuroki, R., & Uchida, S. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00513 · Cited by 146 · cited 1×
“SGLRP provides a nuanced and accurate attribution of relevance, improving the interpretability of model decisions [72].”
Understanding Individual Decisions of CNNs via Contrastive Backpropagation
Gu, J., Yang, Y., & Tresp, V. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-20893-6_8 · Cited by 130
State of the Art of Visual Analytics for eXplainable Deep Learning
Rosa, B. L., Blasilli, G., Bourqui, R., Auber, D., Santucci, G., Capobianco, R., Bertini, E., Giot, R., & Angelini, M. (2023) · Computer Graphics Forum · doi:10.1111/cgf.14733 · Cited by 117 · cited 1×
“La Rosa et al. [31] reported recent contributions of visual analytics and aimed to provide solutions ready to be exploited by DL researchers.”
Visual Analytics for Machine Learning: A Data Perspective Survey
Wang, J., Liu, S., & Zhang, W. Z. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3357065 · Cited by 71 · cited 2×
“To structure our discussion, we adopt the data-centric framework proposed in [32], which has been used to summarize Visual Analytics for Machine Learning (VIS4ML) approaches.”
ConceptExplainer: Interactive Explanation for Deep Neural Networks from a Concept Perspective
Huang, J., Mishra, A., Kwon, B. C., & Bryan, C. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209384 · Cited by 67
Respond-CAM: Analyzing Deep Models for 3D Imaging Data by Visualizations
Zhao, G., Zhou, B., Wang, K., Jiang, R., & Xu, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00928-1_55 · Cited by 55
A Gather-to-Guide Network for Remote Sensing Semantic Segmentation of RGB and Auxiliary Image
Zheng, X., Wu, X., Huan, L., He, W. P., & Zhang, H. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3103517 · Cited by 44 · cited 1×
“Further research has expanded DeconvNets’ applications to image generation [52], unsupervised learning and natural language processing [53], small object detection [54], and semantic image synthesis [55].”
NeuroCartography: Scalable Automatic Visual Summarization of Concepts in Deep Neural Networks
Park, H., Das, N., Duggal, R., Wright, A. P., Shaikh, O., Hohman, F., & Chau, D. H. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114858 · Cited by 39 · cited 1×
“NEUROCARTOGRAPHY [88] enhances concept-based visualization by automating neuron clustering based on semantic similarity.”
Karthik, R., Menaka, R., Hariharan, M., & Kathiresan, G. S. (2021) · IRBM · doi:10.1016/j.irbm.2021.07.002 · Cited by 32 · cited 2×
“Research has also explored XAI’s role in clinical decision support systems [14], medical image analysis [15], [16], trustworthy AI techniques in biomedical imaging [17], [18], [19], AI interpretability in radiology [20], and AI applications in response to COVID-19 [21], [22].”
Kenny, E. M., Delaney, E. D., Greene, D., & Keane, M. T. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-68796-0_2 · Cited by 31 · cited 1×
“One of the most popular topics in the XAI literature is providing post-hoc explanations, which describe an ‘‘after-the-prediction explanation step where some evidence is given to elucidate the predictions made by the AI system’’ [2].”
Ouyang, Y., Wu, Y., Wang, H., Zhang, C., Cheng, F., Jiang, C., Jin, L., Cao, Y., & Li, Q. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3326929 · Cited by 31 · cited 1×
“DiagnosisAssistant [84] leverages Guided Grad-CAM to assist novice physicians in interpreting medical diagnostics as shown in Fig.”
A Visual Designer of Layer‐wise Relevance Propagation Models
Huang, X., Jamonnak, S., Zhao, Y., Wu, T. H., & Xu, W. (2021) · Computer Graphics Forum · doi:10.1111/cgf.14302 · Cited by 25 · cited 2×
“Table 4 summarizes various LRP rules and their respective parameters, providing insights into how different LRP variants propagate relevance scores through the network [75].”
Self-citationStrategyAtlas: Strategy Analysis for Machine Learning Interpretability
Collaris, D., & van Wijk, J. J. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3146806 · Cited by 21 · cited 1×
“STRATEGYATLAS [81] supports perturbation-based visualization by grouping data into ‘‘strategy clusters,’’ allowing domain experts to visually assess global model behavior.”
Bianchi, M., Santis, A. D., Tocchetti, A., & Brambilla, M. (2024) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2024/411 · Cited by 18
Visual Analytics for Efficient Image Exploration and User-Guided Image Captioning
Li, Y., Wang, J., Aboagye, P., Yeh, C. M., Zheng, Y. P., Wang, L., Zhang, W., & Ma, K. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3388514 · Cited by 16
Visual Analytics of Neuron Vulnerability to Adversarial Attacks on Convolutional Neural Networks
Li, Y., Wang, J., Fujiwara, T., & Ma, K. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3587470 · Cited by 11 · cited 1×
“Li et al. [80] introduce a system that enhances adversarial analysis through interactive visualizations.”
Research works printed but not linked: 17
- Nguyen, 2020, Artificial intelligence in the battle against coronavirus (COVID-19): A survey and future research directions, arXiv:2008.07343 · arXiv
- 2025, Google Scholar
- 2025, IEEE Xplore Digital Library
- 2025, Association for Computing Machinery (ACM), 23
- 2025, Springer—International Publisher Science, Technology, Medicine
- 2025, MDPI—Multidisciplinary Digital Publishing Institute
- 2025, Elsevier—Academic Publishing Company
- 2025, Scopus—Abstract and Citation Database
- 2025, ScienceDirect—Elsevier Journal and Book Platform
- 2025, ArXiv—Open-access Preprint Repository
- 2025, Inceptionism: Going Deeper into Neural Networks—Google AI Blog
- 2025, DeepDream Implementation—GitHub Repository by M. Fatin Akarsu
- 2025, DeepDream - dream.ipynb—GitHub Repository by Google
- 2015, Journey Through the Layers of the Mind—Memo Akten
- 2025, Inside Deep Dreams: How Google Made Its Computers Go Crazy—Wired
- 10.1109/ICCV.2019.00304 · DOI
- Anders, 2021, Software for dataset-wide XAI: From local explanations to global insights with zennit, CoRelAy, and ViRelAy, arXiv:2106.13200 · arXiv
Cited by
Osman, K. A. Y. A., Shah, A. F. M. S., Karabulut, M. A., Karahan, S. N., Osmanca, M. S., & Acır, N. (2026) · IEEE Access · doi:10.1109/access.2026.3663161 · Cited by 36
Almegdadi, O., Marcelino, J., Fakhreddine, S., Manso, J., & Marques, N. C. (2025) · Ecological Informatics · doi:10.1016/j.ecoinf.2025.103542 · Cited by 10
“Explainable AI (XAI) methods such as SHAP and LIME complement this perspective by providing post-hoc explanations for models whose internal structure is not directly interpretable (Bhati et al., 2025; Vale et al., 2022; Rudin, 2019; Huang et al., 2024; Taskin et al., 2024; Mia et al., 2023).”
Chen, T., Chiu, M.-C., & Lin, Y.-C. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2610447 · Cited by 6
“In addition, XAI methods can also be divided into several categories such as visualization XAI methods (Bhati et al., 2025; So, 2020), attribute importance evaluation XAI methods, and relationship approximation XAI methods (Chen, 2024).”
Explainable AI: enhancing decision-making in the detection of cyber threats
Prasad, P. W. C., Sayeed, M. S., Nguyen, D.-M., Hutabarat, D. P., & Mohiuddin, G. M. (2026) · Frontiers in Computer Science · doi:10.3389/fcomp.2026.1762332 · Cited by 16
Reimagining cancer treatments in the era of generative AI
Derbal, Y. (2025) · DIGITAL HEALTH · doi:10.1177/20552076251394631 · Cited by 4
“90,91 However, it has been argued that explanation of GenAI outputs alone is not suf fi - cient to engender trust and that efforts to improve transparency should be focused on the socio-technical system (e.g., treatment delivery system) that embed the AI models.”
Metadata
- Authors
- Bhati, D., Amiruzzaman, M., Zhao, Y., Guercio, A., & Le, T.
- Affiliations
- Kent State University (US)
- West Chester University (US)
- Venue
- IEEE Access
- Published
- 2025
- Volume
- 13
- Pages
- 120785-120806
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W4411406312
- Cited by
- 24
- References
- 76
- Citations by year
- 2026: 21 · 2025: 3
- FWCI
- 9.59
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Kent State University’s Open Access APC Support Fund
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- explainable AI, post-hoc explanation, visual analytics, deep learning interpretability, perturbation-based explanation, gradient-based explanation, concept-based explanations, feature attribution, global explanations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.