Cited by 1,164Explainable Artificial Intelligence (XAI)Advanced Neural Network ApplicationsVisual Attention and Saliency Detection

The DOI matches its Crossref record; 47 of the 51 references it cites were found in a registry or an index.

Evaluating the Visualization of What a Deep Neural Network Has Learned

Samek, Wojciech, Binder, Alexander, Montavon, Gregoire, Lapuschkin, Sebastian, Muller, Klaus-Robert·2017·IEEE Transactions on Neural Networks and Learning Systems

DOI 10.1109/tnnls.2016.2599820

Open this paper in CoChat — free account

Abstract

Deep neural networks (DNNs) have demonstrated impressive performance in complex machine learning tasks such as image classification or speech recognition. However, due to their multilayer nonlinear structure, they are not transparent, i.e., it is hard to grasp what makes them arrive at a particular classification or recognition decision, given a new unseen data sample. Recently, several approaches have been proposed enabling one to understand and interpret the reasoning embodied in a DNN for a single test image. These methods quantify the “importance” of individual pixels with respect to the classification decision and allow a visualization in terms of a heatmap in pixel/input space. While the usefulness of heatmaps can be judged subjectively by a human, an objective quality measure is missing. In this paper, we present a general methodology based on region perturbation for evaluating ordered collections of pixels such as heatmaps. We compare heatmaps computed by three different methods on the SUN397, ILSVRC2012, and MIT Places data sets. Our main result is that the recently proposed layer-wise relevance propagation algorithm qualitatively and quantitatively provides a better explanation of what made a DNN arrive at a particular classification decision than the sensitivity-based approach or the deconvolution method. We provide theoretical arguments to explain this result and discuss its practical implications. Finally, we investigate the use of heatmaps for unsupervised assessment of the neural network performance.

Integrity signals

Registry

Crossref · 10.1109/tnnls.2016.2599820

Cites

47 of 47 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,889
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381
An introduction to variable and feature selection
GuyonIsabelle, & ElisseeffAndré (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944968 · Cited by 8,354
Domain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143

Show all 51 references

Research works printed but not linked: 1

jia, 2013, Caffe An Open Source Convolutional Architecture for Fast Feature Embedding

Sorted by citations

Items 601–700 of 1,000

Explain to Not Forget: Defending Against Catastrophic Forgetting with XAI
Ede, S., Baghdadlian, S., Weber, L., Nguyen, A., Zanca, D., Samek, W., & Lapuschkin, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-14463-9_1 · Cited by 10
Self-citation
Multi-touch Attribution for Complex B2B Customer Journeys using Temporal Convolutional Networks
Agrawal, A., Sheoran, N., Suman, S., & Sinha, G. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3487553.3524670 · Cited by 9
Lessons From Deep Neural Networks for Studying the Coding Principles of Biological Neural Networks
Bae, H., Kim, S. J., & Kim, C.-E. (2021) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2020.615129 · Cited by 10
Quantifying Explainability of Saliency Methods in Deep Neural NetworksPreprint
Tjoa, E., & Guan, C. (2020) · arXiv (Cornell University) · Cited by 9
Rethinking the Role of Gradient-Based Attribution Methods for Model InterpretabilityPreprint
Srinivas, S., & Fleuret, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.09128 · Cited by 12
Explainable Deep Learning for Video Recognition Tasks: A Framework & RecommendationsPreprint
Hiley, L., Preece, A. D., & Hicks, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.05667 · Cited by 17
Deep Learning Detection and Recognition of Spot Elevations on Historical Topographic Maps
Arundel, S. T., Morgan, T. P., & Thiem, P. T. (2022) · Frontiers in Environmental Science · doi:10.3389/fenvs.2022.804155 · Cited by 8
Can We Trust Explainable AI Methods on ASR? An Evaluation on Phoneme Recognition
Wu, X., Bell, P., & Rajan, A. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10445989 · Cited by 8
Interpretable A-posteriori error indication for graph neural network surrogate models
Barwey, S., Kim, H., & Maulik, R. (2024) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2024.117509 · Cited by 15
Feasibility of a clouds and edge computing scheme for internal due date assignment in advanced manufacturing systems
Chen, T., & Lin, C. (2025) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-024-14980-z · Cited by 8
Effects of mixed sample data augmentation on interpretability of neural networks
Won, S., Bae, S., & Kim, S. T. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107611 · Cited by 8
To pretrain or not? A systematic analysis of the benefits of pretraining in diabetic retinopathy
Srinivasan, V., Strodthoff, N., Ma, J., Binder, A., Müller, K., & Samek, W. (2022) · PLoS ONE · doi:10.1371/journal.pone.0274291 · Cited by 16
Self-citation
Extending Logic Explained Networks to Text Classification
Jain, R., Ciravegna, G., Barbiero, P., Giannini, F., Buffelli, D., & Lió, P. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.604 · Cited by 8
Visualization of feature evolution during convolutional neural network training
Punjabi, A., & Katsaggelos, A. K. (2017) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco.2017.8081219 · Cited by 9
PEARL
Fu, T., Gao, T., Xiao, C., Ma, T., & Sun, J. (2019) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3307339.3342159 · Cited by 8
Human-In-The-Loop Construction of Decision Tree Classifiers with Parallel Coordinates
Estivill‐Castro, V., Gilmore, E., & Hexel, R. (2020) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc42975.2020.9283240 · Cited by 8
Explainable Hopfield Neural Networks Using an Automatic Video-Generation System
Rubio-Manzano, C., Segura-Navarrete, A., Martínez-Araneda, C., & Vidal‐Castro, C. (2021) · Applied Sciences · doi:10.3390/app11135771 · Cited by 8
Training Convolutional Neural Networks with Differential Evolution using Concurrent Task Apportioning on Hybrid CPU-GPU Architectures
Venkat, R. A., Oussalem, Z., & Bhattacharya, A. K. (2021) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec45853.2021.9504878 · Cited by 8
Evaluation of the Explanatory Power Of Layer-wise Relevance Propagation using Adversarial Examples
Dieter, T. R., & Zisgen, H. (2023) · Neural Processing Letters · doi:10.1007/s11063-023-11166-8 · Cited by 8
Hypericons for interpretability: decoding abstract concepts in visual data
Pandiani, D. S. M., Lazzari, N., van Erp, M., & Presutti, V. (2023) · International Journal of Digital Humanities · doi:10.1007/s42803-023-00077-8 · Cited by 11
Explainable Deep Learning for Fault Prognostics in Complex Systems: A Particle Accelerator Use-Case
Felsberger, L., Apollonio, A., Cartier-Michaud, T., Müller, A., Todd, B., & Kranzlmüller, D. A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-57321-8_8 · Cited by 8
Leveraging conditional generative models in a general explanation framework of classifier decisions
Charachon, M., Cournède, P.-H., Hudelot, C., & Ardon, R. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2022.02.020 · Cited by 10
A Scoping Review of Artificial Intelligence Algorithms in Clinical Decision Support Systems for Internal Medicine Subspecialties
Narindrarangkura, P., Kim, M. S., & Boren, S. A. (2021) · ACI Open · doi:10.1055/s-0041-1735470 · Cited by 8
Technologies for Trustworthy Machine Learning: A Survey in a Socio-Technical ContextPreprint
Toreini, E., Aitken, M., Coopamootoo, K., Elliott, K., González-Zelaya, V., Missier, P., Ng, M., & van Moorsel, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.08911 · Cited by 24
Improving Interpretability in Medical Imaging Diagnosis using Adversarial TrainingPreprint
Margeloiu, A., Simidjievski, N., Jamnik, M., & Weller, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.01166 · Cited by 14
User Effects on Mobile Phone Antennas: Review and Potential Future Solutions
Syrytsin, I., Pedersen, G. F., & Zhang, S. (2023) · IEEE Open Journal of Antennas and Propagation · doi:10.1109/ojap.2023.3337494 · Cited by 7
Monitoring an Ecosystem in Crisis: Measuring Seagrass Meadow Loss Using Deep Learning in Mosquito Lagoon, Florida
Insalaco, S. A., Herrero, H. V., Limber, R., Oliver, C., & Wolfson, W. B. (2024) · Photogrammetric Engineering & Remote Sensing · doi:10.14358/pers.24-00001r2 · Cited by 7
Artificial Intelligence in Genetics
Sanders, M. A. (2024) · Elsevier eBooks · doi:10.1016/b978-0-443-12128-9.00006-1 · Cited by 7
Trustworthy human computation: a survey
Kashima, H., Oyama, S., Arai, H., & Mori, J. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10974-1 · Cited by 9
Dynamic Quasi-Hyperbolic Momentum Iterative Attack With Small Perturbation for 4D-Flight Trajectory Prediction
Zhao, Z., Wang, B., Yao, X., Tian, J., Dong, R., & Zhu, P. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3554800 · Cited by 7
Pruning by Explaining Revisited: Optimizing Attribution Methods to Prune CNNs and Transformers
Hatefi, S. M. V., Dreyer, M., Achtibat, R., Wiegand, T., Samek, W., & Lapuschkin, S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-92648-8_10 · Cited by 15
Self-citation
An interpretable machine learning framework with data-informed imaging biomarkers for diagnosis and prediction of Alzheimer’s disease
Kang, W., Li, B., Jiskoot, L. C., Deyn, P. P. D., Biessels, G. J., Koek, H. L., Claassen, J. A. H. R., Middelkoop, H. A. M., van der Flier, W. M., Jansen, W. J., et al. (2026) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2026.102722 · Cited by 8
Local interpretation of supervised learning models based on high dimensional model representation
Zhang, X., Ling, W., & Li, Z. (2021) · Information Sciences · doi:10.1016/j.ins.2021.01.079 · Cited by 7
Adding Context to Source Code Representations for Deep Learning
Tian, F., & Treude, C. (2022) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme55016.2022.00042 · Cited by 9
Classification of Low-Strain Foundation Pile Testing Signal Using Recurrent Neural Network
Wang, H., Zhang, S., Li, J., Yuan, Y., & Zhang, F. (2023) · Buildings · doi:10.3390/buildings13051228 · Cited by 7
Improving Interpretability via Explicit Word Interaction Graph Layer
Sekhon, A., Chen, H., Shrivastava, A., Wang, Z., Ji, Y., & Qi, Y. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26586 · Cited by 7
Ensembling to Leverage the Interpretability of Medical Image Analysis Systems
Zafeiriou, A., Kallipolitis, A., & Maglogiannis, I. G. (2023) · IEEE Access · doi:10.1109/access.2023.3291610 · Cited by 7
The Manifold Hypothesis for Gradient-Based Explanations
Bordt, S., Upadhyay, U., Akata, Z., & Ulrike Von Luxburg, A. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00378 · Cited by 10
Robustness of Visual Explanations to Common Data Augmentation Methods
Tětková, L., & Hansen, L. K. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00381 · Cited by 8
Explaining Image Classifiers with Multiscale Directional Image Representation
Kolek, S., Windesheim, R., Andrade-Loarca, H., Gitta Kutyniok, B., & Levie, R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01784 · Cited by 9
Explaining Deep Face Algorithms Through Visualization: A Survey
John, T. A., Balasubramanian, V. N., & Jawahar, C. V. (2023) · IEEE Transactions on Biometrics Behavior and Identity Science · doi:10.1109/tbiom.2023.3319837 · Cited by 7
An Empirical Analysis of User Preferences Regarding XAI Metrics
Darias, J. M., Bayrak, B., Caro-Martínez, M., Díaz-Agudo, B., & Recio-García, J. A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-63646-2_7 · Cited by 9
Beyond Deep Learning: Charting the Next Frontiers of Affective Computing
Triantafyllopoulos, A., Christ, L., Gebhard, A., Jing, X., Kathan, A., Milling, M., Tsangko, I., Amiriparian, S., & Schuller, B. W. (2024) · Intelligent Computing · doi:10.34133/icomputing.0089 · Cited by 12
ConvNext as a Basis for Interpretability in Coffee Leaf Rust Classification
Chavarro, A., Renza, D., & Moya-Albor, E. (2024) · Mathematics · doi:10.3390/math12172668 · Cited by 7
Salient Explanation for Fine-Grained Classification
Oh, K., Kim, S., & Oh, I.-S. (2020) · IEEE Access · doi:10.1109/access.2020.2980742 · Cited by 7
Shallow2Deep: Restraining Neural Networks Opacity Through Neural Architecture Search
Agiollo, A., Ciatto, G., & Omicini, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-82017-6_5 · Cited by 7
Interpretable Research Replication Prediction via Variational Contextual Consistency Sentence Masking
Luo, T., Meng, R., Wang, X., & Liu, Y. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.305 · Cited by 10
Construction of a competing endogenous RNA network to analyse glucose-6-phosphate dehydrogenase dysregulation in hepatocellular carcinoma
Wang, P., Yang, X., Liu, D., Yang, Y., Zhang, Y., & Wang, G. (2022) · Bioscience Reports · doi:10.1042/bsr20220674 · Cited by 6
DecompX: Explaining Transformers Decisions by Propagating Token Decomposition
Modarressi, A., Fayyaz, M., Aghazadeh, E., Yaghoobzadeh, Y., & Pilehvar, M. T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.149 · Cited by 11
Voice Presentation Attack Detection Using Convolutional Neural Networks
Himawan, I., Madikeri, S. R., Motlíček, P., Cerňak, M., Sridharan, S., & Fookes, C. (2019) · Advances in computer vision and pattern recognition · doi:10.1007/978-3-319-92627-8_17 · Cited by 7
Evaluating Attribution Methods using White-Box LSTMs
Hao, Y. (2020) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2020.blackboxnlp-1.28 · Cited by 7
Neural network training fingerprint: visual analytics of the training process in classification neural networks
Ferreira, M. D., Cantareira, G. D., de Mello, R. F., & Paulovich, F. V. (2021) · Journal of Visualization · doi:10.1007/s12650-021-00809-4 · Cited by 7
EvalAttAI: A Holistic Approach to Evaluating Attribution Maps in Robust and Non-Robust Models
Nielsen, I. E., Ramachandran, R. P., Bouaynaya, N. C., Fathallah‐Shaykh, H. M., & Rasool, G. (2023) · IEEE Access · doi:10.1109/access.2023.3300242 · Cited by 9
An Interpretable Generative Adversarial Approach to Classification of Latent Entity Relations in Unstructured Sentences
Hsu, S. T., Moon, C., Jones, P., & Samatova, N. F. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11972 · Cited by 7
Aggregating explanation methods for stable and robust explainabilityPreprint
Rieger, L., & Hansen, L. K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.00519 · Cited by 16
Achievements and Challenges in Explaining Deep Learning based Computer-Aided Diagnosis SystemsPreprint
Lucieri, A., Bajwa, M. N., Dengel, A. R., & Ahmed, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.13169 · Cited by 20
Robust Semantic Interpretability: Revisiting Concept Activation VectorsPreprint
Pfau, J., Young, A. T., Wei, J., Wei, M. L., & Keiser, M. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.02768 · Cited by 13
Using Neural Networks to Predict Micro-Spatial Economic GrowthPreprint
Khachiyan, A., Thomas, A. H., Zhou, H., Hanson, G., Cloninger, A., Rosing, T., & Khandelwal, A. K. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3989577 · Cited by 7
Looking Deeper into Deep Learning Model: Attribution-based Explanations of TextCNNPreprint
Xiong, W., Ni’mah, I., Huesca, J. M. G., van Ipenburg, W., Veldsink, J., & Pechenizkiy, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.03970 · Cited by 7
Advancing Attribution-Based Neural Network Explainability through Relative Absolute Magnitude Layer-Wise Relevance Propagation and Multi-Component Evaluation
Vukadin, D., Afrić, P., Šilić, M., & Delač, G. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3649458 · Cited by 7
Deep learning-supported machine vision-based hybrid system combining inhomogeneous 2D and 3D data for the identification of surface defects
Cavaliere, G., Lanz, O., Borgianni, Y., & Savio, E. (2024) · Production & Manufacturing Research · doi:10.1080/21693277.2024.2378199 · Cited by 7
Explainable Quantum Neural Networks: Example-Based and Feature-Based Methods
Tian, J., & Yang, W. (2024) · Electronics · doi:10.3390/electronics13204136 · Cited by 8
AI ethics unwrapped: an empirical investigation of ethical principles in collaborative ideation processes
Cociancig, C., Heuer, H., & Breiter, A. (2024) · AI and Ethics · doi:10.1007/s43681-024-00638-9 · Cited by 6
A systematic review: Brain age gap as a promising early diagnostic biomarker for Alzheimer's disease
Liu, S., Xu, L., Xu, G., Wang, Y., Zhang, G., & He, L. (2025) · Journal of the Neurological Sciences · doi:10.1016/j.jns.2025.123563 · Cited by 7
Evaluating Model Explanations without Ground Truth
Rawal, K., Fu, Z., Delaney, E. D., & Russell, C. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732219 · Cited by 11
Advancing resilience in infrastructure projects through machine learning-driven models
Ojiako, U., Wong, T. C., Smith, C. J., Chipulu, M., Al-Mhdawi, M. K. S., Oyewo, B. M., & Obokoh, L. O. (2025) · Production Planning & Control · doi:10.1080/09537287.2025.2583300 · Cited by 8
Multimodal deep learning model for enhanced early detection of aortic stenosis integrating ECG and chest x-ray with cooperative learning
Nagai, S., Nishimori, M., Shinohara, M., Tanaka, H., & Otake, H. (2025) · Frontiers in Radiology · doi:10.3389/fradi.2025.1698680 · Cited by 6
From physics to intelligence: a review of AI-driven modeling strategies in laser direct energy deposition additive manufacturing
Nabavi, S. F., Nahavandi, S., & Garmestani, H. (2026) · Optics & Laser Technology · doi:10.1016/j.optlastec.2026.114946 · Cited by 7
Faster Capacitated Arc Routing: A Sequence-to-Sequence Approach
Hong, W., & Liu, T. (2022) · IEEE Access · doi:10.1109/access.2022.3140783 · Cited by 6
Impact and Prediction of AI Diagnostic Report Interpretation Type on Patient Trust
Tang, Y., & Cai, J. (2023) · Frontiers in Computing and Intelligent Systems · doi:10.54097/fcis.v3i3.8567 · Cited by 9
On the coherency of quantitative evaluation of visual explanations
Vandersmissen, B., & Jose Oramas, M. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.103934 · Cited by 7
The Reality of High Performing Deep Learning Models: A Case Study on Document Image Classification
Saifullah, S., Agne, S., Dengel, A. R., & Ahmed, S. (2024) · IEEE Access · doi:10.1109/access.2024.3425910 · Cited by 9
Privacy, Trust and Ethical Issues
Shadbolt, N., O’Hara, K., Roure, D. D., & Hall, W. (2019) · Lecture notes in social networks · doi:10.1007/978-3-030-10889-2_4 · Cited by 7
Emotion Recognition Based on Multi-Composition Deep Forest and Transferred Convolutional Neural Network
Liu, X., Xu, Y., Wang, M., Cai, Y., & Qi, G. H. (2019) · Journal of Advanced Computational Intelligence and Intelligent Informatics · doi:10.20965/jaciii.2019.p0883 · Cited by 6
Interpreting a Black-Box Model used for SCADA Attack detection in Gas Pipelines Control System
Rathod, J., Joshi, C., Khochare, J., & Kazi, F. (2020) · IEEE India Conference · doi:10.1109/indicon49873.2020.9342087 · Cited by 6
Ensemble cross‐stage partial attention network for image classification
Lin, H., & Yang, J. (2021) · IET Image Processing · doi:10.1049/ipr2.12335 · Cited by 6
A Review of Interpretable Deep Learning for Neurological Disease Classification
Katarya, R., Sharma, P., Soni, N., & Rath, P. (2022) · International Conference on Advanced Computing and Communication Systems (ICACCS) · doi:10.1109/icaccs54159.2022.9785321 · Cited by 6
Automatic and robust design for multiple self-piercing riveted joints using deep neural network
Zhao, H., Li, H., Liu, Y., & Liu, X. (2022) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-022-09893-8 · Cited by 6
Constructing Explainable Classifiers from the Start—Enabling Human-in-the Loop Machine Learning
Estivill‐Castro, V., Gilmore, E., & Hexel, R. (2022) · Information · doi:10.3390/info13100464 · Cited by 6
Improving explainability results of convolutional neural networks in microscopy images
Kallipolitis, A., Yfantis, P., & Maglogiannis, I. G. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08452-w · Cited by 6
Information Entropy Measures for Evaluation of Reliability of Deep Neural Network Results
Gireesh, E. D., & Gurupur, V. P. (2023) · Entropy · doi:10.3390/e25040573 · Cited by 6
A Novel No-Reference Quality Assessment Metric for Stereoscopic Images with Consideration of Comprehensive 3D Quality Information
Shen, L., Yao, Y., Geng, X., Fang, R., & Wu, D. O. (2023) · Sensors · doi:10.3390/s23136230 · Cited by 6
Harmonizing Feature Attributions Across Deep Learning Architectures: Enhancing Interpretability and Consistency
Kadir, M. A., Addluri, G. K., & Sonntag, D. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-42608-7_8 · Cited by 6
Towards Improved Input Masking for Convolutional Neural Networks
Balasubramanian, S., & Feizi, S. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00178 · Cited by 6
On making sense of neural networks in road analysis
Morris, D., Antoniades, A., & Took, C. C. (2017) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966415 · Cited by 6
Time Series Forecasting with Multiple Deep Learners: Selection from a Bayesian Network
Kobayashi, S., & Shirayama, S. (2017) · Journal of Data Analysis and Information Processing · doi:10.4236/jdaip.2017.53009 · Cited by 6
A Conceptual Framework for Stochastic Neuromorphic Computing
Gaines, B. R. (2020) · IEEE Design and Test · doi:10.1109/mdat.2020.3031857 · Cited by 6
Selection of the Main Control Parameters for the Dst Index Prediction Model Based on a Layer-wise Relevance Propagation Method
Li, Y., Huang, S. Y., Xu, S., Yuan, Z., Jiang, K., Wei, Y. Y., Zhang, J., Xiong, Q. Y., Wang, Z., Lin, R. T., et al. (2022) · The Astrophysical Journal Supplement Series · doi:10.3847/1538-4365/ac616c · Cited by 6
Deep learning with explainability for characterizing age-related intrinsic differences in dynamic brain functional connectivity
Qiao, C., Gao, B., Liu, Y., Hu, X., Hu, W., Calhoun, V. D., & Wang, Y. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102941 · Cited by 7
“What are You Listening to?” Explaining Predictions of Deep Machine Listening Systems
Mishra, S., Sturm, B. L. T., & Dixon, S. (2018) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco.2018.8553178 · Cited by 6
Deep Learning in Gesture Recognition Based on sEMG Signals
Tsinganos, P., Skodras, A. N., Cornelis, B., & Jansen, B. (2018) · VUBIR (Vrije Universiteit Brussel) · doi:10.1201/9780429061141-23 · Cited by 6
Layerwise Knowledge Extraction from Deep Convolutional NetworksPreprint
Odense, S., & Garcez, A. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.09000 · Cited by 10
Towards Interpretable Ensemble Learning for Image-based Malware DetectionPreprint
Lin, Y., & Chang, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.04889 · Cited by 6
Benchmarking Perturbation-based Saliency Maps for Explaining Deep Reinforcement Learning Agents.Preprint
Huber, T., Limmer, B., & André, E. (2021) · arXiv (Cornell University) · Cited by 6
On the Robustness of Pretraining and Self-Supervision for a Deep Learning-based Analysis of Diabetic RetinopathyPreprint
Srinivasan, V., Strodthoff, N., Ma, J., Binder, A., Müller, K., & Samek, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.13497 · Cited by 6
Self-citation
Characterizing personalized neuropathology in dementia and mild cognitive impairment with explainable artificial intelligencePreprint
Leonardsen, E. H., Persson, K., Grødem, E., Dinsdale, N. K., Schellhorn, T., Roe, J. M., Dídac Vidal-piñeiro, A., Sørensen, Ø., Kaufmann, T., Westman, E., et al. (2023) · medRxiv · doi:10.1101/2023.06.22.23291592 · Cited by 6
XAI-based Data Visualization in Multimodal Medical DataPreprint
Sharma, S., Singh, M., Mcdaid, L. J., & Bhattacharyya, S. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.07.11.664302 · Cited by 7
Discriminative attribution from paired images
Eckstein, N., Bukhari, H., Bates, A. S., Jefferis, G. S. X. E., & Funke, J. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-25069-9_27 · Cited by 5
Explainable AI Model Reveals Informative Mutational Signatures for Cancer-Type Classification
Wagner, J., Oldenburg, J., Nath, N., & Simm, S. (2025) · Cancers · doi:10.3390/cancers17111731 · Cited by 5
CLARiC: Contrastive Learning and Root-Cause Inference With Causality for Explainable Smart Manufacturing
Lin, C., Tseng, T.-L., & Tsai, T.-H. (2025) · IEEE Access · doi:10.1109/access.2025.3607848 · Cited by 5

Authors: Samek, W., Binder, A., Montavon, G., Lapuschkin, S., & Muller, K.-R.
Affiliations: Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (DE)
Singapore University of Technology and Design (SG)
Technische Universität Berlin, Berlin, Germany
ORCID: Samek, Wojciech
Binder, Alexander
Lapuschkin, Sebastian
Venue: IEEE Transactions on Neural Networks and Learning Systems
Publisher: Institute of Electrical and Electronics Engineers (IEEE)
Published: 2017-11
Volume: 28
Issue: 11
Pages: 2660-2673
ISSN: 2162-237X (ISSN-L)
2162-2388
DOI: 10.1109/tnnls.2016.2599820
OpenAlex: W2240067561
Cited by: 1,164
References: 51
Citations by year: 2026: 120 · 2025: 105 · 2024: 139 · 2023: 163 · 2022: 147 · 2021: 171 · 2020: 123 · 2019: 92 · 2018: 66 · 2017: 26
FWCI: 53.43
Type: journal article
Language: en
Open access: closed
Licence: ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html (vor)
Locations: IEEE Transactions on Neural Networks and Learning Systems (publishedVersion)
PubMed (publishedVersion)
Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion)
Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
Funders: Brain Korea 21 Plus Program through the National Research Foundation of Korea Funded by the Ministry of Education
DFG (MU 987/17-1)
German Ministry for Education and Research as Berlin Big Data Center (01IS14013A)
Topics: Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
Keywords: deep neural network interpretability, layer-wise relevance propagation, deconvolution, explainable AI, image classification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Evaluating the Visualization of What a Deep Neural Network Has Learned

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Citations

Cites

Cited by

Related

Metadata

Record

About these checks