Cited by 1,164Explainable Artificial Intelligence (XAI)Advanced Neural Network ApplicationsVisual Attention and Saliency Detection

The DOI matches its Crossref record; 47 of the 51 references it cites were found in a registry or an index.

Evaluating the Visualization of What a Deep Neural Network Has Learned

Samek, Wojciech, Binder, Alexander, Montavon, Gregoire, Lapuschkin, Sebastian, Muller, Klaus-Robert·2017·IEEE Transactions on Neural Networks and Learning Systems

DOI 10.1109/tnnls.2016.2599820

Open this paper in CoChat — free account

Abstract

Deep neural networks (DNNs) have demonstrated impressive performance in complex machine learning tasks such as image classification or speech recognition. However, due to their multilayer nonlinear structure, they are not transparent, i.e., it is hard to grasp what makes them arrive at a particular classification or recognition decision, given a new unseen data sample. Recently, several approaches have been proposed enabling one to understand and interpret the reasoning embodied in a DNN for a single test image. These methods quantify the “importance” of individual pixels with respect to the classification decision and allow a visualization in terms of a heatmap in pixel/input space. While the usefulness of heatmaps can be judged subjectively by a human, an objective quality measure is missing. In this paper, we present a general methodology based on region perturbation for evaluating ordered collections of pixels such as heatmaps. We compare heatmaps computed by three different methods on the SUN397, ILSVRC2012, and MIT Places data sets. Our main result is that the recently proposed layer-wise relevance propagation algorithm qualitatively and quantitatively provides a better explanation of what made a DNN arrive at a particular classification decision than the sensitivity-based approach or the deconvolution method. We provide theoretical arguments to explain this result and discuss its practical implications. Finally, we investigate the use of heatmaps for unsupervised assessment of the neural network performance.

Integrity signals

Registry

Crossref · 10.1109/tnnls.2016.2599820

Cites

47 of 47 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,889
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381
An introduction to variable and feature selection
GuyonIsabelle, & ElisseeffAndré (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944968 · Cited by 8,354
Domain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143

Flagged first, then by citations

Show all 51 references

4 of the 51 references couldn’t be matched to a known paper.

Research works printed but not linked: 1

jia, 2013, Caffe An Open Source Convolutional Architecture for Fast Feature Embedding

Sorted by citations

Items 301–400 of 1,000

Opti-CAM: Optimizing saliency maps for interpretability
Zhang, H., Torres, F., Sicre, R., Avrithis, Y., & Ayache, S. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104101 · Cited by 48
The Computational Complexity of Understanding Binary Classifier Decisions
Wäldchen, S., Macdonald, J., Hauch, S., & Gitta Kutyniok, B. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12359 · Cited by 44
Surrogate Model-Based Explainability Methods for Point Cloud NNs
Tan, H., & Kotthaus, H. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00298 · Cited by 33
Review of white box methods for explanations of convolutional neural networks in image classification tasks
Ayyar, M. P., Benois‐Pineau, J., & Zemmari, A. (2021) · Journal of Electronic Imaging · doi:10.1117/1.jei.30.5.050901 · Cited by 35
What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods.
Colin, J., Fel, T., Cadène, R., & Serre, T. (2022) · PubMed · Cited by 33
What Do You See? Evaluation of Explainable Artificial Intelligence (XAI) Interpretability through Neural BackdoorsPreprint
Lin, Y., Lee, W., & Celik, Z. B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.10639 · Cited by 33
Convolutional network architectures for super-resolution/sub-pixel mapping of drone-derived images
Arun, P. V., Herrmann, I., Buddhiraju, K. M., & Karnieli, A. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2018.11.033 · Cited by 32
Understanding Patch-Based Learning of Video Data by Explaining Predictions
Anders, C. J., Montavon, G., Samek, W., & Müller, K. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_16 · Cited by 32
Self-citation
Towards Explainable Classifiers Using the Counterfactual Approach - Global Explanations for Discovering Bias in Data
Mikołajczyk, A., Grochowski, M., & Kwasigroch, A. (2020) · Journal of Artificial Intelligence and Soft Computing Research · doi:10.2478/jaiscr-2021-0004 · Cited by 35
Interpretable deep learning for the remote characterisation of ambulation in multiple sclerosis using smartphones
Creagh, A. P., Lipsmeier, F., Lindemann, M., & Vos, M. D. (2021) · Scientific Reports · doi:10.1038/s41598-021-92776-x · Cited by 33
Artificial intelligence combined with high-throughput calculations to improve the corrosion resistance of AlMgZn alloy
Ji, Y., Fu, X., Ding, F., Xu, Y., He, Y., Ao, M., Xiao, F., Chen, D., Dey, P., Qin, W., et al. (2024) · Corrosion Science · doi:10.1016/j.corsci.2024.112062 · Cited by 32
Advancements in deep learning and explainable artificial intelligence for enhanced medical image analysis: A comprehensive survey and future directions
Mir, A. N., & Rizvi, D. R. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111413 · Cited by 34
VINet: A Visually Interpretable Image Diagnosis Network
Gu, D., Li, Y., Jiang, F., Wen, Z., Liu, S., Shi, W., Lu, G., & Zhou, C. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2971170 · Cited by 30
Road safety performance function analysis with visual feature importance of deep neural nets
Pan, G., Fu, L., Chen, Q., Yu, M., & Muresan, M. (2020) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2020.1003108 · Cited by 31
Understanding the decisions of CNNs: An in-model approach
Rio-Torto, I., Fernandes, K., & Teixeira, L. F. (2020) · Pattern Recognition Letters · doi:10.1016/j.patrec.2020.04.004 · Cited by 35
A method to evaluate task-specific importance of spatio-temporal units based on explainable artificial intelligence
Cheng, X., Wang, J., Li, H., Zhang, Y., Wu, L., & Liu, Y. (2020) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2020.1805116 · Cited by 31
The Simultaneous Prediction of Soil Properties and Vegetation Coverage from Vis-NIR Hyperspectral Data with a One-Dimensional Convolutional Neural Network: A Laboratory Simulation Study
Zhang, F., Wang, C., Pan, K., Guo, Z., Liu, J., Xu, A., Ma, H., & Pan, X. (2022) · Remote Sensing · doi:10.3390/rs14020397 · Cited by 31
Evaluations and Methods for Explanation through Robustness AnalysisPreprint
Hsieh, C.-Y., Yeh, C., Liu, X., Ravikumar, P., Kim, S., Kumar, S., & Hsieh, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.00442 · Cited by 55
Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity AnalysisPreprint
Fel, T., Cadène, R., Chalvidal, M., Cord, M., Vigouroux, D., & Serre, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.04138 · Cited by 39
An adversarial attack approach for eXplainable AI evaluation on deepfake detection models
Gowrisankar, B., & Thing, V. L. L. (2024) · Computers & Security · doi:10.1016/j.cose.2023.103684 · Cited by 30
Understanding Integrated Gradients with SmoothTaylor for Deep Neural Network Attribution
Goh, G. S. W., Lapuschkin, S., Weber, L., Samek, W., & Binder, A. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9413242 · Cited by 30
Self-citation
Objective Diagnosis for Histopathological Images Based on Machine Learning Techniques: Classical Approaches and New Trends
Elazab, N., Soliman, H., El-Sappagh, S., Islam, S. M. R., & Elmogy, M. M. (2020) · Mathematics · doi:10.3390/math8111863 · Cited by 36
A Confusion Matrix for Evaluating Feature Attribution Methods
Arias-Duart, A., Mariotti, E., García-Gasulla, D., & Alonso, J. M. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00380 · Cited by 30
Attention-based GCN integrates multi-omics data for breast cancer subtype classification and patient-specific gene marker identification
Guo, H., Lv, X., Li, Y., & Li, M. (2023) · Briefings in Functional Genomics · doi:10.1093/bfgp/elad013 · Cited by 28
Unlocking the black box of CNNs: Visualising the decision-making process with PRISM
Szandała, T. (2023) · Information Sciences · doi:10.1016/j.ins.2023.119162 · Cited by 29
Prediction of equivalent sand-grain size and identification of drag-relevant scales of roughness – a data-driven approach
Yang, J., Stroh, A., Lee, S.-S., Bagheri, S., Frohnapfel, B., & Forooghi, P. (2023) · Journal of Fluid Mechanics · doi:10.1017/jfm.2023.881 · Cited by 29
Decoding Human Facial Emotions: A Ranking Approach Using Explainable AI
Punuri, S. B., Kuanar, S. K., Mishra, T. K., Rao, V. V. R. M., & Reddy, S. S. (2024) · IEEE Access · doi:10.1109/access.2024.3474012 · Cited by 30
Enhancing the Value of Histopathological Assessment of Allograft Biopsy Monitoring
Wood, M. A., Lesniak, A. J., & Demetris, A. J. (2019) · Transplantation · doi:10.1097/tp.0000000000002656 · Cited by 29
Analysis and Visualization of Deep Neural Networks in Device-Free Wi-Fi Indoor Localization
Liu, S.-J., Chang, R. Y., & Chien, F. (2019) · IEEE Access · doi:10.1109/access.2019.2918714 · Cited by 29
A Study on Development of the Camera-Based Blind Spot Detection System Using the Deep Learning Methodology
Kwon, D., Malaiya, R. K., Yoon, G., Ryu, J.-T., & Pi, S.-Y. (2019) · Applied Sciences · doi:10.3390/app9142941 · Cited by 31
Easy Perturbation EEG Algorithm for Spectral Importance (easyPEASI)
Nahmias, D. O., & Kontson, K. L. (2020) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3394486.3403289 · Cited by 29
Awakening Latent Grounding from Pretrained Language Models for Semantic Parsing
Liu, Q., Yang, D., Zhang, J., Guo, J., Zhou, B., & Lou, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.100 · Cited by 32
NoiseGrad — Enhancing Explanations by Introducing Stochasticity to Model Weights
Bykov, K., Hedström, A., Nakajima, S., & Höhne, M. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i6.20561 · Cited by 34
Novel technical and privacy-preserving technology for artificial intelligence in ophthalmology
Lim, J. S., Hong, M., Lam, W. S. T., Zhang, Z., Teo, Z. L., Liu, Y., Ng, W. Y., Foo, L. L., & Ting, D. S. W. (2022) · Current Opinion in Ophthalmology · doi:10.1097/icu.0000000000000846 · Cited by 31
Explaining the Predictions of Unsupervised Learning Models
Montavon, G., Kauffmann, J., Samek, W., & Müller, K. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-04083-2_7 · Cited by 29
Self-citation
Regional Multi-Scale Approach for Visually Pleasing Explanations of Deep Neural Networks
Seo, D., Oh, K., & Oh, I.-S. (2019) · IEEE Access · doi:10.1109/access.2019.2963055 · Cited by 31
Transparency: Motivations and ChallengesPreprint
Weller, A. (2017) · arXiv (Cornell University) · doi:10.17863/cam.101505 · Cited by 89
Improving deep learning with prior knowledge and cognitive models: A survey on enhancing explainability, adversarial robustness and zero-shot learning
Mumuni, F., & Mumuni, A. (2023) · Cognitive Systems Research · doi:10.1016/j.cogsys.2023.101188 · Cited by 34
Applications of interpretable deep learning in neuroimaging: A comprehensive review
Munroe, L., da Silva, M., Heidari, F., Grigorescu, I., Dahan, S., Robinson, E. C., Deprez, M., & So, P. (2024) · Imaging Neuroscience · doi:10.1162/imag_a_00214 · Cited by 33
State-of-the-Art in Responsible, Explainable, and Fair AI for Medical Image Analysis
Amirian, S., Gao, F., Littlefield, N. G., Hill, J. H., Yates, A. J., Plate, J. F., Pantanowitz, L., Rashidi, H. H., & Tafti, A. P. (2025) · IEEE Access · doi:10.1109/access.2025.3555543 · Cited by 34
Explainable AI for Sensor Signal Interpretation to Revolutionize Human Health Monitoring: A Review
Alharthi, A. S., Alqurashi, A., Alharbi, T. E. A., Alammar, M. M., Aldosari, N., Bouchekara, H. R. E. H., Sha’aban, Y. A., Shahriar, M. S., & Ayidh, A. A. (2025) · IEEE Access · doi:10.1109/access.2025.3585764 · Cited by 36
Multiple Wavelet Convolutional Neural Network for Short-Term Load Forecasting
Liao, Z., Pan, H., Fan, X., Zhang, Y., & Kuang, L. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3026733 · Cited by 29
Deep learning connects DNA traces to transcription to reveal predictive features beyond enhancer–promoter contact
Rajpurkar, A. R., Mateo, L. J., Murphy, S. E., & Boettiger, A. N. (2021) · Nature Communications · doi:10.1038/s41467-021-23831-4 · Cited by 27
How Good is your Explanation? Algorithmic Stability Measures to Assess the Quality of Explanations for Deep Neural Networks
Fel, T., Vigouroux, D., Cadène, R., & Serre, T. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00163 · Cited by 37
Tackling the Accuracy-Interpretability Trade-off: Interpretable Deep Learning Models for Satellite Image-based Real Estate Appraisal
Kucklick, J.-P., & Müller, O. (2022) · ACM Transactions on Management Information Systems · doi:10.1145/3567430 · Cited by 28
Toward Transparent Load Disaggregation—A Framework for Quantitative Evaluation of Explainability Using Explainable AI
Batic, D., Stanković, V., & Stanković, L. (2023) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2023.3300530 · Cited by 28
Diagnostic test accuracy of machine learning algorithms for the detection intracranial hemorrhage: a systematic review and meta-analysis study
Maghami, M., Sattari, S. A., Tahmasbi, M., Panahi, P., Mozafari, J., & Shirbandi, K. (2023) · BioMedical Engineering OnLine · doi:10.1186/s12938-023-01172-1 · Cited by 34
Neural Response Interpretation through the Lens of Critical Pathways
Khakzar, A., Baselizadeh, S., Khanduja, S., Rupprecht, C., Kim, S. T., & Navab, N. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01332 · Cited by 27
Benchmarking Perturbation-Based Saliency Maps for Explaining Atari Agents
Huber, T., Limmer, B., & André, E. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.903875 · Cited by 27
Novel interpretable mechanism of neural networks based on network decoupling method
Duan, D., Wu, X., & Si, S. (2021) · Frontiers of Engineering Management · doi:10.1007/s42524-021-0169-x · Cited by 26
Improving explainable AI with patch perturbation-based evaluation pipeline: a COVID-19 X-ray image analysis case study
Sun, J., Shi, W., Giuste, F., Vaghani, Y. S., Tang, L., & Wang, M. D. (2023) · Scientific Reports · doi:10.1038/s41598-023-46493-2 · Cited by 30
Between homomorphic signal processing and deep neural networks: Constructing deep algorithms for polyphonic music transcription
Su, L. (2017) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipa.2017.8282170 · Cited by 27
Revealing Microstructural Modifications of Graphene Oxide-Modified Cement via Deep Learning and Nanoporosity Mapping: Implications for Structural Materials’ Performance
Wang, W., Chen, S.-J., Duan, W., Sagoe‐Crentsil, K., Pathirage, C. S. N., Li, L., & Lin, J. (2022) · ACS Applied Nano Materials · doi:10.1021/acsanm.2c01044 · Cited by 26
Towards explainability for AI-based edge wireless signal automatic modulation classification
Xu, B., Bhatti, U., Tang, H., Yan, J., Wu, S., Sarhan, N., Awwad, E. M., S., S. M., & Ghadi, Y. (2024) · Journal of Cloud Computing · doi:10.1186/s13677-024-00590-3 · Cited by 23
Visual Analytics of a Computer-Aided Diagnosis System for Pancreatic Lesions
Dmitriev, K., Marino, J., Baker, K. D., & Kaufman, A. E. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2947037 · Cited by 26
Calibrating Human-AI Collaboration: Impact of Risk, Ambiguity and Transparency on Algorithmic Bias
Schmidt, P., & Biessmann, F. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-57321-8_24 · Cited by 26
Topological measurement of deep neural networks using persistent homology
Watanabe, S., & Yamana, H. (2021) · Annals of Mathematics and Artificial Intelligence · doi:10.1007/s10472-021-09761-3 · Cited by 31
An explainable deep vision system for animal classification and detection in trail-camera images with automatic post-deployment retraining
Moallem, G., Pathirage, D., Reznick, J., Gallagher, J. F., & Sari‐Sarraf, H. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.106815 · Cited by 29
Scrutinizing XAI using linear ground-truth data with suppressor variables
Wilming, R., Budding, C., Müller, K., & Haufe, S. (2022) · Machine Learning · doi:10.1007/s10994-022-06167-y · Cited by 30
Bridging Adversarial Robustness and Gradient InterpretabilityPreprint
Kim, B., Seo, J., & Jeon, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.11626 · Cited by 26
XEdgeAI: A human-centered industrial inspection framework with data-centric Explainable Edge AI approach
Nguyen, T. T. H., Nguyen, L. P. T., & Cao, H. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102782 · Cited by 34
Layer-Wise Relevance Propagation for Explainable Deep Learning Based Speech Recognition
Bharadhwaj, H. (2018) · IEEE International Symposium on Signal Processing and Information Technology (ISSPIT) · doi:10.1109/isspit.2018.8642691 · Cited by 25
Interpretable brain disease classification and relevance-guided deep learning
Tinauer, C., Heber, S., Pirpamer, L., Damulina, A., Schmidt, R., Stollberger, R., Ropele, S., & Langkammer, C. (2022) · Scientific Reports · doi:10.1038/s41598-022-24541-7 · Cited by 26
A deep learning framework for identifying Alzheimer's disease using fMRI-based brain network
Wang, R., He, Q., Han, C., Wang, H., Shi, L., & Che, Y. (2023) · Frontiers in Neuroscience · doi:10.3389/fnins.2023.1177424 · Cited by 25
Evading text based emotion detection mechanism via adversarial attacks
Bajaj, A., & Vishwakarma, D. K. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126787 · Cited by 26
IROF: a low resource evaluation metric for explanation methodsPreprint
Rieger, L., & Hansen, L. K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.08747 · Cited by 49
Gradient-Based Instance-Specific Visual Explanations for Object Specification and Object Discrimination
Zhao, C., Hsiao, J. H., & Chan, A. B. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3380604 · Cited by 34
What is the Optimal Attribution Method for Explainable Ophthalmic Disease Classification?
Singh, A., Sengupta, S., Balaji, J. J., Mohammed, A. R., Faruq, I., Jayakumar, V., Zelek, J., & Lakshminarayanan, V. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-63419-3_3 · Cited by 23
Through the looking glass: evaluating post hoc explanations using transparent models
Velmurugan, M., Ouyang, C., Sindhgatta, R., & Moreira, C. (2023) · International Journal of Data Science and Analytics · doi:10.1007/s41060-023-00445-1 · Cited by 26
A New Perspective on Evaluation Methods for Explainable Artificial Intelligence (XAI)
Speith, T., & Langer, M. (2023) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew57809.2023.00061 · Cited by 25
Interpretable prediction models for widespread m6A RNA modification across cell lines and tissues
Zhang, Y., Wang, Z., Zhang, Y., Li, S., Guo, Y., Song, J., & Yu, D. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad709 · Cited by 24
Overinterpretation reveals image classification model pathologiesPreprint
Carter, B., Jain, S., Mueller, J. W., & Gifford, D. K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.08907 · Cited by 30
Deep Feature Selection Framework for Quality Prediction in Injection Molding Process
Kim, I., Na, J., Yun, J. P., & Lee, S. (2023) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2023.3268421 · Cited by 23
Towards a Better Understanding of Evaluating Trustworthiness in AI Systems
Kemmerzell, N., Schreiner, A., Khalid, H., Schalk, M., & Bordoli, L. (2025) · ACM Computing Surveys · doi:10.1145/3721976 · Cited by 34
Explainability and vision foundation models: A survey
Kazmierczak, R., Berthier, É., Frehse, G., & Franchi, G. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103184 · Cited by 45
Deep neural networks
Vieira, S., Pinaya, W. H. L., Garcia‐Dias, R., & Mechelli, A. (2019) · Elsevier eBooks · doi:10.1016/b978-0-12-815739-8.00009-2 · Cited by 23
Don't Lie to Me! Robust and Efficient Explainability with Verified Perturbation Analysis
Fel, T., Ducoffe, M., Vigouroux, D., Cadène, R., Capelle, M., Nicodème, C., & Serre, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01550 · Cited by 35
Evaluating feature attribution methods in the image domain
Gevaert, A., Rousseau, A.-J., Becker, T., Valkenborg, D., Bie, T. D., & Saeys, Y. (2024) · Machine Learning · doi:10.1007/s10994-024-06550-x · Cited by 35
Controlling explanatory heatmap resolution and semantics via decomposition depth
Lapuschkin, S., Binder, A., Müller, K., & Samek, W. (2016) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2016.7532763 · Cited by 23
Self-citation
Evaluation for Weakly Supervised Object Localization: Protocol, Metrics, and Datasets
Choe, J., Oh, S. J., Chun, S., Lee, S., Akata, Z., & Shim, H. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3169881 · Cited by 22
Convolutional Neural Network for Blur Images Detection as an Alternative for Laplacian Method
Szandała, T. (2020) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci47803.2020.9308594 · Cited by 22
Interpreting mental state decoding with deep learning models
Thomas, A. W., Ré, C., & Poldrack, R. A. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.07.003 · Cited by 24
A Generalized Explanation Framework for Visualization of Deep Learning Model Predictions
Wang, P., & Vasconcelos, N. M. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3241106 · Cited by 21
Design of an Intelligent Educational Evaluation System Using Deep Learning
Pei, Y., & Lu, G. (2023) · IEEE Access · doi:10.1109/access.2023.3260979 · Cited by 26
FunnyBirds: A Synthetic Vision Dataset for a Part-Based Analysis of Explainable AI Methods
Hesse, R., Schaub-Meyer, S., & Roth, S. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00368 · Cited by 22
Visualizing and Quantifying Discriminative Features for Face Recognition
Castanon, G. D., & Byrne, J. (2018) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2018.00013 · Cited by 22
Towards computational fluorescence microscopy: Machine learning-based integrated prediction of morphological and molecular tumor profilesPreprint
Binder, A., Bockmayr, M. L., Hägele, M., Wienert, S., Heim, D., Hellweg, K., Stenzinger, A., Parlow, L., Budczies, J., Goeppert, B. F., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.11178 · Cited by 30
Self-citation
Shortcomings of Top-Down Randomization-Based Sanity Checks for Evaluations of Deep Neural Network Explanations
Binder, A., Weber, L., Lapuschkin, S., Montavon, G., Müller, K.-R., & Samek, W. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01549 · Cited by 21
Self-citation
MIMIC-IF: Interpretability and Fairness Evaluation of Deep Learning Models on MIMIC-IV Dataset
Meng, C., Trinh, L., Xu, N., & Liu, Y. (2021) · doi:10.21203/rs.3.rs-402058/v1 · Cited by 21
Development of machine learning-based quantitative structure–activity relationship models for predicting plasma half-lives of drugs in six common food animal species
Wu, P.-Y., Chou, W.-C., Wu, X. W., Kamineni, V. N., Kuchimanchi, Y., Tell, L. A., Maunsell, F. P., & Lin, Z. (2024) · Toxicological Sciences · doi:10.1093/toxsci/kfae125 · Cited by 21
Transparency and insurance professionals: a study of Swedish insurance practice attitudes and future development
Dexe, J., Franke, U., & Rad, A. (2021) · The Geneva Papers on Risk and Insurance Issues and Practice · doi:10.1057/s41288-021-00207-9 · Cited by 21
Crowdsourcing Evaluation of Saliency-Based XAI Methods
Lu, X., Tolmachev, A., Yamamoto, T., Takeuchi, K., Okajima, S., Takebayashi, T., Maruhashi, K., & Kashima, H. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86517-7_27 · Cited by 21
Deep Learning Techniques for OFDM Systems
Meenalakshmi, M., Chaturvedi, S., & Dwivedi, V. K. (2021) · IETE Journal of Research · doi:10.1080/03772063.2021.1986154 · Cited by 21
Interpret what a Convolutional Neural Network learns for fault detection and diagnosis in process systems
Ye, L., Wu, H., Chen, Y., & Fei, Z. (2023) · Journal of Process Control · doi:10.1016/j.jprocont.2023.103086 · Cited by 22
A Comprehensive Review of Computer-Aided Diagnosis of Major Mental and Neurological Disorders and Suicide: A Biostatistical Perspective on Data Mining
Mansourian, M., Khademi, S., & Marateb, H. R. (2021) · Diagnostics · doi:10.3390/diagnostics11030393 · Cited by 21
Deep Learning for Soft Sensor Design
Graziani, S., & Xibilia, M. G. (2019) · Studies in computational intelligence · doi:10.1007/978-3-030-31764-5_2 · Cited by 20
Deep Learning in LncRNAome: Contribution, Challenges, and Perspectives
Alam, T., Al-Absi, H. R. H., & Schmeier, S. (2020) · Non-Coding RNA · doi:10.3390/ncrna6040047 · Cited by 21
Explaining Deep Neural Network using Layer-wise Relevance Propagation and Integrated Gradients
Čík, I., Rasamoelina, A. D., Mach, M., & Sinčák, P. (2021) · IEEE World Symposium on Applied Machine Intelligence and Informatics (SAMI) · doi:10.1109/sami50585.2021.9378686 · Cited by 21
Spatiotemporal Analysis by Deep Learning of Gait Signatures From Floor Sensors
Alharthi, A. S., Casson, A. J., & Ozanyan, K. B. (2021) · IEEE Sensors Journal · doi:10.1109/jsen.2021.3078336 · Cited by 20
Multi-objective Feature Attribution Explanation For Explainable Machine Learning
Wang, Z., Huang, C., Li, Y., & Yao, X. (2023) · ACM Transactions on Evolutionary Learning and Optimization · doi:10.1145/3617380 · Cited by 21

Authors: Samek, W., Binder, A., Montavon, G., Lapuschkin, S., & Muller, K.-R.
Affiliations: Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (DE)
Singapore University of Technology and Design (SG)
Technische Universität Berlin, Berlin, Germany
ORCID: Samek, Wojciech
Binder, Alexander
Lapuschkin, Sebastian
Venue: IEEE Transactions on Neural Networks and Learning Systems
Publisher: Institute of Electrical and Electronics Engineers (IEEE)
Published: 2017-11
Volume: 28
Issue: 11
Pages: 2660-2673
ISSN: 2162-237X (ISSN-L)
2162-2388
DOI: 10.1109/tnnls.2016.2599820
OpenAlex: W2240067561
Cited by: 1,164
References: 51
Citations by year: 2026: 120 · 2025: 105 · 2024: 139 · 2023: 163 · 2022: 147 · 2021: 171 · 2020: 123 · 2019: 92 · 2018: 66 · 2017: 26
FWCI: 53.43
Type: journal article
Language: en
Open access: closed
Licence: ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html (vor)
Locations: IEEE Transactions on Neural Networks and Learning Systems (publishedVersion)
PubMed (publishedVersion)
Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion)
Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
Funders: Brain Korea 21 Plus Program through the National Research Foundation of Korea Funded by the Ministry of Education
DFG (MU 987/17-1)
German Ministry for Education and Research as Berlin Big Data Center (01IS14013A)
Topics: Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
Keywords: deep neural network interpretability, layer-wise relevance propagation, deconvolution, explainable AI, image classification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Evaluating the Visualization of What a Deep Neural Network Has Learned

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Citations

Cites

Cited by

Related

Metadata

Record

About these checks