Cited by 1,164Explainable Artificial Intelligence (XAI)Advanced Neural Network ApplicationsVisual Attention and Saliency Detection

The DOI matches its Crossref record; 47 of the 51 references it cites were found in a registry or an index.

Evaluating the Visualization of What a Deep Neural Network Has Learned

Samek, Wojciech, Binder, Alexander, Montavon, Gregoire, Lapuschkin, Sebastian, Muller, Klaus-Robert·2017·IEEE Transactions on Neural Networks and Learning Systems

DOI 10.1109/tnnls.2016.2599820

Open this paper in CoChat — free account

Abstract

Deep neural networks (DNNs) have demonstrated impressive performance in complex machine learning tasks such as image classification or speech recognition. However, due to their multilayer nonlinear structure, they are not transparent, i.e., it is hard to grasp what makes them arrive at a particular classification or recognition decision, given a new unseen data sample. Recently, several approaches have been proposed enabling one to understand and interpret the reasoning embodied in a DNN for a single test image. These methods quantify the “importance” of individual pixels with respect to the classification decision and allow a visualization in terms of a heatmap in pixel/input space. While the usefulness of heatmaps can be judged subjectively by a human, an objective quality measure is missing. In this paper, we present a general methodology based on region perturbation for evaluating ordered collections of pixels such as heatmaps. We compare heatmaps computed by three different methods on the SUN397, ILSVRC2012, and MIT Places data sets. Our main result is that the recently proposed layer-wise relevance propagation algorithm qualitatively and quantitatively provides a better explanation of what made a DNN arrive at a particular classification decision than the sensitivity-based approach or the deconvolution method. We provide theoretical arguments to explain this result and discuss its practical implications. Finally, we investigate the use of heatmaps for unsupervised assessment of the neural network performance.

Integrity signals

Registry

Crossref · 10.1109/tnnls.2016.2599820

Cites

47 of 47 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,889
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381
An introduction to variable and feature selection
GuyonIsabelle, & ElisseeffAndré (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944968 · Cited by 8,354
Domain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143

Flagged first, then by citations

Show all 51 references

4 of the 51 references couldn’t be matched to a known paper.

Research works printed but not linked: 1

jia, 2013, Caffe An Open Source Convolutional Architecture for Fast Feature Embedding

Sorted by citations

Items 401–500 of 1,000

Disentangled Explanations of Neural Network Predictions by Finding Relevant Subspaces
Chormai, P., Herrmann, J., Müller, K., & Montavon, G. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3388275 · Cited by 36
Self-citation
Constructing personalized characterizations of structural brain aberrations in patients with dementia using explainable artificial intelligence
Leonardsen, E. H., Persson, K., Grødem, E., Dinsdale, N. K., Schellhorn, T., Roe, J. M., Dídac Vidal-piñeiro, A., Sørensen, Ø., Kaufmann, T., Westman, E., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01123-7 · Cited by 28
Biased Embeddings from Wild Data: Measuring, Understanding and Removing
Sutton, A., Lansdall-Welfare, T., & Cristianini, N. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01768-2_27 · Cited by 21
A Neural Network Reveals Motoric Effects of Maternal Preconception Exposure to Nicotine on Rat Pup Behavior: A New Approach for Movement Disorders Diagnosis
Torabi, R., Jenkins, S., Harker, A., Whishaw, I. Q., Gibb, R. L., & Luczak, A. (2021) · Frontiers in Neuroscience · doi:10.3389/fnins.2021.686767 · Cited by 17
Explainable deep learning: concepts, methods, and new developments
Samek, W. (2023) · Elsevier eBooks · doi:10.1016/b978-0-32-396098-4.00008-9 · Cited by 20
Self-citation
Informed machine learning methods for application in engineering: A review
Mackay, C. T., & Nowell, D. (2023) · Proceedings of the Institution of Mechanical Engineers Part C Journal of Mechanical Engineering Science · doi:10.1177/09544062231164575 · Cited by 25
Deep learning referral suggestion and tumour discrimination using explainable artificial intelligence applied to multiparametric MRI
Shin, H., Park, J. E., Jun, Y., Eo, T., Lee, J., Kim, J. E., Lee, D. H., Moon, H. H., Park, S. I., Kim, S., et al. (2023) · European Radiology · doi:10.1007/s00330-023-09710-0 · Cited by 22
DISCOVER: 2-D multiview summarization of Optical Coherence Tomography Angiography for automatic diabetic retinopathy diagnosis
Daho, M. E. H., Li, Y., Zeghlache, R., Boité, H. L., Deman, P., Borderie, L., Ren, H., Manivannan, N., Lepicard, C., Cochener, B., et al. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102803 · Cited by 20
Multi-stream convolutional neural network-based fault diagnosis for variable frequency drives in sustainable manufacturing systems
Grezmak, J. E., Zhang, J., Wang, P. E., & Gao, R. X. (2020) · Procedia Manufacturing · doi:10.1016/j.promfg.2020.02.181 · Cited by 19
Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and Retraining
Madsen, A., Meade, N., Adlakha, V., & Reddy, S. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.125 · Cited by 22
How Much Can I Trust You? -- Quantifying Uncertainties in Explaining Neural NetworksPreprint
Bykov, K., Höhne, M., Müller, K., Nakajima, S., & Kloft, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.09000 · Cited by 35
Synthetic Benchmarks for Scientific Research in Explainable Machine LearningPreprint
Liu, Y., Khandagale, S., White, C., & Neiswanger, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.12543 · Cited by 25
One Explanation is Not Enough: Structured Attention Graphs for Image ClassificationPreprint
Shitole, V., Li, F., Kahng, M., Tadepalli, P. V., & Fern, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.06733 · Cited by 18
Interpretable machine learning comprehensive human gait deterioration analysis
Alharthi, A. S. (2024) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2024.1451529 · Cited by 18
Pixel-wise feature fusion in gully susceptibility: A comparison of feed-forward neural networks and ensemble (voting, stacking) models
Nwazelibe, V. E., Agbasi, J. C., Ayejoto, D. A., & Egbueri, J. C. (2025) · Journal of African Earth Sciences · doi:10.1016/j.jafrearsci.2025.105633 · Cited by 18
Understanding deep neural networks performance for radar-based human motion recognition
Amin, M. G., & Erol, B. (2018) · IEEE Radar Conference (RadarConf) · doi:10.1109/radar.2018.8378780 · Cited by 18
It’s All in the Name: A Character-Based Approach to Infer Religion
Chaturvedi, R., & Chaturvedi, S. (2023) · Political Analysis · doi:10.1017/pan.2023.6 · Cited by 26
Have We Learned to Explain?: How Interpretability Methods Can Learn to Encode Predictions in their Interpretations
Jethani, N., Sudarshan, M., Aphinyanaphongs, Y., & Ranganath, R. (2021) · PubMed · doi:10.48550/arxiv.2103.01890 · Cited by 21
Energy Efficiency Optimization for SWIPT-Based D2D-Underlaid Cellular Networks Using Multiagent Deep Reinforcement Learning
Muy, S., Ron, D., & Lee, J.-R. (2021) · IEEE Systems Journal · doi:10.1109/jsyst.2021.3098860 · Cited by 18
Exploiting auto-encoders and segmentation methods for middle-level explanations of image classification systems
Apicella, A., Giugliano, S., Isgrò, F., & Prevete, R. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109725 · Cited by 18
An Adaptive Hybrid Attention Based Convolutional Neural Net for Intelligent Transportation Object Recognition
Chen, Q., Pan, G., Zhao, L., Fan, J., Chen, W., & Zhang, A. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3227245 · Cited by 18
Novel methods for elucidating modality importance in multimodal electrophysiology classifiers
Ellis, C. A., Sendi, M. S. E., Zhang, R., Carbajal, D. A., Wang, M. D., Miller, R. L., & Calhoun, V. D. (2023) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2023.1123376 · Cited by 18
Explainability and transparency in the realm of digital humanities: toward a historian XAI
El-Hajj, H., Eberle, O., Merklein, A., Siebold, A., Shlomi, N., Büttner, J., Martinetz, J., Müller, K., Montavon, G., & Valleriani, M. (2023) · International Journal of Digital Humanities · doi:10.1007/s42803-023-00070-1 · Cited by 24
Self-citation
Enhancing the confidence of deep learning classifiers via interpretable saliency maps
Tjoa, E., Khok, H. J., Chouhan, T. S., & Guan, C. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126825 · Cited by 31
Efficient Offline Reinforcement Learning With Relaxed Conservatism
Huang, L., Dong, B., & Zhang, W. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3364844 · Cited by 18
An Interpretable Breast Ultrasound Image Classification Algorithm Based on Convolutional Neural Network and Transformer
Meng, X., Ma, J., Liu, F., Chen, Z., & Zhang, T. (2024) · Mathematics · doi:10.3390/math12152354 · Cited by 19
Comparing the Decision-Making Mechanisms by Transformers and CNNs via Explanation Methods
Jiang, M., Khorram, S., & Li, F. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00912 · Cited by 19
c-Eval: A Unified Metric to Evaluate Feature-based Explanations via Perturbation
Vu, M. N., Nguyen, T. D., Phan, N., Gera, R., & Thai, M. T. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9671895 · Cited by 23
A Novel Explainable Deep Belief Network Framework and Its Application for Feature Importance Analysis
Chen, Q., Pan, G., Chen, W., & Wu, P. (2021) · IEEE Sensors Journal · doi:10.1109/jsen.2021.3084846 · Cited by 18
Keep CALM and Improve Visual Feature Attribution
Kim, J. M., Choe, J., Akata, Z., & Oh, S. J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00824 · Cited by 18
A Century of Portraits: A Visual Historical Record of American High School YearbooksPreprint
Ginosar, S., Rakelly, K., Sachs, S., Yin, B., Lee, C., Krähenbühl, P., & Efros, A. A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.02575 · Cited by 18
Building trust in deep learning-based immune response predictors with interpretable explanations
Borole, P., & Rajan, A. (2024) · Communications Biology · doi:10.1038/s42003-024-05968-2 · Cited by 19
The level of strength of an explanation: A quantitative evaluation technique for post-hoc XAI methods
Bello, M., Amador, R., García, M.-M., Ser, J. D., Mesejo, P., & Cordón, Ó. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.111221 · Cited by 17
A comprehensive analysis of perturbation methods in explainable AI feature attribution validation for neural time series classifiers
Šimić, I., Veas, E. E., & Sabol, V. (2025) · Scientific Reports · doi:10.1038/s41598-025-09538-2 · Cited by 23
Do Explanations Explain? Model Knows Best
Khakzar, A., Khorsandi, P., Nobahari, R., & Navab, N. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01000 · Cited by 19
Can local explanation techniques explain linear additive models?
Rahnama, A. H. A., Bütepage, J., Geurts, P., & Boström, H. (2023) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-023-00971-3 · Cited by 17
Explaining through Transformer Input Sampling
Englebert, A., Stassin, S., Nanfack, G., Mahmoudi, S. A., Siebert, X., Cornu, O., & Vleeschouwer, C. D. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00088 · Cited by 17
Efficient analysis of deep neural networks for vision via biologically-inspired receptive field angles: An in-depth survey
Ma, Y., Meng, Y., Lin, H., Liu, C., Hu, M., & Song, Q. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102582 · Cited by 18
Comparing the Interpretability of Deep Networks via Network Dissection
Zhou, B., Bau, D., Oliva, A., & Torralba, A. B. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_12 · Cited by 17
An Interactive Visualization for Feature Localization in Deep Neural Networks
Zurowietz, M., & Nattkemper, T. W. (2020) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.00049 · Cited by 18
On the Lack of Robust Interpretability of Neural Text Classifiers
Zafar, M. B., Donini, M., Slack, D., Archambeau, C., Das, S. R., & Kenthapadi, K. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.327 · Cited by 23
Towards Measuring Bias in Image Classification
Schaaf, N., de Mitri, O., Kim, H. B., Windberger, A., & Huber, M. F. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86365-4_35 · Cited by 17
A Rate-Distortion Framework for Explaining Neural Network DecisionsPreprint
Macdonald, J., Wäldchen, S., Hauch, S., & Gitta Kutyniok, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.11092 · Cited by 30
What Did You Think Would Happen? Explaining Agent Behaviour Through Intended OutcomesPreprint
Yau, H., Russell, C., & Hadfield, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.05064 · Cited by 17
Understanding Human Cognition Through Computational Modeling
Hsiao, J. H. (2024) · Topics in Cognitive Science · doi:10.1111/tops.12737 · Cited by 18
Heatmap Assisted Accuracy Score Evaluation Method for Machine-Centric Explainable Deep Neural Networks
Lee, J., Cho, H., Pyun, Y. J., Kang, S., & Nam, H. (2022) · IEEE Access · doi:10.1109/access.2022.3184453 · Cited by 18
An Improved Explainable Point Cloud Classifier (XPCC)
Arnold, N. I., Angelov, P. P., & Atkinson, P. M. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3150647 · Cited by 16
Visualization of Deep Convolutional Neural Networks to Investigate Porous Nanocomposites for Electromagnetic Interference Shielding
Shi, M., Feng, C., Tu, Y., Shi, G.-S., He, P., Zhang, Y., Zhang, J., Li, J., & Guo, S. (2023) · ACS Applied Materials & Interfaces · doi:10.1021/acsami.3c04557 · Cited by 16
RETRACTED ARTICLE: Development of speech recognition system for remote vocal music teaching based on Markov model
Xu, F., & Xia, Y. (2023) · Soft Computing · doi:10.1007/s00500-023-08277-8 · Cited by 16
Retracted
Feature perturbation augmentation for reliable evaluation of importance estimators in neural networks
Brocki, L., & Chung, N. C. (2023) · Pattern Recognition Letters · doi:10.1016/j.patrec.2023.10.012 · Cited by 21
Stable feature selection utilizing Graph Convolutional Neural Network and Layer-wise Relevance Propagation for biomarker discovery in breast cancer
Chereda, H., Leha, A., & Beißbarth, T. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102840 · Cited by 21
Contrastive Relevance Propagation for Interpreting Predictions by a Single-Shot Object Detector
Tsunakawa, H., Kameya, Y., Lee, H., Shinya, Y., & Mitsumoto, N. (2019) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8851770 · Cited by 17
Learning Deep Generative Models With Doubly Stochastic Gradient MCMC
Du, C., Zhu, J., & Zhang, B. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2688499 · Cited by 24
Concept Saliency Maps to Visualize Relevant Features in Deep Generative Models
Brocki, L., & Chung, N. C. (2019) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2019.00287 · Cited by 18
Heterogeneous Federated LearningPreprint
Yu, F., Zhang, W., Qin, Z., Xu, Z., Di, W., Liu, C., Tian, Z., & Chen, X. (2020) · arXiv (Cornell University) · doi:10.13016/m2cwh1-kqsf · Cited by 16
Do Input Gradients Highlight Discriminative Features?Preprint
Shah, H., Jain, P., & Netrapalli, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.12781 · Cited by 23
Developing a Fidelity Evaluation Approach for Interpretable Machine LearningPreprint
Velmurugan, M., Ouyang, C., Moreira, C., & Sindhgatta, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.08492 · Cited by 45
BenchXAI: Comprehensive benchmarking of post-hoc explainable AI methods on multi-modal biomedical data
Metsch, J. M., & Hauschild, A.-C. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110124 · Cited by 15
Winsor-CAM: Human-Tunable Visual Explanations From Deep Networks via Layer-Wise Winsorization
Wall, C., Wang, L., Rizk, R., & Santosh, K. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3668075 · Cited by 15
Agreement Between Saliency Maps and Human-Labeled Regions of Interest: Applications to Skin Disease Classification
Singh, N., Lee, K., Coz, D., Angermueller, C., Huang, S. S., Loh, A., & Liu, Y. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00376 · Cited by 16
Measuring Feature Importance of Convolutional Neural Networks
Zhang, X., & Gao, J. (2020) · IEEE Access · doi:10.1109/access.2020.3034625 · Cited by 15
Overview of Visualization Methods for Artificial Neural Networks
Matveev, S. A., Oseledets, I. V., Ponomarev, E., & Chertkov, A. (2021) · Computational Mathematics and Mathematical Physics · doi:10.1134/s0965542521050134 · Cited by 14
Machine Learning Verification and Safety for Unmanned Aircraft - A Literature Study
Torens, C., Jünger, F., Schirmer, S., Schopferer, S., Maienschein, T., & Dauer, J. C. (2022) · AIAA SciTech Forum · doi:10.2514/6.2022-1133 · Cited by 15
RELAX: Representation Learning Explainability
Wickstrøm, K., Trosten, D. J., Løkse, S., Boubekki, A., Mikalsen, K. Ø., Kampffmeyer, M., & Jenssen, R. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01773-2 · Cited by 17
Low-fluctuation nonlinear model using incremental step pulse programming with memristive devices
Lee, G. H., Kim, T., Youn, S., Park, J., Kim, S., & Kim, H. (2023) · Chaos Solitons & Fractals · doi:10.1016/j.chaos.2023.113359 · Cited by 15
A User Interface for Explaining Machine Learning Model Explanations
Kadir, M. A., Selim, A. M., Barz, M., & Sonntag, D. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581754.3584131 · Cited by 16
Saliency Cards: A Framework to Characterize and Compare Saliency Methods
Boggust, A., Suresh, H., Strobelt, H., Guttag, J. V., & Satyanarayan, A. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3593997 · Cited by 15
Interpreting deep learning models for glioma survival classification using visualization and textual explanations
Osadebey, M. E., Liu, Q., Fuster‐García, E., & Emblem, K. E. (2023) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-023-02320-2 · Cited by 18
Diffeomorphic Counterfactuals With Generative Models
Dombrowski, A.-K., Gerken, J. E., Müller, K., & Kessel, P. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3339980 · Cited by 26
An Evolutionary Multiobjective Neural Architecture Search Approach to Advancing Cognitive Diagnosis in Intelligent Education
Yang, S., Ma, H., Bi, Y., Tian, Y., Zhang, L., Jin, Y., & Zhang, X. (2024) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2024.3429180 · Cited by 15
A Century of Portraits: A Visual Historical Record of American High School Yearbooks
Ginosar, S., Rakelly, K., Sachs, S., Yin, B., Lee, C., Krähenbühl, P., & Efros, A. A. (2017) · IEEE Transactions on Computational Imaging · doi:10.1109/tci.2017.2699865 · Cited by 15
Interpretable Deep Neural Network to Predict Estrogen Receptor Status from Haematoxylin-Eosin Images
Seegerer, P., Binder, A., Saitenmacher, R., Bockmayr, M. L., Alber, M., Jurmeister, P. S., Klauschen, F., & Müller, K. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-50402-1_2 · Cited by 15
Self-citation
Explainability for Regression CNN in Fetal Head Circumference Estimation from Ultrasound Images
Zhang, J., Petitjean, C., Yger, F., & Aïnouz, S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-61166-8_8 · Cited by 16
Backdoor Attacks on the DNN Interpretation System
Fang, S., & Choromanska, A. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i1.19935 · Cited by 16
Image-based modelling and visualisation of the relationship between laser-cut edge and process parameters
Tatzel, L., Tamimi, O. A., Haueise, T., & León, F. P. (2021) · Optics & Laser Technology · doi:10.1016/j.optlastec.2021.107028 · Cited by 14
Feature visualization within an automated design assessment leveraging explainable artificial intelligence methods
Schönhof, R., Werner, A., Elstner, J., Zopcsak, B., Awad, R., & Huber, M. F. (2021) · Procedia CIRP · doi:10.1016/j.procir.2021.05.075 · Cited by 15
An Efficient Iterative Approach to Explainable Feature Learning
Vlahek, D., & Mongus, D. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3107049 · Cited by 19
Interpreting Interpretations: Organizing Attribution Methods by Criteria
Wang, Z., Mardziel, P., Datta, A., & Fredrikson, M. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00013 · Cited by 19
Predicting Model Failure using Saliency Maps in Autonomous Driving SystemsPreprint
Mohseni, S., Jagadeesh, A., & Wang, Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.07679 · Cited by 20
Investigating Saturation Effects in Integrated GradientsPreprint
Miglani, V., Kokhlikyan, N., Alsallakh, B., Martin, M., & Reblitz-Richardson, O. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12697 · Cited by 20
Explaining Bayesian Neural NetworksPreprint
Bykov, K., Höhne, M., Creosteanu, A., Müller, K., Klauschen, F., Nakajima, S., & Kloft, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.10346 · Cited by 27
Toward Scalable and Unified Example-Based Explanation and Outlier Detection
Chong, P., Cheung, N.-M., Elovici, Y., & Binder, A. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3127847 · Cited by 14
Self-citation
Explainable Deep Learning for Covid-19 Detection Using Chest X-ray and CT-Scan Images
Mahmoudi, S. A., Stassin, S., Daho, M. E. H., Lessage, X., & Mahmoudi, S. (2021) · EAI/Springer Innovations in Communication and Computing · doi:10.1007/978-3-030-72752-9_16 · Cited by 14
Development of a Novel Transformation of Spiking Neural Classifier to an Interpretable Classifier
Jeyasothy, A., Sundaram, S., Ramasamy, S., & Sundararajan, N. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3181181 · Cited by 14
Review of Nodule Mineral Image Segmentation Algorithms for Deep-Sea Mineral Resource Assessment
Song, W. G., Dong, L., Zhao, X., Xia, J., Liu, T., & Shi, Y. (2022) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2022.027214 · Cited by 15
Development of a deep learning network for Alzheimer’s disease classification with evaluation of imaging modality and longitudinal data
Deatsch, A. E., Perovnik, M., Namías, M., Trošt, M., Jeraj, R., & Initiative, A. D. N. (2022) · Physics in Medicine and Biology · doi:10.1088/1361-6560/ac8f10 · Cited by 15
A Model-Agnostic Feature Attribution Approach to Magnetoencephalography Predictions Based on Shapley Value
Fan, Y., Mao, H., & Li, Q. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3248139 · Cited by 14
Shedding light on the black box of a neural network used to detect prostate cancer in whole slide images by occlusion-based explainability
Gallo, M., Gallo, M., Nenutil, R., Němeček, M., Holub, P., & Brázdil, T. (2023) · New Biotechnology · doi:10.1016/j.nbt.2023.09.008 · Cited by 16
Counterfactual-based Saliency Map: Towards Visual Contrastive Explanations for Neural Networks
Wang, X., Wang, Z., Weng, H., Guo, H., Zhang, Z., Jin, L., Wei, T., & Ren, K. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00195 · Cited by 14
SAFE: Saliency-Aware Counterfactual Explanations for DNN-based Automated Driving Systems
Samadi, A., Shirian, A., Koufos, K., Debattista, K., & Dianati, M. (2023) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc57777.2023.10422102 · Cited by 15
Accurate Explanation Model for Image Classifiers using Class Association Embedding
Xie, R., Chen, J., Jiang, L., Xiao, R., Pan, Y., & Cai, Y. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00180 · Cited by 14
Study on interpretability of artificial neural network models for dynamic load identification
Yang, F., Luo, Y., Du, L., Zhang, Y., & Xie, S. (2025) · Measurement · doi:10.1016/j.measurement.2025.117210 · Cited by 14
Interpreting Convolutional Neural Networks for Device-Free Wi-Fi Fingerprinting Indoor Localization via Information Visualization
Chen, K. M., Chang, R. Y., & Liu, S.-J. (2019) · IEEE Access · doi:10.1109/access.2019.2956187 · Cited by 14
Adversarial attacks on text classification models using layer‐wise relevance propagation
Xu, J., & Du, Q. (2020) · International Journal of Intelligent Systems · doi:10.1002/int.22260 · Cited by 16
Image Recommendation Algorithm Combined with Deep Neural Network Designed for Social Networks
Shaohui, D., Chen, Z., Wu, H., Tang, Y., & Li, Y. (2021) · Complexity · doi:10.1155/2021/5196190 · Cited by 14
Deep Visual Analytics (DVA): Applications, Challenges and Future Directions
Islam, M. R., Prome, S. A., Ratan, M. R., Kamal, A. R. M., & Xu, G. (2021) · Human-Centric Intelligent Systems · doi:10.2991/hcis.k.210704.003 · Cited by 14
Towards greater neuroimaging classification transparency via the integration of explainability methods and confidence estimation approaches
Ellis, C. A., Miller, R. L., & Calhoun, V. D. (2023) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2023.101176 · Cited by 14
ECINN: Efficient Counterfactuals from Invertible Neural Networks
Hvilshøj, F., Iosifidis, A., & Assent, I. (2021) · British Machine Vision Conference (BMVC) · doi:10.5244/c.35.159 · Cited by 14
BatchNorm Decomposition for Deep Neural Network Interpretation
Hui, L. Y. W., & Binder, A. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-20518-8_24 · Cited by 14
Self-citation
Objective Diagnosis for Histopathological Images Based on Machine Learning Techniques: Classical Approaches and New Trends
Elazab, N., Soliman, H., El–Sappagh, S., Islam, S. M. R., & Elmogy, M. M. (2020) · RePEc: Research Papers in Economics · Cited by 14

Authors: Samek, W., Binder, A., Montavon, G., Lapuschkin, S., & Muller, K.-R.
Affiliations: Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (DE)
Singapore University of Technology and Design (SG)
Technische Universität Berlin, Berlin, Germany
ORCID: Samek, Wojciech
Binder, Alexander
Lapuschkin, Sebastian
Venue: IEEE Transactions on Neural Networks and Learning Systems
Publisher: Institute of Electrical and Electronics Engineers (IEEE)
Published: 2017-11
Volume: 28
Issue: 11
Pages: 2660-2673
ISSN: 2162-237X (ISSN-L)
2162-2388
DOI: 10.1109/tnnls.2016.2599820
OpenAlex: W2240067561
Cited by: 1,164
References: 51
Citations by year: 2026: 120 · 2025: 105 · 2024: 139 · 2023: 163 · 2022: 147 · 2021: 171 · 2020: 123 · 2019: 92 · 2018: 66 · 2017: 26
FWCI: 53.43
Type: journal article
Language: en
Open access: closed
Licence: ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html (vor)
Locations: IEEE Transactions on Neural Networks and Learning Systems (publishedVersion)
PubMed (publishedVersion)
Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion)
Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
Funders: Brain Korea 21 Plus Program through the National Research Foundation of Korea Funded by the Ministry of Education
DFG (MU 987/17-1)
German Ministry for Education and Research as Berlin Big Data Center (01IS14013A)
Topics: Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
Keywords: deep neural network interpretability, layer-wise relevance propagation, deconvolution, explainable AI, image classification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Evaluating the Visualization of What a Deep Neural Network Has Learned

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Citations

Cites

Cited by

Related

Metadata

Record

About these checks