Cited by 1,164Explainable Artificial Intelligence (XAI)Advanced Neural Network ApplicationsVisual Attention and Saliency Detection

The DOI matches its Crossref record; 47 of the 51 references it cites were found in a registry or an index.

Evaluating the Visualization of What a Deep Neural Network Has Learned

Samek, Wojciech, Binder, Alexander, Montavon, Gregoire, Lapuschkin, Sebastian, Muller, Klaus-Robert·2017·IEEE Transactions on Neural Networks and Learning Systems

DOI 10.1109/tnnls.2016.2599820

Open this paper in CoChat — free account

Abstract

Deep neural networks (DNNs) have demonstrated impressive performance in complex machine learning tasks such as image classification or speech recognition. However, due to their multilayer nonlinear structure, they are not transparent, i.e., it is hard to grasp what makes them arrive at a particular classification or recognition decision, given a new unseen data sample. Recently, several approaches have been proposed enabling one to understand and interpret the reasoning embodied in a DNN for a single test image. These methods quantify the “importance” of individual pixels with respect to the classification decision and allow a visualization in terms of a heatmap in pixel/input space. While the usefulness of heatmaps can be judged subjectively by a human, an objective quality measure is missing. In this paper, we present a general methodology based on region perturbation for evaluating ordered collections of pixels such as heatmaps. We compare heatmaps computed by three different methods on the SUN397, ILSVRC2012, and MIT Places data sets. Our main result is that the recently proposed layer-wise relevance propagation algorithm qualitatively and quantitatively provides a better explanation of what made a DNN arrive at a particular classification decision than the sensitivity-based approach or the deconvolution method. We provide theoretical arguments to explain this result and discuss its practical implications. Finally, we investigate the use of heatmaps for unsupervised assessment of the neural network performance.

Integrity signals

Registry

Crossref · 10.1109/tnnls.2016.2599820

Cites

47 of 47 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,889
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381
An introduction to variable and feature selection
GuyonIsabelle, & ElisseeffAndré (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944968 · Cited by 8,354
Domain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143

Flagged first, then by citations

Show all 51 references

4 of the 51 references couldn’t be matched to a known paper.

Research works printed but not linked: 1

jia, 2013, Caffe An Open Source Convolutional Architecture for Fast Feature Embedding

Sorted by citations

Items 201–300 of 1,000

Establishing and evaluating trustworthy AI: overview and research challenges
Kowald, D., Scher, S., Pammer-Schindler, V., Müllner, P., Waxnegger, K., Demelius, L., Fessl, A., Toller, M., Mendoza Estrada, I. G., Šimić, I., et al. (2024) · Frontiers in Big Data · doi:10.3389/fdata.2024.1467222 · Cited by 59
On the explainability of convolutional neural networks processing ultrasonic guided waves for damage diagnosis
Lomazzi, L., Fabiano, S., Parziale, M., Giglio, M., & Cadini, F. (2022) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2022.109642 · Cited by 58
FUTURE-AI: Guiding Principles and Consensus Recommendations for Trustworthy Artificial Intelligence in Medical Imaging
Lekadir, K., Osuala, R., Gallin, C., Lazrak, N., Kushibar, K., Tsakou, G., Aussó, S., Alberich, L. C., Marias, K., Tsiknakis, M., et al. (2021) · arXiv · doi:10.48550/arxiv.2109.09658 · Cited by 58
Interpreting Neural Networks to Improve Politeness Comprehension
Aubakirova, M., & Bansal, M. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1216 · Cited by 58
Fed2
Yu, F., Zhang, W., Qin, Z., Xu, Z., Wang, D., Liu, C., Tian, Z., & Chen, X. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467309 · Cited by 58
A review: The detection of cancer cells in histopathology based on machine vision
He, W., Liu, T., Han, Y., Ming, W., Du, J., Liu, Y., Yang, Y. H., Wang, L., Jiang, Z., Wang, Y., et al. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105636 · Cited by 58
VisualBackProp: visualizing CNNs for autonomous driving.Preprint
Bojarski, M., Choromanska, A., Choromański, K., Firner, B., Jackel, L., Müller, U., & Zieba, K. (2016) · arXiv (Cornell University) · Cited by 56
A task-and-technique centered survey on visual analytics for deep learning model engineering
García, R., Telea, A. C., da Silva, B. C., Tørresen, J., & Comba, J. L. D. (2018) · Computers & Graphics · doi:10.1016/j.cag.2018.09.018 · Cited by 55
Improving healthcare access management by predicting patient no-show behaviour
Barrera, D., Brailsford, S., Bravo, C., & Smith, H. K. (2020) · Decision Support Systems · doi:10.1016/j.dss.2020.113398 · Cited by 62
Objective Video Quality Assessment Combining Transfer Learning With CNN
Zhang, Y., Gao, X., He, L., Lu, W. B., & He, R. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2890310 · Cited by 55
Deep Neural Network Attribution Methods for Leakage Analysis and Symmetric Key Recovery
Hettwer, B., Gehrer, S., & Güneysu, T. E. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-38471-5_26 · Cited by 57
A Small Step Toward Generalizability: Training a Machine Learning Scoring Function for Structure-Based Virtual Screening
Scantlebury, J., Vost, L., Carbery, A., Hadfield, T. E., Turnbull, O. M., Brown, N., Chenthamarakshan, V., Das, P., Grosjean, H., von Delft, F., et al. (2023) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.3c00322 · Cited by 54
Exploring Evaluation Methods for Interpretable Machine Learning: A Survey
Alangari, N., Menaï, M. E. B., Mathkour, H. I., & Almosallam, I. A. (2023) · Information · doi:10.3390/info14080469 · Cited by 72
Explanations for Attributing Deep Neural Network Predictions
Fong, R., & Vedaldi, A. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_8 · Cited by 52
Taxonomy of Machine Learning Safety: A Survey and Primer
Mohseni, S., Wang, H., Xiao, C., Yu, Z., Wang, Z., & Yadawa, J. (2023) · ACM Computing Surveys · doi:10.1145/3551385 · Cited by 53
Application of Artificial Intelligence for Surface Roughness Prediction of Additively Manufactured Components
Batu, T., Lemu, H. G., & Gebremedhen, H. S. (2023) · Materials · doi:10.3390/ma16186266 · Cited by 64
Survey of XAI in Digital Pathology
Pocevičiūtė, M., Eilertsen, G., & Lundström, C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-50402-1_4 · Cited by 53
Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement LearningPreprint
Atrey, A., Clary, K., & Jensen, D. D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.05743 · Cited by 53
Why Is Everyone Training Very Deep Neural Network With Skip Connections?
Oyedotun, O. K., Ismaeil, K. A., & Aouada, D. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3131813 · Cited by 52
Clinical validation of saliency maps for understanding deep neural networks in ophthalmology
Ayhan, M. S., Kuemmerle, L. B., Kuehlewein, L., INHOFFEN, W., Aliyeva, G., Ziemssen, F., & Berens, P. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102364 · Cited by 55
Guaranteeing Correctness in Black-Box Machine Learning: A Fusion of Explainable AI and Formal Methods for Healthcare Decision-Making
Khan, N., Nauman, M. A., Almadhor, A. S., Akhtar, N., Alghuried, A., & Alhudhaif, A. (2024) · IEEE Access · doi:10.1109/access.2024.3420415 · Cited by 59
Unsupervised Difference Representation Learning for Detecting Multiple Types of Changes in Multitemporal Remote Sensing Images
Zhang, P., Gong, M., Zhang, H., Liu, J., & Yifang Ban, A. (2018) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2018.2872509 · Cited by 51
Deep relevant representation learning for soft sensing
Yan, X., Wang, J., & Jiang, Q. (2019) · Information Sciences · doi:10.1016/j.ins.2019.11.039 · Cited by 52
Efficient Saliency Maps for Explainable AIPreprint
Mundhenk, T. N., Chen, B. Y., & Friedland, G. H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.11293 · Cited by 111
Expert Level Evaluations for Explainable AI (XAI) Methods in the Medical Domain
Muddamsetty, S. M., Jahromi, M. N. S., & Moeslund, T. B. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-68796-0_3 · Cited by 52
ExplainGAN: Model Explanation via Decision Boundary Crossing Transformations
Samangouei, P., Saeedi, A., Nakagawa, L., & Silberman, N. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01249-6_41 · Cited by 50
Media Forensics Considerations on DeepFake Detection with Hand-Crafted Features
Siegel, D., Kraetzer, C., Seidlitz, S., & Dittmann, J. (2021) · Journal of Imaging · doi:10.3390/jimaging7070108 · Cited by 52
Novel applications of Convolutional Neural Networks in the age of Transformers
Ersavas, T., Smith, M. A., & Mattick, J. S. (2024) · Scientific Reports · doi:10.1038/s41598-024-60709-z · Cited by 64
Artificial intelligence in the diagnosis of Parkinson’s disease from ioflupane-123 single-photon emission computed tomography dopamine transporter scans using transfer learning
Kim, D. H., Wit, H., & Thurston, M. (2018) · Nuclear Medicine Communications · doi:10.1097/mnm.0000000000000890 · Cited by 49
135-GHz D-Band 60-Gbps PAM-8 Wireless Transmission Employing a Joint DNN Equalizer With BP and CMMA
Zhou, W., Zhao, L., Zhang, J., Wang, K., Yu, J., Chen, Y.-W., Shen, S., Shiu, R., & Chang, G. (2020) · Journal of Lightwave Technology · doi:10.1109/jlt.2020.2979070 · Cited by 50
Learning Variational Word Masks to Improve the Interpretability of Neural Text Classifiers
Chen, H., & Ji, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.347 · Cited by 59
A comprehensive study on fidelity metrics for XAI
Miró-Nicolau, M., Jaume-i-Capó, A., & Moyà-Alcover, G. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103900 · Cited by 63
Temporal Task Scheduling of Multiple Delay-Constrained Applications in Green Hybrid Cloud
Yuan, H., Bi, J., & Zhou, M. (2018) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2018.2878561 · Cited by 49
Leveraging Latent Features for Local Explanations
Luss, R., Chen, P., Dhurandhar, A., Sattigeri, P., Zhang, Y., Shanmugam, K., & Tu, C. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467265 · Cited by 71
Towards robust explanations for deep neural networks
Dombrowski, A.-K., Anders, C. J., Müller, K., & Kessel, P. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108194 · Cited by 62
Analysis of a Deep Learning Model for 12-Lead ECG Classification Reveals Learned Features Similar to Diagnostic Criteria
Bender, T., Metsch, J. M., Krefting, D., Müller, C., Dathe, H., Seidler, T., Spicher, N., & Hauschild, A.-C. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3271858 · Cited by 55
Attention gate guided multiscale recursive fusion strategy for deep neural network-based fault diagnosis
Zhang, Z., Zhou, F., Karimi, H. R., Fujita, H., Hu, X., Wen, C., & Wang, T. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107052 · Cited by 48
Malware Classification with Improved Convolutional Neural Network Model
Lad, S. S., & Adamuthe, A. C. (2020) · International Journal of Computer Network and Information Security · doi:10.5815/ijcnis.2020.06.03 · Cited by 55
VisualBackProp: efficient visualization of CNNsPreprint
Bojarski, M., Choromanska, A., Choromański, K., Firner, B., Jackel, L., Müller, U., & Zieba, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.05418 · Cited by 96
Human attention guided explainable artificial intelligence for computer vision models
Liu, G., Zhang, J., Chan, A. B., & Hsiao, J. H. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106392 · Cited by 57
Early seizure detection for closed loop direct neurostimulation devices in epilepsy
Dümpelmann, M. (2019) · Journal of Neural Engineering · doi:10.1088/1741-2552/ab094a · Cited by 47
Evaluation Metrics for XAI: A Review, Taxonomy, and Practical Applications
Kadir, M. A., Mosavi, A. H., & Sonntag, D. (2023) · IEEE International Conference on Intelligent Engineering Systems (INES) · doi:10.1109/ines59282.2023.10297629 · Cited by 53
Neurological state changes indicative of ADHD in children learned via EEG-based LSTM networks
Chang, Y., Stevenson, C., Chen, I., Lin, D., & Ko, L. (2022) · Journal of Neural Engineering · doi:10.1088/1741-2552/ac4f07 · Cited by 46
Predicting High-Risk Patients and High-Risk Outcomes in Heart Failure
Wehbe, R. M., Khan, S. S., Shah, S. J., & Ahmad, F. S. (2020) · Heart Failure Clinics · doi:10.1016/j.hfc.2020.05.002 · Cited by 45
Explaining Image Classifiers by Removing Input Features Using Generative Models
Agarwal, C., & Nguyen, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69544-6_7 · Cited by 44
Explainable Artificial Intelligence for Resilient Security Applications in the Internet of Things
Masud, M. T., Keshk, M., Moustafa, N., Linkov, I., & Emge, D. K. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3413790 · Cited by 50
Understanding Deep Neural Networks for Regression in Leaf Counting
Dobrescu, A., Giuffrida, M. V., & Tsaftaris, S. A. (2019) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00316 · Cited by 44
Investigation of Deep-Learning-Driven Identification of Multiple Sclerosis Patients Based on Susceptibility-Weighted Images Using Relevance Analysis
Lopatina, A., Ropele, S., Sibgatulin, R., Reichenbach, J. R., & Güllmar, D. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.609468 · Cited by 49
Explanation by Progressive ExaggerationPreprint
Singla, S., Pollack, B., Chen, J., & Batmanghelich, K. N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.00483 · Cited by 67
Explainable AI: The Effect of Contradictory Decisions and Explanations on Users’ Acceptance of AI Systems
Ebermann, C., Selisky, M., & Weibelzahl, S. (2022) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2022.2126812 · Cited by 54
Augmented Grad-CAM++: Super-Resolution Saliency Maps for Visual Interpretation of Deep Neural Network
Gao, Y., Liu, J., Li, W., Hou, M. L., Li, Y., & Zhao, H. (2023) · Electronics · doi:10.3390/electronics12234846 · Cited by 46
Attention Guided CAM: Visual Explanations of Vision Transformer Guided by Self-Attention
Leem, S., & Seo, H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i4.28077 · Cited by 48
Assessing fidelity in XAI post-hoc techniques: A comparative study with ground truth explanations datasets
Miró-Nicolau, M., Jaume-i-Capó, A., & Moyà-Alcover, G. (2024) · Artificial Intelligence · doi:10.1016/j.artint.2024.104179 · Cited by 55
CNN-based small object detection and visualization with feature activation mapping
Menikdiwela, M., Nguyen, C. V., Li, H., & Shaw, M. E. (2017) · International Conference on Image and Vision Computing New Zealand (IVCNZ) · doi:10.1109/ivcnz.2017.8402455 · Cited by 42
Robust Malware Detection using Residual Attention Network
Ganesan, S., Ravi, V., Krichen, M., Sowmya, V., Alroobaea, R., & Soman, K. P. (2021) · International Conference on Consumer Electronics · doi:10.1109/icce50685.2021.9427623 · Cited by 42
Interpreting the Predictions of Complex ML Models by Layer-wise Relevance PropagationPreprint
Samek, W., Montavon, G., Binder, A., Lapuschkin, S., & Müller, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.08191 · Cited by 54
Self-citation
Fairwashing Explanations with Off-Manifold DetergentPreprint
Anders, C. J., Pasliev, P., Dombrowski, A.-K., Müller, K., & Kessel, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.09969 · Cited by 44
Evaluating the faithfulness of saliency maps in explaining deep learning models using realistic perturbations
Amorim, J. P., Abreu, P. H., Santos, J. A. M., Cortes, M., & Vila, V. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103225 · Cited by 46
Mutual Explanations for Cooperative Decision Making in Medicine
Schmid, U., & Finzel, B. (2020) · Künstliche Intell. · doi:10.1007/s13218-020-00633-2 · Cited by 42
Relevance aggregation for neural networks interpretability and knowledge discovery on tabular data
Grisci, B. I., Krause, M. J., & Dorn, M. (2021) · Information Sciences · doi:10.1016/j.ins.2021.01.052 · Cited by 41
No-Reference Stereoscopic Image Quality Assessment Based on Image Distortion and Stereo Perceptual Information
Shen, L., Fang, R., Yao, Y., Geng, X., & Wu, D. O. (2018) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2018.2804885 · Cited by 40
Quantum-Chemical Insights from Interpretable Atomistic Neural Networks
Schütt, K. T., Gastegger, M., Tkatchenko, A., & Müller, K. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_17 · Cited by 43
Towards Better Understanding Attribution Methods
Rao, S., Böhle, M., & Schiele, B. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00998 · Cited by 40
Evaluating and Characterizing Human Rationales
Carton, S., Rathore, A., & Tan, C. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.747 · Cited by 52
Bilateral sensitivity analysis: a better understanding of a neural network
Zhang, H., Jiang, Y., Wang, J., Zhang, K., & Pal, N. R. (2022) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-022-01511-z · Cited by 41
Attention-Based Deep Learning System for Classification of Breast Lesions—Multimodal, Weakly Supervised Approach
Bobowicz, M., Rygusik, M., Buler, J., Buler, R., Ferlin, M. A., Kwasigroch, A., Szurowska, E., & Grochowski, M. (2023) · Cancers · doi:10.3390/cancers15102704 · Cited by 47
Weakly supervised lesion localization for age-related macular degeneration detection using optical coherence tomography images
Yang, H.-L., Kim, J. J., Kim, J. H., Kang, Y. K., Park, D. H., Park, H. S., Kim, H. K., & Kim, M.-S. (2019) · PLoS ONE · doi:10.1371/journal.pone.0215076 · Cited by 39
Interpretable Graph Capsule Networks for Object Recognition
Gu, J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16237 · Cited by 38
Memory Recall: A Simple Neural Network Training Framework Against Catastrophic Forgetting
Zhang, B., Guo, Y., Li, Y., He, Y., Wang, H., & Dai, Q. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3099700 · Cited by 42
G -LIME: Statistical learning for local interpretations of deep neural networks using global priors
Li, X., Xiong, H., Li, X., Zhang, X., Liu, J., Jiang, H., Chen, Z., & Dou, D. (2022) · Artificial Intelligence · doi:10.1016/j.artint.2022.103823 · Cited by 43
Towards successful implementation of artificial intelligence in skin cancer care: a qualitative study exploring the views of dermatologists and general practitioners
Sangers, T. E., Wakkee, M., Moolenburgh, F. J., Nijsten, T., & Lugtenberg, M. (2022) · Archives of Dermatological Research · doi:10.1007/s00403-022-02492-3 · Cited by 38
Explainable AI for time series via Virtual Inspection Layers
Vielhaben, J., Lapuschkin, S., Montavon, G., & Samek, W. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110309 · Cited by 38
Self-citation
EEG-Based Drowsiness Detection With Fuzzy Independent Phase-Locking Value Representations Using Lagrangian-Based Deep Neural Networks
Reddy, T. K., Arora, V., Gupta, V., Biswas, R., & Behera, L. (2021) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2021.3113823 · Cited by 38
Interpreting runoff forecasting of long short-term memory network: An investigation using the integrated gradient method on runoff data from the Han River Basin
Xin, J. L., Luo, J., Zuo, G., & Yang, X. (2023) · Journal of Hydrology Regional Studies · doi:10.1016/j.ejrh.2023.101549 · Cited by 40
A Comprehensive Survey on Machine and Deep Learning for Optical Communications
Amirabadi, M. A., Nezamalhosseini, S. A., Kahaei, M. H., & Chen, L. R. (2025) · IEEE Access · doi:10.1109/access.2025.3569559 · Cited by 54
Plagiarism Detection in Computer Programming Using Feature Extraction From Ultra-Fine-Grained Repositories
Ljubovic, V., & Pajic, E. (2020) · IEEE Access · doi:10.1109/access.2020.2996146 · Cited by 38
Why did you predict that? Towards explainable artificial neural networks for travel demand analysis
Alwosheel, A., van Cranenburgh, S., & Chorus, C. G. (2021) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2021.103143 · Cited by 40
Convolutional neural network performance compared to radiologists in detecting intracranial hemorrhage from brain computed tomography: A systematic review and meta-analysis
Jørgensen, M. D., Antulov, R., Hess, S., & Lysdahlgaard, S. (2021) · European Journal of Radiology · doi:10.1016/j.ejrad.2021.110073 · Cited by 40
A Regression-Based Approach to Diabetic Retinopathy Diagnosis Using Efficientnet
Vijayan, M., & Venkatakrishnan, S. (2023) · Diagnostics · doi:10.3390/diagnostics13040774 · Cited by 42
Explainability and Evaluation of Vision Transformers: An In-Depth Experimental Study
Stassin, S., Corduant, V., Mahmoudi, S. A., & Siebert, X. (2023) · Electronics · doi:10.3390/electronics13010175 · Cited by 39
Please Tell Me More: Privacy Impact of Explainability through the Lens of Membership Inference Attack
Liu, H., Wu, Y., Yu, Z., & Zhang, N. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00120 · Cited by 46
Mouse Authentication Without the Temporal Aspect – What Does a 2D-CNN Learn?
Chong, P., Tan, Y., Guarnizo, J., Elovici, Y., & Binder, A. (2018) · IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw.2018.00011 · Cited by 37
Self-citation
On the Soundness of XAI in Prognostics and Health Management (PHM)
Solis‐Martín, D., Galán‐Paez, J., & Borrego‐Díaz, J. (2023) · Information · doi:10.3390/info14050256 · Cited by 40
Explaining Black-Box Classifiers with ILP – Empowering LIME with Aleph to Approximate Non-linear Decisions with Relational Rules
Rabold, J., Siebers, M., & Schmid, U. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-99960-9_7 · Cited by 36
Attention Inspiring Receptive-Fields Network for Learning Invariant Representations
Yang, L., Song, Q., Wu, Y., & Hu, M. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2873722 · Cited by 36
Gradient-Based Vs. Propagation-Based Explanations: An Axiomatic Comparison
Montavon, G. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_13 · Cited by 36
Self-citation
Explaining decisions of deep neural networks used for fish age prediction
Ordoñez, A., Eikvil, L., Salberg, A.-B., Harbitz, A., Murray, S., & Kampffmeyer, M. (2020) · PLoS ONE · doi:10.1371/journal.pone.0235013 · Cited by 39
Software for dataset-wide XAI: From local explanations to global insights with Zennit, CoRelAy, and ViRelAy
Anders, C. J., Neumann, D., Samek, W., Müller, K., & Lapuschkin, S. (2026) · PLoS ONE · doi:10.1371/journal.pone.0336683 · Cited by 62
Self-citation
Identifying Individual Facial Expressions by Deconstructing a Neural Network
Arbabzadah, F., Montavon, G., Müller, K., & Samek, W. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-45886-1_28 · Cited by 36
Self-citation
Input-Cell Attention Reduces Vanishing Saliency of Recurrent Neural NetworksPreprint
Ismail, A. A., Gunady, M. K., Pessoa, L., Bravo, H. C., & Feizi, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.12370 · Cited by 40
Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis
(2017) · doi:10.18653/v1/w17-52 · Cited by 36
Global Aggregations of Local Explanations for Black Box models
van der Linden, I., Haned, H., & Kanoulas, E. (2019) · arXiv · doi:10.48550/arxiv.1907.03039 · Cited by 35
Whole slide imaging (WSI) scanner differences influence optical and computed properties of digitized prostate cancer histology
Duenweg, S. R., Bobholz, S. A., Lowman, A. K., Stebbins, M. A., Winiarz, A., Nath, B., Kyereme, F., Iczkowski, K. A., & LaViolette, P. S. (2023) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2023.100321 · Cited by 35
On the interpretability of Fuzzy Cognitive Maps
Nápoles, G., Ranković, N., & Salgueiro, Y. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111078 · Cited by 35
A genetic algorithm-assisted deep learning approach for crop yield prediction
Bi, L., & Hu, G. (2021) · Soft Computing · doi:10.1007/s00500-021-05995-9 · Cited by 35
Normalization Matters in Weakly Supervised Object Localization
Kim, J., Choe, J., Yun, S., & Kwak, N. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00341 · Cited by 35
Quantifying Explainability of Saliency Methods in Deep Neural Networks With a Synthetic Dataset
Tjoa, E., & Guan, C. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3228834 · Cited by 56
Supporting Deep Neural Network Safety Analysis and Retraining Through Heatmap-Based Unsupervised Learning
Fahmy, H. M., Pastore, F., Bagherzadeh, M., & Briand, L. (2021) · IEEE Transactions on Reliability · doi:10.1109/tr.2021.3074750 · Cited by 40
Group-CAM: Group Score-Weighted Visual Explanations for Deep Convolutional NetworksPreprint
Zhang, Q., Rao, L., & Yang, Y.-B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.13859 · Cited by 34
The privacy-explainability trade-off: unraveling the impacts of differential privacy and federated learning on attribution methods
Saifullah, S., Mercier, D., Lucieri, A., Dengel, A., & Ahmed, S. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1236947 · Cited by 53

Authors: Samek, W., Binder, A., Montavon, G., Lapuschkin, S., & Muller, K.-R.
Affiliations: Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (DE)
Singapore University of Technology and Design (SG)
Technische Universität Berlin, Berlin, Germany
ORCID: Samek, Wojciech
Binder, Alexander
Lapuschkin, Sebastian
Venue: IEEE Transactions on Neural Networks and Learning Systems
Publisher: Institute of Electrical and Electronics Engineers (IEEE)
Published: 2017-11
Volume: 28
Issue: 11
Pages: 2660-2673
ISSN: 2162-237X (ISSN-L)
2162-2388
DOI: 10.1109/tnnls.2016.2599820
OpenAlex: W2240067561
Cited by: 1,164
References: 51
Citations by year: 2026: 120 · 2025: 105 · 2024: 139 · 2023: 163 · 2022: 147 · 2021: 171 · 2020: 123 · 2019: 92 · 2018: 66 · 2017: 26
FWCI: 53.43
Type: journal article
Language: en
Open access: closed
Licence: ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html (vor)
Locations: IEEE Transactions on Neural Networks and Learning Systems (publishedVersion)
PubMed (publishedVersion)
Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion)
Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
Funders: Brain Korea 21 Plus Program through the National Research Foundation of Korea Funded by the Ministry of Education
DFG (MU 987/17-1)
German Ministry for Education and Research as Berlin Big Data Center (01IS14013A)
Topics: Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
Keywords: deep neural network interpretability, layer-wise relevance propagation, deconvolution, explainable AI, image classification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Evaluating the Visualization of What a Deep Neural Network Has Learned

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Citations

Cites

Cited by

Related

Metadata

Record

About these checks