Evaluating the Visualization of What a Deep Neural Network Has Learned
Abstract
Deep neural networks (DNNs) have demonstrated impressive performance in complex machine learning tasks such as image classification or speech recognition. However, due to their multilayer nonlinear structure, they are not transparent, i.e., it is hard to grasp what makes them arrive at a particular classification or recognition decision, given a new unseen data sample. Recently, several approaches have been proposed enabling one to understand and interpret the reasoning embodied in a DNN for a single test image. These methods quantify the “importance” of individual pixels with respect to the classification decision and allow a visualization in terms of a heatmap in pixel/input space. While the usefulness of heatmaps can be judged subjectively by a human, an objective quality measure is missing. In this paper, we present a general methodology based on region perturbation for evaluating ordered collections of pixels such as heatmaps. We compare heatmaps computed by three different methods on the SUN397, ILSVRC2012, and MIT Places data sets. Our main result is that the recently proposed layer-wise relevance propagation algorithm qualitatively and quantitatively provides a better explanation of what made a DNN arrive at a particular classification decision than the sensitivity-based approach or the deconvolution method. We provide theoretical arguments to explain this result and discuss its practical implications. Finally, we investigate the use of heatmaps for unsupervised assessment of the neural network performance.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892 · see citing works
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,890 · see citing works
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · see citing works
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381 · see citing works
Domain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318 · see citing works
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143 · see citing works
An introduction to variable and feature selection
GuyonIsabelle, & ElisseeffAndré (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944968 · Cited by 7,843 · see citing works
Natural Language Processing (almost) from ScratchPreprint
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 6,882 · see citing works
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · see citing works
An Introduction to Variable and Feature Selection.
Guyon, I., & Elisseeff, A. (2003) · Cited by 6,576 · see citing works
3D Convolutional Neural Networks for Human Action Recognition
Ji, S., Xu, W., Yang, M., & Yu, K. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.59 · Cited by 6,222 · see citing works
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · see citing works
Computational modelling of visual attention
Itti, L., & Koch, C. (2001) · Nature Reviews Neuroscience · doi:10.1038/35058500 · Cited by 4,733 · see citing works
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524 · see citing works
Self-citationMulti-column deep neural networks for image classification
Ciresan, D. C., Meier, U., & Schmidhuber, J. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6248110 · Cited by 3,760 · see citing works
SUN database: Large-scale scene recognition from abbey to zoo
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., & Torralba, A. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539970 · Cited by 3,283 · see citing works
Deep neural networks are easily fooled: High confidence predictions for unrecognizable images
Nguyen, A., Yosinski, J., & Clune, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298640 · Cited by 3,258 · see citing works
A saliency-based search mechanism for overt and covert shifts of visual attention
Itti, L., & Koch, C. (2000) · Vision Research · doi:10.1016/s0042-6989(99)00163-7 · Cited by 3,137 · see citing works
(2014) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 2,534 · see citing works
Natural Image Statistics and Neural Representation
Simoncelli, E. P., & Olshausen, B. A. (2001) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.24.1.1193 · Cited by 2,460 · see citing works
Neural Networks: Tricks of the Trade: Second Edition
(2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8 · Cited by 2,116 · see citing works
Self-citationUnderstanding deep image representations by inverting them
Mahendran, A., & Vedaldi, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299155 · Cited by 1,827 · see citing works
Understanding Neural Networks Through Deep VisualizationPreprint
Yosinski, J., Clune, J., Nguyen, A. M., Fuchs, T. J., & Lipson, H. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06579 · Cited by 1,687 · see citing works
Going Deeper with Convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2014) · arXiv · doi:10.48550/arxiv.1409.4842 · Cited by 1,461 · see citing works
Explaining nonlinear classification decisions with deep taylor decomposition
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K.-R. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.11.008 · Cited by 1,357 · see citing works
Self-citationLeCun, Y., Bottou, L., Orr, G., & Müller, K. (1998) · neural information processing systems · Cited by 1,333 · see citing works
(2012) · neural information processing systems · Cited by 1,291 · see citing works
LeCun, Y., Bottou, L., Orr, G., & Müller, K. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-35289-8_3 · Cited by 1,268 · see citing works
Le, Q. V., Zou, W. Y., Yeung, S. Y., & Ng, A. Y. (2011) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2011.5995496 · Cited by 966 · see citing works
Object Detectors Emerge in Deep Scene CNNsPreprint
Zhou, B., Khosla, A., Lapedriza, À., Oliva, A., & Torralba, A. B. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6856 · Cited by 823 · see citing works
How to Explain Individual Classification DecisionsPreprint
Baehrens, D., Schroeter, T. S., Stefan Harmeling, B., Kawanabe, M., Hansen, K., & Mueller, K.-R. (2009) · arXiv (Cornell University) · doi:10.48550/arxiv.0912.1128 · Cited by 743 · see citing works
Machine learning of molecular electronic properties in chemical compound space
Montavon, G., Rupp, M., Gobre, V., Vazquez-Mayagoitia, A., Hansen, K., Tkatchenko, A., Müller, K.-R., & von Lilienfeld, O. A. (2013) · New Journal of Physics · doi:10.1088/1367-2630/15/9/095003 · Cited by 631 · see citing works
Self-citationInverting Visual Representations with Convolutional Networks
Dosovitskiy, A., & Brox, T. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.522 · Cited by 631 · see citing works
Machine learning of molecular electronic properties in chemical compound space
Montavon, G., Rupp, M., Gobre, V. V., Vázquez‐Mayagoitia, Á., Hansen, K., Tkatchenko, A., Müller, K., & von Lilienfeld, O. A. (2013) · MPG.PuRe (Max Planck Society) · doi:10.3929/ethz-b-000071765 · Cited by 414 · see citing works
Self-citationComputational models of cortical visual processing.
Heeger, D. J., Simoncelli, E. P., & Movshon, J. A. (1996) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.93.2.623 · Cited by 280 · see citing works
Analyzing Classifiers: Fisher Vectors and Deep Neural Networks
Lapuschkin, S., Binder, A., Montavon, G., Müller, K., & Samek, W. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.318 · Cited by 212 · see citing works
Self-citationOn Relevant Dimensions in Kernel Feature Spaces
Braun, M. L., Buhmann, J. M., & Müller, K. (2008) · doi:10.5555/1390681.1442795 · Cited by 113 · see citing works
Understanding Deep Image Representations by Inverting ThemPreprint
Mahendran, A., & Vedaldi, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.0035 · Cited by 105 · see citing works
Kernel analysis of deep networks
Montavon, G., Braun, M. L., & Müller, K. (2011) · Cited by 96 · see citing works
Self-citationThe LRP toolbox for artificial neural networks
Lapuschkin, S., Binder, A., Montavon, G., Müller, K., & Samek, W. (2016) · Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) · Cited by 90 · see citing works
Self-citationThe LRP toolbox for artificial neural networks
LapuschkinSebastian, BinderAlexander, MontavonGrégoire, MüllerKlaus-Robert, & SamekWojciech (2016) · Journal of Machine Learning Research · doi:10.5555/2946645.3007067 · Cited by 63 · see citing works
Visual Interpretation of Kernel‐Based Prediction Models
Hansen, K., Baehrens, D., Schroeter, T. S., Rupp, M., & Müller, K. (2011) · Molecular Informatics · doi:10.1002/minf.201100059 · Cited by 55 · see citing works
Montavon, G., Braun, M. L., Krueger, T., & Müller, K. (2013) · IEEE Signal Processing Magazine · doi:10.1109/msp.2013.2249294 · Cited by 43 · see citing works
Self-citationVISUALIZATION OF NONLINEAR CLASSIFICATION MODELS IN NEUROIMAGING - Signed Sensitivity Maps
Rasmussen, P. M., Schmah, T., Madsen, K. H., Lund, T. E., Yourganov, G., Strother, S. C., & Hansen, L. K. (2012) · International Conference on Bio-inspired Systems and Signal Processing · doi:10.5220/0003785602540263 · Cited by 13 · see citing works
Flagged first, then by citations
4 of the 51 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Methods for interpreting and understanding deep neural networks
Montavon, G., Samek, W., & Müller, K.-R. (2018) · Digital Signal Processing · doi:10.1016/j.dsp.2017.10.011 · Cited by 2,665 · see citing works
Self-citationA Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524 · see citing works
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762 · see citing works
Shin, D. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2020.102551 · Cited by 1,548 · see citing works
Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3060483 · Cited by 1,477 · see citing works
Self-citationExplaining nonlinear classification decisions with deep taylor decomposition
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K.-R. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.11.008 · Cited by 1,357 · see citing works
Self-citationShen, C. (2018) · Water Resources Research · doi:10.1029/2018wr022643 · Cited by 1,476 · see citing works
Sanity Checks for Saliency Maps
Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018) · arXiv · doi:10.48550/arxiv.1810.03292 · Cited by 1,294 · see citing works
Trojaning Attack on Neural Networks
Liu, Y., Ma, S., Aafer, Y., Lee, W.-C., Zhai, J., Wang, W., & Zhang, X. (2018) · Proceedings 2018 Network and Distributed System Security Symposium · doi:10.14722/ndss.2018.23291 · Cited by 1,270 · see citing works
Samek, W., Wiegand, T., & Müller, K.-R. (2017) · arXiv · doi:10.48550/arxiv.1708.08296 · Cited by 1,080 · see citing works
Self-citationMachine Learning Methods for Histopathological Image Analysis
Komura, D., & Ishikawa, S. (2018) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2018.01.001 · Cited by 937 · see citing works
A Survey on Neural Network Interpretability
Zhang, Y., Tino, P., Leonardis, A., & Tang, K. (2021) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2021.3100641 · Cited by 923 · see citing works
Layer-Wise Relevance Propagation: An Overview
Montavon, G., Binder, A., Lapuschkin, S., Samek, W., & Müller, K. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_10 · Cited by 825 · see citing works
Self-citationMarkus, A. F., Kors, J. A., & Rijnbeek, P. R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103655 · Cited by 811 · see citing works
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2021) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3387166 · Cited by 785 · see citing works
'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions
Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173951 · Cited by 691 · see citing works
Towards Explainable Artificial Intelligence
Samek, W., & Müller, K. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_1 · Cited by 650 · see citing works
Self-citationYuan, X., Huang, B., Wang, Y., Yang, C., & Gui, W. (2018) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2018.2809730 · Cited by 635 · see citing works
The (Un)reliability of Saliency Methods
Kindermans, P.-J., Hooker, S., Adebayo, J., Alber, M., Schütt, K. T., Dähne, S., Erhan, D., & Kim, B. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_14 · Cited by 652 · see citing works
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · see citing works
“In [103] it is presented a comparison between explanations provided with LRP and more traditional deconvolution and sensitivity analysis.”
MethodNotions of explainability and evaluation approaches for explainable artificial intelligence
Vilone, G., & Longo, L. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.009 · Cited by 598 · see citing works
Nauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023) · ACM Computing Surveys · doi:10.1145/3583558 · Cited by 590 · see citing works
Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers
Hohman, F., Kahng, M., Pienta, R., & Chau, D. H. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2843369 · Cited by 582 · see citing works
Unmasking Clever Hans predictors and assessing what machines really learn
Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., & Müller, K. (2019) · RePEc: Research Papers in Economics · Cited by 580 · see citing works
Self-citationERASER: A Benchmark to Evaluate Rationalized NLP Models
DeYoung, J., Jain, S., Rajani, N. F., Lehman, E., Xiong, C., Socher, R., & Wallace, B. C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.408 · Cited by 561 · see citing works
Explainability Methods for Graph Convolutional Neural Networks
Pope, P. E., Kolouri, S., Rostami, M., Martin, C. E., & Hoffmann, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01103 · Cited by 553 · see citing works
Deep image mining for diabetic retinopathy screening
Quellec, G., Charrière, K., Boudi, Y., Cochener, B., & Lamard, M. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.04.012 · Cited by 526 · see citing works
Explainable machine learning in materials science
Zhong, X., Gallagher, B. J., Liu, S., Kailkhura, B., Hiszpanski, A. M., & Han, T. Y.-J. (2022) · npj Computational Materials · doi:10.1038/s41524-022-00884-7 · Cited by 526 · see citing works
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond
Li, X., Xiong, H., Li, X., Wu, X., Zhang, X., Liu, J., Bian, J., & Dou, D. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01756-8 · Cited by 542 · see citing works
Salahuddin, Z., Woodruff, H. C., Chatterjee, A., & Lambin, P. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.105111 · Cited by 508 · see citing works
Explaining anomalies detected by autoencoders using Shapley Additive Explanations
Antwarg, L., Miller, R. M., Shapira, B., & Rokach, L. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115736 · Cited by 496 · see citing works
Towards Robust Interpretability with Self-Explaining Neural NetworksPreprint
Alvarez-Melis, D., & Alvarez-Melis, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07538 · Cited by 545 · see citing works
Interpretability of deep learning models: A survey of results
Chakraborty, S., Tomsett, R., Raghavendra, R., Harborne, D., Alzantot, M., Cerutti, F., Srivastava, M. B., Preece, A. D., Julier, S., Rao, R., et al. (2017) · IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI) · doi:10.1109/uic-atc.2017.8397411 · Cited by 469 · see citing works
Explainable Deep Learning: A Field Guide for the Uninitiated
Ras, G., Xie, N., van Gerven, M. A. J., & Doran, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13200 · Cited by 486 · see citing works
Deep learning in microscopy image analysis: a survey
Xing, F., Xie, Y., Su, H., Liu, F., & Yang, L. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2766168 · Cited by 460 · see citing works
Layer-Wise Relevance Propagation for Neural Networks with Local Renormalization Layers
Binder, A., Montavon, G., Lapuschkin, S., Müller, K., & Samek, W. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-44781-0_8 · Cited by 455 · see citing works
Self-citationLauritsen, S. M., Kristensen, M., Olsen, M. V., Larsen, M. S., Lauritsen, K. M., Jørgensen, M. J., Lange, J., & Thiesson, B. (2020) · Nature Communications · doi:10.1038/s41467-020-17431-x · Cited by 448 · see citing works
A Benchmark for Interpretability Methods in Deep Neural NetworksPreprint
Hooker, S., Erhan, D., Kindermans, P.-J., & Kim, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.10758 · Cited by 438 · see citing works
Deep Hyperspectral Image Sharpening
Dian, R., Shutao Li, A., Guo, A., & Fang, L. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2798162 · Cited by 414 · see citing works
Analysis of Explainers of Black Box Deep Neural Networks for Computer Vision: A Survey
Buhrmester, V., Münch, D., & Arens, M. (2021) · Machine Learning and Knowledge Extraction · doi:10.3390/make3040048 · Cited by 403 · see citing works
Deep learning in bioinformatics: Introduction, application, and perspective in the big data era
Li, Y., Huang, C., Ding, L., Li, Z., Pan, Y., & Gao, X. (2019) · Methods · doi:10.1016/j.ymeth.2019.04.008 · Cited by 401 · see citing works
Obtaining genetics insights from deep learning via explainable artificial intelligence
Novakovsky, G., Dexter, N., Libbrecht, M. W., Wasserman, W. W., & Mostafavi, S. (2022) · Nature Reviews Genetics · doi:10.1038/s41576-022-00532-2 · Cited by 387 · see citing works
Interpretable deep neural networks for single-trial EEG classification
Sturm, I., Lapuschkin, S., Samek, W., & Müller, K. (2016) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2016.10.008 · Cited by 380 · see citing works
Self-citationDeep learning in mental health outcome research: a scoping review
Su, C., Xu, Z., Pathak, J., & Wang, F. (2020) · Translational Psychiatry · doi:10.1038/s41398-020-0780-3 · Cited by 368 · see citing works
Explaining Recurrent Neural Network Predictions in Sentiment Analysis
Arras, L., Montavon, G., Müller, K.-R., & Samek, W. (2017) · Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis · doi:10.18653/v1/w17-5221 · Cited by 361 · see citing works
Self-citationSpatial Pyramid-Enhanced NetVLAD With Weighted Triplet Loss for Place Recognition
Yu, J., Zhu, C., Zhang, J., Huang, Q., & Tao, D. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2908982 · Cited by 355 · see citing works
Digital Pathology: Advantages, Limitations and Emerging Perspectives
Jahn, S. W., Plass, M., & Moinfar, F. (2020) · Journal of Clinical Medicine · doi:10.3390/jcm9113697 · Cited by 351 · see citing works
HESS Opinions: Incubating deep-learning-powered hydrologic science advances as a community
Shen, C., Laloy, E., ELSHORBAGY, A. A., Albert, A., Bales, J. D., Chang, F., Ganguly, S., Hsu, K., Kifer, D., Fang, Z. N., et al. (2018) · Hydrology and earth system sciences · doi:10.5194/hess-22-5639-2018 · Cited by 344 · see citing works
Chen, C., & Gu, G. X. (2020) · Advanced Science · doi:10.1002/advs.201902607 · Cited by 332 · see citing works
Neural Cognitive Diagnosis for Intelligent Education Systems
Wang, F., Liu, Q., Chen, E., Huang, Z., Chen, Y., Yin, Y., Huang, Z., & Wang, S. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6080 · Cited by 325 · see citing works
Islam, M. R., Ahmed, M. U., Barua, S., & Begum, S. (2022) · Applied Sciences · doi:10.3390/app12031353 · Cited by 339 · see citing works
Deep learning algorithm predicts diabetic retinopathy progression in individual patients
Arcadu, F., Benmansour, F., Maunz, A., Willis, J., Haskova, Z., & Prunotto, M. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0172-3 · Cited by 305 · see citing works
Learning how to explain neural networks: PatternNet and PatternAttributionPreprint
Kindermans, P.-J., Schütt, K. T., Alber, M., Müller, K. R., Erhan, D., Kim, B., & Dähne, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.05598 · Cited by 325 · see citing works
Böhle, M., Eitel, F., Weygandt, M., & Ritter, K. (2019) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2019.00194 · Cited by 291 · see citing works
A comprehensive review of deep learning in colon cancer
Paçal, İ., Karaboğa, D., Baştürk, A., Akay, B., & Nalbantoglu, U. (2020) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.104003 · Cited by 289 · see citing works
Skin lesion segmentation via generative adversarial networks with dual discriminators
Lei, B., Xia, Z., Jiang, F., Jiang, X., Ge, Z., Xu, Y., Qin, J., Chen, S., Wang, T., & Wang, S. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101716 · Cited by 280 · see citing works
On the (In)fidelity and Sensitivity for ExplanationsPreprint
Yeh, C., Hsieh, C.-Y., Suggala, A. S., Inouye, D. I., & Ravikumar, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.09392 · Cited by 291 · see citing works
Explaining the unique nature of individual gait patterns with deep learning
Horst, F., Lapuschkin, S., Samek, W., Müller, K.-R., & Schöllhorn, W. I. (2019) · Scientific Reports · doi:10.1038/s41598-019-38748-8 · Cited by 273 · see citing works
Self-citationExplainable Artificial Intelligence: a Systematic Review
Vilone, G., & Longo, L. (2020) · arXiv · doi:10.48550/arxiv.2006.00093 · Cited by 269 · see citing works
Pruning by explaining: A novel criterion for deep neural network pruning
Yeom, S.-K., Seegerer, P., Lapuschkin, S., Binder, A., Wiedemann, S. M., Müller, K., & Samek, W. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.107899 · Cited by 269 · see citing works
Self-citation"What is relevant in a text document?": An interpretable machine learning approach
Arras, L., Horn, F., Montavon, G., Müller, K., & Samek, W. (2017) · PLoS ONE · doi:10.1371/journal.pone.0181142 · Cited by 268 · see citing works
Self-citationDCNNBiLSTM: An Efficient Hybrid Deep Learning-Based Intrusion Detection System
Hnamte, V., & Hussain, J. (2023) · Telematics and Informatics Reports · doi:10.1016/j.teler.2023.100053 · Cited by 299 · see citing works
Gradient-Based Attribution Methods
Ancona, M., Ceolini, E., Öztireli, C., & Groß, M. H. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_9 · Cited by 263 · see citing works
Februus: Input Purification Defense Against Trojan Attacks on Deep Neural Network Systems
Doan, B. G., Abbasnejad, E., & Ranasinghe, D. C. (2020) · Annual Computer Security Applications Conference · doi:10.1145/3427228.3427264 · Cited by 263 · see citing works
Four principles of explainable artificial intelligence
Phillips, P. J., Hahn, C. A., Fontana, P. C., Yates, A. N., Greene, K., Broniatowski, D. A., & Przybocki, M. A. (2021) · doi:10.6028/nist.ir.8312 · Cited by 262 · see citing works
Higher-Order Explanations of Graph Neural Networks via Relevant Walks
Schnake, T., Eberle, O., Lederer, J., Nakajima, S., Schütt, K. T., Müller, K., & Montavon, G. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3115452 · Cited by 289 · see citing works
Self-citationA Survey on Deep Hashing Methods
Luo, X., Wang, H., Wu, D., Chen, C., Deng, M., Huang, J., & Hua, X.-S. (2023) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3532624 · Cited by 241 · see citing works
Explainability of Deep Vision-Based Autonomous Driving Systems: Review and Challenges
Zablocki, É., Ben-Younes, H., Pérez, P., & Cord, M. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01657-x · Cited by 227 · see citing works
Papadimitroulas, P., Brocki, L., Christopher Chung, N., Marchadour, W., Vermet, F., Gaubert, L., Eleftheriadis, V., Plachouris, D., Visvikis, D., Kagadis, G. C., et al. (2021) · Physica Medica · doi:10.1016/j.ejmp.2021.03.009 · Cited by 226 · see citing works
Zhang, Y., Hong, D., McClement, D. G., Oladosu, O., Pridham, G., & Slaney, G. (2021) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2021.109098 · Cited by 224 · see citing works
Mental Health Prediction Using Machine Learning: Taxonomy, Applications, and Challenges
Chung, J., & Teo, J. (2022) · Applied Computational Intelligence and Soft Computing · doi:10.1155/2022/9970363 · Cited by 223 · see citing works
Analyzing Classifiers: Fisher Vectors and Deep Neural Networks
Lapuschkin, S., Binder, A., Montavon, G., Müller, K., & Samek, W. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.318 · Cited by 212 · see citing works
Self-citationAhmad, O. F., Soares, A. S., Mazomenos, E. B., Brandão, P., Vega, R., Seward, E. W., Stoyanov, D. V., Chand, M., & Lovat, L. B. (2018) · The Lancet. Gastroenterology & hepatology · doi:10.1016/s2468-1253(18)30282-6 · Cited by 215 · see citing works
Towards A Rigorous Evaluation Of XAI Methods On Time Series
Schlegel, U., Arnout, H., El‐Assady, M., Oelke, D., & Keim, D. A. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00516 · Cited by 218 · see citing works
Perturbation-based methods for explaining deep neural networks: A survey
Ivanovs, M., Kadiķis, R., & Ozols, K. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.06.030 · Cited by 207 · see citing works
Noorbakhsh, J., Farahmand, S., Foroughi pour, A., Namburi, S., Caruana, D., Rimm, D., Soltanieh-ha, M., Zarringhalam, K., & Chuang, J. H. (2020) · Nature Communications · doi:10.1038/s41467-020-20030-5 · Cited by 206 · see citing works
Evaluating Weakly Supervised Object Localization Methods Right
Choe, J., Oh, S. J., Lee, S., Chun, S., Akata, Z., & Shim, H. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00320 · Cited by 204 · see citing works
Guidelines and evaluation of clinical explainable AI in medical image analysis
Jin, W., Li, X., Fatehi, M., & Hamarneh, G. (2023) · Medical Image Analysis · doi:10.1016/j.media.2022.102684 · Cited by 199 · see citing works
Transparency: Motivations and Challenges
Weller, A. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28954-6_2 · Cited by 198 · see citing works
A deep learning framework for identifying children with ADHD using an EEG-based brain network
Chen, H., Song, Y., & Xiaoli Li, A. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.04.058 · Cited by 189 · see citing works
Zihni, E., Madai, V. I., Livne, M., Galinovic, I., Khalil, A. A., Fiebach, J. B., & Frey, D. (2020) · PLOS ONE · doi:10.1371/journal.pone.0231166 · Cited by 187 · see citing works
CLEVR-XAI: A benchmark dataset for the ground truth evaluation of neural network explanations
Arras, L., Osman, A., & Samek, W. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.11.008 · Cited by 189 · see citing works
Self-citationComparing Automatic and Human Evaluation of Local Explanations for Text Classification
Nguyen, D. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1097 · Cited by 175 · see citing works
Interpretability and fairness evaluation of deep learning models on MIMIC-IV dataset
Meng, C., Trinh, L., Xu, N., Enouen, J., & Liu, Y. (2022) · Scientific Reports · doi:10.1038/s41598-022-11012-2 · Cited by 199 · see citing works
Interpretation and visualization techniques for deep learning models in medical imaging
Huff, D. T., Weisman, A. J., & Jeraj, R. (2021) · Physics in Medicine & Biology · doi:10.1088/1361-6560/abcd17 · Cited by 172 · see citing works
Eitel, F., Soehler, E., Bellmann-Strobl, J., Brandt, A. U., Ruprecht, K., Giess, R. M., Kuchling, J., Asseyer, S., Weygandt, M., Haynes, J.-D., et al. (2019) · NeuroImage: Clinical · doi:10.1016/j.nicl.2019.102003 · Cited by 171 · see citing works
Layer-Wise Relevance Propagation for Deep Neural Network Architectures
Binder, A., Lapuschkin, S., Montavon, G., Müller, K., & Samek, W. (2016) · Lecture notes in electrical engineering · doi:10.1007/978-981-10-0557-2_87 · Cited by 167 · see citing works
Self-citationNeuralCD: A General Framework for Cognitive Diagnosis
Wang, F., Liu, Q., Chen, E., Huang, Z., Yin, Y., Wang, S., & Su, Y. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3201037 · Cited by 168 · see citing works
Explainable Deep Learning Methods in Medical Image Classification: A Survey
Patrício, C., Neves, J. C., & Teixeira, L. F. (2023) · ACM Computing Surveys · doi:10.1145/3625287 · Cited by 190 · see citing works
Weakly-supervised localization of diabetic retinopathy lesions in retinal fundus images
Gondal, W. M., Köhler, J. M., Grzeszick, R., Fink, G. A., & Hirsch, M. (2017) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2017.8296646 · Cited by 159 · see citing works
Kakogeorgiou, I., & Karantzalos, K. G. (2021) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2021.102520 · Cited by 163 · see citing works
Towards Best Practice in Explaining Neural Network Decisions with LRP
Kohlbrenner, M., Bauer, A., Nakajima, S., Binder, A., Samek, W., & Lapuschkin, S. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206975 · Cited by 159 · see citing works
Self-citationUnderstanding autoencoders with information theoretic concepts
Yu, S., & Príncipe, J. C. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.05.003 · Cited by 155 · see citing works
Black-box Explanation of Object Detectors via Saliency Maps
Petsiuk, V., Jain, R., Manjunatha, V., Morariu, V. I., Mehra, A., Ordóñez, V., & Saenko, K. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01128 · Cited by 156 · see citing works
Enhancing discrete choice models with representation learning
Sifringer, B., Lurkin, V., & Alahi, A. (2020) · Transportation Research Part B Methodological · doi:10.1016/j.trb.2020.08.006 · Cited by 155 · see citing works
Grezmak, J. E., Zhang, J., Wang, P. E., Loparo, K. A., & Gao, R. X. (2019) · IEEE Sensors Journal · doi:10.1109/jsen.2019.2958787 · Cited by 145 · see citing works
Bhati, D., Neha, F., & Amiruzzaman, M. (2024) · Journal of Imaging · doi:10.3390/jimaging10100239 · Cited by 166 · see citing works
Yang, Z., Zhang, A., & Sudjianto, A. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108192 · Cited by 144 · see citing works
On Completeness-aware Concept-Based Explanations in Deep Neural NetworksPreprint
Yeh, C., Kim, B., Arık, S. Ö., Li, C., Pfister, T., & Ravikumar, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.07969 · Cited by 144 · see citing works
- Authors
- Samek, W., Binder, A., Montavon, G., Lapuschkin, S., & Muller, K.-R.
- Affiliations
- Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (DE)
- Singapore University of Technology and Design (SG)
- Technische Universität Berlin, Berlin, Germany
- Venue
- IEEE Transactions on Neural Networks and Learning Systems
- Published
- 2017-11
- Volume
- 28
- Issue
- 11
- Pages
- 2660-2673
- ISSN
- 2162-237X (ISSN-L)
- 2162-2388
- OpenAlex
- W2240067561
- Cited by
- 1,164
- References
- 51
- Citations by year
- 2026: 120 · 2025: 105 · 2024: 139 · 2023: 163 · 2022: 147 · 2021: 171 · 2020: 123 · 2019: 92 · 2018: 66 · 2017: 26 · 2016: 11 · 2015: 1
- FWCI
- 53.43
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Neural Networks and Learning Systems (publishedVersion)
- PubMed (publishedVersion)
- Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion)
- Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
- Funders
- Brain Korea 21 Plus Program through the National Research Foundation of Korea Funded by the Ministry of Education
- DFG (MU 987/17-1)
- German Ministry for Education and Research as Berlin Big Data Center (01IS14013A)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- deep neural network interpretability, layer-wise relevance propagation, deconvolution, explainable AI, image classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
