The DOI matches its Crossref record; 47 of the 51 references it cites were found in a registry or an index.
Evaluating the Visualization of What a Deep Neural Network Has Learned
Abstract
Deep neural networks (DNNs) have demonstrated impressive performance in complex machine learning tasks such as image classification or speech recognition. However, due to their multilayer nonlinear structure, they are not transparent, i.e., it is hard to grasp what makes them arrive at a particular classification or recognition decision, given a new unseen data sample. Recently, several approaches have been proposed enabling one to understand and interpret the reasoning embodied in a DNN for a single test image. These methods quantify the “importance” of individual pixels with respect to the classification decision and allow a visualization in terms of a heatmap in pixel/input space. While the usefulness of heatmaps can be judged subjectively by a human, an objective quality measure is missing. In this paper, we present a general methodology based on region perturbation for evaluating ordered collections of pixels such as heatmaps. We compare heatmaps computed by three different methods on the SUN397, ILSVRC2012, and MIT Places data sets. Our main result is that the recently proposed layer-wise relevance propagation algorithm qualitatively and quantitatively provides a better explanation of what made a DNN arrive at a particular classification decision than the sensitivity-based approach or the deconvolution method. We provide theoretical arguments to explain this result and discuss its practical implications. Finally, we investigate the use of heatmaps for unsupervised assessment of the neural network performance.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,889
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381
An introduction to variable and feature selection
GuyonIsabelle, & ElisseeffAndré (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944968 · Cited by 8,354
Domain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143
Research works printed but not linked: 1
- jia, 2013, Caffe An Open Source Convolutional Architecture for Fast Feature Embedding
Cited by
Sorted by citations
Methods for interpreting and understanding deep neural networks
Montavon, G., Samek, W., & Müller, K.-R. (2018) · Digital Signal Processing · doi:10.1016/j.dsp.2017.10.011 · Cited by 2,665
Self-citationA Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524
Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence
Hassija, V., Chamola, V., Mahapatra, A., Singal, A., Goel, D., Huang, K., Scardapane, S., Spinelli, I., Mahmud, M., & Hussain, A. (2024) · Cognitive Computation · doi:10.1007/s12559-023-10179-8 · Cited by 2,495
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
Shin, D. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2020.102551 · Cited by 1,548
Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3060483 · Cited by 1,477
Self-citationExplaining nonlinear classification decisions with deep taylor decomposition
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K.-R. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.11.008 · Cited by 1,357
Self-citationShen, C. (2018) · Water Resources Research · doi:10.1029/2018wr022643 · Cited by 1,522
Sanity Checks for Saliency Maps
Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018) · arXiv · doi:10.48550/arxiv.1810.03292 · Cited by 1,294
Trojaning Attack on Neural Networks
Liu, Y., Ma, S., Aafer, Y., Lee, W.-C., Zhai, J., Wang, W., & Zhang, X. (2018) · Proceedings 2018 Network and Distributed System Security Symposium · doi:10.14722/ndss.2018.23291 · Cited by 1,270
Samek, W., Wiegand, T., & Müller, K.-R. (2017) · arXiv · doi:10.48550/arxiv.1708.08296 · Cited by 1,080
Self-citationMachine Learning Methods for Histopathological Image Analysis
Komura, D., & Ishikawa, S. (2018) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2018.01.001 · Cited by 937
A Survey on Neural Network Interpretability
Zhang, Y., Tino, P., Leonardis, A., & Tang, K. (2021) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2021.3100641 · Cited by 923
Layer-Wise Relevance Propagation: An Overview
Montavon, G., Binder, A., Lapuschkin, S., Samek, W., & Müller, K. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_10 · Cited by 883
Self-citationMarkus, A. F., Kors, J. A., & Rijnbeek, P. R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103655 · Cited by 811
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2021) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3387166 · Cited by 785
'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions
Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173951 · Cited by 691
Towards Explainable Artificial Intelligence
Samek, W., & Müller, K. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_1 · Cited by 704
Self-citationYuan, X., Huang, B., Wang, Y., Yang, C., & Gui, W. (2018) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2018.2809730 · Cited by 638
The (Un)reliability of Saliency Methods
Kindermans, P.-J., Hooker, S., Adebayo, J., Alber, M., Schütt, K. T., Dähne, S., Erhan, D., & Kim, B. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_14 · Cited by 741
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629
“In [103] it is presented a comparison between explanations provided with LRP and more traditional deconvolution and sensitivity analysis.”
MethodNotions of explainability and evaluation approaches for explainable artificial intelligence
Vilone, G., & Longo, L. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.009 · Cited by 598
Nauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023) · ACM Computing Surveys · doi:10.1145/3583558 · Cited by 590
Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers
Hohman, F., Kahng, M., Pienta, R., & Chau, D. H. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2843369 · Cited by 582
Unmasking Clever Hans predictors and assessing what machines really learn
Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., & Müller, K. (2019) · RePEc: Research Papers in Economics · Cited by 580
Self-citationERASER: A Benchmark to Evaluate Rationalized NLP Models
DeYoung, J., Jain, S., Rajani, N. F., Lehman, E., Xiong, C., Socher, R., & Wallace, B. C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.408 · Cited by 561
Explainability Methods for Graph Convolutional Neural Networks
Pope, P. E., Kolouri, S., Rostami, M., Martin, C. E., & Hoffmann, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01103 · Cited by 553
Deep image mining for diabetic retinopathy screening
Quellec, G., Charrière, K., Boudi, Y., Cochener, B., & Lamard, M. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.04.012 · Cited by 537
Explainable machine learning in materials science
Zhong, X., Gallagher, B. J., Liu, S., Kailkhura, B., Hiszpanski, A. M., & Han, T. Y.-J. (2022) · npj Computational Materials · doi:10.1038/s41524-022-00884-7 · Cited by 566
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond
Li, X., Xiong, H., Li, X., Wu, X., Zhang, X., Liu, J., Bian, J., & Dou, D. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01756-8 · Cited by 579
Salahuddin, Z., Woodruff, H. C., Chatterjee, A., & Lambin, P. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.105111 · Cited by 508
Explaining anomalies detected by autoencoders using Shapley Additive Explanations
Antwarg, L., Miller, R. M., Shapira, B., & Rokach, L. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115736 · Cited by 496
Towards Robust Interpretability with Self-Explaining Neural NetworksPreprint
Alvarez-Melis, D., & Jaakkola, T. S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07538 · Cited by 591
Interpretability of deep learning models: A survey of results
Chakraborty, S., Tomsett, R., Raghavendra, R., Harborne, D., Alzantot, M., Cerutti, F., Srivastava, M. B., Preece, A. D., Julier, S., Rao, R., et al. (2017) · IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI) · doi:10.1109/uic-atc.2017.8397411 · Cited by 502
Explainable Deep Learning: A Field Guide for the Uninitiated
Ras, G., Xie, N., van Gerven, M. A. J., & Doran, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13200 · Cited by 551
Deep learning in microscopy image analysis: a survey
Xing, F., Xie, Y., Su, H., Liu, F., & Yang, L. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2766168 · Cited by 460
Layer-Wise Relevance Propagation for Neural Networks with Local Renormalization Layers
Binder, A., Montavon, G., Lapuschkin, S., Müller, K., & Samek, W. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-44781-0_8 · Cited by 512
Self-citationLauritsen, S. M., Kristensen, M., Olsen, M. V., Larsen, M. S., Lauritsen, K. M., Jørgensen, M. J., Lange, J., & Thiesson, B. (2020) · Nature Communications · doi:10.1038/s41467-020-17431-x · Cited by 448
A Benchmark for Interpretability Methods in Deep Neural NetworksPreprint
Hooker, S., Erhan, D., Kindermans, P.-J., & Kim, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.10758 · Cited by 531
Deep Hyperspectral Image Sharpening
Dian, R., Shutao Li, A., Guo, A., & Fang, L. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2798162 · Cited by 414
Analysis of Explainers of Black Box Deep Neural Networks for Computer Vision: A Survey
Buhrmester, V., Münch, D., & Arens, M. (2021) · Machine Learning and Knowledge Extraction · doi:10.3390/make3040048 · Cited by 403
Deep learning in bioinformatics: Introduction, application, and perspective in the big data era
Li, Y., Huang, C., Ding, L., Li, Z., Pan, Y., & Gao, X. (2019) · Methods · doi:10.1016/j.ymeth.2019.04.008 · Cited by 401
Obtaining genetics insights from deep learning via explainable artificial intelligence
Novakovsky, G., Dexter, N., Libbrecht, M. W., Wasserman, W. W., & Mostafavi, S. (2022) · Nature Reviews Genetics · doi:10.1038/s41576-022-00532-2 · Cited by 415
Interpretable deep neural networks for single-trial EEG classification
Sturm, I., Lapuschkin, S., Samek, W., & Müller, K. (2016) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2016.10.008 · Cited by 396
Self-citationDeep learning in mental health outcome research: a scoping review
Su, C., Xu, Z., Pathak, J., & Wang, F. (2020) · Translational Psychiatry · doi:10.1038/s41398-020-0780-3 · Cited by 368
Explaining Recurrent Neural Network Predictions in Sentiment Analysis
Arras, L., Montavon, G., Müller, K.-R., & Samek, W. (2017) · Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis · doi:10.18653/v1/w17-5221 · Cited by 361
Self-citationSpatial Pyramid-Enhanced NetVLAD With Weighted Triplet Loss for Place Recognition
Yu, J., Zhu, C., Zhang, J., Huang, Q., & Tao, D. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2908982 · Cited by 355
Digital Pathology: Advantages, Limitations and Emerging Perspectives
Jahn, S. W., Plass, M., & Moinfar, F. (2020) · Journal of Clinical Medicine · doi:10.3390/jcm9113697 · Cited by 372
HESS Opinions: Incubating deep-learning-powered hydrologic science advances as a community
Shen, C., Laloy, E., ELSHORBAGY, A. A., Albert, A., Bales, J. D., Chang, F., Ganguly, S., Hsu, K., Kifer, D., Fang, Z. N., et al. (2018) · Hydrology and earth system sciences · doi:10.5194/hess-22-5639-2018 · Cited by 353
Chen, C., & Gu, G. X. (2020) · Advanced Science · doi:10.1002/advs.201902607 · Cited by 339
Neural Cognitive Diagnosis for Intelligent Education Systems
Wang, F., Liu, Q., Chen, E., Huang, Z., Chen, Y., Yin, Y., Huang, Z., & Wang, S. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6080 · Cited by 348
Islam, M. R., Ahmed, M. U., Barua, S., & Begum, S. (2022) · Applied Sciences · doi:10.3390/app12031353 · Cited by 366
Deep learning algorithm predicts diabetic retinopathy progression in individual patients
Arcadu, F., Benmansour, F., Maunz, A., Willis, J., Haskova, Z., & Prunotto, M. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0172-3 · Cited by 305
Learning how to explain neural networks: PatternNet and PatternAttributionPreprint
Kindermans, P.-J., Schütt, K. T., Alber, M., Müller, K. R., Erhan, D., Kim, B., & Dähne, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.05598 · Cited by 359
Böhle, M., Eitel, F., Weygandt, M., & Ritter, K. (2019) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2019.00194 · Cited by 291
A comprehensive review of deep learning in colon cancer
Paçal, İ., Karaboğa, D., Baştürk, A., Akay, B., & Nalbantoglu, U. (2020) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.104003 · Cited by 300
Skin lesion segmentation via generative adversarial networks with dual discriminators
Lei, B., Xia, Z., Jiang, F., Jiang, X., Ge, Z., Xu, Y., Qin, J., Chen, S., Wang, T., & Wang, S. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101716 · Cited by 287
On the (In)fidelity and Sensitivity for ExplanationsPreprint
Yeh, C., Hsieh, C.-Y., Suggala, A. S., Inouye, D. I., & Ravikumar, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.09392 · Cited by 321
Explaining the unique nature of individual gait patterns with deep learning
Horst, F., Lapuschkin, S., Samek, W., Müller, K.-R., & Schöllhorn, W. I. (2019) · Scientific Reports · doi:10.1038/s41598-019-38748-8 · Cited by 295
Self-citationExplainable Artificial Intelligence: a Systematic Review
Vilone, G., & Longo, L. (2020) · arXiv · doi:10.48550/arxiv.2006.00093 · Cited by 269
Pruning by explaining: A novel criterion for deep neural network pruning
Yeom, S.-K., Seegerer, P., Lapuschkin, S., Binder, A., Wiedemann, S. M., Müller, K., & Samek, W. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.107899 · Cited by 277
Self-citation"What is relevant in a text document?": An interpretable machine learning approach
Arras, L., Horn, F., Montavon, G., Müller, K., & Samek, W. (2017) · PLoS ONE · doi:10.1371/journal.pone.0181142 · Cited by 286
Self-citationDCNNBiLSTM: An Efficient Hybrid Deep Learning-Based Intrusion Detection System
Hnamte, V., & Hussain, J. (2023) · Telematics and Informatics Reports · doi:10.1016/j.teler.2023.100053 · Cited by 309
Gradient-Based Attribution Methods
Ancona, M., Ceolini, E., Öztireli, C., & Groß, M. H. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_9 · Cited by 284
Februus: Input Purification Defense Against Trojan Attacks on Deep Neural Network Systems
Doan, B. G., Abbasnejad, E., & Ranasinghe, D. C. (2020) · Annual Computer Security Applications Conference · doi:10.1145/3427228.3427264 · Cited by 263
Four principles of explainable artificial intelligence
Phillips, P. J., Hahn, C. A., Fontana, P. C., Yates, A. N., Greene, K., Broniatowski, D. A., & Przybocki, M. A. (2021) · doi:10.6028/nist.ir.8312 · Cited by 262
Higher-Order Explanations of Graph Neural Networks via Relevant Walks
Schnake, T., Eberle, O., Lederer, J., Nakajima, S., Schütt, K. T., Müller, K., & Montavon, G. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3115452 · Cited by 322
Self-citationA Survey on Deep Hashing Methods
Luo, X., Wang, H., Wu, D., Chen, C., Deng, M., Huang, J., & Hua, X.-S. (2023) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3532624 · Cited by 241
Explainability of Deep Vision-Based Autonomous Driving Systems: Review and Challenges
Zablocki, É., Ben-Younes, H., Pérez, P., & Cord, M. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01657-x · Cited by 227
Papadimitroulas, P., Brocki, L., Christopher Chung, N., Marchadour, W., Vermet, F., Gaubert, L., Eleftheriadis, V., Plachouris, D., Visvikis, D., Kagadis, G. C., et al. (2021) · Physica Medica · doi:10.1016/j.ejmp.2021.03.009 · Cited by 226
Zhang, Y., Hong, D., McClement, D. G., Oladosu, O., Pridham, G., & Slaney, G. (2021) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2021.109098 · Cited by 236
Mental Health Prediction Using Machine Learning: Taxonomy, Applications, and Challenges
Chung, J., & Teo, J. (2022) · Applied Computational Intelligence and Soft Computing · doi:10.1155/2022/9970363 · Cited by 256
Analyzing Classifiers: Fisher Vectors and Deep Neural Networks
Lapuschkin, S., Binder, A., Montavon, G., Müller, K., & Samek, W. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.318 · Cited by 226
Self-citationAhmad, O. F., Soares, A. S., Mazomenos, E. B., Brandão, P., Vega, R., Seward, E. W., Stoyanov, D. V., Chand, M., & Lovat, L. B. (2018) · The Lancet. Gastroenterology & hepatology · doi:10.1016/s2468-1253(18)30282-6 · Cited by 213
Towards A Rigorous Evaluation Of XAI Methods On Time Series
Schlegel, U., Arnout, H., El‐Assady, M., Oelke, D., & Keim, D. A. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00516 · Cited by 243
Perturbation-based methods for explaining deep neural networks: A survey
Ivanovs, M., Kadiķis, R., & Ozols, K. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.06.030 · Cited by 235
Noorbakhsh, J., Farahmand, S., Foroughi pour, A., Namburi, S., Caruana, D., Rimm, D., Soltanieh-ha, M., Zarringhalam, K., & Chuang, J. H. (2020) · Nature Communications · doi:10.1038/s41467-020-20030-5 · Cited by 206
Evaluating Weakly Supervised Object Localization Methods Right
Choe, J., Oh, S. J., Lee, S., Chun, S., Akata, Z., & Shim, H. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00320 · Cited by 215
Guidelines and evaluation of clinical explainable AI in medical image analysis
Jin, W., Li, X., Fatehi, M., & Hamarneh, G. (2023) · Medical Image Analysis · doi:10.1016/j.media.2022.102684 · Cited by 199
Transparency: Motivations and Challenges
Weller, A. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28954-6_2 · Cited by 198
A deep learning framework for identifying children with ADHD using an EEG-based brain network
Chen, H., Song, Y., & Li, X. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.04.058 · Cited by 192
Zihni, E., Madai, V. I., Livne, M., Galinovic, I., Khalil, A. A., Fiebach, J. B., & Frey, D. (2020) · PLOS ONE · doi:10.1371/journal.pone.0231166 · Cited by 187
CLEVR-XAI: A benchmark dataset for the ground truth evaluation of neural network explanations
Arras, L., Osman, A., & Samek, W. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.11.008 · Cited by 226
Self-citationComparing Automatic and Human Evaluation of Local Explanations for Text Classification
Nguyen, D. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1097 · Cited by 175
Interpretability and fairness evaluation of deep learning models on MIMIC-IV dataset
Meng, C., Trinh, L., Xu, N., Enouen, J., & Liu, Y. (2022) · Scientific Reports · doi:10.1038/s41598-022-11012-2 · Cited by 213
Interpretation and visualization techniques for deep learning models in medical imaging
Huff, D. T., Weisman, A. J., & Jeraj, R. (2021) · Physics in Medicine & Biology · doi:10.1088/1361-6560/abcd17 · Cited by 172
Eitel, F., Soehler, E., Bellmann-Strobl, J., Brandt, A. U., Ruprecht, K., Giess, R. M., Kuchling, J., Asseyer, S., Weygandt, M., Haynes, J.-D., et al. (2019) · NeuroImage: Clinical · doi:10.1016/j.nicl.2019.102003 · Cited by 171
Layer-Wise Relevance Propagation for Deep Neural Network Architectures
Binder, A., Lapuschkin, S., Montavon, G., Müller, K., & Samek, W. (2016) · Lecture notes in electrical engineering · doi:10.1007/978-981-10-0557-2_87 · Cited by 169
Self-citationNeuralCD: A General Framework for Cognitive Diagnosis
Wang, F., Liu, Q., Chen, E., Huang, Z., Yin, Y., Wang, S., & Su, Y. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3201037 · Cited by 182
Explainable Deep Learning Methods in Medical Image Classification: A Survey
Patrício, C., Neves, J. C., & Teixeira, L. F. (2023) · ACM Computing Surveys · doi:10.1145/3625287 · Cited by 207
Weakly-supervised localization of diabetic retinopathy lesions in retinal fundus images
Gondal, W. M., Köhler, J. M., Grzeszick, R., Fink, G. A., & Hirsch, M. (2017) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2017.8296646 · Cited by 159
Kakogeorgiou, I., & Karantzalos, K. G. (2021) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2021.102520 · Cited by 176
Towards Best Practice in Explaining Neural Network Decisions with LRP
Kohlbrenner, M., Bauer, A., Nakajima, S., Binder, A., Samek, W., & Lapuschkin, S. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206975 · Cited by 177
Self-citationUnderstanding autoencoders with information theoretic concepts
Yu, S., & Príncipe, J. C. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.05.003 · Cited by 162
Black-box Explanation of Object Detectors via Saliency Maps
Petsiuk, V., Jain, R., Manjunatha, V., Morariu, V. I., Mehra, A., Ordóñez, V., & Saenko, K. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01128 · Cited by 170
Enhancing discrete choice models with representation learning
Sifringer, B., Lurkin, V., & Alahi, A. (2020) · Transportation Research Part B Methodological · doi:10.1016/j.trb.2020.08.006 · Cited by 180
Grezmak, J. E., Zhang, J., Wang, P. E., Loparo, K. A., & Gao, R. X. (2019) · IEEE Sensors Journal · doi:10.1109/jsen.2019.2958787 · Cited by 151
Bhati, D., Neha, F., & Amiruzzaman, M. (2024) · Journal of Imaging · doi:10.3390/jimaging10100239 · Cited by 177
Yang, Z., Zhang, A., & Sudjianto, A. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108192 · Cited by 144
Metadata
- Authors
- Samek, W., Binder, A., Montavon, G., Lapuschkin, S., & Muller, K.-R.
- Affiliations
- Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (DE)
- Singapore University of Technology and Design (SG)
- Technische Universität Berlin, Berlin, Germany
- Venue
- IEEE Transactions on Neural Networks and Learning Systems
- Published
- 2017-11
- Volume
- 28
- Issue
- 11
- Pages
- 2660-2673
- ISSN
- 2162-237X (ISSN-L)
- 2162-2388
- OpenAlex
- W2240067561
- Cited by
- 1,164
- References
- 51
- Citations by year
- 2026: 120 · 2025: 105 · 2024: 139 · 2023: 163 · 2022: 147 · 2021: 171 · 2020: 123 · 2019: 92 · 2018: 66 · 2017: 26
- FWCI
- 53.43
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Neural Networks and Learning Systems (publishedVersion)
- PubMed (publishedVersion)
- Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion)
- Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
- Funders
- Brain Korea 21 Plus Program through the National Research Foundation of Korea Funded by the Ministry of Education
- DFG (MU 987/17-1)
- German Ministry for Education and Research as Berlin Big Data Center (01IS14013A)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- deep neural network interpretability, layer-wise relevance propagation, deconvolution, explainable AI, image classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
