Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Open accessThe DOI matches its Crossref record; 224 of the 230 references it cites were found in a registry or an index.
Abstract
With the broader and highly successful usage of machine learning (ML) in industry and the sciences, there has been a growing demand for explainable artificial intelligence (XAI). Interpretability and explanation methods for gaining a better understanding of the problem-solving abilities and strategies of nonlinear ML, in particular, deep neural networks, are, therefore, receiving increased attention. In this work, we aim to: 1) provide a timely overview of this active emerging field, with a focus on “post hoc” explanations, and explain its theoretical foundations; 2) put interpretability algorithms to a test both from a theory and comparative evaluation perspective using extensive simulations; 3) outline best practice aspects, i.e., how to best include interpretation methods into the standard usage of ML; and 4) demonstrate successful usage of XAI in a representative selection of application scenarios. Finally, we discuss challenges and possible future directions of this exciting foundational field of ML.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“While XAI techniques exist [47], their application in complex, real-time energy systems, and providing meaningful explanations to non-experts, remain challenging”
From Addressing Challenges for the Effective Adoption of Artificial Intelligence in the Energy Sector (2025) · cites this paper as [47]
In the citing paper
“Despite previous attempts to address the challenges surrounding model interpretability, most solutions have been implemented post hoc (such as the attention mechanism) focusing on explanations in certain domains, such as vision [184].”
From Human-centred artificial intelligence for mobile health sensing: challenges and opportunities (2023) · cites this paper as [184]
In the citing paper
“LIME produces an explanation list that shows the contribution of individual features to the prediction.”
From Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2022) · cites this paper as [64]
In the citing paper
“This local explanation allows the end-user to determine which feature is important for the precise prediction and how the perturbation in feature affects the prediction results.”
From Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2022) · cites this paper as [64]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Items 201–224 of 224
Gradient-Based Vs. Propagation-Based Explanations: An Axiomatic Comparison
Montavon, G. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_13 · Cited by 48
Self-citationQuantum-Chemical Insights from Interpretable Atomistic Neural Networks
Schütt, K. T., Gastegger, M., Tkatchenko, A., & Müller, K. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_17 · Cited by 47
Producing Radiologist-Quality Reports for Interpretable Deep Learning
Gale, W., Oakden-Rayner, L., Carneiro, G., Palmer, L. J., & Bradley, A. P. (2019) · 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019) · doi:10.1109/isbi.2019.8759236 · Cited by 44 · cited 1×
“Further applications of DNN explanation techniques for medical images include the detection of lesions in diabetic retinopathy data [145], pixelwise analysis of microscopy images from global image annotations [98], the validation of predictions in dermatology [199], and analysis of X-ray images [48].”
BackgroundExplaining Deep Neural Networks with a Polynomial Time Algorithm for Shapley Values Approximation
Ancona, M., Öztireli, C., & Groß, M. H. (2019) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000342737 · Cited by 42
A Rate-Distortion Framework for Explaining Neural Network DecisionsPreprint
Macdonald, J., Wäldchen, S., Hauch, S., & Gitta Kutyniok, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.11092 · Cited by 41 · cited 2×
“Other frameworks such as rate distortion theory [118] have also been proposed.”
BackgroundMethodLearning Global Pairwise Interactions with Bayesian Neural Networks
Cui, T., Marttinen, P., & Kaski, S. (2020) · Frontiers in artificial intelligence and applications · doi:10.3233/faia200205 · Cited by 41 · cited 1×
“While steps have already been taken in XAI to identify interacting variables [34], [42], [115], [162], [188], it will be necessary for a broader usage to generalize the approach, e.”
BackgroundComputationally Efficient Measures of Internal Neuron ImportancePreprint
Shrikumar, A., Su, J., & Kundaje, A. B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.09946 · Cited by 40
Understanding Patch-Based Learning of Video Data by Explaining Predictions
Anders, C. J., Montavon, G., Samek, W., & Müller, K. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_16 · Cited by 39 · cited 1×
“Furthermore, initial steps have been taken to use XAI beyond validation to arrive at better and more predictive models e.g. [152], [6], [8], [7].”
MethodSelf-citationMouse Authentication Without the Temporal Aspect – What Does a 2D-CNN Learn?
Chong, P., Tan, Y., Guarnizo, J., Elovici, Y., & Binder, A. (2018) · IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw.2018.00011 · Cited by 38 · cited 1×
“The work by [87] uses a gradient-based explanation technique to understand the activity prediction across chromosomes, whereas [31] uses the LRP explanation technique for un-”
MethodExplaining Explanations: Axiomatic Feature Interactions for Deep Networks
Janizek, J. D., Sturmfels, P., & Lee, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.04138 · Cited by 37 · cited 2×
“Other higher order methods include GNN-LRP that explains graph neural networks in terms of collections of edges [162] or integrated Hessians [81], which can be seen as the second-order extension of integrated gradients.”
MethodFully interpretable deep learning model of transcriptional control
Liu, Y., Barr, K., & Reinitz, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btaa506 · Cited by 35 · cited 1×
“Other works [114] showed that explanations in the gene analysis lead to interpretable patterns consistent with literature knowledge.”
ResultThe Taylor Decomposition: A Unified Generalization of the Oaxaca Method to Nonlinear ModelsPreprint
Stephen Bazen, A., & Joutard, X. (2013) · RePEc: Research Papers in Economics · Cited by 30
The Whole Is More Than Its Parts? From Explicit to Implicit Pose Normalization
Simon, M., Rodner, E., Darrell, T., & Denzler, J. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2885764 · Cited by 28 · cited 1×
“, different layers of the neural networks) to arrive at a more informative explanation of the prediction process [162], [173], [203].”
BackgroundThe Detection of Errors in Multivariate Data Using Principal Components
Hawkins, D. M. (1974) · Journal of the American Statistical Association · doi:10.2307/2285654 · Cited by 27
Learning Global Pairwise Interactions with Bayesian Neural NetworksPreprint
Cui, T., Marttinen, P., & Kaski, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.08361 · Cited by 18
Structuring Neural Networks for More Explainable Predictions
Rieger, L., Chormai, P., Montavon, G., Hansen, L. K., & Müller, K. (2018) · The Springer series on challenges in machine learning · doi:10.1007/978-3-319-98131-4_5 · Cited by 15 · cited 1×
“This can be explained by VGG-16 having a more explicit structure (standard pooling operations for VGG-16 versus strided convolution for ResNet-50), which better supports the process of relevance propagation (see also [149] for a discussion of the effect of structure on the performance of explanation methods).”
BackgroundSelf-citationOpening the Black Box: Revealing Interpretable Sequence Motifs in Kernel-Based Learning Algorithms
Vidovic, M. M.-C., Görnitz, N., Müller, K., Rätsch, G., & Kloft, M. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-23525-7_9 · Cited by 13 · cited 1×
“At the same time, these methods were able to handle different types of data, including images [13], speech [20], text [10], [38], and structured data, such as molecules [162], [165] or genetic sequences [191].”
MethodContextual Prediction Difference Analysis for Explaining Individual Image ClassificationsPreprint
Gu, J., & Volker Tresp, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.09086 · Cited by 9
Applying machine learning methods to detect convection using GOES-16 ABI dataPreprint
Lee, Y., Kummerow, C. D., & Ebert‐Uphoff, I. (2020) · doi:10.5194/amt-2020-420 · Cited by 8 · cited 1×
“[109] predict various meteorological values, such as convection, or MRMS radar, from satellite imagery, and the latter prediction is then attributed to the individual pixels.”
BackgroundContextual Prediction Difference Analysis.Preprint
Gu, J., & Volker Tresp, C. (2019) · arXiv (Cornell University) · Cited by 7
Don't Paint It Black: White-Box Explanations for Deep Learning in Computer Security.Preprint
Warnecke, A., Arp, D. J., Wressnegger, C., & Rieck, K. (2019) · arXiv (Cornell University) · Cited by 6
Interpretable Text-to-Image Synthesis with Hierarchical Semantic Layout Generation
Hong, S., Yang, D., Choi, J., & Lee, H. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_5 · Cited by 5 · cited 1×
“Others visualize the learned representations and try to make the model itself more interpretable [75].”
BackgroundApplying machine learning methods to detect convection using GOES-16 ABI data
(2021) · Atmospheric measurement techniques · doi:10.5194/amt-2020-420-rc2 · Cited by 4
Chen, T., Wang, Z., Li, L., Li, D., Li, Z., Chang, X., Bian, P., Liang, G., Wang, Q., & Xie, T. (2025) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.ecoop.2025.6
Research works printed but not linked: 13
- Arras, 2020, Ground truth evaluation of neural network explanations with CLEVR-XAI, arXiv:2003.07258 · arXiv
- 10.1146/annurev.neuro.26.041002.131047 · DOI · Paper page
- Chollet, 2015, Keras
- 10.1109/ICCV.2019.00304 · DOI
- Kokhlikyan, 2019, Pytorch Captum
- Lapuschkin, 2016, The layer-wise relevance propagation toolbox for artificial neural networks, J. Mach. Learn. Res., 17, 1
- 10.48550/arXiv.1312.4400 · DOI · Paper page
- 10.5555/2969033.2969153 · doi:10.5555/2969033.2969153
- 10.5555/3157382.3157477 · doi:10.5555/3157382.3157477
- Okuta, CuPy: A numpy-compatible library for NVIDIA GPU calculations, 151
- Oliphant, 2006, A Guide to NumPy, 1
- 10.48550/ARXIV.1404.7828 · DOI
- 10.5555/3045118.3045336 · doi:10.5555/3045118.3045336
Cited by
Combining Machine Learning and Computational Chemistry for Predictive Insights Into Chemical Systems
Keith, J. A., Vassilev-Galindo, V., Cheng, B., Chmiela, S., Gastegger, M., Müller, K.-R., & Tkatchenko, A. (2021) · Chemical Reviews · doi:10.1021/acs.chemrev.1c00107 · Cited by 849
Self-citationEdge artificial intelligence for 6g: vision, enabling technologies, and applications
Letaief, K. B., Shi, Y., Lu, J., & Lu, J. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3126076 · Cited by 829
“Specifically, the developed AI models and algorithms must be privacy-preserving, adversarial-resilient, robust, fair, optimal and interpretable [95].”
BackgroundExplainable AI (XAI): A systematic meta-survey of current challenges and future opportunities
Saeed, W., & Omlin, C. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110273 · Cited by 783
Albahri, A. S., Duhaim, A. M., Fadhel, M. A., Alnoor, A., Baqer, N. S., Alzubaidi, L., Albahri, O. S., Alamoodi, A. H., Bai, J., Salhi, A., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.03.008 · Cited by 716
Longo, L., Brcic, M., Cabitza, F., Choi, J., Confalonieri, R., Ser, J. D., Guidotti, R., Hayashi, Y., Herrera, F., Holzinger, A., et al. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102301 · Cited by 626
“A lot of work exists on explaining the decisions of a classifier with attribution methods [44].”
BackgroundSelf-citation
Metadata
- Authors
- Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R.
- Affiliations
- Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (DE)
- BIFOLD✔Berlin Institute for the Foundations of Learning and Data, Berlin, Germany
- Berlin Institute for the Foundations of Learning and Data (DE)
- ORCID
- Venue
- Proceedings of the IEEE
- Published
- 2021-03
- Volume
- 109
- Issue
- 3
- Pages
- 247-278
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W3132191748
- Cited by
- 1,477
- References
- 230
- Citations by year
- 2026: 305 · 2025: 343 · 2024: 326 · 2023: 273 · 2022: 152 · 2021: 74
- FWCI
- 72.25
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Proceedings of the IEEE (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion, other-oa)
- Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korea Government (2017-0-00451, 2019-0-00079)
- German Ministry for Education and Research (01IS14013A-E, 01GQ1115, 01GQ0850, 01IS18025A, 01IS18037A)
- German Research Foundation (Math+, EXC 2046/1, 390685689)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- explainable AI, interpretability, deep neural network, post-hoc explanation, interpretability evaluation, nonlinear machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.