Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Open accessThe DOI matches its Crossref record; 224 of the 230 references it cites were found in a registry or an index.
Abstract
With the broader and highly successful usage of machine learning (ML) in industry and the sciences, there has been a growing demand for explainable artificial intelligence (XAI). Interpretability and explanation methods for gaining a better understanding of the problem-solving abilities and strategies of nonlinear ML, in particular, deep neural networks, are, therefore, receiving increased attention. In this work, we aim to: 1) provide a timely overview of this active emerging field, with a focus on “post hoc” explanations, and explain its theoretical foundations; 2) put interpretability algorithms to a test both from a theory and comparative evaluation perspective using extensive simulations; 3) outline best practice aspects, i.e., how to best include interpretation methods into the standard usage of ML; and 4) demonstrate successful usage of XAI in a representative selection of application scenarios. Finally, we discuss challenges and possible future directions of this exciting foundational field of ML.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“While XAI techniques exist [47], their application in complex, real-time energy systems, and providing meaningful explanations to non-experts, remain challenging”
From Addressing Challenges for the Effective Adoption of Artificial Intelligence in the Energy Sector (2025) · cites this paper as [47]
In the citing paper
“Despite previous attempts to address the challenges surrounding model interpretability, most solutions have been implemented post hoc (such as the attention mechanism) focusing on explanations in certain domains, such as vision [184].”
From Human-centred artificial intelligence for mobile health sensing: challenges and opportunities (2023) · cites this paper as [184]
In the citing paper
“LIME produces an explanation list that shows the contribution of individual features to the prediction.”
From Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2022) · cites this paper as [64]
In the citing paper
“This local explanation allows the end-user to determine which feature is important for the precise prediction and how the perturbation in feature affects the prediction results.”
From Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2022) · cites this paper as [64]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 12×
“This modified gradient view on LRP can also be leveraged to achieve a simpler and more general implementation of LRP based on ‘forward hooks’, which we describe in the second part of Appendix B, and which we use to apply LRP on VGG-16 [175] and ResNet-50 [65] in Section IV.”
BackgroundMethodInfluentialVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 6×
“This modified gradient view on LRP can also be leveraged to achieve a simpler and more general implementation of LRP based on ‘forward hooks’, which we describe in the second part of Appendix B, and which we use to apply LRP on VGG-16 [175] and ResNet-50 [65] in Section IV.”
BackgroundMethodInfluentialImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 4×
“Consequently, we also exchange the neural network architecture and use an AlexNet [99] model, which is able to process the transformed input data using 2-D-convolution operators.”
BackgroundMethodInfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“, [23], [52], [70], [107], [108], and [161]) have gained highest popularity.”
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 9×
“A comparison of performance on the Adience benchmark of the original model (pretrained on ImageNet only) and the improved model is given in the table below.”
MethodInfluentialRegression Models and Life-Tables
Cox, D. R. (1972) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1972.tb00899.x · Cited by 39,793
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 8×
“A comparison of performance on the Adience benchmark of the original model (pretrained on ImageNet only) and the improved model is given in the table below.”
BackgroundInfluentialThe Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 1×
“In games such as the arcade game Atari Breakout, the computer player would progressively learn strategies commonly employed by human players, such as “tunneldigging” [124], [200].”
Background
Research works printed but not linked: 13
- Arras, 2020, Ground truth evaluation of neural network explanations with CLEVR-XAI, arXiv:2003.07258 · arXiv
- 10.1146/annurev.neuro.26.041002.131047 · DOI · Paper page
- Chollet, 2015, Keras
- 10.1109/ICCV.2019.00304 · DOI
- Kokhlikyan, 2019, Pytorch Captum
- Lapuschkin, 2016, The layer-wise relevance propagation toolbox for artificial neural networks, J. Mach. Learn. Res., 17, 1
- 10.48550/arXiv.1312.4400 · DOI · Paper page
- 10.5555/2969033.2969153 · doi:10.5555/2969033.2969153
- 10.5555/3157382.3157477 · doi:10.5555/3157382.3157477
- Okuta, CuPy: A numpy-compatible library for NVIDIA GPU calculations, 151
- Oliphant, 2006, A Guide to NumPy, 1
- 10.48550/ARXIV.1404.7828 · DOI
- 10.5555/3045118.3045336 · doi:10.5555/3045118.3045336
Cited by
Combining Machine Learning and Computational Chemistry for Predictive Insights Into Chemical Systems
Keith, J. A., Vassilev-Galindo, V., Cheng, B., Chmiela, S., Gastegger, M., Müller, K.-R., & Tkatchenko, A. (2021) · Chemical Reviews · doi:10.1021/acs.chemrev.1c00107 · Cited by 849
Self-citationEdge artificial intelligence for 6g: vision, enabling technologies, and applications
Letaief, K. B., Shi, Y., Lu, J., & Lu, J. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3126076 · Cited by 829
“Specifically, the developed AI models and algorithms must be privacy-preserving, adversarial-resilient, robust, fair, optimal and interpretable [95].”
BackgroundExplainable AI (XAI): A systematic meta-survey of current challenges and future opportunities
Saeed, W., & Omlin, C. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110273 · Cited by 783
Albahri, A. S., Duhaim, A. M., Fadhel, M. A., Alnoor, A., Baqer, N. S., Alzubaidi, L., Albahri, O. S., Alamoodi, A. H., Bai, J., Salhi, A., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.03.008 · Cited by 716
Longo, L., Brcic, M., Cabitza, F., Choi, J., Confalonieri, R., Ser, J. D., Guidotti, R., Hayashi, Y., Herrera, F., Holzinger, A., et al. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102301 · Cited by 626
“A lot of work exists on explaining the decisions of a classifier with attribution methods [44].”
BackgroundSelf-citationMachine Learning for Electrocatalyst and Photocatalyst Design and Discovery
Mai, H., Le, T. C., Chen, D., Winkler, D. A., & Caruso, R. A. (2022) · Chemical Reviews · doi:10.1021/acs.chemrev.2c00061 · Cited by 552
Quo vadis artificial intelligence?
Jiang, Y., Li, X., Luo, H., Yin, S., & Kaynak, O. (2022) · Discover Artificial Intelligence · doi:10.1007/s44163-022-00022-8 · Cited by 544
“Theoretical guarantee is needed in terms of why it works and what the target is [125–128].”
BackgroundInterpretable deep learning: interpretation, interpretability, trustworthiness, and beyond
Li, X., Xiong, H., Li, X., Wu, X., Zhang, X., Liu, J., Bian, J., & Dou, D. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01756-8 · Cited by 497
“From previous reviews and outlooks for the interpretations [29, 47, 81, 107, 144], we summarize the most important desiderata for the interpretation algorithms, i.”
MethodExplainable AI Methods - A Brief Overview
Holzinger, A., Saranti, A., Molnar, C., Biecek, P., & Samek, W. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-04083-2_2 · Cited by 379
“Therefore, LIME is also often referred to as surrogate-based explanation technique [70].”
BackgroundSelf-citationThe role of deep learning in urban water management: A critical review
Fu, G., Jin, Y., Sun, S., Yuan, Z., & Butler, D. (2022) · Water Research · doi:10.1016/j.watres.2022.118973 · Cited by 433
“Compared to conventional machine learning algorithms, developing explainable deep learning has unique difficulties arising from (Samek et al., 2021): 1) multi-scale and distributed nature of network representations, 2) instability from the high depth of networks, 3) searching a reference data point…”
BackgroundDeep learning for water quality
Zhi, W., Appling, A. P., Golden, H. E., Podgorski, J., & Li, L. (2024) · Nature Water · doi:10.1038/s44221-024-00202-z · Cited by 364
“XDL includes model-agnostic and model-specific approaches that identify and rank important features, relationships and mechanisms that contribute to model predictions 92 .”
Explainable, trustworthy, and ethical machine learning for healthcare: A survey
Rasheed, K., Qayyum, A., Ghaly, M., Al-Fuqaha, A., Razi, A., & Qadir, J. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.106043 · Cited by 372
A critical review of RNN and LSTM variants in hydrological time series predictions
Waqas, M., & Humphries, U. W. (2024) · MethodsX · doi:10.1016/j.mex.2024.102946 · Cited by 354
SpookyNet: Learning force fields with electronic degrees of freedom and nonlocal effects
Unke, O. T., Chmiela, S., Gastegger, M., Schütt, K. T., Sauceda, H. E., & Müller, K.-R. (2021) · Nature Communications · doi:10.1038/s41467-021-27504-0 · Cited by 344
“Obtaining such an understanding of how ML models, 79 here SpookyNet, solve a prediction problem is crucially important in the sciences as a low test set error 33 alone can not rule out that a model may overfit or for example capitalize on various artifacts in data 80 or show “Clever Hans” effects.”
BackgroundPruning by explaining: A novel criterion for deep neural network pruning
Yeom, S.-K., Seegerer, P., Lapuschkin, S., Binder, A., Wiedemann, S., Müller, K.-R., & Samek, W. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.107899 · Cited by 263
Self-citationHigher-Order Explanations of Graph Neural Networks via Relevant Walks
Schnake, T., Eberle, O., Lederer, J., Nakajima, S., Schutt, K. T., Muller, K.-R., & Montavon, G. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3115452 · Cited by 259
“For a more comprehensive set of related works, we refer the reader to the review papers [17], [18] for XAI and [2], [19] for GNNs.”
BackgroundSelf-citationDeep learning as a tool for ecology and evolution
Borowiec, M. L., Dikow, R. B., Frandsen, P. B., McKeeken, A., Valentini, G., & White, A. E. (2022) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13901 · Cited by 249
“Samek et al. (2021) describe 46 different methods for generating useful interpretations of neural networks (see Table 3 in Samek et al. (2021)).”
MethodZhao, S., Zhang, B., Yang, J., Zhou, J., & Xu, Y. (2024) · Nature Reviews Methods Primers · doi:10.1038/s43586-024-00346-y · Cited by 213
“A common combination is to use deep learning models such as convolutional NNs or recurrent NNs to extract features from data such as images or text 91 and then use LDA to reduce and classify the extracted features.”
Deep Reinforcement Learning for Smart Grid Operations: Algorithms, Applications, and Prospects
Li, Y., Yu, C., Shahidehpour, M., Yang, T., Zeng, Z., & Chai, T. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2023.3303358 · Cited by 203
“Recently, the rapid development of DL has aroused great interests in industry and academia [11], [12].”
Marey, A., Arjmand, P., Alerab, A. D. S., Eslami, M. J., Saad, A. M., Sanchez, N., & Umair, M. (2024) · Egyptian Journal of Radiology and Nuclear Medicine · doi:10.1186/s43055-024-01356-2 · Cited by 196
“This ambiguity can undermine healthcare practitioners’ confidence in relying on AI predictions, particularly when they conflict with conventional clinical judgment [40].”
BackgroundA survey on the interpretability of deep learning in medical diagnosis
Teng, Q., Liu, Z., Song, Y., Han, K., & Lu, Y. (2022) · Multimedia Systems · doi:10.1007/s00530-022-00960-4 · Cited by 198
“In contrast to the sensitivity analysis method, occlusion sensitivity [30, 39] is considered as another perturbationbased interpretability method, which measures the important features by occluding parts of the input.”
MethodGhasemi, A., Hashtarkhani, S., Schwartz, D. L., & Shaban‐Nejad, A. (2024) · Cancer Innovation · doi:10.1002/cai2.136 · Cited by 193
Recent Applications of Explainable AI (XAI): A Systematic Literature Review
Saarela, M., & Podgorelec, V. (2024) · Applied Sciences · doi:10.3390/app14198884 · Cited by 183
CLEVR-XAI: A benchmark dataset for the ground truth evaluation of neural network explanations
Arras, L., Osman, A., & Samek, W. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.11.008 · Cited by 179
“The field of explainable AI (XAI) [8, 9] has recently developed various techniques to uncover the decision making process of the model and help unmasking such Clever Hans(1) 10”
Self-citationFrom attribution maps to human-understandable explanations through Concept Relevance Propagation
Achtibat, R., Dreyer, M., Eisenbraun, I., Bosse, S., Wiegand, T., Samek, W., & Lapuschkin, S. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00711-8 · Cited by 181
“In addition, Supplementary Note 9 provides an example on how CRP can be leveraged to identify systematically learned biases in male versus female face classification and Supplementary Note 10 demonstrates the applicability of CRP to time-series data.”
BackgroundMethodInfluentialSelf-citationMachine learning for numerical weather and climate modelling: a review
de Burgh-Day, C. O., & Leeuwenburg, T. (2023) · Geoscientific Model Development · doi:10.5194/gmd-16-6433-2023 · Cited by 176
Nascita, A., Montieri, A., Aceto, G., Ciuonzo, D., Persico, V., & Pescapé, A. (2021) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2021.3098157 · Cited by 177
WavCapsNet: An Interpretable Intelligent Compound Fault Diagnosis Method by Backward Tracking
Li, W., Lan, H., Chen, J., Feng, K., & Huang, R. (2023) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2023.3282664 · Cited by 172
“required of interpretability [25], [26].”
Deep learning in food authenticity: Recent advances and future trends
Deng, Z., Wang, T., Zheng, Y., Zhang, W., & Yun, Y.-H. (2024) · Trends in Food Science & Technology · doi:10.1016/j.tifs.2024.104344 · Cited by 173
Naiseh, M., Al-Thani, D., Jiang, N., & Ali, R. (2023) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102941 · Cited by 170
Explainable artificial intelligence for cybersecurity: a literature survey
Charmet, F., Tanuwidjaja, H. C., Ayoubi, S., Gimenez, P.-F., Han, Y., Jmila, H., Blanc, G., Takahashi, T., & Zhang, Z. (2022) · Annals of Telecommunications · doi:10.1007/s12243-022-00926-7 · Cited by 167
“The XAI has been broadly surveyed in several studies [2, 29, 47, 105].”
BackgroundNanehkaran, Y. A., Chen, B., Cemiloglu, A., Chen, J., Anwar, S., Azarafza, M., & Derakhshani, R. (2023) · Water · doi:10.3390/w15152707 · Cited by 163
“The cost is frequently defined as a statistic whose value can only be approximated [71].”
Mind the gap: Performance metric evaluation in brain‐age prediction
de Lange, A. G., Anatürk, M., Rokicki, J., Han, L. K. M., Franke, K., Alnæs, D., Ebmeier, K. P., Draganski, B., Kaufmann, T., Westlye, L. T., et al. (2022) · Human Brain Mapping · doi:10.1002/hbm.25837 · Cited by 160
“…assessment can be used to identify the MRI measures that are most prominently used in the model (Salih et al., 2021; Amoroso et al., 2019; Samek, Montavon, Lapuschkin, Anders, & Müller, 2021; Vercio et al., 2020), and partial dependence plots (Friedman, 2001; Zhao & Hastie, 2021) can…”
Kakogeorgiou, I., & Karantzalos, K. (2021) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2021.102520 · Cited by 158
“…noise to the original input and averages the calculated attributions: 𝛷𝛷 𝑺𝑺𝑰𝑰 ( 𝑓𝑓 𝑐𝑐 , 𝑥𝑥 ) = 𝔼𝔼 𝜀𝜀 ~ 𝒩𝒩�0 , 𝜎𝜎 𝑰𝑰� [ 𝛷𝛷 ( 𝑓𝑓 𝑐𝑐 , 𝑥𝑥 + 𝜀𝜀 )] This approach can be understood as an averaging process that makes the initial explanation smoother (Samek et al., 2021).”
BackgroundMethodResultInfluentialCarbon emission prediction models: A review
Jin, Y., Sharifi, A., Li, Z., Chen, S., Zeng, S., & Zhao, S. (2024) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2024.172319 · Cited by 180
Adak, A., Pradhan, B., & Shukla, N. (2022) · Foods · doi:10.3390/foods11101500 · Cited by 194
“Deep neural networks lack interpretability, and the model features that drive the outcome are difficult to understand [17,52–57].”
BackgroundSpecies delimitation 4.0: integrative taxonomy meets artificial intelligence
Karbstein, K., Kösters, L. M., Hodač, L., Hofmann, M. R., Hörandl, E., Tomasello, S., Wagner, N. D., Emerson, B. C., Albach, D. C., Scheu, S., et al. (2024) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2023.11.002 · Cited by 177
“Although new explainable AI (XAI) approaches are emerging to visualize and detect biological features learned by the ML model [117], previous delimitation results from integrative taxonomy are needed, as well as experienced taxonomists to control and validate ML-based species classi fi cation or…”
BackgroundA critical review on diagnosis of diabetic retinopathy using machine learning and deep learning
Das, D., Biswas, S. K., & Bandyopadhyay, S. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-12642-4 · Cited by 180
“Therefore, eXplainable AI(XAI) [123] has been introduced to provide an explanation to complex, non-linear, accountable, and augmented behaviors of DL models and shallow ML models, to make them reliable, adoptable and predictable.”
BackgroundBeyond explaining: Opportunities and challenges of XAI-based model improvement
Weber, L., Lapuschkin, S., Binder, A., & Samek, W. (2023) · Information Fusion · doi:10.1016/j.inffus.2022.11.013 · Cited by 143
Self-citationArtificial intelligence-based methods for renewable power system operation
Li, Y., Ding, Y., He, S., Hu, F., Duan, J., Wen, G., Geng, H., Wu, Z., Gooi, H. B., Zhao, Y., et al. (2024) · Nature Reviews Electrical Engineering · doi:10.1038/s44287-024-00018-9 · Cited by 189
Toward Explainable Artificial Intelligence for Precision Pathology
Klauschen, F., Dippel, J., Keyl, P., Jurmeister, P., Bockmayr, M., Mock, A., Buchstab, O., Alber, M., Ruff, L., Montavon, G., et al. (2024) · Annual Review of Pathology: Mechanisms of Disease · doi:10.1146/annurev-pathmechdis-051222-113147 · Cited by 139
Self-citationEarly warning of complex climate risk with integrated artificial intelligence
Reichstein, M., Benson, V., Blunk, J., Camps‐Valls, G., Creutzig, F., Fearnley, C. J., Han, B., Kornhuber, K., Rahaman, N., Schölkopf, B., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-57640-w · Cited by 241
Alvarado, R. (2023) · Science and Engineering Ethics · doi:10.1007/s11948-023-00451-3 · Cited by 137
“Unlike other more conventional machine learning algorithms which work through the application of linear regression analysis, deep neural networks consist of several layers of non-linear regressions (Samek et al, 2021; Sarle, 1994).”
MethodXAIR: A Systematic Metareview of Explainable AI (XAI) Aligned to the Software Development Process
Clement, T., Kemmerzell, N., Abdelaal, M., & Amberg, M. (2023) · Machine Learning and Knowledge Extraction · doi:10.3390/make5010006 · Cited by 131
Artificial intelligence (AI)—it’s the end of the tox as we know it (and I feel fine)*
Kleinstreuer, N., & Härtung, T. (2024) · Archives of Toxicology · doi:10.1007/s00204-023-03666-2 · Cited by 176
“The field of explainable AI (xAI) aims to address these interpretability issues (Samek et al. 2021). xAI refers to methods for producing explainable models while also enabling human-understandable explanations to be generated for individual predictions.”
BackgroundToward Explainable Artificial Intelligence for Regression Models: A methodological perspective
Letzgus, S., Wagner, P., Lederer, J., Samek, W., Müller, K., & Montavon, G. (2022) · IEEE Signal Processing Magazine · doi:10.1109/msp.2022.3153277 · Cited by 166
“For more general recommendations on XAI we refer to [64].”
BackgroundMethodInfluentialSelf-citationLi, S., Li, T., Sun, C., Yan, R., & Chen, X. (2023) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2023.05.027 · Cited by 137
Vale, D., El-Sharif, A., & Ali, M. (2022) · AI and Ethics · doi:10.1007/s43681-022-00142-y · Cited by 124
“The instability of some post-hoc explainability methods such as LIME further calls into question their use.”
BackgroundInfluentialAyano, Y. M., Schwenker, F., Dufera, B. D., & Debelee, T. G. (2022) · Diagnostics · doi:10.3390/diagnostics13010111 · Cited by 168
“Having fewer noises around the target class and the capacity to show the part of a signal that negatively contributes to the output, LRP is superior over gradient-based explanation techniques [133,134].”
BackgroundToward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks
Räuker, T., Ho, A., Casper, S., & Hadfield-Menell, D. (2023) · 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00039 · Cited by 122
“ours [3], [65], [67], [75], [108], [134], [152], [175], [197]– [199], [236], [243], [246], [247].”
BackgroundMethodFinding and removing Clever Hans: Using explanation methods to debug and improve deep models
Anders, C. J., Weber, L., Neumann, D., Samek, W., Müller, K.-R., & Lapuschkin, S. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.07.015 · Cited by 119
“[14, 15] for an overview) in the emerging field of eXplainable Artificial Intelligence (XAI).”
BackgroundMethodSelf-citationDeep learning for power quality
de Oliveira, R. A., & Bollen, M. (2022) · Electric Power Systems Research · doi:10.1016/j.epsr.2022.108887 · Cited by 152
Guo, R., Liu, H., Xie, G., Zhang, Y. M., & Liu, D. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3181692 · Cited by 132
“Since interpretability is a domain-specific notion, there can be no all-purpose definition [12].”
BackgroundFrom Clustering to Cluster Explanations via Neural Networks
Kauffmann, J., Esders, M., Ruff, L., Montavon, G., Samek, W., & Müller, K. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3185901 · Cited by 162
“Such methods include, among others, integrated gradients (IGs) [29], or layerwise relevance propagation (LRP) [17], [22], [58].”
MethodInfluentialSelf-citationPIP-Net: Patch-Based Intuitive Prototypes for Interpretable Image Classification
Nauta, M., Schlötterer, J., van Keulen, M., & Seifert, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00269 · Cited by 227
“Deep neural networks are dominant in computer vision, but there is a high demand for understanding the reasoning of such complex models [23,30].”
BackgroundMachine Learning Methods for Weather Forecasting: A Survey
Zhang, H., Liu, Y., Zhang, C., & Li, N. (2025) · Atmosphere · doi:10.3390/atmos16010082 · Cited by 113
Explainable AI improves task performance in human–AI collaboration
Senoner, J., Schallmoser, S., Kratzwald, B., Feuerriegel, S., & Netland, T. H. (2024) · Scientific Reports · doi:10.1038/s41598-024-82501-9 · Cited by 202
Social Media Hate Speech Detection Using Explainable Artificial Intelligence (XAI)
Mehta, H., & Passi, K. (2022) · Algorithms · doi:10.3390/a15080291 · Cited by 106
Explainable Artificial Intelligence for Fault Diagnosis of Industrial Processes
Jang, K., Pilario, K. E., Lee, N., Moon, I., & Na, J. (2023) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2023.3240601 · Cited by 148
“To solve this problem, there have been an increasing number of studies on ML concerned with addressing this issue, which is known as explainable artificial intelligence (XAI) [18], [19].”
Iterative integration of deep learning in hybrid Earth surface system modelling
Chen, M., Qian, Z., Boers, N., Jakeman, A. J., Kettner, A. J., Brandt, M., Kwan, M., Batty, M., Li, W., Zhu, R., et al. (2023) · Nature Reviews Earth & Environment · doi:10.1038/s43017-023-00452-7 · Cited by 133
“Explainable or interpretable artificial intelligence using explanatory approaches (such as layer-wise relevance propagation, integrated gradients and occlusion analysis) do however allow users to understand internal mechanics of deep neural networks 146 .”
BackgroundXiong, P., Lee, S. M., & Chan, G. (2022) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2022.860032 · Cited by 141
“In Samek et al. (148), four post-hoc explanation techniques have been summarized as follows: interpretable local surrogates, occlusion analysis, gradient-based techniques, and layerwise relevance propagation (LRP).”
MethodExplainability and causability in digital pathology
Plass, M., Kargl, M., Kiehl, T., Regitnig, P., Geißler, C., Evans, T., Zerbe, N., Carvalho, R., Holzinger, A., & Müller, H. (2023) · The Journal of Pathology Clinical Research · doi:10.1002/cjp2.322 · Cited by 99
Delivering Trustworthy AI through Formal XAI
Marques-Silva, J. P., & Ignatiev, A. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21499 · Cited by 222
“These can be broadly categorized as local or global (Guidotti et al. 2019; Samek et al. 2019; Molnar 2020; Samek et al. 2021).”
BackgroundMultiomics Research: Principles and Challenges in Integrated Analysis
Luo, Y., Zhao, C., & Chen, F. (2024) · BioDesign Research · doi:10.34133/bdr.0059 · Cited by 120
“Supervised learning meth-ods, such as random forests (RFs) [ 111 ], support vector machines (SVMs) [ 112 ], and deep neural networks (DNNs) [ 113 ], are employed for classification and regression tasks and provide powerful predictive capabilities.”
Li, S., Li, T., Sun, C., Chen, X., & Yan, R. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3282599 · Cited by 99
“D CNNs have presented, the poor interpretability and noise robustness are still the main factors limiting their application in industry [15], [16].”
BackgroundSpringenberg, M., Frommholz, A., Wenzel, M. A., Weicken, E., Ma, J., & Strodthoff, N. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102809 · Cited by 134
“To gain insights into the opaque decision making of the complex DL models, we draw on the growing body of research literature in the field of XAI, see (Montavon et al., 2018; Samek and Müller, 2019; Samek et al., 2021) for reviews.”
MethodStenzinger, A., Alber, M., Allgäuer, M., Jurmeister, P. S., Bockmayr, M. L., Budczies, J., Lennerz, J. K., Eschrich, J., Kazdal, D. N., Schirmacher, P., et al. (2021) · Seminars in Cancer Biology · doi:10.1016/j.semcancer.2021.02.011 · Cited by 102
Wang, C., Cheng, X., Luo, K. H., Nandakumar, K., Wang, Z., Ni, M., Bi, X. T., Zhang, J., & Wang, C. (2025) · Materials Science and Engineering R Reports · doi:10.1016/j.mser.2025.101010 · Cited by 97
Opening the Black Box: A systematic review on explainable artificial intelligence in remote sensing
Höhl, A., Obadić, I., Fernández‐Torres, M., Najjar, H., Oliveira, D. A. B., Akata, Z., Dengel, A. R., & Zhu, X. X. (2024) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2024.3467001 · Cited by 141
Making Sense of Machine Learning: A Review of Interpretation Techniques and Their Applications
Tursunalieva, A., Alexander, D. L., Dunne, R. A., Li, J., Riera, L., & Zhao, Y. (2024) · Applied Sciences · doi:10.3390/app14020496 · Cited by 119
Yang, Y. Q., & Li, H. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.112649 · Cited by 86
Comesana, A. E., Huntington, T. T., Scown, C. D., Niemeyer, K. E., & Rapp, V. H. (2022) · Fuel · doi:10.1016/j.fuel.2022.123836 · Cited by 90
Liu, X., Ren, M., Yang, Z., Yan, G., Guo, Y., Cheng, L., & Chengke, W. (2022) · Energy · doi:10.1016/j.energy.2022.124857 · Cited by 97
Song, H., Liu, X., & Song, M. (2023) · Applied Energy · doi:10.1016/j.apenergy.2023.121077 · Cited by 88
Zou, Y., Lin, Z., Li, D., & Liu, Z. (2023) · Journal of Energy Storage · doi:10.1016/j.est.2023.109069 · Cited by 93
Varadharajan, C., Appling, A., Arora, B., Christianson, D., Hendrix, V., Kumar, V., Junior, A. R. L., Müller, J., Oliver, S. K., Ombadi, M., et al. (2022) · Hydrological Processes · doi:10.1002/hyp.14565 · Cited by 94
“Recent advances in explainable AI such as local interpretable model-agnostic explanation (LIME; Ribeiro et al., 2016) or Shapley additive explanations based on occlusion analysis (SHAP; Lundberg & Lee, 2017) can explain individual predictions by many ML models (Samek et al., 2021).”
BackgroundOster, S., Breese, P. P., Ulbricht, A., Mohr, G., & Altenburg, S. J. (2023) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-023-02117-0 · Cited by 97
“Therefore, the training process is computationally cheap (training time below 3 min on a Nvidia Geforce RTX3090 GPU) and the model’s small layer depth (only twelve model layers) and dimension benefit its interpretability (Samek et al., 2021).”
Explaining Machine Learning Models for Clinical Gait Analysis
Slijepcevic, D., Horst, F., Lapuschkin, S., Horsak, B., Raberger, A.-M., Kranzl, A., Samek, W., Breiteneder, C., Schöllhorn, W. I., & Zeppelzauer, M. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3474121 · Cited by 82
“In contrast to images, which are more frequently subject to explainability studies [2, 19, 58, 59], the evaluation of explanations becomes particularly challenging when the input signals are more abstract and thus not straightforward to interpret, as often is the case with biomedical signals.”
BackgroundSelf-citationDeep Learning in Neuroimaging: Promises and challenges
Yan, W., Qu, G., Hu, W., Abrol, A., Cai, B., Qiao, C., Plis, S., Wang, Y., Jing Sui (Beijing Normal University), my correct affiliation is beijing normal university, N. Q. U. O. S. A. T., & Calhoun, V. D. (2022) · IEEE Signal Processing Magazine · doi:10.1109/msp.2021.3128348 · Cited by 86
Liu, W., Zhao, F., Shankar, A., Maple, C. R., Peter, J. D., Kim, B., Słowik, A., Parameshachari, B. D., & Lv, J. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3336721 · Cited by 92
“Machine learning explainability can be classified into intrinsic and post-explainability [15].”
Wysocka, M., Wysocki, O., Zufferey, M., Landers, D., & Freitas, A. (2023) · BMC Bioinformatics · doi:10.1186/s12859-023-05262-8 · Cited by 103
“[7, 8, 9, 10, 11, 12, 13]), XAI in the field of genomics [14, 15, 16] and medicine [17, 18, 19, 20, 21, 22], we highlight a much more specific subfield, aiming to link explainability and biological interpretability.”
BackgroundPetersen, E., Potdevin, Y., Mohammadi, E., Zidowitz, S., Breyer, S., Nowotka, D., Henn, S., Pechmann, L., Leucker, M., Rostalski, P., et al. (2022) · IEEE Access · doi:10.1109/access.2022.3178382 · Cited by 112
“For a comprehensive review and taxonomy of explainability methods and their current challenges, refer to, e.g., Arrieta et al. [191] or Samek et al. [386]; for a review of explainability within the MML context, see Tjoa and Guan [260].”
BackgroundTime Series Forecasting using LSTM and ARIMA
Albeladi, K., Zafar, B. A., & Mueen, A. (2023) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2023.0140133 · Cited by 143
“When dealing with real world’s problems shallow neural networks need to be sufficiently expressive to predict the task optimally while deep neural networks (DNNs) have been proposed as a way of producing more predictive models [12].”
BackgroundDeep image captioning: A review of methods, trends and future challenges
Xu, L., Tang, Q., Lv, J., Zheng, B., Zeng, X., & Li, W. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126287 · Cited by 95
AudioMNIST: Exploring Explainable Artificial Intelligence for audio analysis on a simple benchmark
Becker, S., Vielhaben, J., Ackermann, M., Müller, K.-R., Lapuschkin, S., & Samek, W. (2024) · Journal of the Franklin Institute · doi:10.1016/j.jfranklin.2023.11.038 · Cited by 73
“In order to understand their inner workings and classfication strategies, we can emply post-hoc explanation methods, see [8, 26] for a recent overview of current approaches.”
BackgroundMethodSelf-citationStatistical Deep Learning for Spatial and Spatiotemporal Data
Wikle, C. K., & Zammit‐Mangion, A. (2023) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-033021-112628 · Cited by 114
The challenges of integrating explainable artificial intelligence into GeoAI
Jin, X., & Sieber, R. E. (2023) · Transactions in GIS · doi:10.1111/tgis.13045 · Cited by 105
“SHAP provides a snapshot of the challenges introduced by a facile integration of XAI and GeoAI but it is far from the only XAI approach that illustrates a reliance on an appropriate reference against which to understand what is activated within a DNN (Samek et al., 2021 ).”
BackgroundMethodInfluentialMachine Learning for Emergency Management: A Survey and Future Outlook
Kyrkou, C., Kolios, P., Theocharides, T., & Polycarpou, M. M. (2022) · Proceedings of the IEEE · doi:10.1109/jproc.2022.3223186 · Cited by 100
“To this end, ML systems for disaster and risk management need to not only give a prediction or recommendation but also a set of quantitative reasons as to why a certain outcome is reached [39].”
BackgroundB-cos Networks: Alignment is All We Need for Interpretability
Böhle, M., Fritz, M., & Schiele, B. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01008 · Cited by 139
Enhancing transparency in data-driven urban pluvial flood prediction using an explainable CNN model
Gao, W., Liao, Y., Chen, Y., Lai, C., He, S., & Wang, Z. (2024) · Journal of Hydrology · doi:10.1016/j.jhydrol.2024.132228 · Cited by 83
Keyl, J., Keyl, P., Montavon, G., Hosch, R., Brehmer, A., Mochmann, L. H., Jurmeister, P. S., Dernbach, G., Kim, M., Koitka, S., et al. (2025) · Nature Cancer · doi:10.1038/s43018-024-00891-1 · Cited by 98
“Together with recent methods for understanding the decision-making of such models, referred to as explainable artificial intelligence (xAI), this makes it possible to assess individual patient prognosis and unravel the contribution of each variable 10–15 .”
Self-citationMohamed, E., Sirlantzis, K., & Howells, G. (2022) · Displays · doi:10.1016/j.displa.2022.102239 · Cited by 67
“Model-agnostic methods such as Shapley values [23,24] and Anchors [25] are beyond the scope of this review because they are well-covered by Molnar [26] and Samek et al. [27].”
BackgroundMethodSingle-cell gene regulatory network prediction by explainable AI
Keyl, P., Bischoff, P., Dernbach, G., Bockmayr, M. L., Fritz, R., Horst, D. A., Blüthgen, N., Montavon, G., Müller, K., & Klauschen, F. (2023) · Nucleic Acids Research · doi:10.1093/nar/gkac1212 · Cited by 80
Self-citationA Review of Indoor Positioning Systems for UAV Localization with Machine Learning Algorithms
Sandamini, C., Maduranga, M. W. P., Tilwari, V., Yahaya, J. B., Qamar, F., Nguyễn, Q. N., & Ibrahim, S. R. A. (2023) · Electronics · doi:10.3390/electronics12071533 · Cited by 76
“Numerous linked neurons make up an ANN.”
Interpretable Machine Learning for Discovery: Statistical Challenges and Opportunities
Allen, G. I., Gan, L., & Zheng, L. (2023) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-040120-030919 · Cited by 115
Nascita, A., Montieri, A., Aceto, G., Ciuonzo, D., Persico, V., & Pescapè, A. (2023) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2023.3246794 · Cited by 77
“Therefore, in response to the needs arising from DL adoption for effective TC, the field of XAI is providing tools to link the outcome of the classification to the structure of the DL model and the input, making the model explainable , in a number of ways [8].”
MethodWang, D., Honnorat, N., Fox, P. T., Ritter, K., Eickhoff, S. B., Seshadri, S. S., & Habes, M. (2023) · NeuroImage · doi:10.1016/j.neuroimage.2023.119929 · Cited by 78
A review and benchmark of feature importance methods for neural networks
Mandler, H., & Weigand, B. (2024) · ACM Computing Surveys · doi:10.1145/3679012 · Cited by 77
“Apart from trustworthiness of an AM, it also needs to be computationally efficient in order to be employed [117].”
Westerlund, A. M., Hawe, J. S., Heinig, M., & Schunkert, H. (2021) · International Journal of Molecular Sciences · doi:10.3390/ijms221910291 · Cited by 75
LSTM-GAN-AE: A Promising Approach for Fault Diagnosis in Machine Health Monitoring
Liu, H., Zhao, H., Wang, J., Yuan, S., & Feng, W. (2021) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2021.3135328 · Cited by 61
“DBN and CNN can adaptively capture spatial features and achieve powerful generalization performance [20], [21].”
Background
Metadata
- Authors
- Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R.
- Affiliations
- Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (DE)
- BIFOLD✔Berlin Institute for the Foundations of Learning and Data, Berlin, Germany
- Berlin Institute for the Foundations of Learning and Data (DE)
- ORCID
- Venue
- Proceedings of the IEEE
- Published
- 2021-03
- Volume
- 109
- Issue
- 3
- Pages
- 247-278
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W3132191748
- Cited by
- 1,477
- References
- 230
- Citations by year
- 2026: 305 · 2025: 343 · 2024: 326 · 2023: 273 · 2022: 152 · 2021: 74
- FWCI
- 72.25
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Proceedings of the IEEE (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion, other-oa)
- Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korea Government (2017-0-00451, 2019-0-00079)
- German Ministry for Education and Research (01IS14013A-E, 01GQ1115, 01GQ0850, 01IS18025A, 01IS18037A)
- German Research Foundation (Math+, EXC 2046/1, 390685689)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- explainable AI, interpretability, deep neural network, post-hoc explanation, interpretability evaluation, nonlinear machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.