Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Open accessThe DOI matches its Crossref record; 224 of the 230 references it cites were found in a registry or an index.
Abstract
With the broader and highly successful usage of machine learning (ML) in industry and the sciences, there has been a growing demand for explainable artificial intelligence (XAI). Interpretability and explanation methods for gaining a better understanding of the problem-solving abilities and strategies of nonlinear ML, in particular, deep neural networks, are, therefore, receiving increased attention. In this work, we aim to: 1) provide a timely overview of this active emerging field, with a focus on “post hoc” explanations, and explain its theoretical foundations; 2) put interpretability algorithms to a test both from a theory and comparative evaluation perspective using extensive simulations; 3) outline best practice aspects, i.e., how to best include interpretation methods into the standard usage of ML; and 4) demonstrate successful usage of XAI in a representative selection of application scenarios. Finally, we discuss challenges and possible future directions of this exciting foundational field of ML.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“While XAI techniques exist [47], their application in complex, real-time energy systems, and providing meaningful explanations to non-experts, remain challenging”
From Addressing Challenges for the Effective Adoption of Artificial Intelligence in the Energy Sector (2025) · cites this paper as [47]
In the citing paper
“Despite previous attempts to address the challenges surrounding model interpretability, most solutions have been implemented post hoc (such as the attention mechanism) focusing on explanations in certain domains, such as vision [184].”
From Human-centred artificial intelligence for mobile health sensing: challenges and opportunities (2023) · cites this paper as [184]
In the citing paper
“LIME produces an explanation list that shows the contribution of individual features to the prediction.”
From Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2022) · cites this paper as [64]
In the citing paper
“This local explanation allows the end-user to determine which feature is important for the precise prediction and how the perturbation in feature affects the prediction results.”
From Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2022) · cites this paper as [64]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 12×
“This modified gradient view on LRP can also be leveraged to achieve a simpler and more general implementation of LRP based on ‘forward hooks’, which we describe in the second part of Appendix B, and which we use to apply LRP on VGG-16 [175] and ResNet-50 [65] in Section IV.”
BackgroundMethodInfluentialVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 6×
“This modified gradient view on LRP can also be leveraged to achieve a simpler and more general implementation of LRP based on ‘forward hooks’, which we describe in the second part of Appendix B, and which we use to apply LRP on VGG-16 [175] and ResNet-50 [65] in Section IV.”
BackgroundMethodInfluentialImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 4×
“Consequently, we also exchange the neural network architecture and use an AlexNet [99] model, which is able to process the transformed input data using 2-D-convolution operators.”
BackgroundMethodInfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“, [23], [52], [70], [107], [108], and [161]) have gained highest popularity.”
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 9×
“A comparison of performance on the Adience benchmark of the original model (pretrained on ImageNet only) and the improved model is given in the table below.”
MethodInfluentialRegression Models and Life-Tables
Cox, D. R. (1972) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1972.tb00899.x · Cited by 39,793
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 8×
“A comparison of performance on the Adience benchmark of the original model (pretrained on ImageNet only) and the improved model is given in the table below.”
BackgroundInfluentialThe Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 1×
“In games such as the arcade game Atari Breakout, the computer player would progressively learn strategies commonly employed by human players, such as “tunneldigging” [124], [200].”
Background
Research works printed but not linked: 13
- Arras, 2020, Ground truth evaluation of neural network explanations with CLEVR-XAI, arXiv:2003.07258 · arXiv
- 10.1146/annurev.neuro.26.041002.131047 · DOI · Paper page
- Chollet, 2015, Keras
- 10.1109/ICCV.2019.00304 · DOI
- Kokhlikyan, 2019, Pytorch Captum
- Lapuschkin, 2016, The layer-wise relevance propagation toolbox for artificial neural networks, J. Mach. Learn. Res., 17, 1
- 10.48550/arXiv.1312.4400 · DOI · Paper page
- 10.5555/2969033.2969153 · doi:10.5555/2969033.2969153
- 10.5555/3157382.3157477 · doi:10.5555/3157382.3157477
- Okuta, CuPy: A numpy-compatible library for NVIDIA GPU calculations, 151
- Oliphant, 2006, A Guide to NumPy, 1
- 10.48550/ARXIV.1404.7828 · DOI
- 10.5555/3045118.3045336 · doi:10.5555/3045118.3045336
Cited by
Items 401–500 of 1,002
Schnake, T., Jafari, F. R., Lederer, J., Xiong, P., Nakajima, S., Gugler, S., Montavon, G., & Müller, K. (2025) · Information Fusion · doi:10.1016/j.inffus.2024.102923 · Cited by 27
“Popular first-order feature methods include Layer-wise Relevance Propagation (LRP) [5, 84, 85, 97], Integrated Gradients (IG) [112], SHAP [78], and others [130, 13, 93, 89].”
BackgroundMethodSelf-citationFarmanesh, A., Sanchis, R. G., & Ordieres-Meré, J. B. (2025) · Sensors · doi:10.3390/s25103048 · Cited by 17
“Explainable AI for industrial quality inspection is gaining importance as regulatory requirements and safety considerations demand transparent decision-making [116].”
Guo, G., Qamar, F., Kazmi, S. H. A., & Rehman, M. H. U. (2025) · Internet of Things · doi:10.1016/j.iot.2025.101686 · Cited by 12
Evaluating deep transfer learning for whole-brain cognitive decoding
Thomas, A. W., Lindenberger, U., Samek, W., & Müller, K. (2023) · Journal of the Franklin Institute · doi:10.1016/j.jfranklin.2023.07.015 · Cited by 17
Self-citationBenchmarking explanation methods for mental state decoding with deep learning models
Thomas, A. W., Ré, C., & Poldrack, R. A. (2023) · NeuroImage · doi:10.1016/j.neuroimage.2023.120109 · Cited by 19
“…a set of mental states from brain activity, researchers often make use of explanation methods from explainable artificial intelligence research (XAI; Samek et al., 2021; Doshi-Velez and Kim, 2017) to gain insights into the models’ learned mappings between mental states and brain activity, seeking…”
BackgroundMethodInfluentialDeep Learning Methods for Daily Wildfire Danger ForecastingPreprint
Prapas, I., Kondylatos, S., Papoutsis, I., Camps‐Valls, G., Ronco, M., Fernández‐Torres, M., Guillem, M. P., & Carvalhais, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.02736 · Cited by 61
A Survey on Multi-Objective Based Parameter Optimization for Deep Learning
Chakraborty, M., Pal, W., Bandyopadhyay, S., & Maulik, U. (2023) · Computer Science · doi:10.7494/csci.2023.24.3.5479 · Cited by 16
Maximizing the carbon sink function of paddy systems in China with machine learning
Wang, J., Wu, Q., He, Y., Li, Y., Xu, J., & Jiang, Q. (2023) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2023.168542 · Cited by 13
Enhancing Going Concern Prediction With Anchor Explainable AI and Attention-Weighted XGBoost
Thanathamathee, P., Sawangarreerak, S., & Nizam, D. N. M. (2024) · IEEE Access · doi:10.1109/access.2024.3401007 · Cited by 11
“Consequently, LIME is commonly known as a surrogate-based explanation method [35].”
MethodZhang, Z., Wang, Y., Zhang, D., Zhao, D., Shi, H., Yan, H., Zhou, X., Feng, X., & Yang, C. (2024) · Green Chemical Engineering · doi:10.1016/j.gce.2024.06.002 · Cited by 14
Unifying Qualitative and Quantitative Safety Verification of DNN-Controlled Systems
Zhi, D., Wang, P., Liu, S., Ong, C.-H. L., & Zhang, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-65630-9_20 · Cited by 27
Hybrid neuro fuzzy inference systems for simulating catchment sediment yield
Sedighkia, M., Jahanshahloo, M., & Datta, B. (2024) · International Journal of Sediment Research · doi:10.1016/j.ijsrc.2024.02.004 · Cited by 11
Logic-Based Explainability: Past, Present and Future
Marques-Silva, J. P. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-75387-9_12 · Cited by 49
Explaining predictive uncertainty by exposing second-order effects
Bley, F., Lapuschkin, S., Samek, W., & Montavon, G. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.111171 · Cited by 22
“Understanding a model’s prediction in terms of input features, a common theme in Explainable AI [8, 9, 10], has been extensively studied for classification [11, 12], and extensions to e.g. regression have been proposed [13].”
Self-citationKozielski, M., Sikora, M., & Wawrowski, Ł. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113092 · Cited by 15
“In the first case, the explanation method is dedicated to a specific type of model, typically neural network architecture [10].”
MethodTowards an interpretable deep learning model of cancer
Nilsson, A., Meimetis, N., & Lauffenburger, D. A. (2025) · npj Precision Oncology · doi:10.1038/s41698-025-00822-y · Cited by 14
D2CN: Distributed Deep Convolutional Network
Ding, Y., Shi, J., Yang, Z., Zhang, Z., Liu, Y., & Li, X. (2025) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2025.3534177 · Cited by 11
“As a popular technique in network interpretability [13], exploring networks from mathematical perspectives has been widely used to tackle network challenges.”
Artificial intelligence in the management of patient-ventilator asynchronies: A scoping review
Muñoz, J., Ruíz-Cacho, R., Fernández-Araujo, N. J., Candela, A., Visedo, L. C., & Muñoz-Visedo, J. (2025) · Heart & Lung · doi:10.1016/j.hrtlng.2025.05.003 · Cited by 16
When Mathematical Methods Meet Artificial Intelligence and Mobile Edge Computing
Liang, Y., Bi, X., Shen, R., He, Z., Wang, Y., Xu, J., Zhang, Y., & Fan, X. (2025) · Mathematics · doi:10.3390/math13111779 · Cited by 17
Majstorović, J., Giffard‐Roisin, S., & Poli, P. (2022) · Geophysical Journal International · doi:10.1093/gji/ggac369 · Cited by 16
RoverNet: Vision-Based Adaptive Human-to-Robot Object Handovers
Mavsar, M., & Ude, A. (2022) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids53995.2022.10000200 · Cited by 11
“utilized, as they have proven to be efficient in modelling a variety of nonlinear processes [1], [2].”
MethodAutomatic identification of chemical moieties
Lederer, J., Gastegger, M., Schütt, K. T., Kampffmeyer, M., Müller, K., & Unke, O. T. (2023) · Physical Chemistry Chemical Physics · doi:10.1039/d3cp03845a · Cited by 11
Towards Self-explainable Classifiers and Regressors in Neuroimaging with Normalizing Flows
Wilms, M., Mouchès, P., Bannister, J. J., Rajashekar, D., Langner, S., & Forkert, N. D. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87586-2_3 · Cited by 13
Wilkinson, C., Yawney, J., & Gadsden, S. A. (2026) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2026.200647 · Cited by 38
Javaran, M. R. H., Rajabi, M. M., Kamali, N., Fahs, M., & Belfort, B. (2023) · Water · doi:10.3390/w15162890 · Cited by 17
STEG-XAI: explainable steganalysis in images using neural networks
Kuchumova, E., Monterrubio, S. M. M., & Recio-García, J. A. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17483-3 · Cited by 11
“Another neuralnetworkspecificmethod,Layer-WiseRelevancePropagation,computestherelevance of each pixel and its neural representation in the network’s layers, allowing it to represent its global contribution to the image classification [49].”
An Empirical Study on Low- and High-Level Explanations of Deep Learning Misbehaviours
Zohdinasab, T., Riccio, V., & Tonella, P. (2023) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem56168.2023.10304866 · Cited by 15
“Samek et al. [40] provided a survey on explanatory techniques applicable to DNNs. Unlike our work, they did not evaluate the similarity among explanations produced by different techniques and they analysed human interpretability of explanations without conducting any human assessment involving…”
BackgroundAttractor Inspired Deep Learning for Modelling Chaotic Systems
Dutta, A., Harshith, J., Ramamoorthy, A., & Lakshmanan, K. (2023) · Human-Centric Intelligent Systems · doi:10.1007/s44230-023-00045-z · Cited by 16
“Physics-based machine learning, dynamical understanding using deep neural networks [32], and por-trayal learning for studying continual system dynamics and biochemical factors have all benefited from this integration.”
Yang, M., Liu, S., Hao, T., Ma, C., Chen, H., Li, Y., Wu, C., Xie, J., Qiu, H., Li, J., et al. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102785 · Cited by 10
Xu, D., Zhou, H., Quan, W., Ugbolue, U. C., Fekete, G., & Gu, Y. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e26052 · Cited by 11
An Explainable Machine Learning Approach for Anomaly Detection in Satellite Telemetry Data
Kricheff, S., Maxwell, E., Plaks, C., & Simon, M. (2024) · IEEE Aerospace Conference (AERO) · doi:10.1109/aero58975.2024.10521300 · Cited by 14
“It works by calculating relevance values for each value in an input sequence by back propagating through the neurons in each layer of a neural network, starting from the output [38].”
BackgroundKoh, E. K., Dananjaya, P. A., Liu, L., Lee, C. X. X., Lim, G. J., You, Y. S., & Lew, W. S. (2024) · ACS Nano · doi:10.1021/acsnano.4c07454 · Cited by 12
“85 As the number of NN layers increases, so does the requisite number of memristors needed to simulate each weight connection, and consequently, the likelihood of compounded errors also increases due to error aggregation through the network.”
Sadakane, T., Nakata, K., Suda, K., & Yokogawa, D. (2024) · Bulletin of the Chemical Society of Japan · doi:10.1093/bulcsj/uoae121 · Cited by 10
Transformer-Based Ionospheric Prediction and Explainability Analysis for Enhanced GNSS Positioning
Wang, H.-S., Jwo, D., & Lee, Y.-H. (2024) · Remote Sensing · doi:10.3390/rs17010081 · Cited by 10
Zaboli, M., Karimi, N., Ajarostaghi, S. S. M., & Saedodin, S. (2025) · Renewable Energy · doi:10.1016/j.renene.2025.122407 · Cited by 11
Wanpeng, C. A. O., Danyang, M. E. I., Yongmei, G. U. O., & Ghorbani, H. (2025) · Measurement · doi:10.1016/j.measurement.2025.117144 · Cited by 10
Wang, D., Xia, Y., Pedrycz, W., Li, Z., & Yu, Z. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127553 · Cited by 10
Explainability Feature Bands Adaptive Selection for Hyperspectral Image Classification
Liu, J., Lan, J., Zeng, Y., Luo, W., Zhuang, Z., & Zou, J. (2025) · Remote Sensing · doi:10.3390/rs17091620 · Cited by 10
Lin, J.-Q., Yan, X., Wang, E.-Z., Zhang, Q., Zhang, K., Liu, P., & Zhang, L. (2025) · Petroleum Science · doi:10.1016/j.petsci.2025.07.008 · Cited by 15
Multi-Sensor Fusion and Deep Learning for Predictive Lubricant Health Assessment
Chen, Y., Shen, J., Zhou, F., Li, H., Yang, K., & Wang, L. (2025) · Lubricants · doi:10.3390/lubricants13080364 · Cited by 10
Explainable Natural Language Processing
Søgaard, A. (2021) · Synthesis lectures on human language technologies · doi:10.2200/s01118ed1v01y202107hlt051 · Cited by 38
Machine Learning for Ischemic Heart Disease Diagnosis Aided by Evolutionary Computing
Alsaffar, M. S., Alshammari, A., Alshammari, G., Aljaloud, S., Almurayziq, T. S., Abdoon, F. M., & Abebaw, S. (2021) · Applied Bionics and Biomechanics · doi:10.1155/2021/6718029 · Cited by 11
“Still following the concepts presented by Flores [8 – 27], it can be said that the tests performed allow to de fi ne the real capacity presented by the tool to present positive results, for patients who really have ischemic heart disease (sensitivity); speci fi city refers to measuring the tool ’ s…”
RetractedExplainable AI for Non-Experts: Energy Tariff Forecasting
Ma, H., McAreavey, K., McConville, R., & Liu, W. (2022) · International Conference on Automation and Computing (ICAC) · doi:10.1109/icac55051.2022.9911105 · Cited by 20
“This is especially noticeable when the system is making high-stake decisions [6] where human understanding of those decisions is vital.”
BackgroundDeep Learning Method for Evaporation Duct Inversion Based on GPS Signal
Cui, M., & Zhang, Y. (2022) · Atmosphere · doi:10.3390/atmos13122091 · Cited by 10
Analysing cerebrospinal fluid with explainable deep learning: From diagnostics to insights
Schweizer, L., Seegerer, P., Kim, H., Saitenmacher, R., Muench, A., Barnick, L., Osterloh, A., Dittmayer, C., Jödicke, R., Pehl, D., et al. (2023) · Neuropathology and Applied Neurobiology · doi:10.1111/nan.12866 · Cited by 15
“Abbreviations: A, lymphocyte; B, activated lymphocyte; C, plasma cell; E, activated monocyte/macrophage; J, tumour cell; M, mitosis Propagation [34, 35].”
Ufer, C., & Blank, H. (2023) · Language Cognition and Neuroscience · doi:10.1080/23273798.2023.2166679 · Cited by 12
“The representational hierarchy of speech processing was apparent in NNs with intermediate network layers mapping best to neural activity in PAC, while later layers predicted activity in adjacent regions along the temporal plane best (Kell et al., 2018).”
InfluentialIMG-NILM: A Deep learning NILM approach using energy heatmaps
Edmonds, J., & Abdallah, Z. S. (2023) · ACM Symposium on Applied Computing · doi:10.1145/3555776.3578604 · Cited by 10
Fu, Y., Huang, C.-C., Lin, Z., Lee, C.-C., & Wang, J. (2023) · IEEE Transactions on Electron Devices · doi:10.1109/ted.2023.3283944 · Cited by 10
“R ECENTLY, an increasing requirement for pattern recognition and decision-making in computing systems has led to the development of artificial neural network (ANN) [1], [2].”
BackgroundTesting methods of neural systems understanding
Lindsay, G. W., & Bau, D. (2023) · Cognitive Systems Research · doi:10.1016/j.cogsys.2023.101156 · Cited by 26
Chadaga, K., Prabhu, S., Sampathila, N., Chadaga, R., Bairy, G. M., & Swathi, K. S. (2023) · Journal of Technology in Behavioral Science · doi:10.1007/s41347-023-00343-0 · Cited by 16
“Further, we used a deep neural network to predict child sexual abuse awareness prediction (Samek et al., 2021).”
Selecting Correct Methods to Extract Fuzzy Rules from Artificial Neural Network
Tan, X., Zhou, Y. H., Ding, Z., & Liu, Y. (2021) · Mathematics · doi:10.3390/math9111164 · Cited by 12
“Hence, in this paper, we focus on XAI via fuzzy rules.”
MethodInfluentialUnveiling MIMETIC: Interpreting Deep Learning Traffic Classifiers via XAI Techniques
Nascita, A., Montieri, A., Aceto, G., Ciuonzo, D., Persico, V., & Pescapè, A. (2021) · IEEE International Conference on Cyber Security and Resilience (CSR) · doi:10.1109/csr51186.2021.9527948 · Cited by 10
“Accordingly, different families of approaches have emerged with the aim of shedding light on ML/DL outcomes via post-hoc explanations [5].”
BackgroundLearning domain invariant representations by joint Wasserstein distance minimization
Andéol, L., Kawakami, Y., Wada, Y., Kanamori, T., Müller, K., & Montavon, G. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.07.028 · Cited by 15
“Lastly, we inspect the learned invariant representation using UMAP embeddings [40] and ‘explainable AI’ (cf. [52, 56]).”
MethodSelf-citationTrustworthy artificial intelligence in healthcare
Subaşı, A., Özaltın, Ö., Mitra, A., Subasi, M. E., & Sarirete, A. (2023) · Elsevier eBooks · doi:10.1016/b978-0-443-15299-3.00015-4 · Cited by 9
Neural module networks: A review
Fashandi, H. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126518 · Cited by 13
Ghajari, Y. E., Kaveh, M., & Martín, D. R. (2023) · Mathematics · doi:10.3390/math11194145 · Cited by 9
Energy-informed graph transformer model for solid mechanical analyses
Feng, B., & Zhou, X. (2024) · Communications in Nonlinear Science and Numerical Simulation · doi:10.1016/j.cnsns.2024.108103 · Cited by 12
Optimizing credit card fraud detection: a deep learning approach to imbalanced datasets
Ndama, O., Bensassi, I., & En-Naimi, E. M. (2024) · International Journal of Power Electronics and Drive Systems/International Journal of Electrical and Computer Engineering · doi:10.11591/ijece.v14i4.pp4802-4814 · Cited by 9
“Every neuron in this tightly linked layer calculates the weighted total of its inputs, which is then sent through a rectified linear unit (ReLU) activation function [19].”
BackgroundRibeiro, E., Cárdenas, D. A. C., Dias, F. M., Krieger, J. E., & Gutierrez, M. A. (2024) · European Heart Journal - Digital Health · doi:10.1093/ehjdh/ztae045 · Cited by 13
Li, Z., Jin, Y., Tian, Y., Liu, J., Wang, M., Wei, X., Zhao, L., & Liu, C. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108890 · Cited by 11
Qu, J., Yousef, S., Faney, T., de Hemptinne, J.-C., & Gallinari, P. (2024) · Applied Energy · doi:10.1016/j.apenergy.2024.124025 · Cited by 13
Eberle, O., Büttner, J., El-Hajj, H., Montavon, G., Müller, K., & Valleriani, M. (2024) · Science Advances · doi:10.1126/sciadv.adj1719 · Cited by 23
“In order to evaluate the ML model itself and the features that are used for a certain prediction, the field of explainable AI [141, 142, 143, 49] has developed techniques to make models transparent and reveal their inner logic.”
Self-citationYu, R., Li, Y., Zhao, K., & Fan, F. (2024) · Sleep and Biological Rhythms · doi:10.1007/s41105-024-00563-8 · Cited by 12
“It maps the features of existing spatial samples to another feature space through layer-by-layer feature mapping, allowing the network to learn a better representation of the existing inputs [59].”
Hofmann, S., Goltermann, O., Scherf, N., Müller, K., Loeffler, M., Villringer, A., Gaebler, M., Witte, A. V., & Beyer, F. (2025) · Imaging Neuroscience · doi:10.1162/imag_a_00497 · Cited by 12
“We applied layer-wise relevance propagation (LRP; Bach et al., 2015; Lapuschkin et al., 2019; Montavon et al., 2018; Samek et al., 2021) to study which features drive the performance of our BA prediction models.”
Floreale, G., Baraldi, P., Zio, E., & Fink, O. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110518 · Cited by 11
Prospecting artificial microbial consortia toward lignin valorization
Hu, J., Gao, B.-C., Liu, Z., Li, X., Yuan, Y., & Li, B. (2025) · Chemical Engineering Journal · doi:10.1016/j.cej.2025.162375 · Cited by 15
Kim, E., & Hau, Y. S. (2025) · International Journal of Green Energy · doi:10.1080/15435075.2025.2553864 · Cited by 12
“By stacking multiple hidden layers, DNN can learn more complex data patterns (Samek et al. 2021).”
Tiong, J. Y., Hasikin, K. B., Ngui, R., Divis, P. C. S., Loo, C. K., Lai, K. W., Cheong, F. W., & Sulaiman, W. Y. W. (2025) · Acta Tropica · doi:10.1016/j.actatropica.2025.107842 · Cited by 10
Bonneau, K., Lederer, J., Templeton, C., Rosenberger, D., Giambagli, L., Müller, K., & Clementi, C. (2025) · Nature Communications · doi:10.1038/s41467-025-65863-0 · Cited by 22
“As a response, the new area of “ Explainable Arti fi cial Intelligence (XAI) ” has emerged to begin providing tools to tackle the interpretation of arti fi cial neural networks 29 .”
Liu, J., Yu, W., Yang, M., & Liu, F. (2025) · Safety Science · doi:10.1016/j.ssci.2025.107090 · Cited by 12
The Rise of AI Avatars: Legal Personhood, Rights and Liabilities in an Evolving Metaverse
Cheong, B. C. (2024) · Journal of Digital Technologies and Law · doi:10.21202/jdtl.2024.42 · Cited by 16
Siva Viknesh, S., Tatari, Y., Christenson, C., & Arzani, A. (2026) · Physical Review Research · doi:10.1103/dwkk-5g2h · Cited by 37
Stroke Risk Prediction and Prevention: Traditional versus Machine Learning Approaches
Singh, M. S., Thongam, K., Choudhary, P., & Bhagat, P. K. (2025) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-025-10406-5 · Cited by 20
PredDiff: Explanations and interactions from conditional expectations
Blücher, S., Vielhaben, J., & Strodthoff, N. (2022) · Artificial Intelligence · doi:10.1016/j.artint.2022.103774 · Cited by 26
“To this end, the Explainable AI (XAI) community has put forward a plethora of different attribution methods, see [6, 21, 25, 26, 29] for reviews.”
Background[Retracted] Music Emotion Analysis Based on PSO‐BP Neural Network and Big Data Analysis
Chen, X. (2021) · Computational Intelligence and Neuroscience · doi:10.1155/2021/6592938 · Cited by 10
“Samek et al. have verified the effectiveness of the music emotion evaluation system in the PSO-BP neural network through practice, and the results show that the innovative PSO-BP neural network analysis method can effectively perform high accuracy on different types of databases [14].”
MethodRetractedPalta, P., Kaur, P., & Mann, K. S. (2022) · Journal of Microwave Power and Electromagnetic Energy · doi:10.1080/08327823.2022.2103630 · Cited by 17
A deep learning-based framework for accurate identification and crop estimation of olive trees
Khan, U. M., Maqsood, M., Gillani, S. A., Durrani, M. Y., Mehmood, I., & Seo, S. (2022) · The Journal of Supercomputing · doi:10.1007/s11227-022-04738-3 · Cited by 10
“The reduced feature vector is then classified using a fully connected network with backpropagation into several output classes [34].”
MethodMoore, J. A., Tuladhar, A., Ismail, Z., Mouchès, P., Wilms, M., & Forkert, N. D. (2022) · Neuroinformatics · doi:10.1007/s12021-022-09602-6 · Cited by 14
“Saliency maps are frequently used in computer vision tasks to enhance understanding around which parts of an input stimuli a DCNN focuses on to arrive at a classification (Samek et al., 2021).”
BackgroundReliability of Deep Neural Networks for an End-to-End Imitation Learning-Based Lane Keeping
Liu, S., & Müller, S. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3299229 · Cited by 11
“…PF can achieve a reliable performance of evaluating the saliency map quality through calculating the decay of the DNN’s inference ability during recursively destroying the input features according to their relevances ranked by the saliency map algorithms [28], and is thus adopted by this work.”
BackgroundMethodKim, T., Ha, B., & Hwangbo, S. (2023) · Machine Learning with Applications · doi:10.1016/j.mlwa.2023.100505 · Cited by 12
NeuralCMS: A deep learning approach to study Jupiter’s interior
Ziv, M., Galanti, E., Sheffer, A., Howard, S., Guillot, T., & Kaspi, Y. (2024) · Astronomy and Astrophysics · doi:10.1051/0004-6361/202450223 · Cited by 9
From mechanism-based to data-driven approaches in materials science
Hiemer, S., & Zapperi, S. (2021) · Materials Theory · doi:10.1186/s41313-021-00027-3 · Cited by 10
“Hiemer and ZapperiMaterials Theory (2021) 5:4 Page 8 of 9”
BackgroundInfluentialLocal Interpretable Classifier Explanations with Self-generated Semantic Features
Angiulli, F., Fassetti, F., & Nisticò, S. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88942-5_31 · Cited by 9
Investigating Explanatory Factors of Machine Learning Models for Plant Classification
Wöber, W., Mehnen, L., Sykacek, P., & Meimberg, H. (2021) · Plants · doi:10.3390/plants10122674 · Cited by 11
Explain to Not Forget: Defending Against Catastrophic Forgetting with XAI
Ede, S., Baghdadlian, S., Weber, L., Nguyen, A., Zanca, D., Samek, W., & Lapuschkin, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-14463-9_1 · Cited by 18
Self-citationLiu, J., Yuan, G., Yang, C., Song, H., & Luo, L. (2022) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2022.023195 · Cited by 15
Wang, Y., Wang, G., Yao, G., Yang, L. G., Li, J., & He, S. (2023) · Advanced Theory and Simulations · doi:10.1002/adts.202200656 · Cited by 9
Flexible Pressure Sensors and Machine Learning Algorithms for Human Walking Phase Monitoring
Hải, N. T., Ngo, B.-V., Nguyen, T.-N., & Vu, C. C. (2023) · Micromachines · doi:10.3390/mi14071411 · Cited by 10
“This system can classify the human walking phases using two machine learning models, including a simple deep neural network [33] and a random forest [34].”
MethodArtificial Neural Networks and Data Science
Bihl, T., Young, W. A., Moyer, A. C., & Frimel, S. (2023) · IGI Global eBooks · doi:10.4018/978-1-7998-9220-5.ch052 · Cited by 13
A Hybrid Extreme Gradient Boosting and Long Short-Term Memory Algorithm for Cyber Threats Detection
Amin, R., Eltaweel, G., Ali, A. F., & Tahoun, M. (2023) · MENDEL · doi:10.13164/mendel.2023.2.307 · Cited by 8
Predicting Heat Capacity of Molecular Fluids Using Interpretable Machine Learning Model
Li, S., He, H., & Wang, Y. (2024) · Industrial & Engineering Chemistry Research · doi:10.1021/acs.iecr.4c02495 · Cited by 8
Explainable machine learning model of disorganisation in swarms of drones
Gackowska, M., & Cofta, P. (2024) · Scientific Reports · doi:10.1038/s41598-024-73220-2 · Cited by 8
“Explainability and its importance was reviewed by Samek et al. 27 .”
BackgroundHousehold Power Consumption Analysis using Machine Learning
Raviprabhakaran, V., Pranay, P., Nendralla, B., & Pranay, L. S. (2024) · IEEE International Conference on Sustainable Energy and Future Electric Transportation (SEFET) · doi:10.1109/sefet61574.2024.10718254 · Cited by 8
Pradhan, A., Adams, K. H., Chandrasekaran, V., Liu, Z., Reager, J. T., Stuart, A. M., & Turmon, M. (2024) · Journal of Geophysical Research Machine Learning and Computation · doi:10.1029/2024jh000322 · Cited by 15
Ai, J., Han, X., Chen, L., He, H., Li, X., Tan, Y., Xie, T., & Tang, X. (2025) · International Journal of Remote Sensing · doi:10.1080/01431161.2024.2448309 · Cited by 11
“These deep learning (DL) methods have powerful feature learning capabilities and the ability to represent highly complex functions, demonstrating better generalization compared to traditional machine learning methods (Rezaee et al. 2018; Samek et al. 2021).”
Roy, D. K., & Datta, B. (2025) · Journal of Environmental Management · doi:10.1016/j.jenvman.2025.124592 · Cited by 11
Eren, K. K., Küçük, K., Özyurt, F., & Alhazmi, O. H. (2025) · IEEE Access · doi:10.1109/access.2025.3547642 · Cited by 14
“DL models, such as convolutional neural networks (CNNs), long short-term memory networks (LSTMs), and autoen-coders, have garnered particular attention due to their capacity to handle complex data proving their success in areas such as image recognition and natural language processing [13], [14].”
Gayathri, R. M., Sangeetha, S., Sangeetha, R., Mary, G. L. R., Mathivanan, S. K., & Moorthy, U. (2025) · IEEE Access · doi:10.1109/access.2025.3564971 · Cited by 12
“Looking for further tips on techniques and applications for interpreting deep neural networks [5], [6], [7] are analyzed to reveal the potential of AI in personalized medicine.”
Zhou, R., Bull-Vulpe, E. F., Pan, Y., & Paesani, F. (2025) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.5c00474 · Cited by 12
Morik, M., Hashemi, A. B., Müller, K., Haufe, S., & Nakajima, S. (2025) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3594724 · Cited by 17
Metadata
- Authors
- Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R.
- Affiliations
- Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (DE)
- BIFOLD✔Berlin Institute for the Foundations of Learning and Data, Berlin, Germany
- Berlin Institute for the Foundations of Learning and Data (DE)
- ORCID
- Venue
- Proceedings of the IEEE
- Published
- 2021-03
- Volume
- 109
- Issue
- 3
- Pages
- 247-278
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W3132191748
- Cited by
- 1,477
- References
- 230
- Citations by year
- 2026: 305 · 2025: 343 · 2024: 326 · 2023: 273 · 2022: 152 · 2021: 74
- FWCI
- 72.25
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Proceedings of the IEEE (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion, other-oa)
- Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korea Government (2017-0-00451, 2019-0-00079)
- German Ministry for Education and Research (01IS14013A-E, 01GQ1115, 01GQ0850, 01IS18025A, 01IS18037A)
- German Research Foundation (Math+, EXC 2046/1, 390685689)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- explainable AI, interpretability, deep neural network, post-hoc explanation, interpretability evaluation, nonlinear machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.