Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Open accessThe DOI matches its Crossref record; 224 of the 230 references it cites were found in a registry or an index.
Abstract
With the broader and highly successful usage of machine learning (ML) in industry and the sciences, there has been a growing demand for explainable artificial intelligence (XAI). Interpretability and explanation methods for gaining a better understanding of the problem-solving abilities and strategies of nonlinear ML, in particular, deep neural networks, are, therefore, receiving increased attention. In this work, we aim to: 1) provide a timely overview of this active emerging field, with a focus on “post hoc” explanations, and explain its theoretical foundations; 2) put interpretability algorithms to a test both from a theory and comparative evaluation perspective using extensive simulations; 3) outline best practice aspects, i.e., how to best include interpretation methods into the standard usage of ML; and 4) demonstrate successful usage of XAI in a representative selection of application scenarios. Finally, we discuss challenges and possible future directions of this exciting foundational field of ML.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“While XAI techniques exist [47], their application in complex, real-time energy systems, and providing meaningful explanations to non-experts, remain challenging”
From Addressing Challenges for the Effective Adoption of Artificial Intelligence in the Energy Sector (2025) · cites this paper as [47]
In the citing paper
“Despite previous attempts to address the challenges surrounding model interpretability, most solutions have been implemented post hoc (such as the attention mechanism) focusing on explanations in certain domains, such as vision [184].”
From Human-centred artificial intelligence for mobile health sensing: challenges and opportunities (2023) · cites this paper as [184]
In the citing paper
“LIME produces an explanation list that shows the contribution of individual features to the prediction.”
From Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2022) · cites this paper as [64]
In the citing paper
“This local explanation allows the end-user to determine which feature is important for the precise prediction and how the perturbation in feature affects the prediction results.”
From Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2022) · cites this paper as [64]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 12×
“This modified gradient view on LRP can also be leveraged to achieve a simpler and more general implementation of LRP based on ‘forward hooks’, which we describe in the second part of Appendix B, and which we use to apply LRP on VGG-16 [175] and ResNet-50 [65] in Section IV.”
BackgroundMethodInfluentialVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 6×
“This modified gradient view on LRP can also be leveraged to achieve a simpler and more general implementation of LRP based on ‘forward hooks’, which we describe in the second part of Appendix B, and which we use to apply LRP on VGG-16 [175] and ResNet-50 [65] in Section IV.”
BackgroundMethodInfluentialImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 4×
“Consequently, we also exchange the neural network architecture and use an AlexNet [99] model, which is able to process the transformed input data using 2-D-convolution operators.”
BackgroundMethodInfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“, [23], [52], [70], [107], [108], and [161]) have gained highest popularity.”
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 9×
“A comparison of performance on the Adience benchmark of the original model (pretrained on ImageNet only) and the improved model is given in the table below.”
MethodInfluentialRegression Models and Life-Tables
Cox, D. R. (1972) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1972.tb00899.x · Cited by 39,793
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 8×
“A comparison of performance on the Adience benchmark of the original model (pretrained on ImageNet only) and the improved model is given in the table below.”
BackgroundInfluentialThe Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 1×
“In games such as the arcade game Atari Breakout, the computer player would progressively learn strategies commonly employed by human players, such as “tunneldigging” [124], [200].”
Background
Research works printed but not linked: 13
- Arras, 2020, Ground truth evaluation of neural network explanations with CLEVR-XAI, arXiv:2003.07258 · arXiv
- 10.1146/annurev.neuro.26.041002.131047 · DOI · Paper page
- Chollet, 2015, Keras
- 10.1109/ICCV.2019.00304 · DOI
- Kokhlikyan, 2019, Pytorch Captum
- Lapuschkin, 2016, The layer-wise relevance propagation toolbox for artificial neural networks, J. Mach. Learn. Res., 17, 1
- 10.48550/arXiv.1312.4400 · DOI · Paper page
- 10.5555/2969033.2969153 · doi:10.5555/2969033.2969153
- 10.5555/3157382.3157477 · doi:10.5555/3157382.3157477
- Okuta, CuPy: A numpy-compatible library for NVIDIA GPU calculations, 151
- Oliphant, 2006, A Guide to NumPy, 1
- 10.48550/ARXIV.1404.7828 · DOI
- 10.5555/3045118.3045336 · doi:10.5555/3045118.3045336
Cited by
Items 601–700 of 1,002
Li, Z., Wang, C., Chen, X., Yu, N., Xing, L., Xuan, J., Han, M., & Ni, M. (2024) · Energy Conversion and Management · doi:10.1016/j.enconman.2024.118886 · Cited by 9
Explainable Computer Vision Techniques and Deep Transfer Learning for Tomato Leaf Disease Diagnosis
Drake, V., Henry, S., Namuwaya, W., Kayongo, N., Lule, E., & Marvin, G. (2024) · International Conference on Image Processing and Capsule Networks (ICIPCN) · doi:10.1109/icipcn63822.2024.00101 · Cited by 6
“To mitigate the opacity of deep neural networks, various methodologies with different levels of effectiveness were proposed [20].”
BackgroundChimeric U-Net – Modifying the standard U-Net towards explainability
Schulze, K., Peppert, F., Schütte, C., & Sunkara, V. (2024) · Artificial Intelligence · doi:10.1016/j.artint.2024.104240 · Cited by 7
Patel, A. K., Trujillo-Rivera, E. A., Chamberlain, J. M., Morizono, H., & Pollack, M. M. (2024) · PLoS ONE · doi:10.1371/journal.pone.0288233 · Cited by 7
Srinivasan, V., Strodthoff, N., Ma, J., Binder, A., Müller, K., & Samek, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.13497 · Cited by 6
“To obtain explanation heatmaps, we utilize Layer-wise Relevance Propagation (LRP) with α 1 β 0 rule [65,69].”
Self-citationOblivion: Poisoning Federated Learning by Inducing Catastrophic Forgetting
Zhang, C., Zhou, B., He, Z., Liu, Z., Chen, Y., Xu, W., & Li, B. (2023) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom53939.2023.10228981 · Cited by 10
“According to explanatory analysis on learning models [23], [24], only a fraction of all weights is critical in determining the prediction results, while most of the weights in a large-scale learning model are redundant.”
BackgroundHong, B., & Burghout, W. (2024) · Emission Control Science and Technology · doi:10.1007/s40825-024-00241-8 · Cited by 5
“This drawback may become increasingly evident with black-box models which, when encountering real-world scenarios beyond their training data, may lead to model performance characterized by poor generalization and overfitting [20].”
Exploring the Reliability of SHAP Values in Reinforcement Learning
Engelhardt, R. C., Lange, M., Wiskott, L., & Konen, W. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-63800-8_9 · Cited by 8
Mouakher, A., & Kononov, R. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02421-y · Cited by 11
“However, creating a comprehensive XAI validation framework remains a significant challenge [71].”
Classification of flotation froth images using neural networks with explanations
Borys, D., & Kozielski, M. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.09.510 · Cited by 5
Solving the enigma: Enhancing faithfulness and comprehensibility in explanations of deep networks
Mamalakis, M., Mamalakis, A., Agartz, I., Mørch-Johnsen, L. E., Murray, G. K., Suckling, J., & Lió, P. (2025) · AI Open · doi:10.1016/j.aiopen.2025.02.001 · Cited by 10
“Post-hoc techniques encompass model-specific approaches designed to address specific nonlinear model behaviors and model-agnostic approaches to investigate data complexity (Samek et al., 2021; van der Velden, 2023).”
Lawson, C. G. R., Englezos, E., & Rourke, M. F. (2025) · Plants People Planet · doi:10.1002/ppp3.70007 · Cited by 9
“…Al-Ansari et al., 2024; Hassija et al., 2024; Ali et al., 2023; Saranya & Subhashini, 2023; Lisboa et al., 2023; Yu, 2023; Li et al., 2022; Samek et al., 2021; Burkart & Huber, 2021; Confalonieri et al., 2021; Vassiliades et al., 2021; Alicioglu & Sun, 2022; Das & Rad, 2020; Arrieta et…”
A segment-based framework for explainability in animal affective computing
Boneh-Shitrit, T., Finka, L. R., Mills, D. S., Luna, S. P. L., Costa, E. D., Zamansky, A., & Bremhorst, A. (2025) · Scientific Reports · doi:10.1038/s41598-025-96634-y · Cited by 6
A comprehensive multimodal benchmark of neuromorphic training frameworks for spiking neural networks
Cheng, Y., Hu, W., He, Y., & Huang, J. Z. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111543 · Cited by 10
Hammer, A., Goettling, M., Malberg, H., Linke, A. H.-P., Richter, S., Mangner, N., & Schmidt, M. U. (2025) · npj Artificial Intelligence · doi:10.1038/s44387-025-00022-w · Cited by 11
“Weused Spearman ’ s rank correlation analysis as the weighted interconnection in neurons enables NNs to learn complex, non-linear relationships 58 .”
Mtfsfn: a multi-view time-frequency-space fusion network for EEG-based emotion recognition
Wang, Z., & Gao, S. (2025) · Cognitive Neurodynamics · doi:10.1007/s11571-025-10342-x · Cited by 6
“At the same time, deep learning brings forth deep neural networks capable of learning more abstract, higher-level features to tackle complex problems (Samek et al. 2021).”
Herlambang, S. W., Dewanta, F., & Purwanto, Y. (2025) · doi:10.1109/icoict66265.2025.11192987 · Cited by 8
“DNN is part of machine learning and deep learning, which is designed to mimic the way the human brain processes information [16].”
Explainable AI (XAI) in Healthcare: Building Trust in Medical Diagnosis Systems
Khade, M. P. S. (2025) · International Journal of Scientific Research in Science and Technology · doi:10.32628/ijsrst25126288 · Cited by 5
Alarcon, G. M., & Capiola, A. (2025) · Frontiers in Computer Science · doi:10.3389/fcomp.2025.1662185 · Cited by 13
Large Language Models in Bio-Ontology Research: A Review
Manda, P. (2025) · Bioengineering · doi:10.3390/bioengineering12111260 · Cited by 16
“Without transparent explanations, curators may struggle to evaluate why a given term was proposed or how a mapping was derived, limiting trust and adoption in high-stakes biomedical settings [97].”
Fu, A., An, T., Li, W., & Qiao, J. (2025) · Engineering Research Express · doi:10.1088/2631-8695/ae1dc9 · Cited by 5
Maram, P., Parvathi, R., & Pattabiraman, V. (2025) · Results in Engineering · doi:10.1016/j.rineng.2025.108489 · Cited by 8
Makhoabenyane, M., Guo, S., & Leburu, E. (2025) · Sustainability · doi:10.3390/su172411330 · Cited by 5
Guo, Z., Zhong, M., Duan, S., & Wang, L. (2025) · IEEE Signal Processing Letters · doi:10.1109/lsp.2025.3644313 · Cited by 7
“Traditional handcrafted-feature detectors such as HOG [9] or texture descriptors [10] perform poorly in UAV scenarios, motivating the shift toward deep learning–based methods [11].”
Vasiliki, M., Serdaris, P., Antoniadis, I. I., & Spinthiropoulos, K. G. (2025) · Digital · doi:10.3390/digital6010001 · Cited by 7
“This inference legitimizes the use of AI in marketing, turning trust into proactive engagement with ethically perceived brands [39].”
Rahman, M. M., Calhoun, V. D., & Plis, S. (2026) · Imaging Neuroscience · doi:10.1162/imag.a.1129 · Cited by 16
Ling, M., Guan, Y., Lian, Y., Sun, X., Gao, Y., & Ren, Y. (2026) · Journal of Hydrology · doi:10.1016/j.jhydrol.2026.135231 · Cited by 7
Abass, S. A. E., Altamimi, A. S. A., Aldakhil, T., & Prajapati, P. (2026) · Critical Reviews in Analytical Chemistry · doi:10.1080/10408347.2026.2694610 · Cited by 6
Chen, H., & Martel, A. L. (2024) · International Workshop on Breast Imaging (IWBI) · doi:10.1117/12.3025375 · Cited by 6
Research on Time Series Data Prediction Based on Machine Learning Algorithms
Zheng, Z., Yang, Y., Zhou, J., & Gu, F. (2024) · IEEE International Conference on Control, Electronics and Computer Technology (ICCECT) · doi:10.1109/iccect60629.2024.10546215 · Cited by 6
“…memory networks (LSTM) and gated recurrent units (GRU) are able to capture complex long-term and short-term periodic patterns in time series as well as the interactions between sequences, and have thus achieved significant effects in multiple auxiliary tasks of natural language processing [15-19].”
BackgroundWhat-If XAI Framework (WiXAI): From Counterfactuals towards Causal Understanding
Kshetry, N., & Kantardzic, M. M. (2024) · Journal of Computer and Communications · doi:10.4236/jcc.2024.126011 · Cited by 7
Explainable, Deep Reinforcement Learning–Based Decision Making for Operations and Maintenance
Spangler, R. M., Raeisinezhad, M., & Cole, D. G. (2024) · Nuclear Technology · doi:10.1080/00295450.2024.2377034 · Cited by 5
“A representation of this process can be seen in Fig.”
Interpretability as Approximation: Understanding Black-Box Models by Decision Boundary
Dong, H., Liu, B., Ye, D., & Liu, G. (2024) · Electronics · doi:10.3390/electronics13224339 · Cited by 9
Kamarian, S., Khalvandi, A., Heidarizadi, E., Saber‐Samandari, S., & Song, J. (2024) · Mechanics of Advanced Materials and Structures · doi:10.1080/15376494.2024.2418975 · Cited by 5
Provenance-Enabled Explainable AI
Zhang, J., Zhou, W., & Ujcich, B. E. (2024) · Proceedings of the ACM on Management of Data · doi:10.1145/3698826 · Cited by 12
Uncertainty Quantification of Satellite-Based Essential Climate Variables Derived from Deep Learning
Gou, J., Salberg, A.-B., Shahvandi, M. K., Tourian, M. J., Meyer, U., Boergens, E., Waldeland, A. U., Velicogna, I., Dahl, F., Jäggi, A., et al. (2026) · Surveys in Geophysics · doi:10.1007/s10712-025-09919-2 · Cited by 13
Toward Faithful Neural Network Intrinsic Interpretation With Shapley Additive Self-Attribution
Sun, Y., Zhu, H., & Xiong, H. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3570692 · Cited by 9
“In recent years, e ff orts have been made to provide intrinsic interpretation of neural networks without relying on post hoc interpretation algorithms [17], [18], [19].”
Revesai, Z., & Kogeda, O. P. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-96997-3_14 · Cited by 6
Zhang, B., Gao, R., Gao, P., Wang, Y., & Yang, Z. (2025) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2025.3586230 · Cited by 5
“Given the substantial advantages and broad application potential of GNN [28], [29], [30] in processing graph-structured data, there has been a surge of research on resource allocation leveraging GNNs.”
Evaluating contrastive explanations for AI planning with non-experts: a smart home battery scenario
Shi, Y., McAreavey, K., & Liu, W. (2022) · International Conference on Automation and Computing (ICAC) · doi:10.1109/icac55051.2022.9911125 · Cited by 5
“The authors of [5], [6] identify several challenges faced by XAI research.”
BackgroundDiscriminative attribution from paired images
Eckstein, N., Bukhari, H., Bates, A. S., Jefferis, G. S. X. E., & Funke, J. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-25069-9_27 · Cited by 5
“A large number of approaches for feature attribution have been proposed in recent years (for a recent review see [27] and related work below) and they remain a popular choice for practitioners due to their ease of use, availability in popular Deep Learning frameworks, and intuitive outputs.”
BackgroundTian, W., Liu, Y., Xie, J., Huang, W., Chen, W., Tao, T., & Xin, K. (2023) · Water · doi:10.3390/w15101960 · Cited by 5
DIP-CBML: A New Classification of Thai Dragon Fruit Species from Images
Yusamran, N., & Hiransakolwong, N. (2023) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2023.0140523 · Cited by 5
“The model gives high predictive performance [25] by mimicking the operation of human neurons by building a neural network with multiple nonlinear processing layers [14, 26].”
BackgroundSurvey of Neuromorphic Computing: A Data Science Perspective
Clark, K., & Wu, Y. (2023) · International Conference on Computer Communication and Artificial Intelligence (CCAI) · doi:10.1109/ccai57533.2023.10201289 · Cited by 5
“The neurons on each layer are mapped through a process called error backpropagation [30] [31].”
MethodExplainable Deep Learning Based-System for Multilabel Classification of 12-Lead ECG
Azzem, Y. C. H., & Harrag, F. (2023) · International Conference on Networking and Advanced Systems (ICNAS) · doi:10.1109/icnas59892.2023.10330482 · Cited by 5
“In the context of medical diagnosis, this is particularly valuable as it enables the identification of whether the model has learned irrelevant features or noise in the input data, which it may mistakenly associate with the diagnostic task [13].”
BackgroundWang, X., Chai, Z., Li, S., Liu, Y., Li, C., Jiang, Y., & Liu, Q. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae063 · Cited by 6
“While DL-based approaches employ state-of-the-art neural network-based algorithms, including Artificial Neural Networks (Yang 2008), Convolutional Neural Networks (Yamashita et al. 2018), Deep Neural Networks (Samek et al. 2021), Capsule Neural Networks (Sabour et al. 2017), and Graph Neural Networks (Zhou et al. 2020…”
MethodData-driven real-time predictive control for industrial heating loads
Wu, C., Zhou, Y., & Wu, J. (2024) · Electric Power Systems Research · doi:10.1016/j.epsr.2024.110420 · Cited by 5
“Nowadays, data-driven methods, such as artificial neural networks (ANN) [12], deep neural networks (DNN) [13], and deep reinforcement learning (DRL) [14], have opened up the possibility for real-time predictive control in complex industrial processes.”
Tang, Y., Lin, Z., Zhou, L., Wang, W., Wen, L., Zhou, Y., Ge, Z., Chen, Z., Dai, W., Yang, Z., et al. (2024) · Journal Of Big Data · doi:10.1186/s40537-024-00989-4 · Cited by 7
“Recent years have witnessed significant advances in artificial intelligence (AI), particularly in deep learning (DL) [1], which has been successfully applied in various domains [2].”
BackgroundAutomating field‐based floral surveys with machine learning
Sookhan, N., Sookhan, S., Grewal, D., & MacIvor, J. S. (2024) · Ecological Solutions and Evidence · doi:10.1002/2688-8319.12393 · Cited by 6
Neural network embedding of functional microconnectome
Shirakami, A., Hase, T., Yamaguchi, Y., & Shimono, M. (2025) · Network Neuroscience · doi:10.1162/netn_a_00424 · Cited by 4
“This approach leverages the advancements in artificial neural networks to automatically extract features from network architectures, minimizing subjective researcher bias (LeCun et al., 2015; Samek et al., 2021; van der Laak et al., 2021).”
Establishing Transparency in Artificial Intelligence Systems
Quakulinski, L., Koumpis, A., & Beyan, O. (2022) · International Conference on Transdisciplinary AI (TransAI) · doi:10.1109/transai54797.2022.00027 · Cited by 6
Scheduling BCG and IL-2 Injections for Bladder Cancer Immunotherapy Treatment
Yaniv‐Rosenfeld, A., Savchenko, E. V., Rosenfeld, A., & Lazebnik, T. (2023) · Mathematics · doi:10.3390/math11051192 · Cited by 6
Explainability as a Method for Learning From Computers
Klimo, M., Kopčan, J., & Králik, Ľ. (2023) · IEEE Access · doi:10.1109/access.2023.3265582 · Cited by 5
“In this paper, we have ingested two representative methods for computing feature relevance: a gradient backpropagation-based method and a relevance backpropagation-based method [41], [42].”
MethodHernández, L. J. R., Domínguez, H. D. J. O., Villegas, O. O. V., Sánchez, V. G. C., Sossa, H., & Polanco-González, J. (2023) · Pattern Recognition Letters · doi:10.1016/j.patrec.2023.05.005 · Cited by 5
Wang, Z. (2023) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2023.01406102 · Cited by 5
“…in industry and science, there is a growing need for interpretable artificial intelligence, and therefore the problem of better understanding nonlinear ML, and in particular DNNs, interpretable and explainable methods for solving capabilities and strategies are receiving increasing attention [5].”
BackgroundOdle, E., Hsueh, Y.-J., & Lin, P. (2023) · Electronics · doi:10.3390/electronics12194185 · Cited by 6
Machine learning methods for autism spectrum disorder classification
Damianos, L., Vlachas, C., Kollias, K.-F., Asimopoulos, N., & Fragulis, G. F. (2023) · AIP conference proceedings · doi:10.1063/5.0182539 · Cited by 6
Interpretable Deep Learning for Diagnosis of Maize Streak Disease
Kalyango, M. F., & Ntanda, K. M. (2023) · International Conference on the Advancements of Artificial Intelligence in African Context (AAIAC) · doi:10.1109/aaiac60008.2023.10465315 · Cited by 5
“Deep Learning models have exhibited a high accuracy in a number of AI tasks where they are applied, their acceptability has however been low due to lack of means of interpretation by users of such models [10], [11].”
BackgroundA Framework for Autonomous AI-Driven Drug DiscoveryPreprint
Selinger, D. W., Wall, T. R., Stylianou, E. S., Khalil, E. M., Gaetz, J., & Levy, O. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.12.17.629024 · Cited by 10
“Neural networks in particular, including graph neural networks and deep learning models, typically operate as "black boxes," making it difficult to trace the evidence supporting their predictions ( 26 ) .”
Tian, J., & Yang, W. (2024) · Entropy · doi:10.3390/e26110902 · Cited by 6
“Despite significant advancements in the field of eXplainable Artificial Intelligence (XAI) [23,25,38–41], the development of Quantum eXplainable Artificial Intelligence (QXAI) has been comparatively gradual.”
Li, Y., Ge, X., & Liu, B. (2024) · Chemical Engineering Research and Design · doi:10.1016/j.cherd.2024.12.015 · Cited by 5
Sheikh, M. M., Balachandra, M., Narendra, V. G., & Maiya, A. G. (2025) · IEEE Access · doi:10.1109/access.2025.3535590 · Cited by 7
Model checking deep neural networks: opportunities and challenges
Sbaï, Z. (2025) · Frontiers in Computer Science · doi:10.3389/fcomp.2025.1557977 · Cited by 4
Interacting with fallible AI: is distrust helpful when receiving AI misclassifications?
Peters, T. M., & Scharlau, I. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1574809 · Cited by 9
“Numerous studies on explainable AI (XAI; e.g., Samek et al., 2021; Mohseni et al., 2021) are currently investigating various methods to improve interactions with AI by incorporating explainability methods.”
Geographically informed graph neural networks
Ma, X., Zhang, Z., & Song, Y. (2025) · Spatial Statistics · doi:10.1016/j.spasta.2025.100920 · Cited by 5
Lv, X., Li, H., Li, Y., Zhuo, R., Yue, Y., Wang, Y., Zheng, X., & Gao, H. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-03143-z · Cited by 4
“It measures a model’s ability to correctly predict the order of event occurrences, assessing whether the model accurately ranks event times for pairs of samples [28].”
Xuan, Z., Zheng, F., Xia, J., & Zhu, J. (2025) · Geophysical Research Letters · doi:10.1029/2025gl118319 · Cited by 4
“The deep feature hierarchy enables DNN to effectively extract nonlinear multivariate coupling features from prior prediction and observation at the analysis step (Samek et al., 2021; Sze et al., 2017).”
Dissanayake, T., Müller, K., & von Lühmann, A. (2025) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2025.3617858 · Cited by 9
Niu, Z., Wu, Y., Lan, S., Wu, C., Han, X., & Chen, J. (2025) · International Journal of Electrical Power & Energy Systems · doi:10.1016/j.ijepes.2025.111373 · Cited by 6
MeshAFN: An efficient attention-based Fourier network for self-supervised mesh generation
Wang, M., Zhang, H., Li, H., & Li, N. (2025) · Physics of Fluids · doi:10.1063/5.0294377 · Cited by 4
Alzate‐Ricaurte, S., Osorio, F. O., Costa-Roig, A., Gómez-Ayala, M. C., Gomez, I. E., Arnold, M., Campos, C. I., Sánchez-Sosa, S., Lovell, M. A., Cruz, G., et al. (2026) · Journal of Pediatric Surgery · doi:10.1016/j.jpedsurg.2026.162932 · Cited by 5
Hasan, E., Fuad, M. T. H., Sharif, O., & Wagler, A. E. (2026) · Journal of Imaging · doi:10.3390/jimaging12030098 · Cited by 5
Villa, C. F. H., Arango, D. A. G., Giraldo, L. F. G., Romero, R. A. M., Valencia-Arias, A., & Ochoa, J. A. V. (2026) · Information · doi:10.3390/info17050456 · Cited by 4
Liu, X., Chen, Z., Zhang, H., Li, J., Jiang, Q., Ren, L., & Luo, Y. (2023) · Pest Management Science · doi:10.1002/ps.7566 · Cited by 6
Basics of Auditable AI Systems
Gräupner, O. S., Pawlaszczyk, D., & Hummert, C. (2023) · Congress in Computer Science, Computer Engineering, and Applied Computing (CSCE) · doi:10.1109/csce60160.2023.00382 · Cited by 6
“Well-known XAI methods including Sensitivity Analysis [7], Layer-wise Relevance Propagation (LRP) [8], Interpretable Local Sur-rogates [9], Occlusion Analysis [9] and Integrated Gradi-ents/SmoothGrad [9].”
MethodEvolutionary Approach to Optimal Oil Skimmer Assignment for Oil Spill Response: A Case Study
Kim, Y.-H., Kim, H.-J., Cho, D.-H., & Yoon, Y. (2024) · Biomimetics · doi:10.3390/biomimetics9060330 · Cited by 4
“Therefore, in our second study, we propose a surrogate model that employs a deep neural network (DNN) [7,8] to expedite resource allocation optimization.”
MethodPelaez, M. B. L., Dhondt, J., Bertels, J., Hauwermeiren, D. V., & Verwaeren, J. (2024) · International Journal of Pharmaceutics · doi:10.1016/j.ijpharm.2024.125012 · Cited by 4
Cross-Domain Adversarial Attacks and Robust Defense Mechanisms for Multimodal Neural Networks
Barach, J. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-83796-8_23 · Cited by 11
Tsoulos, I. G., & Charilogis, V. (2025) · AppliedMath · doi:10.3390/appliedmath5010027 · Cited by 4
“The proposed method could be also applied on deep neural networks used in the recent literature [55,56].”
AI algorithms in visual communication design: enhancing design creativity and efficiency
Cao, Q. (2025) · International Journal for Simulation and Multidisciplinary Design Optimization · doi:10.1051/smdo/2025008 · Cited by 5
When Can You Trust Your Explanations? A Robustness Analysis on Feature Importances
Vascotto, I., Rodríguez, Á., Bonaita, A., & Bortolussi, L. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-08327-2_11 · Cited by 16
“As we will be discussing results obtained from neural networks, we can further distinguish between perturbation-based and gradient-based approaches [26].”
HUDD: A tool to debug DNNs for safety analysis
Fahmy, H. M., Pastore, F., & Briand, L. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion55297.2022.9793750 · Cited by 4
Sim, S., EunKyung, L., Seo, M., Seong, N.-H., Jeong, D., Woo, J., & Han, K. (2022) · Remote Sensing Letters · doi:10.1080/2150704x.2022.2138620 · Cited by 4
“Deep neural network (DNN) is an artificial neural network consisting of multiple hidden layers between the input and output layers; they have high modelling and prediction accuracy for nonlinear relationships (Samek et al. 2021).”
BackgroundExplainable Deep Learning for Brain-Computer Interfaces through Layerwise Relevance Propagation
Mun, V., & Abibullaev, B. (2023) · International Winter Conference on Brain-Computer Interface (BCI) · doi:10.1109/bci57258.2023.10078678 · Cited by 4
“To tackle the black-box property of deep learning, several methods were proposed with varying degrees of functionality [21].”
MethodReducing Overhead of Feature Importance Visualization via Static GradCAM Computation
Bhat, A., & Raychowdhury, A. (2023) · IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS) · doi:10.1109/aicas57966.2023.10168594 · Cited by 5
“However, their lack of explainability limits their widespread adoption in safety-critical tasks where the enduser requires transparency and trust [1].”
BackgroundZhao, Y., Vaddi, P. K., Pietrykowski, M. C., Khafizov, M., & Smidts, C. S. (2023) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2023.109592 · Cited by 5
The Green Notebook - A Co-Creativity Partner for Facilitating Sustainability Reflection
Deng, Y., Jääskeläinen, P., & Popova, V. (2023) · ACM International Conference on Interactive Media Experiences · doi:10.1145/3573381.3596465 · Cited by 7
“AI art tools such as DALL-E 21 and Stable Diffusion2 have become more accessible to the general public, and since many AI methods require a large amount of computational power, e.g. time-consuming training of DeepNeural Network (DNN) [16], the global energy consumption of AI-related technology is expected to grow with…”
BackgroundAn interpretable neural network for robustly determining the location and number of cluster centers
Xie, X., Pu, Y., Zhang, H., Mańdziuk, J., El-Alfy, E.-S. M., & Wang, J. (2023) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-023-01978-4 · Cited by 4
“In addition, these neural networks-based methods are still not interpretable and the reliability of the obtained results cannot be guaranteed since the introduced neural networks are black-box models [32–34].”
Kim, Y.-H., Kim, W., Yoon, J., Chung, S. K., & Kim, D. (2024) · Information · doi:10.3390/info15030134 · Cited by 7
Comparative evaluation of interpretation methods in surface-based age prediction for neonates
Wu, X., Xie, C., Cheng, F., Li, Z., Li, R., Xu, D., Kim, H., Zhang, J., Liu, H., & Liu, M. (2024) · NeuroImage · doi:10.1016/j.neuroimage.2024.120861 · Cited by 6
“In response to this challenge, researchers have been developing methods to uncover the enigmatic "black box" of deep learning, thus rendering the decision-making process of these models more transparent, compre-hensible, and explainable (Mueller et al., 2013; Nazir et al., 2023; Samek et al., 2021).”
BackgroundEnhancing Robotic Grasp Failure Prediction Using A Pre-hoc Explainability Framework*
Acun, C., Ashary, A., Popa, D. O., & Nasraoui, O. (2024) · IEEE International Conference on Automation Science and Engineering (CASE) · doi:10.1109/case59546.2024.10711481 · Cited by 4
“However, this integration has also raised significant challenges regarding the trustworthiness and interpretability of the underlying algorithms [2], [3].”
BackgroundMazhar, F., & Javed, A. (2024) · Journal of Fluids and Structures · doi:10.1016/j.jfluidstructs.2024.104200 · Cited by 5
Explainability in AI-Based Applications – A Framework for Comparing Different Techniques
Grobrügge, A., Mishra, N., Jakubik, J., & Satzger, G. (2024) · IEEE Conference on Business Informatics · doi:10.1109/cbi62504.2024.00018 · Cited by 5
“This multi-method approach not only provides a safety net by cross-verifying results but also helps in uncovering any potential biases or systematic errors present in individual methods.”
Nguyen, D. D., Nguyen, M. D., Nguyen, T. V., Cao, C. T., Phong, T. V., Duc, D. M., Bien, T. X., Prakash, I., Le, H. V., & Pham, B. T. (2024) · Journal of the Indian Society of Remote Sensing · doi:10.1007/s12524-024-02055-1 · Cited by 10
“Notably, each layer utilizes the output of the preceding layer as its input, contributing to the hierarchical representation of features (Samek et al., 2021; Vieira et al., 2020).”
Rajanandhini, V. M., Alruwais, N., Mazroa, A. A., & Alkharashi, A. (2024) · Matéria (Rio de Janeiro) · doi:10.1590/1517-7076-rmat-2024-0569 · Cited by 4
“HRPR: A positive correlation of 0.37 with High-Range Plasticity Reducer (HRPR) suggests that as PFA increases, HRPR tends to increase as well [38].”
InfluentialMetabolic nanoscopy enhanced by experimental and computational approaches
Jang, H., Wu, S., Li, Y., Li, Z., & Shi, L. (2024) · npj Imaging · doi:10.1038/s44303-024-00062-y · Cited by 5
“Tosolvethisproblem,explainableNNshavebeenstudied 41,42 .”
Chen, L., Nasif, K. F. A., Deng, B., Niu, S., & Xie, C. Y. (2024) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai62512.2024.00083 · Cited by 7
Verbel, A. A., Schutesky, M. E., Gregory, D. D., & Verbel, A. (2025) · Journal of Geochemical Exploration · doi:10.1016/j.gexplo.2025.107701 · Cited by 4
Huang, Y., Xin, Z., Liao, G., Huang, P., Hou, G., & Zou, R. (2025) · Remote Sensing · doi:10.3390/rs17050840 · Cited by 4
Explainable Artificial Intelligence for Business and Economics: Methods, Applications and Challenges
Lyu, Q., & Wu, S. (2025) · Expert Systems · doi:10.1111/exsy.70017 · Cited by 10
“However, there is no well-agreed understanding of what exactly XAI is or what an optimal explanation should be (Samek et al. 2021; Islam et al. 2024).”
Metadata
- Authors
- Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R.
- Affiliations
- Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (DE)
- BIFOLD✔Berlin Institute for the Foundations of Learning and Data, Berlin, Germany
- Berlin Institute for the Foundations of Learning and Data (DE)
- ORCID
- Venue
- Proceedings of the IEEE
- Published
- 2021-03
- Volume
- 109
- Issue
- 3
- Pages
- 247-278
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W3132191748
- Cited by
- 1,477
- References
- 230
- Citations by year
- 2026: 305 · 2025: 343 · 2024: 326 · 2023: 273 · 2022: 152 · 2021: 74
- FWCI
- 72.25
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Proceedings of the IEEE (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion, other-oa)
- Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korea Government (2017-0-00451, 2019-0-00079)
- German Ministry for Education and Research (01IS14013A-E, 01GQ1115, 01GQ0850, 01IS18025A, 01IS18037A)
- German Research Foundation (Math+, EXC 2046/1, 390685689)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- explainable AI, interpretability, deep neural network, post-hoc explanation, interpretability evaluation, nonlinear machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.