Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Open accessThe DOI matches its Crossref record; 224 of the 230 references it cites were found in a registry or an index.
Abstract
With the broader and highly successful usage of machine learning (ML) in industry and the sciences, there has been a growing demand for explainable artificial intelligence (XAI). Interpretability and explanation methods for gaining a better understanding of the problem-solving abilities and strategies of nonlinear ML, in particular, deep neural networks, are, therefore, receiving increased attention. In this work, we aim to: 1) provide a timely overview of this active emerging field, with a focus on “post hoc” explanations, and explain its theoretical foundations; 2) put interpretability algorithms to a test both from a theory and comparative evaluation perspective using extensive simulations; 3) outline best practice aspects, i.e., how to best include interpretation methods into the standard usage of ML; and 4) demonstrate successful usage of XAI in a representative selection of application scenarios. Finally, we discuss challenges and possible future directions of this exciting foundational field of ML.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“While XAI techniques exist [47], their application in complex, real-time energy systems, and providing meaningful explanations to non-experts, remain challenging”
From Addressing Challenges for the Effective Adoption of Artificial Intelligence in the Energy Sector (2025) · cites this paper as [47]
In the citing paper
“Despite previous attempts to address the challenges surrounding model interpretability, most solutions have been implemented post hoc (such as the attention mechanism) focusing on explanations in certain domains, such as vision [184].”
From Human-centred artificial intelligence for mobile health sensing: challenges and opportunities (2023) · cites this paper as [184]
In the citing paper
“LIME produces an explanation list that shows the contribution of individual features to the prediction.”
From Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2022) · cites this paper as [64]
In the citing paper
“This local explanation allows the end-user to determine which feature is important for the precise prediction and how the perturbation in feature affects the prediction results.”
From Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2022) · cites this paper as [64]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 12×
“This modified gradient view on LRP can also be leveraged to achieve a simpler and more general implementation of LRP based on ‘forward hooks’, which we describe in the second part of Appendix B, and which we use to apply LRP on VGG-16 [175] and ResNet-50 [65] in Section IV.”
BackgroundMethodInfluentialVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 6×
“This modified gradient view on LRP can also be leveraged to achieve a simpler and more general implementation of LRP based on ‘forward hooks’, which we describe in the second part of Appendix B, and which we use to apply LRP on VGG-16 [175] and ResNet-50 [65] in Section IV.”
BackgroundMethodInfluentialImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 4×
“Consequently, we also exchange the neural network architecture and use an AlexNet [99] model, which is able to process the transformed input data using 2-D-convolution operators.”
BackgroundMethodInfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“, [23], [52], [70], [107], [108], and [161]) have gained highest popularity.”
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 9×
“A comparison of performance on the Adience benchmark of the original model (pretrained on ImageNet only) and the improved model is given in the table below.”
MethodInfluentialRegression Models and Life-Tables
Cox, D. R. (1972) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1972.tb00899.x · Cited by 39,793
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 8×
“A comparison of performance on the Adience benchmark of the original model (pretrained on ImageNet only) and the improved model is given in the table below.”
BackgroundInfluentialThe Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 1×
“In games such as the arcade game Atari Breakout, the computer player would progressively learn strategies commonly employed by human players, such as “tunneldigging” [124], [200].”
Background
Research works printed but not linked: 13
- Arras, 2020, Ground truth evaluation of neural network explanations with CLEVR-XAI, arXiv:2003.07258 · arXiv
- 10.1146/annurev.neuro.26.041002.131047 · DOI · Paper page
- Chollet, 2015, Keras
- 10.1109/ICCV.2019.00304 · DOI
- Kokhlikyan, 2019, Pytorch Captum
- Lapuschkin, 2016, The layer-wise relevance propagation toolbox for artificial neural networks, J. Mach. Learn. Res., 17, 1
- 10.48550/arXiv.1312.4400 · DOI · Paper page
- 10.5555/2969033.2969153 · doi:10.5555/2969033.2969153
- 10.5555/3157382.3157477 · doi:10.5555/3157382.3157477
- Okuta, CuPy: A numpy-compatible library for NVIDIA GPU calculations, 151
- Oliphant, 2006, A Guide to NumPy, 1
- 10.48550/ARXIV.1404.7828 · DOI
- 10.5555/3045118.3045336 · doi:10.5555/3045118.3045336
Cited by
Items 101–200 of 1,002
Towards Trustworthy AI in Dentistry
Ma, J., Schneider, L., Lapuschkin, S., Achtibat, R., Duchrau, M., Krois, J., Schwendicke, F., & Samek, W. (2022) · Journal of Dental Research · doi:10.1177/00220345221106086 · Cited by 61
“In recent years, many approaches to explain predictions of AI models have been developed in the field of XAI (Samek et al. 2021).”
BackgroundSelf-citationCharfeddine, M., Kammoun, H. M., Hamdaoui, B., & Guizani, M. (2024) · IEEE Access · doi:10.1109/access.2024.3367792 · Cited by 107
“5 architecture, can generate human-like responses and engage in meaningful conversations using deep learning techniques [5].”
BackgroundHofmann, S., Beyer, F., Lapuschkin, S., Goltermann, O., Loeffler, M., Müller, K., Villringer, A., Samek, W., & Witte, A. V. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119504 · Cited by 78
Self-citationChadaga, K., Prabhu, S., Sampathila, N., Chadaga, R., Umakanth, S., Bhat, D., & S, S. K. G. (2024) · Scientific Reports · doi:10.1038/s41598-024-52428-2 · Cited by 66
“A DNN consists of multiple input, hidden and output layers 39 .”
BackgroundWhat kind of trust does AI deserve, if any?
Alvarado, R. (2023) · AI and Ethics · doi:10.1007/s43681-022-00224-x · Cited by 60
Subramanian, H. V., Canfield, C., & Shank, D. B. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102780 · Cited by 84
Can we trust explainable artificial intelligence in wind power forecasting?
Liao, W., Fang, J., Ye, L., Bak‐Jensen, B., Yang, Z., & Porté‐Agel, F. (2024) · Applied Energy · doi:10.1016/j.apenergy.2024.124273 · Cited by 68
Establishing and evaluating trustworthy AI: overview and research challenges
Kowald, D., Scher, S., Pammer-Schindler, V., Müllner, P., Waxnegger, K., Demelius, L., Fessl, A., Toller, M., Mendoza Estrada, I. G., Šimić, I., et al. (2024) · Frontiers in Big Data · doi:10.3389/fdata.2024.1467222 · Cited by 59
“The term scope refers to whether XAI methods can produce global or local explanations (Samek et al., 2021).”
Mouchès, P., Wilms, M., Rajashekar, D., Langner, S., & Forkert, N. D. (2022) · Human Brain Mapping · doi:10.1002/hbm.25805 · Cited by 68
On Tackling Explanation Redundancy in Decision Trees
Izza, Y., Ignatiev, A., & Marques-Silva, J. P. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13575 · Cited by 120
“There is an extensive body of work on non-formal XAI approaches to XAI (Adadi and Berrada, 2018; Montavon et al., 2018; Samek et al., 2019; Guidotti et al., 2019; Samek et al., 2021; Tjoa and Guan, 2021; Holzinger et al., 2022, 2020; Ras et al., 2022).”
BackgroundLomazzi, L., Fabiano, S., Parziale, M., Giglio, M., & Cadini, F. (2022) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2022.109642 · Cited by 62
Physics-informed neural networks with hard linear equality constraints
Chen, H., Constante‐Flores, G. E., & Li, C. (2024) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2024.108764 · Cited by 109
Bottieau, J., Wang, Y., Grève, Z. D., Vallée, F., & Toubeau, J. (2022) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2022.3195970 · Cited by 66
“…approach, which consist in analyzing an already trained (black-box) model by, e.g., interpretable local surrogates, gradient-based or perturbation-based methods [35], and ii) an intrinsic approach, in which the architecture of the model is directly designed with interpretable components [36], [37].”
Rise of artificial general intelligence: risks and opportunities
Buttazzo, G. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1226990 · Cited by 58
“To address these issues a new research branch of machine learning, referred to as Explainable AI, has emerged to provide understandable interpretations of the output provided by neural networks and detect possible biases in the dataset (Das and Rad, 2020; Samek et al., 2020).”
BackgroundVisser, R., Peters, T. M., Scharlau, I., & Hammer, B. (2025) · Cognitive Systems Research · doi:10.1016/j.cogsys.2025.101357 · Cited by 58
Chemello, G., Salvatori, B., Morettini, M., & Tura, A. (2022) · Biosensors · doi:10.3390/bios12110985 · Cited by 70
Kinney, M. N., Anastasiadou, M., Naranjo-Zolotov, M., & Santos, V. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e28562 · Cited by 97
“…as decision trees, linear regressions, rule-based models, tree-based ensemble models, and classification rules are readily understood and reproducible, it is not possible to know how the output of the system is reached with so-called black-box models without additional measures being taken [32].”
BackgroundAn Explainable AI Approach for Breast Cancer Metastasis Prediction Based on Clinicopathological Data
Maouche, I., Terrissa, L. S., Benmohammed, K., & Zerhouni, N. (2023) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2023.3282840 · Cited by 60
“An interesting review provided by Samek et al. [37] discusses thoughtfully the technical foundations of XAI, and introduces existing post-hoc XAI algorithms for deep neural networks with examples of its application for imaging datasets.”
BackgroundRobustness, Stability, and Fidelity of Explanations for a Deep Skin Cancer Classification Model
Saarela, M., & Geogieva, L. (2022) · Applied Sciences · doi:10.3390/app12199545 · Cited by 66
“However, this advantage comes with the trade-off that deep learning models, with their many kinds of processing layers and multitudes of weights, are also reckoned to be one of the least interpretable machine learning models [18].”
BackgroundClare, M., Sonnewald, M., Lguensat, R., Deshayes, J., & Balaji, V. (2022) · Journal of Advances in Modeling Earth Systems · doi:10.1029/2022ms003162 · Cited by 74
“XAI techniques can be used to explain the skill of neural networks (Samek et al., 2019, 2021; Arrieta et al., 2020), but there has been little work combining explainability with uncertainty analysis in part because the distributions in BNNs add extra complexity.”
InfluentialAn explainable transfer learning framework for multi-classification of lung diseases in chest X-rays
Patel, A. N., Murugan, R., Srivastava, G., Maddikunta, P. K. R., Yenduri, G., Gadekallu, T. R., & Chengoden, R. (2024) · Alexandria Engineering Journal · doi:10.1016/j.aej.2024.04.072 · Cited by 63
Rezazade Mehrizi, M. H., Mol, F., Peter, M., Ranschaert, E., Dos Santos, D. P., Shahidi, R., Fatehi, M., & Dratsch, T. (2023) · Scientific Reports · doi:10.1038/s41598-023-36435-3 · Cited by 54
“Ideally, professionals engage in “reflective practices” through which they directly inspect medical images and informational inputs, and consider various medical scenarios, compared to the situation in which they do not have such suggestions 13 .”
BackgroundAghaeipoor, F., Sabokrou, M., & Fernández, A. (2023) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2023.3243935 · Cited by 73
Reflective artificial intelligence
Lewis, P. R., & Sarkadi, Ş. (2024) · Minds and Machines · doi:10.1007/s11023-024-09664-2 · Cited by 53
“Post-hoc methods to interpret and ‘explain’ ANN-based models, such as LIME and SHAP [25], result not in an explanation of the internal mechanics of ANNs, but approximations in the form of equivalent interpretable models.”
Current status and future directions of explainable artificial intelligence in medical imaging
Saw, S. N., Yan, Y. Y., & Ng, K. H. (2024) · European Journal of Radiology · doi:10.1016/j.ejrad.2024.111884 · Cited by 81
Recognition of EEG Signals from Imagined Vowels Using Deep Learning Methods
Sarmiento, L. C., Villamizar, S., Vargas, O. L., Collazos, A. S. C., Sarmiento, J., & Bacca, J. (2021) · Sensors · doi:10.3390/s21196503 · Cited by 58
“Among the advantages of DL is the ability to automatically identify the characteristics of a signal [38,39].”
BackgroundRecent progress of artificial intelligence for liquid-vapor phase change heat transfer
Suh, Y., Chandramowlishwaran, A. M., & Won, Y. (2024) · npj Computational Materials · doi:10.1038/s41524-024-01223-8 · Cited by 68
“In practice, shallow ANNs may not be suf fi ciently expressive and have limited learnability 108 .”
Structure-based, deep-learning models for protein-ligand binding affinity prediction
Wang, D. D., Wu, W., & Wang, R. (2024) · Journal of Cheminformatics · doi:10.1186/s13321-023-00795-9 · Cited by 75
Novel Vision Transformer–Based Bi-LSTM Model for LU/LC Prediction—Javadi Hills, India
Mohanrajan, S. N., & Loganathan, A. (2022) · Applied Sciences · doi:10.3390/app12136387 · Cited by 53
Tian, W., Liao, Z., Zhang, Z., Wu, H., & Xin, K. (2022) · Water Resources Research · doi:10.1029/2021wr030939 · Cited by 57
“…a large number of studies in the context of mathematics, cybernetics, and machine learning have considered the existence of a linear structure as a reflection of interpretability (Liu et al., 2021; Mardt et al., 2018; Montavon et al., 2017; Samek et al., 2021; Tu et al., 2014; Zhang et al., 2021).”
BackgroundConceptExplainer: Interactive Explanation for Deep Neural Networks from a Concept Perspective
Huang, J., Mishra, A., Kwon, B. C., & Bryan, C. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209384 · Cited by 47
“The popularity of deep learning in today’s society has led to increased demands for XAI techniques that are friendly to non-expert users [27].”
MethodInfluentialGenerating and evaluating synthetic data in digital pathology through diffusion models
Pozzi, M., Noei, S., Robbi, E., Cima, L., Moroni, M., Munari, E., Torresani, E., & Jurman, G. (2024) · Scientific Reports · doi:10.1038/s41598-024-79602-w · Cited by 61
“Explainable artificial intelligence is a rapidly growing sub field within the machine learning community 72 .”
Towards robust explanations for deep neural networks
Dombrowski, A.-K., Anders, C. J., Müller, K., & Kessel, P. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108194 · Cited by 94
Self-citationMachine Learning for Health: Algorithm Auditing & Quality Control
Oala, L., Murchison, A. G., Balachandran, P., Choudhary, S., Fehr, J., Leite, A. W., Goldschmidt, P. G., Johner, C., Schörverth, E. D. M., Nakasi, R., et al. (2021) · Journal of Medical Systems · doi:10.1007/s10916-021-01783-y · Cited by 48
“These include model-oriented metrics, such as predictive performance, robustness [33, 34], interpretability [1, 35], disparity [36] or uncertainty [13, 24, 37] but also dataoriented metrics related to sample size determination [38], sparseness [39], bias [40] distribution mismatch [41, 42] and label quality [7].”
Self-citationBender, T., Metsch, J. M., Krefting, D., Müller, C., Dathe, H., Seidler, T., Spicher, N., & Hauschild, A.-C. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3271858 · Cited by 69
“To address this need, frameworks and methods from the field of Explainable Artificial Intelligence (XAI) are developed and evaluated [19].”
MethodMachine learning for modern power distribution systems: Progress and perspectives
Marković, M., Bossart, M., & Hodge, B. S. (2023) · Journal of Renewable and Sustainable Energy · doi:10.1063/5.0147592 · Cited by 60
Explaining Deep Learning for ECG Analysis: Building Blocks for Auditing and Knowledge Discovery
Wagner, P., Mehari, T., Haverkamp, W., & Strodthoff, N. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108525 · Cited by 48
“This led to the emergence of explainable AI (XAI) as a subfield of machine learning, which tries to shed light on the decision process implemented by neural networks, see [15, 16, 17, 18] for general reviews.”
BackgroundOn the Use of Deep Learning for Video Classification
Rehman, A. U., Belhouari, S. B., Kabir, M. A., & Khan, A. (2023) · Applied Sciences · doi:10.3390/app13032007 · Cited by 55
Explainable AI for industrial fault diagnosis: A systematic review
Cação, J., Santos, J., & Antunes, M. (2025) · Journal of Industrial Information Integration · doi:10.1016/j.jii.2025.100905 · Cited by 85
This looks More Like that: Enhancing Self-Explaining Models by Prototypical Relevance Propagation
Gautam, S., Höhne, M., Hansen, S., Jenssen, R., & Kampffmeyer, M. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.109172 · Cited by 71
Improving the interpretability of deep reinforcement learning in urban drainage system operation
Tian, W., Fu, G., Xin, K., Zhang, Z., & Liao, Z. (2023) · Water Research · doi:10.1016/j.watres.2023.120912 · Cited by 52
Characterization of anomalous diffusion classical statistics powered by deep learning (CONDOR)
Gentili, A., & Volpe, G. (2021) · Journal of Physics A Mathematical and Theoretical · doi:10.1088/1751-8121/ac0c5d · Cited by 46
“A deeper understanding of the relative relevance of different inputs for the deep learning architecture can be gained by employing advanced interpretability algorithms [54].”
BackgroundCalderon-Ramirez, S., Yang, S., Moemeni, A., Colreavy‐Donnelly, S., Elizondo, D. A., Oala, L., Rodríguez‐Capitán, J., Jiménez-Navarro, M. F., López‐Rubio, E., & Molina‐Cabello, M. A. (2021) · IEEE Access · doi:10.1109/access.2021.3085418 · Cited by 51
“Model uncertainty estimation is a common approach to increase model interpretability and safety in use [13], [15].”
BackgroundThe promise of explainable deep learning for omics data analysis: Adding new discovery tools to AI
Santorsola, M., & Lescai, F. (2023) · New Biotechnology · doi:10.1016/j.nbt.2023.06.002 · Cited by 49
Weighting Factors Optimization for FCS-MPC in PMSM Drives Using Aggregated Residual Network
Yao, C., Ma, G., Sun, Z., Luo, J., Ren, G., & Xu, S. (2023) · IEEE Transactions on Power Electronics · doi:10.1109/tpel.2023.3323627 · Cited by 48
“According to [30] and [31], it is beneficial to employ ReLU as activate function for alleviating the gradient vanishing and overfitting.”
MethodRavindran, A. S., & Contreras-Vidal, J. L. (2023) · Scientific Reports · doi:10.1038/s41598-023-43871-8 · Cited by 71
“The scope of the paper will therefore be limited to the visualization approach as this is also the most extensively developed explainability method in other domains as well 31, 32 .”
MethodAn explainable AI framework for robust and transparent data-driven wind turbine power curve models
Letzgus, S., & Müller, K. (2023) · Energy and AI · doi:10.1016/j.egyai.2023.100328 · Cited by 60
“Moreover, we address the challenge of how to best present XAI attributions to domain experts for maximal benefit, an active field of research across domains [18, 17].”
InfluentialxxAI - Beyond Explainable Artificial Intelligence
Holzinger, A., Goebel, R. G., Fong, R., Moon, T., Müller, K., & Samek, W. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-04083-2_1 · Cited by 56
Self-citationLogic-Based Explainability in Machine Learning
Marques-Silva, J. P. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-31414-8_2 · Cited by 122
Anomaly diagnosis of connected autonomous vehicles: A survey
Fang, Y., Min, H., Wu, X., Wang, W., Zhao, X., Martínez‐Pastor, B., & Teixeira, R. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102223 · Cited by 57
Linka, K., Reiter, N., Würges, J., Schicht, M., Bräuer, L., Cyron, C. J., Paulsen, F. P., & Budday, S. (2021) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2021.704738 · Cited by 51
“Using relevance propagation, a concept of explainable arti fi cial intelligence (Samek et al., 2021), we identify the importance of the different tissue constituents for the mechanical response of human brain tissue, where quasi-elastic and viscous effects show distinct regional trends.”
Yao, C., Xu, S., Ren, G., Wu, S., Li, G., Sun, Z., & Ma, G. (2024) · IEEE Transactions on Power Electronics · doi:10.1109/tpel.2024.3351911 · Cited by 47
“As the simple structure of ReLU realizes fewer calculations and faster convergence speed [13], the activation layer is commonly selected as ReLU and the mathematical expression of ReLU can be formulated as (1) Furthermore, the max pooling layer reduces the size of featured maps and remove the…”
MethodLi, B., Wang, G., Chen, L., Sun, F., Wang, R., Liao, M., Xu, H. L., Li, S., & Kang, Y. (2024) · Engineering Geology · doi:10.1016/j.enggeo.2024.107803 · Cited by 48
Learning Patch-Channel Correspondence for Interpretable Face Forgery Detection
Hua, Y., Shi, R., Wang, P., & Ge, S. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3246793 · Cited by 57
“Model interpretability [31] aims to describe the internal working mechanism of deep neural networks in understandable terms to human beings.”
BackgroundRegulative development as a model for origin of life and artificial life studies
Fields, C., & Levin, M. (2023) · Biosystems · doi:10.1016/j.biosystems.2023.104927 · Cited by 40
Anand, G., Kumari, S., & Pulle, R. (2023) · International Journal of Computer Science and Engineering · doi:10.14445/23488387/ijcse-v10i5p101 · Cited by 42
“Comparative Analysis The comparative evaluation of the developed method is estimated with several previous methods like Support Vector Machine (SVM) (MTD1) [7], multilayer perception (MLP) classifier (MTD2)[8], light gradient-boosting machine (Light GBM) (MTD3) [9], Decision tree classifier (MTD3)[10], and Deep Neural…”
MethodInfluentialXia, X., Wu, W., Li, Z., Han, X., Xue, X., Xiao, G. G., & Guo, T. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3390696 · Cited by 61
“As we know, although more hidden layers can fit more complex nonlinear functions and theoretically improve the model, too many layers may also cause overfitting problems, increase the difficulty of training, and make the model difficult to converge [38].”
BackgroundChen, S., Liu, Y., Zhang, Q., Shao, Z., & Wang, Z. (2025) · Advanced Intelligent Systems · doi:10.1002/aisy.202400898 · Cited by 116
Junjia, Y., Alias, A. H., Haron, N. A., & Bakar, N. A. (2023) · Sustainability · doi:10.3390/su151511803 · Cited by 61
Semnani, P., Bogojeski, M., Bley, F., Zhang, Z., Wu, Q., Kneib, T., Herrmann, J., Weisser, C., Patcas, F., & Müller, K. (2024) · The Journal of Physical Chemistry C · doi:10.1021/acs.jpcc.4c05332 · Cited by 51
Machine and Deep Learning-based XSS Detection Approaches: A Systematic Literature Review
Thajeel, I. K., Samsudin, K. B., Hashim, S. J., & Hashim, F. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.101628 · Cited by 58
“The network’s weights and biases are repeatedly fine-tuned using techniques such as stochastic gradient descent to reduce error or costs (Samek et al., 2021).”
Using Landsat-5 for Accurate Historical LULC Classification: A Comparison of Machine Learning Models
Кривогуз, Д., Chernyi, S., Zinchenko, E., Silkin, A. A., & Zinchenko, A. A. (2023) · Data · doi:10.3390/data8090138 · Cited by 49
Altham, C., Zhang, H., & Pereira, E. (2024) · PLoS ONE · doi:10.1371/journal.pone.0303644 · Cited by 55
Wearable MOF biosensors: A new frontier in real-time health monitoring
Rabiee, N. (2025) · TrAC Trends in Analytical Chemistry · doi:10.1016/j.trac.2025.118156 · Cited by 43
Explainable AI for time series via Virtual Inspection Layers
Vielhaben, J., Lapuschkin, S., Montavon, G., & Samek, W. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110309 · Cited by 38
Self-citationExtraction of Fetal Electrocardiogram by Combining Deep Learning and SVD-ICA-NMF Methods
Ziani, S., Farhaoui, Y., & Moutaib, M. (2023) · Big Data Mining and Analytics · doi:10.26599/bdma.2022.9020035 · Cited by 38
An ensemble learning method based on deep neural network and group decision making
Zhou, X., He, J., & Yang, C. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107801 · Cited by 43
Artificial Intelligence in Diagnostic Dermatology: Challenges and the Way Forward
Sengupta, D. (2023) · Indian Dermatology Online Journal · doi:10.4103/idoj.idoj_462_23 · Cited by 46
“[24] However, in the recent transformer‑based models, the attention mechanism can itself be interpreted as a form of explainability, though recently, new techniques have been proposed beyond simple attention visualization.”
Characterization of anomalous diffusion classical statistics powered by deep learning (CONDOR)
Gentili, A., & Volpe, G. (2021) · UCL Discovery (University College London) · Cited by 37
Explain and improve: LRP-inference fine-tuning for image captioning models
Sun, J., Lapuschkin, S., Samek, W., & Binder, A. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.07.008 · Cited by 57
“With explanation methods [26], we have a deeper understanding of image captioning models beyond visualizing the attention.”
MethodSelf-citationArtificial Intelligence in Cardiovascular Imaging: “Unexplainable” Legal and Ethical Challenges?
Lang, M., Bernier, A., & Knoppers, B. M. (2022) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2021.10.009 · Cited by 37
Zhang, S., Zhang, S., Yuan, W., & Quek, T. Q. S. (2023) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2023.3240163 · Cited by 35
“The NTNs not only bridge the digital divide in rural areas but also support the Intelligent Transportation System (ITS) in smart cities by providing vehicle-related services, e.g., [5], [6], [7], [8], [9], [10].”
BackgroundExplainable AI reveals Clever Hans effects in unsupervised learning models
Kauffmann, J., Dippel, J., Ruff, L., Samek, W., Müller, K., & Montavon, G. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01000-2 · Cited by 86
“Each score R j can be interpreted as the contribution of the training point u j to the anomaly of x .”
Self-citationOuifak, H., & Idri, A. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130602 · Cited by 48
An uncertainty-aware, shareable, and transparent neural network architecture for brain-age modeling
Hahn, T., Ernsting, J., Winter, N. R., Holstein, V., Leenings, R., Beisemann, M., Fisch, L., Sarink, K., Emden, D., Opel, N., et al. (2022) · Science Advances · doi:10.1126/sciadv.abg9471 · Cited by 47
Deepfakes: evolution and trends
Gil, R., Virgili‐Gomà, J., López, J. M., & García, R. (2023) · Soft Computing · doi:10.1007/s00500-023-08605-y · Cited by 50
“The approach is thus completely agnostic regarding the interpretation of these keywords because they are highly contextual, like in the case of neural networks methods and applications (Samek et al. 2021).”
BackgroundEffects of explainable artificial intelligence in neurology decision support
Gombolay, G. Y., Silva, A., Schrum, M., Gopalan, N., Hallman‐Cooper, J., Dutt, M., & Gombolay, M. (2024) · Annals of Clinical and Translational Neurology · doi:10.1002/acn3.52036 · Cited by 45
Adversarial Attacks in Explainable Machine Learning: A Survey of Threats Against Models and Humans
Vadillo, J., Santana, R., & Lozano, J. A. (2024) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1567 · Cited by 60
Wilms, M., Bannister, J. J., Mouchès, P., MacDonald, M. E., Rajashekar, D., Langner, S., & Forkert, N. D. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3161947 · Cited by 40
“As a fully invertible bidirectional model, our model is also able to estimate an unseen subject’s brain age given an image while being self-explainable (see [1] for definitions).”
BackgroundCXAI: Explaining Convolutional Neural Networks for Medical Imaging Diagnostic
Rguibi, Z., Hajami, A., Zitouni, D., Elqaraoui, A., & Bedraoui, A. (2022) · Electronics · doi:10.3390/electronics11111775 · Cited by 58
“This may be important for two reasons: first, so that the doctor can verify that the prediction is correct; and second, so that the doctor can understand which aspects of the image were most important in making the diagnosis [7].”
BackgroundOpti-CAM: Optimizing saliency maps for interpretability
Zhang, H., Torres, F., Sicre, R., Avrithis, Y., & Ayache, S. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104101 · Cited by 64
Sun, J., Shen, Y., Wang, Y., & Wang, Y. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3539842 · Cited by 37
“With the deepening understanding of the structure of biological neural networks and their corresponding mechanisms, artificial neural networks have witnessed significant advancements [6], [7], [8], [9].”
Patient-level proteomic network prediction by explainable artificial intelligence
Keyl, P., Bockmayr, M. L., Heim, D., Dernbach, G., Montavon, G., Müller, K., & Klauschen, F. (2022) · npj Precision Oncology · doi:10.1038/s41698-022-00278-4 · Cited by 45
Self-citationAn illustration of model agnostic explainability methods applied to environmental data
Wikle, C. K., Datta, A., Hari, B. V., Boone, E. L., Sahoo, I., Indulekha, K., Castruccio, S., Simmons, S. J., Burr, W. S., & Chang, W. (2022) · Environmetrics · doi:10.1002/env.2772 · Cited by 40
Ripple Knowledge Graph Convolutional Networks for Recommendation Systems
Li, C., Cao, Y., Zhu, Y., Cheng, D., Li, C., & Morimoto, Y. (2024) · Machine Intelligence Research · doi:10.1007/s11633-023-1440-x · Cited by 45
“2 Neural Networks in Recommendation Systems Neural networks have been successful in image classification, speech recognition, machine translation, and many other research fields [22].”
BackgroundMumali, F., & Kałkowska, J. (2024) · Computers & Industrial Engineering · doi:10.1016/j.cie.2024.110272 · Cited by 45
Anders, C. J., Neumann, D., Samek, W., Müller, K., & Lapuschkin, S. (2026) · PLoS ONE · doi:10.1371/journal.pone.0336683 · Cited by 79
“…approach used by LRP [9] in a simple and intuitive manner: The provision of an easy to modify and flexible implementation of LRP is paramount for obtaining excellent results, by optimally aligning the method to the characteristics of the model (or parts thereof) to be analyzed [7, 10, 12].”
InfluentialSelf-citationMachine learning approaches for biomolecular, biophysical, and biomaterials research
Rickert, C. A., & Lieleg, O. (2022) · Biophysics Reviews · doi:10.1063/5.0082179 · Cited by 34
Legal Means of Providing the Principle of Transparency of the Artificial Intelligence
Kharitonova, Y. S. (2023) · Journal of Digital Technologies and Law · doi:10.21202/jdtl.2023.14 · Cited by 46
A review of deep learning models to detect malware in Android applications
Mbunge, E., Muchemwa, B., Batani, J., & Mbuyisa, N. (2023) · Cyber Security and Applications · doi:10.1016/j.csa.2023.100014 · Cited by 44
Artificial intelligence for diagnosis of mild–moderate COVID-19 using haematological markers
Chadaga, K., Prabhu, S., Vivekananda Bhat, K., Sampathila, N., Umakanth, S., & Chadaga, R. (2023) · Annals of Medicine · doi:10.1080/07853890.2023.2233541 · Cited by 34
“While Ml needs feature selection, Dl automatically performs feature mining and modelling steps during training. it can also generate new features, handle unstructured data, perform self-learning tasks and support parallel processing. complex irregular patterns can be modelled by DNNs [45]. its main job is to manage us…”
BackgroundHuman-centred artificial intelligence for mobile health sensing: challenges and opportunities
Dang, T., Spathis, D., Ghosh, A., & Mascolo, C. (2023) · Royal Society Open Science · doi:10.1098/rsos.230806 · Cited by 30
“Despite previous attempts to address the challenges surrounding model interpretability, most solutions have been implemented post hoc (such as the attention mechanism) focusing on explanations in certain domains, such as vision [184].”
BackgroundUncertainties in the application of artificial neural networks in ocean engineering
Juan, N. P., Matutano, C., & Valdecantos, V. N. (2023) · Ocean Engineering · doi:10.1016/j.oceaneng.2023.115193 · Cited by 32
Sanchís, J., García-Ponsoda, S., Teruel, M. Á., Trujillo, J. C., & Song, I. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e26028 · Cited by 32
Deep Learning Methods in Soft Robotics: Architectures and Applications
Čakurda, T., Trojanová, M., Pomin, P., & Hošovský, A. (2024) · Advanced Intelligent Systems · doi:10.1002/aisy.202400576 · Cited by 34
Safety Implications of Explainable Artificial Intelligence in End-to-End Autonomous Driving
Atakishiyev, S., Salameh, M., & Goebel, R. G. (2025) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2025.3574738 · Cited by 44
“Meanwhile, recent attempts to explain the behavior of deep neural networks [16] have been a motivation for automotive researchers to present intelligible information to target groups so that relevant people understand the behavior of the black-box system in end-to-end driving.”
Modeling Wildfire Spread with an Irregular Graph Network
Jiang, W., Wang, F., Su, G., Li, X., Wang, G., Zheng, X., Wang, T., & Meng, Q. (2022) · Fire · doi:10.3390/fire5060185 · Cited by 30
B-Cos Alignment for Inherently Interpretable CNNs and Vision Transformers
Böhle, M., Singh, N., Fritz, M., & Schiele, B. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3355155 · Cited by 58
“However, gains in the visual quality of the explanations could come at the cost of their model-faithfulness [6].”
BackgroundHarnessing Ensemble Machine Learning Models for Improved Salinity Prediction in Large Basin Scales
Mahmoud, M. F., Arabi, M., & Pallickara, S. (2024) · Journal of Hydrology · doi:10.1016/j.jhydrol.2025.132691 · Cited by 38
Cheng, C., Aruchunan, E., & Aziz, N. (2025) · Scientific Reports · doi:10.1038/s41598-025-85440-1 · Cited by 37
“Utilising their layered design, DNNs proficiently capture non-linear correlations in extensive datasets 35 , hence enhancing the calibration and forecasting precision of epidemiological models 36 .”
Metadata
- Authors
- Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R.
- Affiliations
- Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (DE)
- BIFOLD✔Berlin Institute for the Foundations of Learning and Data, Berlin, Germany
- Berlin Institute for the Foundations of Learning and Data (DE)
- ORCID
- Venue
- Proceedings of the IEEE
- Published
- 2021-03
- Volume
- 109
- Issue
- 3
- Pages
- 247-278
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W3132191748
- Cited by
- 1,477
- References
- 230
- Citations by year
- 2026: 305 · 2025: 343 · 2024: 326 · 2023: 273 · 2022: 152 · 2021: 74
- FWCI
- 72.25
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Proceedings of the IEEE (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion, other-oa)
- Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korea Government (2017-0-00451, 2019-0-00079)
- German Ministry for Education and Research (01IS14013A-E, 01GQ1115, 01GQ0850, 01IS18025A, 01IS18037A)
- German Research Foundation (Math+, EXC 2046/1, 390685689)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- explainable AI, interpretability, deep neural network, post-hoc explanation, interpretability evaluation, nonlinear machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.