Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Open accessThe DOI matches its Crossref record; 224 of the 230 references it cites were found in a registry or an index.
Abstract
With the broader and highly successful usage of machine learning (ML) in industry and the sciences, there has been a growing demand for explainable artificial intelligence (XAI). Interpretability and explanation methods for gaining a better understanding of the problem-solving abilities and strategies of nonlinear ML, in particular, deep neural networks, are, therefore, receiving increased attention. In this work, we aim to: 1) provide a timely overview of this active emerging field, with a focus on “post hoc” explanations, and explain its theoretical foundations; 2) put interpretability algorithms to a test both from a theory and comparative evaluation perspective using extensive simulations; 3) outline best practice aspects, i.e., how to best include interpretation methods into the standard usage of ML; and 4) demonstrate successful usage of XAI in a representative selection of application scenarios. Finally, we discuss challenges and possible future directions of this exciting foundational field of ML.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“While XAI techniques exist [47], their application in complex, real-time energy systems, and providing meaningful explanations to non-experts, remain challenging”
From Addressing Challenges for the Effective Adoption of Artificial Intelligence in the Energy Sector (2025) · cites this paper as [47]
In the citing paper
“Despite previous attempts to address the challenges surrounding model interpretability, most solutions have been implemented post hoc (such as the attention mechanism) focusing on explanations in certain domains, such as vision [184].”
From Human-centred artificial intelligence for mobile health sensing: challenges and opportunities (2023) · cites this paper as [184]
In the citing paper
“LIME produces an explanation list that shows the contribution of individual features to the prediction.”
From Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2022) · cites this paper as [64]
In the citing paper
“This local explanation allows the end-user to determine which feature is important for the precise prediction and how the perturbation in feature affects the prediction results.”
From Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2022) · cites this paper as [64]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 12×
“This modified gradient view on LRP can also be leveraged to achieve a simpler and more general implementation of LRP based on ‘forward hooks’, which we describe in the second part of Appendix B, and which we use to apply LRP on VGG-16 [175] and ResNet-50 [65] in Section IV.”
BackgroundMethodInfluentialVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 6×
“This modified gradient view on LRP can also be leveraged to achieve a simpler and more general implementation of LRP based on ‘forward hooks’, which we describe in the second part of Appendix B, and which we use to apply LRP on VGG-16 [175] and ResNet-50 [65] in Section IV.”
BackgroundMethodInfluentialImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 4×
“Consequently, we also exchange the neural network architecture and use an AlexNet [99] model, which is able to process the transformed input data using 2-D-convolution operators.”
BackgroundMethodInfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“, [23], [52], [70], [107], [108], and [161]) have gained highest popularity.”
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 9×
“A comparison of performance on the Adience benchmark of the original model (pretrained on ImageNet only) and the improved model is given in the table below.”
MethodInfluentialRegression Models and Life-Tables
Cox, D. R. (1972) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1972.tb00899.x · Cited by 39,793
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 8×
“A comparison of performance on the Adience benchmark of the original model (pretrained on ImageNet only) and the improved model is given in the table below.”
BackgroundInfluentialThe Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 1×
“In games such as the arcade game Atari Breakout, the computer player would progressively learn strategies commonly employed by human players, such as “tunneldigging” [124], [200].”
Background
Research works printed but not linked: 13
- Arras, 2020, Ground truth evaluation of neural network explanations with CLEVR-XAI, arXiv:2003.07258 · arXiv
- 10.1146/annurev.neuro.26.041002.131047 · DOI · Paper page
- Chollet, 2015, Keras
- 10.1109/ICCV.2019.00304 · DOI
- Kokhlikyan, 2019, Pytorch Captum
- Lapuschkin, 2016, The layer-wise relevance propagation toolbox for artificial neural networks, J. Mach. Learn. Res., 17, 1
- 10.48550/arXiv.1312.4400 · DOI · Paper page
- 10.5555/2969033.2969153 · doi:10.5555/2969033.2969153
- 10.5555/3157382.3157477 · doi:10.5555/3157382.3157477
- Okuta, CuPy: A numpy-compatible library for NVIDIA GPU calculations, 151
- Oliphant, 2006, A Guide to NumPy, 1
- 10.48550/ARXIV.1404.7828 · DOI
- 10.5555/3045118.3045336 · doi:10.5555/3045118.3045336
Cited by
Items 201–300 of 1,002
NoiseGrad — Enhancing Explanations by Introducing Stochasticity to Model Weights
Bykov, K., Hedström, A., Nakajima, S., & Höhne, M. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i6.20561 · Cited by 53
Glassboxing Deep Learning to Enhance Aircraft Detection from SAR Imagery
Luo, R., Jin, X., Chen, L., Pan, Z., Cai, X., Li, Z., Wang, J., & Ford, A. (2021) · Remote Sensing · doi:10.3390/rs13183650 · Cited by 32
“Another challenge lies in the customization of XAI techniques for object detection in SAR image analytics, because most of these techniques have been designed for classification [20].”
MethodExplaining the Predictions of Unsupervised Learning Models
Montavon, G., Kauffmann, J., Samek, W., & Müller, K. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-04083-2_7 · Cited by 41
“The runtime advantage of LRP facilitates explanation of large models and datasets (e.g. GPU implementations of LRP can achieve hundreds of image classification explanations per second [1, 40]).”
BackgroundMethodSelf-citationEvaluating Surprise Adequacy for Deep Learning System Testing
Kim, J.-H., FELDT, R. H., & Yoo, S. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3546947 · Cited by 55
Farrag, A., Gad, G., Fadlullah, Z. M., Fouda, M. M., & Alsabaan, M. (2023) · IEEE Access · doi:10.1109/access.2023.3330465 · Cited by 37
“Subsequently, we conduct a quantitative assessment of all the explanatory methods by assessing image entropy and constructing pixel-flipping graphs, following a similar approach as seen in prior research [38], [56], [57].”
MethodA novel deep learning model to predict the soil nutrient levels (N, P, and K) in cabbage cultivation
Sajindra, H., Abekoon, T., Jayakody, A., & Rathnayake, U. (2023) · Smart Agricultural Technology · doi:10.1016/j.atech.2023.100395 · Cited by 43
Wei, L., Li, Q., Song, Y., Stefanov, S., Dong, R., Fu, N., Siriwardane, E. M. D., Chen, F., & Hu, J. (2024) · Advanced Science · doi:10.1002/advs.202304305 · Cited by 49
Sen, B., Saha, S., Kumar, R., Jhala, R., Patil, N., Mahapatro, A., Bhowmik, A., & Santhosh, A. (2025) · Results in Engineering · doi:10.1016/j.rineng.2025.104457 · Cited by 28
A semantic analysis-driven customer requirements mining method for product conceptual design
Wu, X., Hong, Z., Feng, Y., Li, M.-D., Lou, S., & Tan, J. (2022) · Scientific Reports · doi:10.1038/s41598-022-14396-3 · Cited by 39
“Research on the interpretability of deep neural networks shows that the shallow layers of the networks extract general features, while the deep layers of networks extract high-level semantic information related to specific tasks 39 .”
BackgroundMethodInfluentialExplainable Artificial Intelligence: An Updated Perspective
Krajna, A., Kovač, M., Brčić, M., & Šarčević, A. (2022) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.23919/mipro55190.2022.9803681 · Cited by 36
Messner, W. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110729 · Cited by 35
Dreyer, M., Achtibat, R., Wiegand, T., Samek, W., & Lapuschkin, S. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00397 · Cited by 45
“In order to ensure robust decompositions and stable heatmaps, several purposed LRP rules have been introduced in literature [43, 52].”
BackgroundMethodInfluentialSelf-citationExploration of materials fatigue influence factors using interpretable machine learning
Frie, C., Durmaz, A. R., & Eberl, C. (2024) · Fatigue & Fracture of Engineering Materials & Structures · doi:10.1111/ffe.14315 · Cited by 34
Poly-cam: high resolution class activation map for convolutional neural networks
Englebert, A., Cornu, O., & Vleeschouwer, C. D. (2024) · Machine Vision and Applications · doi:10.1007/s00138-024-01567-7 · Cited by 38
“A detailed review of the approaches that have been investigated to explain a CNN is provided in [6].”
Explainable AI for Sensor Signal Interpretation to Revolutionize Human Health Monitoring: A Review
Alharthi, A. S., Alqurashi, A., Alharbi, T. E. A., Alammar, M. M., Aldosari, N., Bouchekara, H. R. E. H., Sha’aban, Y. A., Shahriar, M. S., & Ayidh, A. A. (2025) · IEEE Access · doi:10.1109/access.2025.3585764 · Cited by 44
“…in the previous layer which is received from R k , for the k th neuron in the lower layer, where the relevance conservation principle is satisfied [30] as : Equation (9) shows that total relevance is conserved across layers, ensuring the sum of relevance scores equals the model’s final output for…”
An Analysis Method for Interpretability of Convolutional Neural Network in Bearing Fault Diagnosis
Guo, L., Gu, X., Yu, Y., Duan, A., & Gao, H. (2023) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2023.3334350 · Cited by 30
“Early in the development of deep learning methods, fully connected layer-based or other approximate structures of classifiers generally are characterized by large weight links and parametric quantities [23].”
BackgroundFang, J., Guo, X., Liu, Y., Chang, X., Fujita, H., & Wu, J. (2023) · Computers & Industrial Engineering · doi:10.1016/j.cie.2023.109667 · Cited by 27
3D microstructural generation from 2D images of cement paste using generative adversarial networks
Zhao, X. W., Wang, L., Li, Q., Chen, H., Liu, S., Hou, P., Ye, J., Pei, Y., Wu, X., Yuan, J., et al. (2024) · Cement and Concrete Research · doi:10.1016/j.cemconres.2024.107726 · Cited by 29
“Recently, deep learning has undergone significant advancements, demonstrating potent learning capabilities essential for capturing complex information [26, 27].”
Scrutinizing XAI using linear ground-truth data with suppressor variables
Wilming, R., Budding, C., Müller, K., & Haufe, S. (2022) · Machine Learning · doi:10.1007/s10994-022-06167-y · Cited by 49
“Lundberg and Lee, 2017), which are based on a model’s output only, and methods that are tailored to a specific class of models (e.g., neural networks, Zeiler and Fergus, 2014; Springenberg et al., 2015; Bach et al., 2015; Binder et al., 2016; Montavon et al., 2017; Kim et al., 2018; Montavon et al., 2018; Samek et al.…”
MethodHerwig, N., Peng, Z., & Borghesani, P. (2023) · Tribology International · doi:10.1016/j.triboint.2023.108670 · Cited by 28
Schreiber, A., & Schreiber, I. (2024) · Information and Computer Security · doi:10.1108/ics-10-2023-0199 · Cited by 43
Wilczok, D. (2025) · Aging · doi:10.18632/aging.206190 · Cited by 39
Wang, B., Xu, T., Zheng, B., Kai, Y., & Zhang, K. (2025) · eTransportation · doi:10.1016/j.etran.2025.100477 · Cited by 28
Interpretable generative deep learning: an illustration with single cell gene expression data
Treppner, M., Binder, H., & Hess, M. (2022) · Human Genetics · doi:10.1007/s00439-021-02417-6 · Cited by 33
“It is possible to employ the best performing approach and only in the second step care about the interpretations (Samek et al. 2021).”
BackgroundImportance of Artificial Intelligence in Neural Network through using MediaPipe
Mohamed, N. A. (2022) · International Conference on Electronics, Communication and Aerospace Technology (ICECA) · doi:10.1109/iceca55336.2022.10009513 · Cited by 26
“Convolut ional neural networks are a type o f neural network that is part icu larly well-suited fo r working with images [4].”
BackgroundChung, S., & Zhang, Y. (2023) · Applied Sciences · doi:10.3390/app13126937 · Cited by 32
Alqadhi, S. D., Mallick, J., Alkahtani, M. Q., Ahmad, I., Alqahtani, D. A., & Hằng, H. T. (2023) · Natural Hazards · doi:10.1007/s11069-023-06357-4 · Cited by 37
“Drawing inspiration from the intricacies of the human brain, they demonstrate exceptional proficiency in discerning complex patterns within datasets (Samek et al. 2021).”
A survey of post-hoc xai methods from a visualization perspective: Challenges and opportunities
Bhati, D., Amiruzzaman, M., Zhao, Y., Guercio, A., & Le, T. (2025) · IEEE Access · doi:10.1109/access.2025.3581136 · Cited by 24
“Samek et al. [25], [26] have made significant contributions to understanding XAI in image classification, particularly through their work on saliency maps and the development of post-hoc interpretation methods.”
On clustering and interpreting with rules by means of mathematical optimization
Carrizosa, E., Kurishchenko, K., Marı́n, A., & Morales, D. R. (2023) · Computers & Operations Research · doi:10.1016/j.cor.2023.106180 · Cited by 25
I-FLASH: Interpretable Fake News Detector Using LIME and SHAP
Dua, V., Rajpal, A., Rajpal, S., Agarwal, M., & Kumar, N. (2023) · Wireless Personal Communications · doi:10.1007/s11277-023-10582-2 · Cited by 33
“As a result, we no longer need to sacrifice predictability for interpretability, and we can use strong nonlinear ML to its full potential in real-world applications [52].”
BackgroundLi, Z., Yu, J., Wang, C., Bello, I. T., Yu, N., Chen, X., Zheng, K., Han, M., & Ni, M. (2024) · Applied Energy · doi:10.1016/j.apenergy.2024.123236 · Cited by 28
Artificial intelligence in bioinformatics: a survey
Jiang, J., Li, Y., Cao, S., Shan, Y., Liu, Y., Fei, T., Yu, Y., Feng, Y., Li, Y., Li, Y., et al. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf576 · Cited by 33
Gómez, P. M., Carrillo, O. J. J., Kuffer, M., Thomson, D. R., Quiroz, J. L. O., Villaseñor, E., Vanhuysse, S. G., Abascal, Á., Oluoch, I., Nagenborg, M., et al. (2021) · Sustainability · doi:10.3390/su132212640 · Cited by 27
“In addition to uncertainty quantification, there is a growing interest in making machine learning and deep learning models more interpretable and understandable, aiming at neural networks that provide understandable justifications for their output, leading to insights about the inner workings [89,90].”
BackgroundCausal deep learning models for studying the Earth system
Tesch, T., Kollet, S. J., & Garcke, J. (2023) · Geoscientific model development · doi:10.5194/gmd-16-2149-2023 · Cited by 32
“Molnar, 2019; Samek et al., 2021) to analyze the relations learned by the models (Roscher et al., 2020) The approach has been applied in several recent studies (Ham et al., 2019; Gagne et al., 2019; McGovern et al., 2019; Toms et al., 2020; Ebert-Uphoff and Hilburn, 2020; Padarian et al., 2020),…”
MethodSoK: Explainable Machine Learning in Adversarial Environments
Noppel, M., & Wressnegger, C. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00021 · Cited by 50
Deep Feature Selection Framework for Quality Prediction in Injection Molding Process
Kim, I., Na, J., Yun, J. P., & Lee, S. (2023) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2023.3268421 · Cited by 25
“Existing XAI-based applications, such as network pruning [20] and pixel-flipping [21], are important in terms of how well they interpret the optimized deep network; however, feature selection necessitates interpreting the overfitted deep network to eliminate unnecessary features.”
MethodA Review on Artificial Neural Networks for Structural Analysis
Saini, R. (2025) · Journal of Vibration Engineering & Technologies · doi:10.1007/s42417-024-01749-7 · Cited by 33
“As discussed by Samek et al. [15] and Behler [154], ANN performance depends largely on the availability of data sets that properly represent the complexities of real-world structures.”
Towards a Better Understanding of Evaluating Trustworthiness in AI Systems
Kemmerzell, N., Schreiner, A., Khalid, H., Schalk, M., & Bordoli, L. (2025) · ACM Computing Surveys · doi:10.1145/3721976 · Cited by 45
Human Activity Recognition Based on Acceleration Data From Smartphones Using HMMs
Iloga, S., Bordat, A., Kernec, J. L., & Romain, O. (2021) · IEEE Access · doi:10.1109/access.2021.3117336 · Cited by 26
“More details related to this limitation are available in [89] where an overview of explainable arti-ficial intelligence for deep neural networks is proposed.”
BackgroundSchuhmacher, D., Schörner, S., Küpper, C., Großerueschkamp, F., Sternemann, C., Lugnier, C., Kraeft, A., Jütte, H., Tannapfel, A., Reinacher‐Schick, A., et al. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102594 · Cited by 26
Optimization of U-shaped flow channel by RBFNN and NSGA-II
Li, M., Liu, X., Kong, D., Guo, Q., Wang, Y., & Yao, J. Y. (2022) · International Journal of Mechanical Sciences · doi:10.1016/j.ijmecsci.2022.108012 · Cited by 23
The Role of Artificial Intelligence in Eliminating Accounting Errors
Najjar, M. A., Ghanem, M. G., Mahboub, R., & Nakhal, B. (2024) · Journal of risk and financial management · doi:10.3390/jrfm17080353 · Cited by 42
“These tools require sophisticated algorithms that extend beyond linear relationships, enabling users to surpass traditional analytical abilities and enhance tax reporting quality (Samek et al. 2021).”
BackgroundLi, X., Ye, C., Lu, K., Xue, C., Li, X., & Zhang, Y. (2024) · Environmental Science & Technology · doi:10.1021/acs.est.4c02221 · Cited by 26
Incremental Residual Concept Bottleneck Models
Shang, C., Zhou, S., Zhang, H., Ni, X., Yang, Y., & Wang, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01049 · Cited by 78
“Deep neural networks (DNNs) [31, 45] have achieved unprecedented success in a wide range of machine learning tasks, including computer vision [16, 32], natural lan-* Figure 1.”
BackgroundAn organic artificial synaptic memristor for neuromorphic computing
Gao, K., Sun, B. T., Yang, B., Cao, Z., Cui, Y., Wang, M., Kong, C., Zhou, G., Luo, S., Chen, X., et al. (2025) · Applied Materials Today · doi:10.1016/j.apmt.2025.102628 · Cited by 27
Do Transformer Models Show Similar Attention Patterns to Task-Specific Human Gaze?
Eberle, O., Brandl, S., Pilot, J., & Søgaard, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.296 · Cited by 56
Reveal to Revise: An Explainable AI Life Cycle for Iterative Bias Correction of Deep Models
Pahde, F., Dreyer, M., Samek, W., & Lapuschkin, S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43895-0_56 · Cited by 40
Self-citationThe Importance of Distrust in AI
Peters, T. M., & Visser, R. W. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-44070-0_15 · Cited by 44
Hybrid Chaotic Zebra Optimization Algorithm and Long Short-Term Memory for Cyber Threats Detection
Amin, R., Eltaweel, G., Ali, A. F., & Tahoun, M. (2024) · IEEE Access · doi:10.1109/access.2024.3397303 · Cited by 22
“Deep learning algorithms operate on high-dimensional datasets [5], which are significantly impacted by extensive model development processes, redundant data, and decreased performance, all of which make data analysis extremely challenging [6], [7].”
BackgroundExplainability of Vision Transformers: a Comprehensive Review and New PerspectivesPreprint
Kashefi, R., Barekatain, L., Sabokrou, M., & Aghaeipoor, F. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5055345 · Cited by 27
Binder, A., Weber, L., Lapuschkin, S., Montavon, G., Müller, K.-R., & Samek, W. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01549 · Cited by 21
Self-citationA review of machine learning in scanpath analysis for passive gaze-based interaction
Selim, A. M., Barz, M., Bhatti, O. S., Alam, H., & Sonntag, D. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1391745 · Cited by 34
“They mostly work with visual inputs and computer vision tasks to generate heatmaps, i.e., activation maps, that highlight the most important regions in the input image that the model used to make its prediction (Samek et al., 2021).”
BackgroundMethodWang, S., Li, Z., Chen, G., & Yue, Y. (2024) · Scientific Reports · doi:10.1038/s41598-024-71989-w · Cited by 24
ML meets aerospace: challenges of certifying airborne AI
Luettig, B., Akhiat, Y., & Daw, Z. (2024) · Frontiers in Aerospace Engineering · doi:10.3389/fpace.2024.1475139 · Cited by 35
“…in medical assessment because they are crucial for ensuring that AI-driven medical assessments are not only accurate but also transparent and interpretable by doctors and end users, paving the way for more widespread adoption of AI technologies in healthcare (Loh et al., 2022; Samek et al., 2021).”
Chen, J., Wang, R., Chen, Z., Lin, X., Takaoka, M., Li, X., & Yan, J. (2025) · Chemical Engineering Journal · doi:10.1016/j.cej.2025.161959 · Cited by 21
Explainable Artificial Intelligence for State of Charge Estimation of Lithium-Ion Batteries
Chan, H. T. J., Rubeša-Zrim, J., Pichler, F., Salihi, A., Mourad, A., Šimić, I., Časni, K., & Veas, E. E. (2025) · Applied Sciences · doi:10.3390/app15095078 · Cited by 29
Large language models for bioinformatics
Ruan, W., Lyu, Y., Zhang, J., Cai, J., Shu, P., Yang, G., Lu, Y., Gao, S., Wang, Y., Wang, P., et al. (2025) · Quantitative Biology · doi:10.1002/qub2.70014 · Cited by 48
Interpreting mental state decoding with deep learning models
Thomas, A. W., Ré, C., & Poldrack, R. A. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.07.003 · Cited by 30
Research on Traffic Congestion Forecast Based on Deep Learning
Qi, Y., & Cheng, Z. (2023) · Information · doi:10.3390/info14020108 · Cited by 28
Samir, M., Sherief, N. H., & Abdelmoez, W. (2023) · Computers · doi:10.3390/computers12070128 · Cited by 27
Understanding neural network through neuron level visualization
Dou, H., Shen, F., Zhao, J., & Mu, X. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.09.030 · Cited by 30
Insights into the inner workings of transformer models for protein function prediction
Wenzel, M. A., Grüner, E., & Strodthoff, N. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae031 · Cited by 27
“…field of explainable machine learning (Lundberg and Lee 2017, Montavon et al. 2018, Arrieta et al. 2020, Tjoa and Guan 2020, Covert et al. 2021, Samek et al. 2021) aims at developing methods that enable humans to better in-terpret—or to a limited degree: understand—such “opaque,” complex…”
BackgroundRule-Based Detection of Anomalous Patterns in Device Behavior for Explainable IoT Security
Costa, G., Forestiero, A., & Ortale, R. (2023) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2023.3327822 · Cited by 24
“Explainable AI [11], [40], a sub-field of ML, aims to enhance the training, learning and representation processes providing Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
BackgroundYang, H., Yang, S., Meng, D., Hu, C., Wu, C., Yang, B., Nie, P., Si, Y., & Su, X. (2024) · Mathematics · doi:10.3390/math13010121 · Cited by 25
“Compared to Ant Colony Optimization [44] and Particle Swarm Optimization [45], the WOA demonstrates better convergence capability and optimization speed.”
Chen, H., & Martel, A. L. (2025) · Journal of medical imaging · doi:10.1117/1.jmi.12.s2.s22007 · Cited by 28
Explainable Artificial Intelligence: A systematic Review of Progress and Challenges
Mohamed, A., Abdelqader, K., & Shaalan, K. F. (2025) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2025.200595 · Cited by 65
Xu, X., & Feng-hu, L. (2021) · Computational Intelligence and Neuroscience · doi:10.1155/2021/8785127 · Cited by 20
RetractedHigh-Quality Video Watermarking Based on Deep Neural Networks for Video with HEVC Compression
Kaczyński, M., Piotrowski, Z., & Pietrow, D. (2022) · Sensors · doi:10.3390/s22197552 · Cited by 20
Feature fusion based artificial neural network model for disease detection of bean leaves
Önler, E. (2023) · Electronic Research Archive · doi:10.3934/era.2023122 · Cited by 24
Yang, G. G., Wang, Q., Feng, J., He, L., Li, R., Lu, W., Liao, E., & Lai, Z. (2024) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2024.171365 · Cited by 23
Interpretable Object Recognition by Semantic Prototype Analysis
Wan, Q., Wang, R., & Chen, X. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00085 · Cited by 31
“According to the purpose of interpretable methods, they can be grouped into two types typically [16, 27].”
BackgroundMethodAI-Based Anomaly Detection for Clinical-Grade Histopathological Diagnostics
Dippel, J., Prenißl, N., Hense, J., Liznerski, P., Winterhoff, T., Schallenberg, S., Kloft, M., Buchstab, O., Horst, D. A., Alber, M., et al. (2024) · NEJM AI · doi:10.1056/aioa2400468 · Cited by 31
Analyzing Atomic Interactions in Molecules as Learned by Neural Networks
Esders, M., Schnake, T., Lederer, J., Kabylda, A., Montavon, G., Tkatchenko, A., & Müller, K. R. (2025) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.4c01424 · Cited by 33
“Moreover, XAI methods have been used to debug models, to gain novel insights and to capture whether or not suspected/expected structures or knowledge are embodied in the respective ML architectures (see e.g. [19]).”
Self-citationIdentifying Microscopic Augmented Images using Pre-Trained Deep Convolutional Neural Networks
Tiwari, R. G., Misra, A., Khullar, V., Agarwal, A. K., Gupta, S., & Srivastava, A. P. (2021) · International Conference on Technological Advancements and Innovations (ICTAI) · doi:10.1109/ictai53825.2021.9673472 · Cited by 24
Kacprzak, T., & Fluri, J. (2022) · Physical Review X · doi:10.1103/physrevx.12.031029 · Cited by 29
Review on Epilepsy Detection with Explainable Artificial Intelligence
Rathod, P., & Naik, S. T. (2022) · International Conference on Emerging Trends in Engineering and Technology (ICETET) · doi:10.1109/icetet-sip-2254415.2022.9791595 · Cited by 21
Rajapaksha, D., & Bergmeir, C. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21469 · Cited by 27
Explainable online ensemble of deep neural network pruning for time series forecasting
Saadallah, A., Jakobs, M., & Morik, K. J. (2022) · Machine Learning · doi:10.1007/s10994-022-06218-4 · Cited by 27
“A wide variety of explanation methods have been presented in the literature (Camburu, 2020; Samek et al., 2021; Molnar, 2020).”
This Looks Like That There: Interpretable Neural Networks for Image Tasks When Location Matters
Barnes, E. A., Barnes, R. J., Martin, Z., & Rader, J. K. (2022) · Artificial Intelligence for the Earth Systems · doi:10.1175/aies-d-22-0001.1 · Cited by 30
Luo, J., Lan, L., Huang, S., Zeng, X., Qu, X., Li, M., Yang, S., Zhao, W., & Zhou, X. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104310 · Cited by 19
Explainable deep learning: concepts, methods, and new developments
Samek, W. (2023) · Elsevier eBooks · doi:10.1016/b978-0-32-396098-4.00008-9 · Cited by 27
Self-citationOn computing probabilistic abductive explanations
Izza, Y., Huang, X., Ignatiev, A., Narodytska, N., Cooper, M., & Marques-Silva, J. P. (2023) · International Journal of Approximate Reasoning · doi:10.1016/j.ijar.2023.108939 · Cited by 55
Saba, L., Maindarkar, M. A., Khanna, N. N., Johri, A. M., Mantella, L. E., Laird, J. R., Paraskevas, K. I., Ruzsa, Z., Kalra, M. K., Fernandes, J. F. E., et al. (2023) · Frontiers in Bioscience-Landmark · doi:10.31083/j.fbl2810248 · Cited by 21
Disentangled Explanations of Neural Network Predictions by Finding Relevant Subspaces
Chormai, P., Herrmann, J., Müller, K., & Montavon, G. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3388275 · Cited by 58
“Explainable AI has been shown to be a promising approach to extract insights in the data and in the systems or processes that generates this data [2], [6].”
BackgroundSelf-citationData-driven AI algorithms for construction machinery
Liang, K., Zhao, J., Zhang, Z., Guan, W., Pan, M., & Li, M. (2024) · Automation in Construction · doi:10.1016/j.autcon.2024.105648 · Cited by 26
Evaluating Visual Explanations of Attention Maps for Transformer-Based Medical Imaging
Chung, M., Won, J. B., Kim, G., Kim, Y., & Özbulak, U. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-77610-6_11 · Cited by 33
“To enhance the trustworthiness of predictions made by ViTs and to address the aforementioned interpretability challenges, numerous methods have been proposed in the literature [11,24].”
Deep Learning Techniques for Prostate Cancer Analysis and Detection: Survey of the State of the Art
Olawuyi, O., & Viriri, S. (2025) · Journal of Imaging · doi:10.3390/jimaging11080254 · Cited by 22
AI-powered spatial cell phenomics enhances risk stratification in non-small cell lung cancer
Schallenberg, S., Dernbach, G., Ruane, S., Jurmeister, P. S., Böhm, C., Standvoss, K., Ghosh, S., Frentsch, M., Dragomir, M. P., Keyl, P., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-65783-z · Cited by 25
“The lymphocyte, which typically has a minimal diameter of 8 µ m, was used as the lower limit 87,89 .”
Multi-scale digital twins for personalized medicine
Vallée, A. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1753906 · Cited by 24
“Techniques such as counterfactual explanations, SHAP (SHapley Additive exPlanations) values, and interpretable surrogate models should be systematically embedded within MSDT architectures (32).”
Machine Learning Analysis for Phenolic Compound Monitoring Using a Mobile Phone-Based ECL Sensor
Taylor, J. W., Rivera, E. C., Kim, S., Campbell, R., Summerscales, R., & Kwon, H. J. (2021) · Sensors · doi:10.3390/s21186004 · Cited by 18
“The number of neurons of the hidden layer are key parameters which affect the performance of the NN significantly [18].”
BackgroundExplainable nucleus classification using Decision Tree Approximation of Learned Embeddings
Amgad, M. T., Atteya, L. A., Hussein, H., Mohammed, K. H., Hafiz, E. O. A., Elsebaie, M. A., Mobadersany, P., Manthey, D., Gutman, D. A., Elfandy, H., et al. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab670 · Cited by 20
Tian, J., Li, S., Liu, X., & Wang, P. (2022) · Energy Reports · doi:10.1016/j.egyr.2022.05.127 · Cited by 16
DENSITY RESULTS BY DEEP NEURAL NETWORK OPERATORS WITH INTEGER WEIGHTS
Costarellı, D. (2022) · Mathematical Modelling and Analysis · doi:10.3846/mma.2022.15974 · Cited by 22
“In recent years, the theory of multi-layer NNs has been deeply studied, see, e.g., [34], in view of its wide importance in both theoretical and applied fields.”
BackgroundMachine learning algorithm for the shear strength prediction of basalt-driven lateritic soil
Niyogi, A., Ansari, T. A., Sathapathy, S. K., Sarkar, K., & Singh, R. (2023) · Earth Science Informatics · doi:10.1007/s12145-023-00950-8 · Cited by 22
“(Hinton 2012; Samek et al. 2021).”
A spatial correlation prediction model of urban PM2.5 concentration based on deconvolution and LSTM
Zhang, B., Liu, Y., Yong, R., Zou, G., Yang, R., Pan, J., & Li, M. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126280 · Cited by 20
Rahul, M. R., & Chiddarwar, S. S. (2023) · International Journal of Precision Engineering and Manufacturing · doi:10.1007/s12541-023-00875-8 · Cited by 21
“However, deep learning algorithms have emerged as a promising approach to overcoming the difficulties associated with using vision systems for burr detection [31].”
BackgroundExplainability and transparency in the realm of digital humanities: toward a historian XAI
El-Hajj, H., Eberle, O., Merklein, A., Siebold, A., Shlomi, N., Büttner, J., Martinetz, J., Müller, K., Montavon, G., & Valleriani, M. (2023) · International Journal of Digital Humanities · doi:10.1007/s42803-023-00070-1 · Cited by 28
“The above represent a small subset of the fields where XAI has had an impact on model validation and insight generation; for a more detailed review of XAI methods and applications see Samek et al. (2021); Samek (2023).”
InfluentialSelf-citationPrincipled Limitations on Self-Representation for Generic Physical Systems
Fields, C., Glazebrook, J. F., & Levin, M. G. (2024) · Entropy · doi:10.3390/e26030194 · Cited by 27
“These restrictions have clear relevance for the explanation problem [76,77] that besets ANN designers and users, particularly designers and users of multi-layer deep learning systems.”
BackgroundSocial-aware graph contrastive learning for recommender systems
Zhang, Y., Zhu, J., Zhang, Y., Zhu, Y. Q., Zhou, J., & Xie, Y. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.111558 · Cited by 23
Non-technological barriers: the last frontier towards AI-powered intelligent optical networks
Khan, F. N. (2024) · Nature Communications · doi:10.1038/s41467-024-50307-y · Cited by 27
“Recent progresses in explainable AI (XAI) research 56,70 and data debiasing techniques 71,72 will particularly be helpful in addressing these two challenges, respectively.”
Background
Metadata
- Authors
- Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R.
- Affiliations
- Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (DE)
- BIFOLD✔Berlin Institute for the Foundations of Learning and Data, Berlin, Germany
- Berlin Institute for the Foundations of Learning and Data (DE)
- ORCID
- Venue
- Proceedings of the IEEE
- Published
- 2021-03
- Volume
- 109
- Issue
- 3
- Pages
- 247-278
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W3132191748
- Cited by
- 1,477
- References
- 230
- Citations by year
- 2026: 305 · 2025: 343 · 2024: 326 · 2023: 273 · 2022: 152 · 2021: 74
- FWCI
- 72.25
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Proceedings of the IEEE (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion, other-oa)
- Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korea Government (2017-0-00451, 2019-0-00079)
- German Ministry for Education and Research (01IS14013A-E, 01GQ1115, 01GQ0850, 01IS18025A, 01IS18037A)
- German Research Foundation (Math+, EXC 2046/1, 390685689)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- explainable AI, interpretability, deep neural network, post-hoc explanation, interpretability evaluation, nonlinear machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.