Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Open accessThe DOI matches its Crossref record; 224 of the 230 references it cites were found in a registry or an index.
Abstract
With the broader and highly successful usage of machine learning (ML) in industry and the sciences, there has been a growing demand for explainable artificial intelligence (XAI). Interpretability and explanation methods for gaining a better understanding of the problem-solving abilities and strategies of nonlinear ML, in particular, deep neural networks, are, therefore, receiving increased attention. In this work, we aim to: 1) provide a timely overview of this active emerging field, with a focus on “post hoc” explanations, and explain its theoretical foundations; 2) put interpretability algorithms to a test both from a theory and comparative evaluation perspective using extensive simulations; 3) outline best practice aspects, i.e., how to best include interpretation methods into the standard usage of ML; and 4) demonstrate successful usage of XAI in a representative selection of application scenarios. Finally, we discuss challenges and possible future directions of this exciting foundational field of ML.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“While XAI techniques exist [47], their application in complex, real-time energy systems, and providing meaningful explanations to non-experts, remain challenging”
From Addressing Challenges for the Effective Adoption of Artificial Intelligence in the Energy Sector (2025) · cites this paper as [47]
In the citing paper
“Despite previous attempts to address the challenges surrounding model interpretability, most solutions have been implemented post hoc (such as the attention mechanism) focusing on explanations in certain domains, such as vision [184].”
From Human-centred artificial intelligence for mobile health sensing: challenges and opportunities (2023) · cites this paper as [184]
In the citing paper
“LIME produces an explanation list that shows the contribution of individual features to the prediction.”
From Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2022) · cites this paper as [64]
In the citing paper
“This local explanation allows the end-user to determine which feature is important for the precise prediction and how the perturbation in feature affects the prediction results.”
From Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2022) · cites this paper as [64]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 12×
“This modified gradient view on LRP can also be leveraged to achieve a simpler and more general implementation of LRP based on ‘forward hooks’, which we describe in the second part of Appendix B, and which we use to apply LRP on VGG-16 [175] and ResNet-50 [65] in Section IV.”
BackgroundMethodInfluentialVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 6×
“This modified gradient view on LRP can also be leveraged to achieve a simpler and more general implementation of LRP based on ‘forward hooks’, which we describe in the second part of Appendix B, and which we use to apply LRP on VGG-16 [175] and ResNet-50 [65] in Section IV.”
BackgroundMethodInfluentialImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 4×
“Consequently, we also exchange the neural network architecture and use an AlexNet [99] model, which is able to process the transformed input data using 2-D-convolution operators.”
BackgroundMethodInfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“, [23], [52], [70], [107], [108], and [161]) have gained highest popularity.”
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 9×
“A comparison of performance on the Adience benchmark of the original model (pretrained on ImageNet only) and the improved model is given in the table below.”
MethodInfluentialRegression Models and Life-Tables
Cox, D. R. (1972) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1972.tb00899.x · Cited by 39,793
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 8×
“A comparison of performance on the Adience benchmark of the original model (pretrained on ImageNet only) and the improved model is given in the table below.”
BackgroundInfluentialThe Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 1×
“In games such as the arcade game Atari Breakout, the computer player would progressively learn strategies commonly employed by human players, such as “tunneldigging” [124], [200].”
Background
Research works printed but not linked: 13
- Arras, 2020, Ground truth evaluation of neural network explanations with CLEVR-XAI, arXiv:2003.07258 · arXiv
- 10.1146/annurev.neuro.26.041002.131047 · DOI · Paper page
- Chollet, 2015, Keras
- 10.1109/ICCV.2019.00304 · DOI
- Kokhlikyan, 2019, Pytorch Captum
- Lapuschkin, 2016, The layer-wise relevance propagation toolbox for artificial neural networks, J. Mach. Learn. Res., 17, 1
- 10.48550/arXiv.1312.4400 · DOI · Paper page
- 10.5555/2969033.2969153 · doi:10.5555/2969033.2969153
- 10.5555/3157382.3157477 · doi:10.5555/3157382.3157477
- Okuta, CuPy: A numpy-compatible library for NVIDIA GPU calculations, 151
- Oliphant, 2006, A Guide to NumPy, 1
- 10.48550/ARXIV.1404.7828 · DOI
- 10.5555/3045118.3045336 · doi:10.5555/3045118.3045336
Cited by
Items 501–600 of 1,002
Santra, S., & Behera, R. (2025) · Communications in Nonlinear Science and Numerical Simulation · doi:10.1016/j.cnsns.2025.109184 · Cited by 8
Enhanced deep Convolutional Neural Network for SARS-CoV-2 variants classification
Awe, O. I., Obura, H. O., Charles, S., Mudibo, E. O., & Mwanga, M. J. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1512003 · Cited by 31
“Reported saliency results were averaged over multiple runs to ensure robustness and reduce stochastic artifacts (Samek et al., 2021).”
Error-controlled non-additive interaction discovery in machine learning models
Chen, W., Jiang, Y., Noble, W. S., & Lu, Y. Y. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01086-8 · Cited by 12
“While the complexity of ML models contributes to their power, it also makes them challenging to interpret, leaving users with few clues about the mechanisms underlying a given model’s outputs.”
Karkadakattil, A. (2025) · Australian Journal of Multi-Disciplinary Engineering · doi:10.1080/14488388.2025.2570030 · Cited by 8
“Such surface conditions can reduce fatigue life, compromise dimensional accuracy, and limit osseointegration in biomedical implants (Samek et al. 2021; Zhang et al. 2020; Zhou et al. 2019).”
From prediction to intervention: causal digital twins for personalized clinical decision support
Vallée, A. (2026) · Journal of Translational Medicine · doi:10.1186/s12967-026-07895-8 · Cited by 19
“In parallel, interpretability is a critical component in the deployment of causal digital twins, particularly in clinical contexts where decision support must be transparent and actionable [35, 36].”
Finding Local Explanations Through Masking Models
Angiulli, F., Fassetti, F., & Nisticò, S. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-91608-4_46 · Cited by 8
To pretrain or not? A systematic analysis of the benefits of pretraining in diabetic retinopathy
Srinivasan, V., Strodthoff, N., Ma, J., Binder, A., Müller, K., & Samek, W. (2022) · PLoS ONE · doi:10.1371/journal.pone.0274291 · Cited by 17
“The layer-wise conservation principle in LRP assures that the relevances from a higher layer is pre-served when propagated to a lower layer .”
InfluentialSelf-citationStomberg, T. T., Leonhardt, J., Weber, I., & Roscher, R. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1278118 · Cited by 10
Filho, J. D. S. R., Cardoso, L. F. F., da Silva, R. L. S., Carneiro, N., Santos, V. C. A., & Alves, R. C. D. O. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122986 · Cited by 11
Kumar, A., Kumar, M. S., & Goswami, P. (2024) · An International Journal of Optimization and Control Theories & Applications (IJOCTA) · doi:10.11121/ijocta.1424 · Cited by 8
“More advances in the applications of deep neural networks are presented in ( [15], [16]).”
BackgroundMachine Learning for Digital Shadow Design in Health Insurance Sector
Rodríguez-Aguilar, R., Marmolejo-Suacedo, J.-A., Rodríguez-Aguilar, M., & Marmolejo-Saucedo, L. (2024) · Mobile Networks and Applications · doi:10.1007/s11036-023-02289-2 · Cited by 9
“The deep neural networks have several hidden layers with millions of artificial neurons connected to each other [54 ] GBM Gradient Boosting Machine is a direct learning ensemble method.”
An evolutionary algorithm for interpretable molecular representations
Pflüger, P. M., Kühnemund, M., Katzenburg, F., Kuchen, H., & Glorius, F. (2024) · Chem · doi:10.1016/j.chempr.2024.02.004 · Cited by 9
Roy, A., Sharma, R. K., Jash, D., Rao, B. P., Dev, J. A., & Tomson, J. K. (2024) · Journal of Asian Earth Sciences · doi:10.1016/j.jseaes.2024.106093 · Cited by 8
Chen, W., Farag, S., Butt, U., & Al‐Khateeb, H. (2024) · Applied Sciences · doi:10.3390/app14209528 · Cited by 11
“As rental value estimation undergoes rapid transformation, the application of advanced AI-based algorithms, such as ML models or deep learning (DL) methods, has introduced nuanced alternatives and offered promising solutions [7,8].”
BackgroundUser Study on the Effects Explainable AI Visualizations on Non-experts
Schulze‐Weddige, S., & Zylowski, T. (2022) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-030-95531-1_31 · Cited by 14
El-Hajj, H., Zamani, M., Büttner, J., Martinetz, J., Eberle, O., Shlomi, N., Siebold, A., Montavon, G., Müller, K., Kantz, H., et al. (2022) · Datenbank-Spektrum · doi:10.1007/s13222-022-00414-1 · Cited by 13
“[2, 22]), and well-founded approaches have been proposed to attribute the model’s predictions to the input features.”
BackgroundSelf-citationExplainable AI (XAI): A Survey of Current and Future Opportunities
Kumari, M., Chaudhary, A., & Narayan, Y. (2022) · Studies in computational intelligence · doi:10.1007/978-3-031-18292-1_4 · Cited by 10
Interpretation of Dynamic Models Based on Neural Networks in the Form of Integral-Power Series
Fomin, O. O., Polozhaenko, S., Krykun, V., Orlov, A. A., & Лись, Д. А. (2022) · Lecture notes in networks and systems · doi:10.1007/978-3-031-20141-7_24 · Cited by 13
HistoTrust: tracing AI behavior with secure hardware and blockchain technology
Paulin, D., Joud, R., Hennebert, C., Moëllic, P.-A., Franco-Rondisson, T., & Jayles, R. (2023) · Annals of Telecommunications · doi:10.1007/s12243-022-00943-6 · Cited by 13
“A large variety of approaches have been proposed to enlighten the blackbox paradigm of deep NN models [17] even for modern architectures.”
BackgroundGuidotti, D., Pandolfo, L., & Pulina, L. (2023) · Information · doi:10.3390/info14070397 · Cited by 8
Shouny, A. A., Rezk, H., Sayed, E. T., Abdelkareem, M. A., Issa, U. H., Miky, Y. H., & Olabi, A. G. (2023) · Biomimetics · doi:10.3390/biomimetics8070557 · Cited by 8
“Convolutional Neural Networks (CNNs) thrive at image analysis, whereas Recurrent Neural Networks (RNNs) excel at sequential data processing, making them useful for tasks such as time series analysis and natural language processing [31,32].”
Chang, L., Wu, H., & Alshamrani, A. M. (2024) · Measurement · doi:10.1016/j.measurement.2024.114511 · Cited by 8
ECQ$$^{\text {x}}$$: Explainability-Driven Quantization for Low-Bit and Sparse DNNs
Becking, D., Dreyer, M., Samek, W., Müller, K., & Lapuschkin, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-04083-2_14 · Cited by 27
“A common problem within deep neural networks is, that the gradient becomes increasingly noisy with network depth [33], partly a result from gradient shattering [5].”
BackgroundSelf-citationUnderstanding Image Captioning Models beyond Visualizing AttentionPreprint
Sun, J., Lapuschkin, S., Samek, W., & Binder, A. (2020) · arXiv (Cornell University) · Cited by 8
Self-citationDeep Learning for Video Classification: A ReviewPreprint
Rehman, A. U., & Belhouari, S. B. (2021) · doi:10.36227/techrxiv.15172920.v1 · Cited by 10
Transparency in Music-Generative AI: A Systematic Literature ReviewPreprint
Batlle-Roca, R., Gómez, E., Liao, W., Serra, X., & Mitsufuji, Y. (2023) · Research Square · doi:10.21203/rs.3.rs-3708077/v1 · Cited by 18
Moussaoui, H., AKKAD, N. E., & BENSLIMANE, M. (2023) · Research Square · doi:10.21203/rs.3.rs-3195386/v1 · Cited by 8
Non-Uniform Interpolation in Integrated Gradients for Low-Latency Explainable-AI
Bhat, A., & Raychowdhury, A. (2023) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas46773.2023.10181829 · Cited by 13
“These post-hoc techniques can be applied to existing pre-trained models [6].”
MethodFragility, robustness and antifragility in deep learning
Pravin, C., Martino, I., Nicosia, G., & Ojha, V. (2023) · Artificial Intelligence · doi:10.1016/j.artint.2023.104060 · Cited by 10
“Deep neural networks (DNNs) are extensively used in various tasks and domains, achieving noteworthy performances in both research and real-world applications [1, 2].”
BackgroundWasserstein generative adversarial networks for topology optimization
Pereira, L., & Driemeier, L. (2024) · Structures · doi:10.1016/j.istruc.2024.106924 · Cited by 7
Amiri, M., & Nazari, S. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109255 · Cited by 12
Use of artificial intelligence in soybean breeding and production
Singh, A. K., Jones, S. E., der Laan, L. V., Ayanlade, T. T., Raigne, J., Saleem, N., Joshi, S., Arshad, M., Zaremehrjerdi, H., Rairdin, A. M., et al. (2025) · Advances in agronomy · doi:10.1016/bs.agron.2024.11.003 · Cited by 12
Dai, Y., Xu, W., Yan, M., Feng, X., & Zhao, J. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110139 · Cited by 7
Li, W., Chen, X., Liu, S., Zhang, C., & Liu, G. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012778 · Cited by 7
A New Active Sonar Detector Based on Beamformed Deep Neural Network
Peng, C., Wang, L., Gao, J., Zhang, S., & Ji, H. (2025) · IEEE Journal of Oceanic Engineering · doi:10.1109/joe.2025.3535597 · Cited by 8
“Given the success of deep neural network (DNN) in various classification tasks [29], [30], we use DNN in sonar signal processing.”
An Analysis of Deep Learning Algorithms for Detection of Pulmonary Illness
Singh, A. P., Nigam, A., & Kumar, V. (2025) · International Conference on Computational Intelligence, Communication Technology and Networking (CICTN) · doi:10.1109/cictn64563.2025.10932572 · Cited by 10
Ngabu, W., Pramoedyo, H., Iriany, A., & Fitriani, R. (2025) · International Electronics Symposium (IES) · doi:10.1109/ies67184.2025.11161827 · Cited by 9
“It is designed to simulate the brain’s information processing through a network of interconnected units referred to as neurons [23].”
Explainability and importance estimate of time series classifier via embedded neural network
Chan, H. T. J., Šimić, I., & Veas, E. E. (2025) · Scientific Reports · doi:10.1038/s41598-025-17703-w · Cited by 8
“Depending on the scope of the feature importance estimate, they can be grouped into global or local approaches 51 .”
Pan, R., Liu, S., Chen, Y., Dong, W., Jiang, W., Zhang, W., Jiang, Y., Xin, F., & Jiang, M. (2025) · Biotechnology for Biofuels and Bioproducts · doi:10.1186/s13068-025-02715-5 · Cited by 9
“Common artificial intelligence algorithm models include decision trees, linear regression, deep neural networks, and naive Bayes, among others [61].”
Wide-range constitutive modeling with multi-fidelity physics-guided neural network
Huang, Y., Yang, J., & Zhang, Y. (2026) · International Journal of Mechanical Sciences · doi:10.1016/j.ijmecsci.2026.111146 · Cited by 7
Explainability of vision transformers: a comprehensive review and new perspectives
Kashefi, R., Barekatain, L., Sabokrou, M., & Aghaeipoor, F. (2026) · Multimedia Tools and Applications · doi:10.1007/s11042-026-21313-7 · Cited by 17
“XAI offers several methods such as global and local approaches to explain DNN decision-making, providing a better understanding of the internal workings of the models Samek et al. (2021).”
Naeim, M., & Narimani, M. (2026) · Biological Psychology · doi:10.1016/j.biopsycho.2026.109232 · Cited by 7
Variant Approach for Identifying Spurious Relations That Deep Learning Models Learn
Tesch, T., Kollet, S. J., & Garcke, J. (2021) · Frontiers in Water · doi:10.3389/frwa.2021.745563 · Cited by 8
“To address this lack of interpretability, several interpretation methods have been developed (see e.g., Gilpin et al., 2018; Montavon et al., 2018; Zhang and Zhu, 2018; Molnar, 2019; Samek et al., 2021) providing descriptions of the learned function (e.g., measures of feature importance, FI).”
BackgroundMethodAllosteric control of ACE2 peptidase domain dynamics
Trozzi, F., Karki, N., Song, Z., Verma, N., Kraka, E., Zoltowski, B. D., & Tao, P. (2022) · Organic & Biomolecular Chemistry · doi:10.1039/d2ob00606e · Cited by 8
An evolutionary ensemble learning for diagnosing COVID-19 via cough signals
Najaran, M. H. T. (2023) · Intelligent Medicine · doi:10.1016/j.imed.2023.01.001 · Cited by 7
Learning to Select Prototypical Parts for Interpretable Sequential Data Modeling
Zhang, Y., Gao, N., & Ma, C. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i5.25812 · Cited by 12
Principles And Applications of Artificial Intelligence (AI) Algorithms: A Review of The Literature
Wang, J. J. (2023) · Highlights in Science Engineering and Technology · doi:10.54097/hset.v57i.9983 · Cited by 9
“80 manual feature design and selection, deep learning can automatically learn and extract features from data through multilayer neural networks, thus reducing the workload and error of feature engineering [5-6].”
The Manifold Hypothesis for Gradient-Based Explanations
Bordt, S., Upadhyay, U., Akata, Z., & Ulrike Von Luxburg, A. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00378 · Cited by 23
“[49] Wojciech Samek, Grégoire Montavon, Sebastian Lapuschkin, Christopher J Anders, and Klaus-Robert Müller.”
BackgroundExplaining Deep Face Algorithms Through Visualization: A Survey
John, T. A., Balasubramanian, V. N., & Jawahar, C. V. (2023) · IEEE Transactions on Biometrics Behavior and Identity Science · doi:10.1109/tbiom.2023.3319837 · Cited by 8
“Our Contributions: As stated earlier, while there have been earlier efforts on reviewing existing work on machine learning explainability [10], [11], [12], [13], [14], [15], [16], ours is the first to focus on a meta-analysis of face explainability.”
BackgroundJaradat, S., Nayak, R., & Elhenawy, M. M. Z. (2024) · IEEE International Conference on Computing and Machine Intelligence (ICMI) · doi:10.1109/icmi60790.2024.10586184 · Cited by 6
“Interpretable models are designed to provide clear, human-understandable insights into how a model arrives at specific predictions [16].”
BackgroundIntegrating White and Black Box Techniques for Interpretable Machine Learning
Vernon, E. M., Masuyama, N., & Nojima, Y. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-97-3562-4_51 · Cited by 10
An Improved YOLOv5 Algorithm for Bamboo Strip Defect Detection Based on the Ghost Module
Yang, R.-X., Lee, Y.-R., Lee, F., Liang, Z., & Liu, Y. (2024) · Forests · doi:10.3390/f15091480 · Cited by 10
Spatial weighting — An effective incorporation of geological expertise into deep learning models
Wang, W., Zhao, C., & Wu, Y. (2024) · Geochemistry · doi:10.1016/j.chemer.2024.126212 · Cited by 8
Layer-wise relevance propagation for backbone identification in discrete fracture networks
Berrone, S., Santa, F. D., Mastropietro, A., Pieraccini, S., & Vaccarino, F. (2021) · Journal of Computational Science · doi:10.1016/j.jocs.2021.101458 · Cited by 8
Brini, I., Mehri, M., Ingold, R., & Amara, N. E. B. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16014-1_10 · Cited by 9
Pavithra, V., & Jayalakshmi, V. (2022) · Elsevier eBooks · doi:10.1016/b978-0-323-91664-6.00001-2 · Cited by 7
Wagner, P., Strodthoff, N., Wurzel, P., Marbán, A., Scharf, S., Schäfer, H., Seegerer, P., Loth, A. G., Hartmann, S., Klauschen, F., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-18097-9 · Cited by 7
“33 ).”
Self-citationAn efficient pruning and fine-tuning method for deep spiking neural network
Meng, L., Qiao, G., Zhang, X. Y., Bai, J., Zuo, Y., Zhou, P., Liu, Y., & Hu, S. (2023) · Applied Intelligence · doi:10.1007/s10489-023-05056-8 · Cited by 11
Preemptively pruning Clever-Hans strategies in deep neural networks
Linhardt, L., Müller, K., & Montavon, G. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102094 · Cited by 18
“Explainable AI (XAI) [36, 88, 10, 87, 41] is a natural starting point for robustification beyond classical hold-out validation because it places human experts in the loop.”
BackgroundSelf-citationLog-Gaussian gamma processes for training Bayesian neural networks in Raman and CARS spectroscopies
Härkönen, T., Vartiainen∥, E. M., Lensu, L. T., Moores, M. T., & Roininen, L. (2024) · Physical Chemistry Chemical Physics · doi:10.1039/d3cp04960d · Cited by 11
Human-Centered Evaluation of XAI Methods
Tomotaki-Dawoud, K., Samek, W., Eisert, P., Lapuschkin, S., & Bosse, S. (2023) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw60847.2023.00122 · Cited by 18
“TABLE II INFERENCE SPEED SHOWN BY THE NUMBER OF EXPLANATIONS PER SECOND (COMPUTED IN BATCHES OF UP TO 16 SAMPLES ON A GPU AVERAGED OVER TEN RUNS) [21].”
BackgroundMethodInfluentialSelf-citationEarly warning of complex climate risk with integrated artificial intelligencePreprint
Reichstein, M., Frank, D., Benson, V., Camps‐Valls, G., Denzler, J., Kornhuber, K., Schölkopf, B., Vinuesa, R., Han, B., Fearnley, C. J., et al. (2024) · Research Square · doi:10.21203/rs.3.rs-4248340/v1 · Cited by 7
Colas, J. T., O’Doherty, J. P., & Grafton, S. T. (2024) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011950 · Cited by 6
“Guess what I'm doing”: Extending legibility to sequential decision tasks
Faria, M., Melo, F. S., & Paiva, A. (2024) · Artificial Intelligence · doi:10.1016/j.artint.2024.104107 · Cited by 12
“Most approaches to improve AIA transparency have focused on making more transparent technical aspects of AIA such as the reason behind application failures [36, 37, 38] or the inference processes of complex decision process like deep learning approaches [39, 40, 41].”
BackgroundThe explainable structure of deep neural network for recommendation systems
Zanjani, M. D., & Aghdam, M. H. (2024) · Future Generation Computer Systems · doi:10.1016/j.future.2024.05.036 · Cited by 9
Liu, S., Jia, R., Hao, X.-B., Liu, P., Deng, Y.-H., & Sun, H.-M. (2024) · Journal of Earthquake Engineering · doi:10.1080/13632469.2024.2364698 · Cited by 6
“Due to the powerful capability of deep neural networks, they have been increasingly used to solve some artificial intelligence problems (Kita and Kajikawa 2021; Radomirović et al. 2023; Samek et al. 2021 Before training the network, we need to pre-process the data.”
Explainable Quantum Neural Networks: Example-Based and Feature-Based Methods
Tian, J., & Yang, W. (2024) · Electronics · doi:10.3390/electronics13204136 · Cited by 10
Li, Y., Zhou, R., Zhou, Y., & Chen, Z. S. (2025) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2025.110395 · Cited by 9
The Explanation Necessity for Healthcare AI
Mamalakis, M., de Vareilles, H., Murray, G. K., Lió, P., & Suckling, J. (2025) · IEEE Symposium on Trustworthy, Explainable and Responsible Computational Intelligence (CITREx) · doi:10.1109/citrexcompanion65208.2025.10981502 · Cited by 9
“They include both model-specific approaches that address particular nonlinear behaviors and model-agnostic approaches that explore data complexity [2, 6].”
Optimizing Class Imbalance in Facial Expression Recognition Using Dynamic Intra-Class Clustering
Li, Q., Fu, K., Liu, J., Li, Y., Ren, Q., Xu, K., Fu, J., Liu, N., & Yuan, Y. (2025) · Biomimetics · doi:10.3390/biomimetics10050296 · Cited by 14
“However, when the subclass number was further increased to [ 8, 4, 1, 40, 24, 1, 12 ] , the classification accuracy dropped to 91.260%, which may imply that overly granular subclass partitioning could lead to model overfitting, thereby affecting its performance on the test set.”
InfluentialData Fusion for Integrative Species Identification using Deep Learning
Kösters, L. M., Karbstein, K., Hofmann, M. R., Hodač, L., Mäder, P., & Wäldchen, J. (2025) · Systematic Biology · doi:10.1093/sysbio/syaf026 · Cited by 10
Machine Learning for Multi-Class Kidney Tumour Segmentation: Advancing Classification Accuracy
Rajagopal, K., Kumari, V. S., Pillai, N. M., Rekha, A., Gunasekaran, M., & Lourdu Mahimai Doss, P. (2025) · International Conference on Pervasive Computing and Social Networking (ICPCSN) · doi:10.1109/icpcsn65854.2025.11035274 · Cited by 7
“• Improving AI-based models' interpretability in order to increase their clinical acceptability [14].”
Hong, C., & He, Q. (2025) · Applied Sciences · doi:10.3390/app15126893 · Cited by 9
Hao, Y., Han, Y., Huang, J., Hao, C., Yu, B., Wei, S., & Zhou, K. (2025) · Frontiers in Neuroscience · doi:10.3389/fnins.2025.1653693 · Cited by 9
Neural interaction explainable AI predicts drug response across cancers
Keyl, P., Keyl, J., Möck, A., Dernbach, G., Mochmann, L. H., Kiermeyer, N., Jurmeister, P. S., Bockmayr, M. L., Schwarz, R. F., Montavon, G., et al. (2025) · NAR Cancer · doi:10.1093/narcan/zcaf029 · Cited by 6
Self-citationAdvancing resilience in infrastructure projects through machine learning-driven models
Ojiako, U., Wong, T. C., Smith, C. J., Chipulu, M., Al-Mhdawi, M. K. S., Oyewo, B. M., & Obokoh, L. O. (2025) · Production Planning & Control · doi:10.1080/09537287.2025.2583300 · Cited by 8
“� LIME provides local, model-agnostic interpretations for individual predictions (Biecek and Burzykowski, 2021; Ribeiro, Singh, and Guestrin 2016; Samek et al. 2021).”
X-Fake: Juggling Utility Evaluation and Explanation of Simulated SAR Images
Huang, Z., Zhuang, Y., Zhong, Z., Xu, F., Cheng, G., & Han, J. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3634988 · Cited by 11
“Researchers in explainable artificial intelligence (XAI) have developed numerous methods for helping users understand the predictions of complex supervised learning models [39].”
Mining Spatiotemporal Information for Harmful Algal Bloom Forecasting and Mechanism Interpreting
Jia, Q., Xu, C., Jia, H., Velazquez, C., Leng, L., & Yin, D. (2024) · ACS ES&T Water · doi:10.1021/acsestwater.4c00115 · Cited by 7
A Unified Generative Artificial Intelligence Approach for Converting Social Media Content
Bondé, L., & Dembele, S. (2024) · International Conference on Artificial Intelligence, Big Data, Computing and Data Communication Systems (icABCD) · doi:10.1109/icabcd62167.2024.10645267 · Cited by 6
“In Gen AI, these technologies are implemented using Deep Neural Networks (DNNs) [25], RNNs/LSTMs, Region-Based Convolutional Neural Networks (R-CNNs), You Only Look Once (YOLO) [26] and Single Shot MultiBox Detector (SSD) [27], Generative Adversarial Networks (GANs), Capsule Networks [28],…”
MethodUnveiling diagnostic information for type 2 diabetes through interpretable machine learning
Lv, X., Luo, J., Zhang, Y., Guo, H. M., Yang, M., Li, M., Chen, Q., & Jing, R. (2024) · Information Sciences · doi:10.1016/j.ins.2024.121582 · Cited by 6
Fleischmann, S., Dietz, S., Shanbhag, J., Wuensch, A., Nitschke, M., Miehling, J., Wartzack, S. J., Leyendecker, S., Eskofier, B. M., & Koelewijn, A. D. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3702646 · Cited by 7
“Then, the activation score of the output layer is propagated back layer by layer following the principle of relevance conservation until the input layer is reached [46].”
Deep learning-based prediction of initiation jet momentum ratio in jet-induced oblique detonations
Bao, Y., Qiu, R., Lou, J., Han, X. J., & You, Y. (2024) · Aerospace Science and Technology · doi:10.1016/j.ast.2024.109724 · Cited by 7
Chen, H., Alfred, M. C., Brown, A. D., Atinga, A., & Cohen, E. (2024) · JMIR Formative Research · doi:10.2196/59045 · Cited by 8
“Previous work has used post hoc XAI techniques for explaining the prediction of CXR pathology classifiers [4,75]; however, post hoc XAI techniques only approximate XSL • FO RenderX the behavior of the model and, therefore, may not provide faithful explanations [60,76].”
Chin, M. Y., Joy, D. A., Samaddar, M., Rana, A., Chow, J., Miyamoto, T., & Calvert, M. (2025) · SLAS DISCOVERY · doi:10.1016/j.slasd.2025.100208 · Cited by 8
Messner, W. (2025) · Journal of World Business · doi:10.1016/j.jwb.2025.101622 · Cited by 10
Ghorai, A., Nandi, U., Singh, M. M., Changdar, C., Paul, B., Chowdhuri, P., & Pal, P. (2025) · Pattern Analysis and Applications · doi:10.1007/s10044-025-01428-7 · Cited by 6
“Therefore, the research - ers moved to apply Deep Neural Network (DNN) [ 11 ] to develop SLRS where features are extracted automatically using the deep networks with multiple hidden layers and extended sufficient accuracy in recognition task.”
Insulator defect detection based on feature pyramid network and diffusion model
Hien, A. T., Tran-Anh, D., Viet, D. C., Thuy, Q. D. T., & Huu, Q. N. (2025) · Signal Image and Video Processing · doi:10.1007/s11760-025-03960-9 · Cited by 6
From neural activations to concepts: A survey on explaining concepts in neural networks
Lee, J. H., Lanza, S., & Wermter, S. (2024) · Neurosymbolic Artificial Intelligence · doi:10.3233/nai-240743 · Cited by 21
“Some XAI surveys have been published in recent years [15,24,31,32,45,47,49].”
Urban Fine Management of Multisource Spatial Data Fusion Based on Smart City Construction
Long, Y., Zhang, X., & Feng, G. (2021) · Mathematical Problems in Engineering · doi:10.1155/2021/5058791 · Cited by 6
“3ey are space static information data, space static and dynamic information data, and space dynamic information data [21].”
BackgroundRetractedLi, B., & Zhang, L. (2021) · Computational Intelligence and Neuroscience · doi:10.1155/2021/3884587 · Cited by 6
“)e algorithm mainly analyzes the correlation degree between different types of data and uses the method with strong internal correlation degree for information mining and data processing [22].”
MethodRetractedCoding infant engagement in the Face-to-Face Still-Face paradigm using deep neural networks
Faltyn, M., Krzeczkowski, J. E., Cummings, M., Anwar, S., Zeng, T., Zahid, I., Ntow, K. O.-B., & Lieshout, R. J. V. (2023) · Infant Behavior and Development · doi:10.1016/j.infbeh.2023.101827 · Cited by 6
Liu, Z., Wang, X., & Zhang, Y. T. (2023) · IEEE Transactions on Plasma Science · doi:10.1109/tps.2023.3267733 · Cited by 6
“By adjusting the number of hidden layers and neurons in the hidden layers, DNNs can predict arbitrarily complex functions [35], [36].”
BackgroundExplainable audio Classification of Playing Techniques with Layer-wise Relevance Propagation
Wang, C., Lostanlen, V., & Lagrange, M. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095894 · Cited by 6
“Although LRP has led to many publications in image and text processing [22], it has been rarely applied to speech [23, 24] and never to music.”
BackgroundHuman Expectations and Perceptions of Learning in Machine Teaching
Mishra, S., & Rzeszotarski, J. M. (2023) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3565472.3595612 · Cited by 8
“While complex ML algorithms like Stochastic Bandits [1], Transformers [87], and Deep Neural Networks [74], might be more efficient at learning data patterns, interpreting their learning process requires signifi-”
MethodComputer-aided diagnostic model for retinal vascular diseases using graph-based attention mechanism
Sivapriya, G., Devi, R. M., & Keerthika, P. (2023) · The Journal of Supercomputing · doi:10.1007/s11227-023-05581-w · Cited by 6
Kadir, M. A., Addluri, G. K., & Sonntag, D. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-42608-7_8 · Cited by 7
“These techniques are precious for interpreting complex models like deep learning, where the learned representations may be abstract and difficult to interpret directly [18].”
BackgroundDeep Learning Based Feature Selection and Ensemble Learning for Sintering State Recognition
Xu, X., & Zhou, X. (2023) · Sensors · doi:10.3390/s23229217 · Cited by 7
“Deep learning can solve the drawbacks of manual feature extraction by automatically discovering the representations needed for recognition [10–12].”
BackgroundInterpretable parallel channel encoding convolutional neural network for bearing fault diagnosis
Tong, Q., Du, S., Jiang, X., Lu, F., Feng, Z., Liu, R., Xu, J., & Huo, J. (2024) · Measurement Science and Technology · doi:10.1088/1361-6501/ad28b0 · Cited by 7
Impact the Classes’ number on the convolutional neural networks performance for image classification
Ali, A. K., Abdullah, A. M., & Raheem, S. F. (2023) · International Journal of Advanced Science Computing and Engineering · doi:10.62527/ijasce.5.2.132 · Cited by 9
“…is a type of artificial neural network that is particularly well-suited for image processing and analysis tasks, it is based on the idea of using a series of convolutional layers to extract and transform features from input images, which are then used to perform classification or other tasks [24].”
BackgroundA Comprehensive Review and Comparison of Image Super-resolution Techniques
Kumar, L., Rajaan, R., Choudhary, N., & Sharma, A. T. (2024) · International Journal of Advanced engineering Management and Science · doi:10.22161/ijaems.102.5 · Cited by 7
Metadata
- Authors
- Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R.
- Affiliations
- Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (DE)
- BIFOLD✔Berlin Institute for the Foundations of Learning and Data, Berlin, Germany
- Berlin Institute for the Foundations of Learning and Data (DE)
- ORCID
- Venue
- Proceedings of the IEEE
- Published
- 2021-03
- Volume
- 109
- Issue
- 3
- Pages
- 247-278
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W3132191748
- Cited by
- 1,477
- References
- 230
- Citations by year
- 2026: 305 · 2025: 343 · 2024: 326 · 2023: 273 · 2022: 152 · 2021: 74
- FWCI
- 72.25
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Proceedings of the IEEE (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion, other-oa)
- Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korea Government (2017-0-00451, 2019-0-00079)
- German Ministry for Education and Research (01IS14013A-E, 01GQ1115, 01GQ0850, 01IS18025A, 01IS18037A)
- German Research Foundation (Math+, EXC 2046/1, 390685689)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- explainable AI, interpretability, deep neural network, post-hoc explanation, interpretability evaluation, nonlinear machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.