Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Open accessThe DOI matches its Crossref record; 224 of the 230 references it cites were found in a registry or an index.
Abstract
With the broader and highly successful usage of machine learning (ML) in industry and the sciences, there has been a growing demand for explainable artificial intelligence (XAI). Interpretability and explanation methods for gaining a better understanding of the problem-solving abilities and strategies of nonlinear ML, in particular, deep neural networks, are, therefore, receiving increased attention. In this work, we aim to: 1) provide a timely overview of this active emerging field, with a focus on “post hoc” explanations, and explain its theoretical foundations; 2) put interpretability algorithms to a test both from a theory and comparative evaluation perspective using extensive simulations; 3) outline best practice aspects, i.e., how to best include interpretation methods into the standard usage of ML; and 4) demonstrate successful usage of XAI in a representative selection of application scenarios. Finally, we discuss challenges and possible future directions of this exciting foundational field of ML.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“While XAI techniques exist [47], their application in complex, real-time energy systems, and providing meaningful explanations to non-experts, remain challenging”
From Addressing Challenges for the Effective Adoption of Artificial Intelligence in the Energy Sector (2025) · cites this paper as [47]
In the citing paper
“Despite previous attempts to address the challenges surrounding model interpretability, most solutions have been implemented post hoc (such as the attention mechanism) focusing on explanations in certain domains, such as vision [184].”
From Human-centred artificial intelligence for mobile health sensing: challenges and opportunities (2023) · cites this paper as [184]
In the citing paper
“LIME produces an explanation list that shows the contribution of individual features to the prediction.”
From Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2022) · cites this paper as [64]
In the citing paper
“This local explanation allows the end-user to determine which feature is important for the precise prediction and how the perturbation in feature affects the prediction results.”
From Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2022) · cites this paper as [64]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 12×
“This modified gradient view on LRP can also be leveraged to achieve a simpler and more general implementation of LRP based on ‘forward hooks’, which we describe in the second part of Appendix B, and which we use to apply LRP on VGG-16 [175] and ResNet-50 [65] in Section IV.”
BackgroundMethodInfluentialVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 6×
“This modified gradient view on LRP can also be leveraged to achieve a simpler and more general implementation of LRP based on ‘forward hooks’, which we describe in the second part of Appendix B, and which we use to apply LRP on VGG-16 [175] and ResNet-50 [65] in Section IV.”
BackgroundMethodInfluentialImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 4×
“Consequently, we also exchange the neural network architecture and use an AlexNet [99] model, which is able to process the transformed input data using 2-D-convolution operators.”
BackgroundMethodInfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“, [23], [52], [70], [107], [108], and [161]) have gained highest popularity.”
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 9×
“A comparison of performance on the Adience benchmark of the original model (pretrained on ImageNet only) and the improved model is given in the table below.”
MethodInfluentialRegression Models and Life-Tables
Cox, D. R. (1972) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1972.tb00899.x · Cited by 39,793
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 8×
“A comparison of performance on the Adience benchmark of the original model (pretrained on ImageNet only) and the improved model is given in the table below.”
BackgroundInfluentialThe Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 1×
“In games such as the arcade game Atari Breakout, the computer player would progressively learn strategies commonly employed by human players, such as “tunneldigging” [124], [200].”
Background
Research works printed but not linked: 13
- Arras, 2020, Ground truth evaluation of neural network explanations with CLEVR-XAI, arXiv:2003.07258 · arXiv
- 10.1146/annurev.neuro.26.041002.131047 · DOI · Paper page
- Chollet, 2015, Keras
- 10.1109/ICCV.2019.00304 · DOI
- Kokhlikyan, 2019, Pytorch Captum
- Lapuschkin, 2016, The layer-wise relevance propagation toolbox for artificial neural networks, J. Mach. Learn. Res., 17, 1
- 10.48550/arXiv.1312.4400 · DOI · Paper page
- 10.5555/2969033.2969153 · doi:10.5555/2969033.2969153
- 10.5555/3157382.3157477 · doi:10.5555/3157382.3157477
- Okuta, CuPy: A numpy-compatible library for NVIDIA GPU calculations, 151
- Oliphant, 2006, A Guide to NumPy, 1
- 10.48550/ARXIV.1404.7828 · DOI
- 10.5555/3045118.3045336 · doi:10.5555/3045118.3045336
Cited by
Items 301–400 of 1,002
Deep learning to capture leaf shape in plant images: Validation by geometric morphometrics
Hodač, L., Karbstein, K., Kösters, L. M., Rzanny, M., Wittich, H. C., Boho, D., Šubrt, D., Mäder, P., & Wäldchen, J. (2024) · The Plant Journal · doi:10.1111/tpj.17053 · Cited by 24
“:3) an-dez et al., 2023; Holzinger et al., 2022; Li et al., 2022; Samek et al., 2021).”
Optimizing Bearing Fault Detection: CNN-LSTM with Attentive TabNet for Electric Motor Systems
Khawaja, A. U., Shaf, A., Thobiani, F. A., Ali, T. M., Irfan, M., Pirzada, A. R., & Shahkeel, U. (2024) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2024.054257 · Cited by 20
“As a solution to these limitations, Deep Neural Networks (DNNs) emerged as a powerful subset of deep learning models that produce hierarchical representations based on raw data, eliminating the need to manually construct feature representations [21].”
Keep CALM and Improve Visual Feature Attribution
Kim, J. M., Choe, J., Akata, Z., & Oh, S. J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00824 · Cited by 28
“Interpretable AI [25, 40, 24, 52] is becoming an absolute necessity in safety-critical and high-stakes applications of machine learning.”
BackgroundExplainable AI-Aided Feature Selection and Model Reduction for DRL-Based V2X Resource Allocation
Khan, N., Abdallah, A., Celik, A., Eltawil, A. M., & Coleri, S. (2025) · IEEE Transactions on Communications · doi:10.1109/tcomm.2025.3554655 · Cited by 20
“For instance, Saliency Maps and Layer-wise Relevance Propagation methods are well-suited for convolutional neural networks, highlighting important input regions in computer vision tasks [25].”
Intrusion Detection Systems in IoT Based on Machine Learning: A state of the art
BERHILI, M., Chaieb, O., & Benabdellah, M. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.11.089 · Cited by 27
González‐Briones, A., Palomino-Sánchez, J., Vale, Z., Ramos, C., & Corchado, J. M. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110324 · Cited by 20
Explainable transfer learning for subsurface soil moisture prediction
Ye, S., Chai, Y., Li, J., Wang, J., Deng, X.-B., & Ran, Q. (2025) · Journal of Hydrology · doi:10.1016/j.jhydrol.2025.133473 · Cited by 18
Addressing Challenges for the Effective Adoption of Artificial Intelligence in the Energy Sector
Park, C. (2025) · Sustainability · doi:10.3390/su17135764 · Cited by 17
Šimić, I., Veas, E. E., & Sabol, V. (2025) · Scientific Reports · doi:10.1038/s41598-025-09538-2 · Cited by 27
“Depending on the scope, explanations can be either global or local 14 .”
Improving Multimodal Speech Recognition by Data Augmentation and Speech Representations
Oneaţă, D., & Cucu, H. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00504 · Cited by 36
“To better understand the complicated interactions that arise in a multimodal speech recognition network, a future work direction could investigate which parts of the inputs (audio, image, previously predicted tokens) contribute more to the output; modern tools for explainable machine learning [13,15,35] are a possible…”
BackgroundExplainability in Automatic Short Answer Grading
Schlippe, T., Stierstorfer, Q., ten Koppel, M., & Libbrecht, P. (2023) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-19-8040-4_5 · Cited by 20
Mukhamediya, A., Fazli, S., & Zollanvari, A. (2023) · IEEE Access · doi:10.1109/access.2023.3287093 · Cited by 27
“areas [5], [6], [7], [8], [9], [10], and found eminent appli-”
Miss-gradient boosting regression tree: a novel approach to imputing water treatment data
Zhang, W., Li, R., Zhao, J., Wang, J., Meng, X., & Li, Q. (2023) · Applied Intelligence · doi:10.1007/s10489-023-04828-6 · Cited by 27
“, the use of traditional machine learning in imputation) [19] or needing substantial training data to train numerous parameters for deep learning [22, 23].”
BackgroundPafla, M., Larson, K. S., & Hancock, M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642934 · Cited by 24
Arenas, M., Barceló, P., Bertossi, L., & Monet, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08015 · Cited by 63
“We should mention, however, that there are some recent methods to explain the results from a neural network that do not rely only on its input/output relation (c.f. (Samek et al., 2021) for a recent and thorough review).”
MethodHigh-density Integrated Photonic Tensor Processing Unit with a Matrix Multiply CompilerPreprint
Ma, X., Peserico, N., Khaled, A., Guo, Z., Nouri, B. M., Dalir, H., Shastri, B. J., & Sorger, V. J. (2022) · Research Square · doi:10.21203/rs.3.rs-1833027/v1 · Cited by 18
Deep Learning for Video Classification: A Review
Rehman, A., & Belhaouari, S. B. (2021) · doi:10.36227/techrxiv.15172920 · Cited by 17
Pan, J., Wei, Y., Meng, L., & Meng, X. (2025) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00092-5 · Cited by 16
Zhao, C., Li, X., Shao, Z., Yang, H., & Wang, F. (2022) · Connection Science · doi:10.1080/09540091.2022.2061915 · Cited by 16
“…in artificial intelligence and the application of deep learning in multiple domains (Chang et al., 2021; Chen et al., 2021; LeCun et al., 2015; Samek et al., 2021), several researchers have focused on implementing an effective and accurate prediction of metro passenger flow (Liu et al., 2019;…”
BackgroundGreen Supply Chain Optimization Based on BP Neural Network
Wang, H. (2022) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2022.865693 · Cited by 16
Barber, S., Lima, L. A. M., Sakagami, Y., Quick, J., Latiffianti, E., Liu, Y., Ferrari, R., Letzgus, S., Zhang, X., & Hammer, F. (2022) · Energies · doi:10.3390/en15155638 · Cited by 22
Bhachawat, S., Shriram, E., Srinivasan, K., & Hu, Y. (2023) · Diagnostics · doi:10.3390/diagnostics13020288 · Cited by 18
“The most prominent limitation is a lack of suitable datasets.”
BackgroundNtrougkas, M. V., Gkalelis, N., & Mezaris, V. (2022) · IEEE International Symposium on Multimedia (ISM) · doi:10.1109/ism55400.2022.00014 · Cited by 20
“For a more comprehensive review the interested reader is referred to [4], [20].”
BackgroundBrocki, L., & Chung, N. C. (2023) · Pattern Recognition Letters · doi:10.1016/j.patrec.2023.10.012 · Cited by 32
Kiasari, M. M., & Aly, H. H. (2024) · Fire · doi:10.3390/fire7090296 · Cited by 18
Razavi-Termeh, S. V., Sadeghi‐Niaraki, A., Yao, X. A., Naqvi, R. A., & Choi, S.-M. (2024) · Journal of Environmental Management · doi:10.1016/j.jenvman.2024.122361 · Cited by 20
Riyadh, A., & Peleato, N. M. (2024) · Digital Chemical Engineering · doi:10.1016/j.dche.2024.100202 · Cited by 19
Machine learning detection of dust impact signals observed by the Solar Orbiter
Kvammen, A., Wickstrøm, K., Kočiščák, S., Vaverka, J., Nouzák, L., Zaslavsky, A., Babić, K. R., Gjelsvik, A., Píša, D., Soucek, J., et al. (2023) · Annales Geophysicae · doi:10.5194/angeo-41-69-2023 · Cited by 22
Explainable AI guided unsupervised fault diagnostics for high-voltage circuit breakers
Hsu, C.-C., Frusque, G., Forest, F., Macedo, F. T. B., Franck, C. M., & Fink, O. (2025) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2025.111199 · Cited by 19
“Feature attribution method can be divided into three main categories: occlusion-based, gradient-based, and propagation-based [51].”
Applications of explainable artificial intelligence in renewable energy research
Perr‐Sauer, J., Ugirumurera, J., Gafur, J., Bensen, E. A., Nguyen, T., Paul, S., Severino, J., Nag, A., Vijayshankar, S., Gasper, P. J., et al. (2025) · Energy Reports · doi:10.1016/j.egyr.2025.08.046 · Cited by 23
Interpretable Machine Learning: A Comprehensive Review of Foundations, Methods, and the Path Forward
Fridkin, S., & Bendersky, M. (2026) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70075 · Cited by 38
Tang, Q., Cen, X., & Pan, C. (2022) · Mathematical Biosciences & Engineering · doi:10.3934/mbe.2022457 · Cited by 17
Son, R., Ma, P., Wang, H., Rasch, P. J., Wang, S., Kim, H., Jeong, J., Lim, K. S., & Yoon, J. (2022) · Journal of Advances in Modeling Earth Systems · doi:10.1029/2022ms002995 · Cited by 17
“explainable artificial intelligence (XAI), have shown remarkable development in recent years (Samek et al., 2021).”
BackgroundMethodRELAX: Representation Learning Explainability
Wickstrøm, K., Trosten, D. J., Løkse, S., Boubekki, A., Mikalsen, K. Ø., Kampffmeyer, M., & Jenssen, R. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01773-2 · Cited by 26
“The field of XAI have grown significantly in recent years, with a recent review article listing 46 different XAI methods (Samek et al., 2021).”
BackgroundEvolutionary Approaches to Explainable Machine Learning
Zhou, R., & Hu, T. (2023) · Genetic and evolutionary computation · doi:10.1007/978-981-99-3814-8_16 · Cited by 25
“In recent years, many methods have been developed for explaining the internals of deep learning models [43].”
MethodDiffeomorphic Counterfactuals With Generative Models
Dombrowski, A.-K., Gerken, J. E., Müller, K., & Kessel, P. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3339980 · Cited by 44
“Supplying this desired transparency has been the subject of re-centdevelopmentsinthefieldofexplainableAI(XAI)[20],[21], [22],[23],[24],amelioratingtheaforementionedchallenges.”
BackgroundKhan, U., Pao, W. K. S., Pilario, K. E., Sallih, N., & Khan, R. (2023) · International Journal of Numerical Methods for Heat & Fluid Flow · doi:10.1108/hff-09-2023-0526 · Cited by 17
A method to assist designers in optimizing the exterior styling of vehicles based on key features
Hou, X., Gou, B., Chen, D., Chu, J., Ding, N., & Ma, L. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124485 · Cited by 17
Chen, C., Zhang, L., Zhou, C., & Luo, Y. (2025) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2025.115478 · Cited by 15
Wang, B., Pei, W., Xue, B., & Zhang, M. (2025) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2025.3558419 · Cited by 25
“A prominent area of research, known as Explainable Deep Learning (XDL) [16], [17], [18], [19], [20], aims to provide insights into the behaviour of black-box deep learning models.”
Stream-based explainable recommendations via blockchain profiling
Leal, F., Veloso, B., Malheiro, B., Burguillo, J. C., Chis, A. E., & González–Vélez, H. (2021) · Integrated Computer-Aided Engineering · doi:10.3233/ica-210668 · Cited by 23
“Nevertheless, there are post hoc complex explanation techniques that can be used together with deep neural networks [34].”
Explainable machine learning in identifying credit card defaulters
Srinath, T., & Gururaja, H. S. (2022) · Global Transitions Proceedings · doi:10.1016/j.gltp.2022.04.025 · Cited by 19
Big Data Management for Machine Learning from Big Data
Olawoyin, A. M., Leung, C. K.-S., Hryhoruk, C. C. J., & Cuzzocrea, A. (2023) · Lecture notes in networks and systems · doi:10.1007/978-3-031-29056-5_35 · Cited by 16
Explaining Bayesian Neural Networks
Bykov, K., Höhne, M., Creosteanu, A., Müller, K., Klauschen, F., Nakajima, S., & Kloft, M. (2025) · Transactions on machine learning research · doi:10.48550/arxiv.2108.10346 · Cited by 37
“When tackling real-world learning problems, Bayesian models have been helpful to assess the intrinsic uncertainties encountered when predicting.”
Perspectives, challenges and future of artificial intelligence in personalised nutrition research
Brankovic, A., & Hendrie, G. A. (2025) · Proceedings of The Nutrition Society · doi:10.1017/s0029665125100657 · Cited by 17
Application of Deep Learning Networks to Segmentation of Surface of Railway Tracks
Bojarczak, P., & Nowakowski, W. (2021) · Sensors · doi:10.3390/s21124065 · Cited by 14
Disproving XAI Myths with Formal Methods – Initial Results
Marques-Silva, J. P. (2023) · IEEE International Conference on Engineering of Complex Computer Systems (ICECCS) · doi:10.1109/iceccs59891.2023.00012 · Cited by 31
“There are fairly up to date overviews of non-formal explainability approaches [27], [65], [66], [76]–[78].”
BackgroundSensitivity analysis by differential importance measure for unsupervised fault diagnostics
Floreale, G., Baraldi, P., Lu, X., Rossetti, P., & Zio, E. (2023) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2023.109846 · Cited by 14
Liu, G., Wang, P., & Kwon, Y. (2023) · Geophysical Research Letters · doi:10.1029/2023gl106278 · Cited by 16
“The total relevance is conserved during this process through propagation rules, creating a “heatmap” of each pixel's contribution to the network's final decision (Montavon et al., 2019; Samek et al., 2021).”
BackgroundUnleashing Dynamic Pipeline Reconfiguration of P4 Switches for Efficient Network Monitoring
Sadi, A. A., Savi, M., Melis, A., Prandini, M., & Callegati, F. (2024) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2024.3377538 · Cited by 22
“A further step in this domain has been made by Taurus [24], which offloads CNN [35], Deep Neural Networks (DNN) [36], and Support Vector Machine (SVM) [37] models to the data plane exploiting MapReduce [38].”
MethodA framework for strategic discovery of credible neural network surrogate models under uncertainty
Singh, P., Maupin, K. A., & Faghihi, D. (2024) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2024.117061 · Cited by 19
Wang, B., Zou, R., Mao, J., Wu, C., & Xue, D. (2024) · Aerospace Science and Technology · doi:10.1016/j.ast.2024.109918 · Cited by 14
Macrocycle‐based differential sensing: Design strategies and applications
Tian, J., Zheng, Z., Pan, Y., Wang, Y., & Guo, D. (2025) · Responsive materials · doi:10.1002/rpm.20240036 · Cited by 14
Song, Z., Trozzi, F., Tian, H., Yin, C., & Tao, P. (2022) · ACS Physical Chemistry Au · doi:10.1021/acsphyschemau.2c00005 · Cited by 14
Backward recursive Class Activation Map refinement for high resolution saliency map
Englebert, A., Cornu, O., & Vleeschouwer, C. D. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956711 · Cited by 15
“For a review see [5].”
Bradbury, A. W., & Robertson, C. (1993) · PubMed · Cited by 14
Longo, L., Brčić, M., Cabitza, F., Choi, J., Confalonieri, R., Ser, J. D., Guidotti, R., Hayashi, Y., Herrera, F., Holzinger, A., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.19775 · Cited by 14
“A lot of work exists on explaining the decisions of a classifier with attribution methods [44].”
BackgroundSelf-citationCoughLIME: Sonified Explanations for the Predictions of COVID-19 Cough Classifiers
Wullenweber, A., Akman, A., & Schuller, B. W. (2022) · 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC) · doi:10.1109/embc48229.2022.9871291 · Cited by 13
“XAI methods proposed up to this date can be classified into four categories: interpretable local surrogates, occlusion analysis, integrated gradients/smoothGrad, and layer-wise relevance propagation [13].”
Interpretable research of fuzzy methods: A literature survey
Zhang, K., Shao, T., Sun, Y., Xu, X., Zhang, X., Zhou, X., Ding, K., & Huang, S. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103524 · Cited by 26
Wearable smart sensors integration with AI and machine learning for tracking human health
Xianyong, S., Mishra, D. K., Özcan, Ö. Ö., Karahan, M., Panpan, T., Zhongshan, X., & Kumar, P. V. (2025) · Biosensors and Bioelectronics X · doi:10.1016/j.biosx.2025.100711 · Cited by 31
Mechanistically guided residual learning for battery state monitoring throughout life
Che, Y., Zheng, Y., Rhyu, J., Guo, J., Wang, S., Teodorescu, R., & Braatz, R. D. (2026) · Nature Communications · doi:10.1038/s41467-025-67565-z · Cited by 26
Uncertainty explanation of artificial intelligence models by SHAP
Cheng, X., Wang, T., Zhu, D., & Ma, J. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.115437 · Cited by 22
Multi-granularity visual explanations for CNN
Bao, H., Wang, G., Li, S., & Liu, Q. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109474 · Cited by 15
Neural network application for semantic segmentation of fundus
Александрович, П. Р., RAS, I. R. –. B. O. T. F. “. A. P., Mukhin, A., Ilyasova, N. Y., RAS, I. R. –. B. O. T. F. “. A. P., Demin, N. S., & RAS, I. R. –. B. O. T. F. “. A. P. (2022) · Computer Optics · doi:10.18287/2412-6179-co-1010 · Cited by 14
“В последнее время большинство задач интеллек - туального анализа данных решается с помощью нейросетевых алгоритмов [6, 7].”
Perturbation Effect: A Metric to Counter Misleading Validation of Feature Attribution
Šimić, I., Sabol, V., & Veas, E. E. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557418 · Cited by 22
“However, other authors argue in favor of explaining the predictions of deep learning models [4, 22, 27].”
BackgroundMethodA Synergic Approach of Deep Learning towards Digital Additive Manufacturing: A Review
Pratap, A., Sardana, N., Utomo, S., Ayeelyan, J., Karthikeyan, P., & Hsiung, P. (2022) · Algorithms · doi:10.3390/a15120466 · Cited by 16
Genetic Programming for Interpretable and Explainable Machine Learning
Hu, T. (2023) · Genetic and evolutionary computation · doi:10.1007/978-981-19-8460-0_4 · Cited by 18
Guan, X., Li, J., Yang, C., & Xing, W. (2023) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi12040174 · Cited by 16
Yin, L., & Wei, X. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110347 · Cited by 13
Surface wave dispersion curve inversion using mixture density networks
Keil, S., & Wassermann, J. (2023) · Geophysical Journal International · doi:10.1093/gji/ggad227 · Cited by 13
Viljanen, M., Minnema, J., Wassenaar, P. N. H., Rorije, E., & Peijnenburg, W. J. G. M. (2023) · SAR and QSAR in environmental research · doi:10.1080/1062936x.2023.2254225 · Cited by 17
“Neural Networks (NN) are also traditional algorithms that have become very popular, especially by the use of deep Neural Networks (dNN) for complex problems [59].”
BackgroundGarcía-Quintas, A., Roy, A., Barbraud, C., Demarcq, H., Denis, D., & Bertrand, S. (2023) · Ecology and Evolution · doi:10.1002/ece3.10549 · Cited by 20
“…number at 50 km radius Chlorophyll a concentration at 100 km radius Cays number at 10 km radius Sea surface temperature at 10 km radius Sea surface temperature at 50 km radius Sea surface temperature at 100 km radius 0 2 4 6 8 10 Mean decreasing Gini index Va ria bl es Samek et al., 2021).”
BackgroundGlobal geometry of chemical graph neural network representations in terms of chemical moieties
El-Samman, A. M., Incé A. Husain, Huynh, M., Castro, S. D., Morton, B., Baerdemacker, S. D., & Baerdemacker, S. D. (2024) · Digital Discovery · doi:10.1039/d3dd00200d · Cited by 23
Obremski, D., & Wienrich, C. (2024) · Journal of Computer and Communications · doi:10.4236/jcc.2024.124010 · Cited by 15
“Explainability: A special feature of generative AI systems is that even those who develop these instruments are no longer able to reconstruct how certain re-sults have been achieved, as the inputs are processed using highly non-linear and distributed processes (black box problem) [58].”
BackgroundInterpreting and generalizing deep learning in physics-based problems with functional linear models
Arzani, A., Yuan, L., Newell, P., & Bei, W. (2024) · Engineering With Computers · doi:10.1007/s00366-024-01987-z · Cited by 25
“More broadly, the growing field of interpretable and explainable artificial intelligence (XAI) offers a set of tools aimed at making black-box deep learning models understandable and transparent to humans [12, 13].”
BackgroundExplainable deep learning insights into the history and future of net primary productivity in China
Liu, N., Hao, Z., & Zhao, P. (2024) · Ecological Indicators · doi:10.1016/j.ecolind.2024.112394 · Cited by 15
Hajikarimloo, B., Tos, S. M., Alvani, M. S., Rafiei, M. A., Akbarzadeh, D., Eftekhar, M. S., Akhlaghpasand, M., & Habibi, M. A. (2024) · World Neurosurgery · doi:10.1016/j.wneu.2024.10.089 · Cited by 14
From Bytes to Biases: Investigating the Cultural Self-Perception of Large Language Models
Messner, W., Greene, T., & Matalone, J. (2025) · Journal of Public Policy & Marketing · doi:10.1177/07439156251319788 · Cited by 26
“LLMs employ deep learning architectures (Goodfellow, Bengio, and Courville 2016; Najafabadi et al. 2015; Samek et al. 2021), particularly transformer networks (Vaswani et al. 2017), which are designed to encode information about words and their contexts into high-dimensional vectors known as word…”
SoK: Dataset Copyright Auditing in Machine Learning Systems
Du, L., Zhou, X., Chen, M., Zhang, C., Su, Z., Cheng, P., Chen, J., & Zhang, Z. (2025) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp61157.2025.00025 · Cited by 29
A Comprehensive Survey on Self-Interpretable Neural Networks
Ji, Y., Sun, Y., Zhang, Y., Wang, Z., Zhuang, Y., Gong, Z., Shen, D., Qin, C., Zhu, H., & Xiong, H. (2025) · Proceedings of the IEEE · doi:10.1109/jproc.2025.3635153 · Cited by 43
“As summarized in Table 2, numerous studies [4], [7], [30], [33], [34], [35], [58], [59] have addressed specific aspects of self-interpretation.”
[Retracted] Compensated Fuzzy Neural Network‐Based Music Teaching Ability Assessment Model
Chen, X. (2021) · Computational Intelligence and Neuroscience · doi:10.1155/2021/3865190 · Cited by 12
“Samek et al. [20] have verified the results by practice, and the results show that the teaching scheme of grouped multiple music teaching in colleges and universities has better teaching effect and is suitable for teaching music to junior and senior students.”
ResultRetractedModeling Health Data Using Machine Learning Techniques Applied to Financial Management Predictions
Sanz, R. L., & Sanz, P. L. (2022) · Applied Sciences · doi:10.3390/app122312148 · Cited by 19
Ogrezeanu, I. A., Suciu, C., & Lucian Mihai, I. T. U. (2024) · Mediterranean Conference on Control and Automation (MED) · doi:10.1109/med61351.2024.10566174 · Cited by 16
“For the classification task, utilizing the WaRP-C subset, a series of deep neural networks (DNNs) [9] and convolutional neural networks (CNN) [10] were trained.”
MethodTowards Ground Truth Evaluation of Visual ExplanationsPreprint
Osman, A., Arras, L., & Samek, W. (2020) · arXiv (Cornell University) · Cited by 16
Self-citationREM: An Integrative Rule Extraction Methodology for Explainable Data Analysis in HealthcarePreprint
Shams, Z., Dimanov, B., Kola, S., Simidjievski, N., Andrés-Terré, H., Scherer, P., Matjašec, U., Abraham, J., Jamnik, M., & Lió, P. (2021) · medRxiv · doi:10.1101/2021.01.25.21250459 · Cited by 15
State-of-the-Art and Challenges of Engineering ML- Enabled Software Systems in the Deep Learning Era
Assres, G. M., Bhandari, G. P., Shalaginov, A., Grønli, T., & Ghinea, G. (2025) · ACM Computing Surveys · doi:10.1145/3731597 · Cited by 21
“For example, one of the most commonly used deinitions of AI is stated as "the simulation of human intelligence in computers that are programmed to think like humans and mimic their actions" [110].”
A Multi-Criteria Decision-Making Approach for the Selection of Explainable AI Methods
Matejová, M., & Paralič, J. (2025) · Machine Learning and Knowledge Extraction · doi:10.3390/make7040158 · Cited by 17
“A related metric is speed [60], which represents one of the most practically important, but less formalized in the literature, criteria for evaluating methods (XAI).”
VISUAL STORYTELLING AND EXPLAINABLE INTELLIGENCE IN ORGANIZATIONAL CHANGE COMMUNICATION
Kashyap, S., Purohit, S., Kumar, A., Jawaid, F. I. M., Kumar, J. R. R., & Ajani, S. N. (2025) · ShodhKosh Journal of Visual and Performing Arts · doi:10.29121/shodhkosh.v6.i5s.2025.6965 · Cited by 13
Bridging the Black Box: A Survey on Mechanistic Interpretability in AI
Somvanshi, S., Islam, M. M., Rafe, A., Tusti, A. G., Chakraborty, A., Baitullah, A., Chowdhury, T. I., Alnawmasi, N., Dutta, A. K., & Das, S. (2026) · ACM Computing Surveys · doi:10.1145/3787104 · Cited by 50
“Explainable-by-design models embed transparency into their architecture, as seen in decision trees, linear regression, symbolic reasoning, and rule-based systems [121, 122].”
Imaging bridges pathology and radiology
Hansmann, M., Klauschen, F., Samek, W., Müller, K., Donnadieu, E., Scharf, S., Hartmann, S., Koch, I., Ackermann, J., Pantanowitz, L., et al. (2023) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2023.100298 · Cited by 12
“Besides the analysis of raw data, machine learning methods are more and more used for post-processing tasks.(18,24) Modern machine learning algorithms are employed to identify and verify new essentials and correlations of data sets at a scale that may exceed human capabilities.”
MethodSelf-citationLomazzi, L., Morin, D., Cadini, F., Manes, A., & Aune, V. (2023) · International Journal of Protective Structures · doi:10.1177/20414196231198259 · Cited by 14
“Finally, the most in fl uential parameters from which the DNNs derive FSI predictions were identi fi ed through the explainability approach (Do ˇ silovi´c et al., 2018; Lomazzi et al., 1989; Samek et al., 2021).”
Rodriguez, O., Taylor, J. M., & Pardo, D. (2023) · Geophysical Journal International · doi:10.1093/gji/ggad362 · Cited by 12
Zhang, Y., Shao, Y., Luo, R., Xiong, L. Y., & Zhang, J. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3321056 · Cited by 12
“Nowadays deep learning has been successfully applied in pattern recognition since it can learn the features and classifi-cation boundaries jointly from the raw data [17].”
BackgroundAdding attention to the neural ordinary differential equation for spatio-temporal prediction
Wang, P., Zhang, T., Zhang, H., Cheng, S., & Wang, W. (2023) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2023.2275160 · Cited by 12
“To ensure the interpretability of the model, we propose a spatio-temporal attention module to fuse k hidden states for capturing the long-term temporal dependence in data because existing studies have shown that the attention mechanism is an inter-pretable data structure (Samek et al .”
MethodTian, W., Zhang, Z., Bouffard, D., Wu, H., Xin, K., Gu, X., & Liao, Z. (2023) · Water Research · doi:10.1016/j.watres.2023.120996 · Cited by 13
Explainable Artificial Intelligence as a Cybersecurity Aid
Doshi, R., & Hiran, K. K. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/978-1-6684-6361-1.ch003 · Cited by 13
Delivering Inflated Explanations
Izza, Y., Ignatiev, A., Stuckey, P. J., & Marques-Silva, J. P. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i11.29170 · Cited by 34
“…and Guestrin 2016; Lund-berg and Lee 2017; Ribeiro, Singh, and Guestrin 2018), heuristic learning of saliency maps (and their variants) (Bach et al. 2015; Samek and M¨uller 2019; Samek et al. 2019, 2021), but also proposals of intrinsic interpretability (Rudin 2019; Molnar 2020; Rudin et al. 2022).”
BackgroundT-TAME: Trainable Attention Mechanism for Explaining Convolutional Networks and Vision Transformers
Ntrougkas, M. V., Gkalelis, N., & Mezaris, V. (2024) · IEEE Access · doi:10.1109/access.2024.3405788 · Cited by 15
“For a more comprehensive review of the different taxonomies of explanation methods and the different approaches therein, the interested reader is referred to [9], [10], [36], [37].”
BackgroundMethodParziale, M., Silva, P. H., Giglio, M., & Cadini, F. (2024) · Structures · doi:10.1016/j.istruc.2024.107583 · Cited by 16
Zhong, X., Huang, J., Li, Y., Chen, J., Tian, Y., & Wu, Z. (2024) · Physica Scripta · doi:10.1088/1402-4896/ada20a · Cited by 14
Metadata
- Authors
- Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R.
- Affiliations
- Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (DE)
- BIFOLD✔Berlin Institute for the Foundations of Learning and Data, Berlin, Germany
- Berlin Institute for the Foundations of Learning and Data (DE)
- ORCID
- Venue
- Proceedings of the IEEE
- Published
- 2021-03
- Volume
- 109
- Issue
- 3
- Pages
- 247-278
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W3132191748
- Cited by
- 1,477
- References
- 230
- Citations by year
- 2026: 305 · 2025: 343 · 2024: 326 · 2023: 273 · 2022: 152 · 2021: 74
- FWCI
- 72.25
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Proceedings of the IEEE (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion, other-oa)
- Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korea Government (2017-0-00451, 2019-0-00079)
- German Ministry for Education and Research (01IS14013A-E, 01GQ1115, 01GQ0850, 01IS18025A, 01IS18037A)
- German Research Foundation (Math+, EXC 2046/1, 390685689)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- explainable AI, interpretability, deep neural network, post-hoc explanation, interpretability evaluation, nonlinear machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.