Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Open accessThe DOI matches its Crossref record; 223 of the 230 references it cites were found in a registry or an index.
Abstract
With the broader and highly successful usage of machine learning (ML) in industry and the sciences, there has been a growing demand for explainable artificial intelligence (XAI). Interpretability and explanation methods for gaining a better understanding of the problem-solving abilities and strategies of nonlinear ML, in particular, deep neural networks, are, therefore, receiving increased attention. In this work, we aim to: 1) provide a timely overview of this active emerging field, with a focus on “post hoc” explanations, and explain its theoretical foundations; 2) put interpretability algorithms to a test both from a theory and comparative evaluation perspective using extensive simulations; 3) outline best practice aspects, i.e., how to best include interpretation methods into the standard usage of ML; and 4) demonstrate successful usage of XAI in a representative selection of application scenarios. Finally, we discuss challenges and possible future directions of this exciting foundational field of ML.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“While XAI techniques exist [47], their application in complex, real-time energy systems, and providing meaningful explanations to non-experts, remain challenging”
From Addressing Challenges for the Effective Adoption of Artificial Intelligence in the Energy Sector (2025) · cites this paper as [47]
In the citing paper
“Despite previous attempts to address the challenges surrounding model interpretability, most solutions have been implemented post hoc (such as the attention mechanism) focusing on explanations in certain domains, such as vision [184].”
From Human-centred artificial intelligence for mobile health sensing: challenges and opportunities (2023) · cites this paper as [184]
In the citing paper
“LIME produces an explanation list that shows the contribution of individual features to the prediction.”
From Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2022) · cites this paper as [64]
In the citing paper
“This local explanation allows the end-user to determine which feature is important for the precise prediction and how the perturbation in feature affects the prediction results.”
From Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2022) · cites this paper as [64]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 12×
“This modified gradient view on LRP can also be leveraged to achieve a simpler and more general implementation of LRP based on ‘forward hooks’, which we describe in the second part of Appendix B, and which we use to apply LRP on VGG-16 [175] and ResNet-50 [65] in Section IV.”
BackgroundMethodInfluentialVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 6×
“This modified gradient view on LRP can also be leveraged to achieve a simpler and more general implementation of LRP based on ‘forward hooks’, which we describe in the second part of Appendix B, and which we use to apply LRP on VGG-16 [175] and ResNet-50 [65] in Section IV.”
BackgroundMethodInfluentialImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 4×
“Consequently, we also exchange the neural network architecture and use an AlexNet [99] model, which is able to process the transformed input data using 2-D-convolution operators.”
BackgroundMethodInfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“, [23], [52], [70], [107], [108], and [161]) have gained highest popularity.”
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 9×
“A comparison of performance on the Adience benchmark of the original model (pretrained on ImageNet only) and the improved model is given in the table below.”
MethodInfluentialRegression Models and Life-Tables
Cox, D. R. (1972) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1972.tb00899.x · Cited by 39,793
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 8×
“A comparison of performance on the Adience benchmark of the original model (pretrained on ImageNet only) and the improved model is given in the table below.”
BackgroundInfluentialThe Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 1×
“In games such as the arcade game Atari Breakout, the computer player would progressively learn strategies commonly employed by human players, such as “tunneldigging” [124], [200].”
BackgroundRegression Models and Life-Tables
Cox, D. R. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-4380-9_37 · Cited by 26,297
Semi-supervised classification with graph convolutional networks
Kipf, T. N., & Welling, M. (2016) · arXiv · doi:10.48550/arxiv.1609.02907 · Cited by 24,853 · cited 2×
“Graph neural networks [91], [160] multiply the input connectivity matrix multiple times and, consequently, also exhibit product structures.”
BackgroundGrad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
(1967) · Cited by 23,719
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 14×
“SHAP and Kernel SHAP [116] are practical algorithms to approximate Shapley values that sample a few occlusions according to the probability distribution used to compute Shapley values and then fit a linear surrogate model that correctly predicts the effect of these occlusions on the output.”
BackgroundMethodInfluentialIntroduction to Solid State Physics
Kittel, C., & Fan, H. Y. (1957) · American Journal of Physics · doi:10.1119/1.1934457 · Cited by 22,240
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570 · cited 1×
“, [23], [52], [70], [107], [108], and [161]) have gained highest popularity.”
Neural networks for pattern recognition
(1994) · Choice Reviews Online · doi:10.5860/choice.31-5500 · Cited by 18,538
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“More complex architectures involving attention mechanisms were also proposed [15], [105], [195], and inspection of the attention mechanism itself can also deliver useful insights into the model prediction.”
BackgroundVisualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 5×
“(3) can also be interpreted as a gradient that has been biased to positive values, an idea also found in methods such as DeconvNet [201] or Guided Backprop [178].”
BackgroundMethodInfluentialIntroduction to Solid State Physics
Kittel, C., & Dexter, D. L. (1953) · American Journal of Physics · doi:10.1119/1.1933590 · Cited by 13,240
Gauriat, C.-M., Pencolé, Y., Ribot, P., & Brouillet, G. (2024) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.4230/oasics.dx.2024.27 · Cited by 12,586
Neural Networks for Pattern Recognition
Bishop, C. M. (1995) · doi:10.1093/oso/9780198538493.001.0001 · Cited by 12,287
The perceptron: A probabilistic model for information storage and organization in the brain.
Rosenblatt, F. (1958) · Psychological Review · doi:10.1037/h0042519 · Cited by 12,280 · cited 1×
“Note that, in numerous scientific applications, it has been most common so far to use linear models [151], favoring interpretability often at the expense of predictivity (see [63] and [117]).”
BackgroundGaussian Processes for Machine Learning
Rasmussen, C. E., & Williams, C. K. I. (2005) · doi:10.7551/mitpress/3206.001.0001 · Cited by 11,705
Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond
Schölkopf, B., & Smola, A. J. (2001) · The MIT Press eBooks · doi:10.7551/mitpress/4175.001.0001 · Cited by 11,703
(2016) · Cited by 11,679
Learning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428 · cited 1×
“, neural networks with an explicit top-level sumpooling structure [25], [28], [111], [143], [206].”
BackgroundCaffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · cited 5×
“In order to understand the learned prediction strategies of our model and to verify that it uses meaningful features in the training data, we test different explanation methods, first, a simple occlusion and gradient analysis, and we also take an off-the-shelve explanation software, the LRP Toolbox [102] for Caffe [83…”
MethodInfluentialThe Nature of Statistical Learning Theory
Vapnik, V. N. (2000) · Springer eBooks · doi:10.1007/978-1-4757-3264-1 · Cited by 10,696 · cited 1×
“, [132], [163], [164], [190], and [194]), and particularly during the last decade, deep learning methods (e.”
BackgroundOn Estimation of a Probability Density Function and Mode
Parzen, E. (1962) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177704472 · Cited by 10,585 · cited 2×
“While a possible approach to explaining would be to train a surrogate neural network to match the output of the unsupervised model, it was found that some unsupervised models, such as k-means clustering [119] or kernel density estimation [141], can be directly rewritten as a neural network (or “neuralized”), without r…”
BackgroundRudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 2×
“, in [154]), their applicability becomes more limited when the goal is to explain the strategy of some existing black-box model.”
BackgroundFrom local explanations to global understanding with explainable AI for trees
Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., & Lee, S.-I. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-019-0138-9 · Cited by 10,551 · cited 4×
“sex of the participant (figure is adapted from [115]).”
BackgroundMethodInfluentialExplaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381 · cited 1×
“The last difficulty comes from the challenge of searching for a root point x on which to base the explanation, which is both close to the data and not an adversarial example [53], [135].”
BackgroundThe graph neural network model
Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., & Monfardini, G. (2009) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2008.2005605 · Cited by 9,582 · cited 2×
“Graph neural networks [91], [160] multiply the input connectivity matrix multiple times and, consequently, also exhibit product structures.”
BackgroundTowards Evaluating the Robustness of Neural Networks
Carlini, N., & Wagner, D. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.49 · Cited by 8,859 · cited 1×
“measurement artifacts or adversarial perturbations [182]; 2) resilience to drifting data distributions [49]; 3) ability to accurately assess the confidence of its own predictions [135], [139]; 4) safety and security aspects [21], [26], [84], [193]; 5) legal requirements or adherence to social norms [54], [60]; 6) abil…”
BackgroundShow, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143
Nonlinear Component Analysis as a Kernel Eigenvalue Problem
Schölkopf, B., Smola, A., & Müller, K.-R. (1998) · Neural Computation · doi:10.1162/089976698300017467 · Cited by 8,084 · cited 2×
“, centroid-based [119], PCA-based [64], kernelbased [141], or combinations of them [37], [72], [164])”
Background(2011) · Cited by 6,993
17. A Value for n-Person Games
Shapley, L. S. (1953) · Contributions to the Theory of Games (AM-28), Volume II · doi:10.1515/9781400881970-018 · Cited by 6,950 · cited 4×
“Here, we present three frameworks: the Shapley values [116], [169], [179] that come from game theory, the Taylor expansions [13], [19], and the deep Taylor decomposition [127], which applies Taylor expansions repeatedly at each layer of a DNN.”
BackgroundMethodInfluential“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Neural Collaborative Filtering
He, X., Liao, L., Zhang, H., Nie, L., Hu, X., & Chua, T.-S. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052569 · Cited by 6,652 · cited 1×
“Neural networks with product structures commonly occur for relational tasks, such as comparing images [122] or collaborative filtering [66].”
BackgroundSqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., & Keutzer, K. (2016) · arXiv · doi:10.48550/arxiv.1602.07360 · Cited by 6,201 · cited 2×
“Given that most ML models are differentiable, this method is widely applicable also for neural networks with complex structures, such as ResNets [65] or SqueezeNets [79].”
MethodLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 1×
“Orthogonal to the quest for better and more holistic ML models, explainable artificial intelligence (XAI) [18], [73], [112], [128], [159] has developed as a subfield of ML that seeks to augment the training process, the learned representations, and the decisions with human-interpretable explanations.”
BackgroundDeep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 3×
“For instance, the works [134], [174]— also known in this context as “deep dreams”—highlighted surprising effects when analyzing the inner behavior of deep image classification models by synthesizing meaningful preferred stimuli.”
BackgroundMethodExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments
Huang, G. B., Mattar, M., Berg, T. L., & Miller, E. G. (2008) · HAL · Cited by 4,822
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524 · cited 9×
“At the same time, these methods were able to handle different types of data, including images [13], speech [20], text [10], [38], and structured data, such as molecules [162], [165] or genetic sequences [191].”
BackgroundMethodInfluentialSelf-citationLearning With Kernels: Support Vector Machines, Regularization, Optimization, and Beyond
Williams, C. K. I. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s269 · Cited by 4,281
Axiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087 · cited 5×
“Integrated gradients [183] explain by integrating the gradient ∇f(x) along some trajectory in input space connecting some root point x to the data point x.”
BackgroundMethodInfluentialCaffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1408.5093 · Cited by 3,752 · cited 5×
“In order to understand the learned prediction strategies of our model and to verify that it uses meaningful features in the training data, we test different explanation methods, first, a simple occlusion and gradient analysis, and we also take an off-the-shelve explanation software, the LRP Toolbox [102] for Caffe [83…”
MethodInfluentialA survey on concept drift adaptation
Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014) · ACM Computing Surveys · doi:10.1145/2523813 · Cited by 3,735 · cited 1×
“measurement artifacts or adversarial perturbations [182]; 2) resilience to drifting data distributions [49]; 3) ability to accurately assess the confidence of its own predictions [135], [139]; 4) safety and security aspects [21], [26], [84], [193]; 5) legal requirements or adherence to social norms [54], [60]; 6) abil…”
BackgroundAn introduction to kernel-based learning algorithms
Muller, K.-R., Mika, S., Ratsch, G., Tsuda, K., & Scholkopf, B. (2001) · IEEE Transactions on Neural Networks · doi:10.1109/72.914517 · Cited by 3,473 · cited 1×
“, [132], [163], [164], [190], and [194]), and particularly during the last decade, deep learning methods (e.”
BackgroundStriving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447 · cited 2×
“(3) can also be interpreted as a gradient that has been biased to positive values, an idea also found in methods such as DeconvNet [201] or Guided Backprop [178].”
MethodMachine Learning Applications in Cancer Prognosis and Prediction
Kourou, K., Exarchos, T. P., Exarchos, K. P., Karamouzis, M. V., & Fotiadis, D. I. (2015) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2014.11.005 · Cited by 3,338 · cited 1×
“In a medical scenario, the function may receive as input an array of clinical variables, and the output of the function may represent the evidence for a certain medical condition [96].”
BackgroundDeep neural networks are easily fooled: High confidence predictions for unrecognizable images
Nguyen, A., Yosinski, J., & Clune, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298640 · Cited by 3,258 · cited 3×
“The last difficulty comes from the challenge of searching for a root point x on which to base the explanation, which is both close to the data and not an adversarial example [53], [135].”
BackgroundTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 2×
“Application-driven evaluations account for these additional factors, however, they are also hard to implement in practice [41].”
BackgroundArtificial intelligence and the future of work: Human-AI symbiosis in organizational decision making
Jarrahi, M. H. (2018) · Business Horizons · doi:10.1016/j.bushor.2018.03.007 · Cited by 2,772 · cited 1×
“measurement artifacts or adversarial perturbations [182]; 2) resilience to drifting data distributions [49]; 3) ability to accurately assess the confidence of its own predictions [135], [139]; 4) safety and security aspects [21], [26], [84], [193]; 5) legal requirements or adherence to social norms [54], [60]; 6) abil…”
BackgroundLearning Important Features Through Propagating Activation Differences
Shrikumar, A., Greenside, P., & Kundaje, A. (2017) · arXiv · doi:10.48550/arxiv.1704.02685 · Cited by 2,761 · cited 1×
“Furthermore, a number of other propagation techniques have been proposed [100], [170], [171], [202] with some of the rules overlapping with LRP for certain choices of parameters.”
BackgroundImageNet: A large-scale hierarchical image database
Deng, J. H., Dong, W. F., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvprw.2009.5206848 · Cited by 2,714
Deep Clustering for Unsupervised Learning of Visual Features
Caron, M., Bojanowski, P., Joulin, A., & Douze, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_9 · Cited by 2,674
Methods for interpreting and understanding deep neural networks
Montavon, G., Samek, W., & Müller, K.-R. (2018) · Digital Signal Processing · doi:10.1016/j.dsp.2017.10.011 · Cited by 2,665 · cited 4×
“P R A C T I C A L M E T H O D S F O R E X P L A I N I N G D N N s In view of the multiple challenges posed by analyzing deep neural network functions, building robust and practical methods to explain their decisions has developed into an own research area [59], [128], [159], and an abundance of methods has been propos…”
BackgroundMethodInfluentialSelf-citationKhan, J., Wei, J. S., Ringnér, M., Saal, L. H., Ladanyi, M., Westermann, F., Berthold, F., Schwab, M., Antonescu, C. R., Peterson, C., et al. (2001) · Nature Medicine · doi:10.1038/89044 · Cited by 2,644 · cited 1×
“measurement artifacts or adversarial perturbations [182]; 2) resilience to drifting data distributions [49]; 3) ability to accurately assess the confidence of its own predictions [135], [139]; 4) safety and security aspects [21], [26], [84], [193]; 5) legal requirements or adherence to social norms [54], [60]; 6) abil…”
BackgroundSocial GAN: Socially Acceptable Trajectories with Generative Adversarial Networks
Gupta, A., Johnson, J., Fei-Fei, L., Savarese, S., & Alahi, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00240 · Cited by 2,466
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 1×
“Other methods that explain by building a local surrogate include LORE [58] and Anchors [148].”
MethodEuropean Union Regulations on Algorithmic Decision Making and a “Right to Explanation”
Goodman, B., & Flaxman, S. (2017) · AI Magazine · doi:10.1609/aimag.v38i3.2741 · Cited by 2,131 · cited 1×
“measurement artifacts or adversarial perturbations [182]; 2) resilience to drifting data distributions [49]; 3) ability to accurately assess the confidence of its own predictions [135], [139]; 4) safety and security aspects [21], [26], [84], [193]; 5) legal requirements or adherence to social norms [54], [60]; 6) abil…”
BackgroundOptimizing Spatial filters for Robust EEG Single-Trial Analysis
Blankertz, B., Tomioka, R., Lemm, S., Kawanabe, M., & Muller, K.-R. (2008) · IEEE Signal Processing Magazine · doi:10.1109/msp.2008.4408441 · Cited by 2,080 · cited 2×
“However, in contrast to the patterns computed with conventional approaches [24], [63], which only allows visualizing the aggregated information (average activity) per class, the explanations computed with LRP are available for every single input of the deep learning classifier, i.”
BackgroundIntroduction to Solid State Physics
Aharony, A., & Entin-Wohlman, O. (2018) · WORLD SCIENTIFIC eBooks · doi:10.1142/11041 · Cited by 1,917
Neural networks for pattern recognition
Marshall, J. A. (1995) · Neural Networks · doi:10.1016/0893-6080(95)90002-0 · Cited by 1,889
Causability and explainability of artificial intelligence in medicine
Holzinger, A., Langs, G., Denk, H., Zatloukal, K., & Müller, H. (2019) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1312 · Cited by 1,885 · cited 1×
“An important application area is in medicine [74].”
BackgroundRuff, L., Vandermeulen, R. A., Görnitz, N., Deecke, L., Siddiqui, S. A., Binder, A., Müller, E., & Kloft, M. (2018) · International Conference on Machine Learning · Cited by 1,719
Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks
Katz, G., Barrett, C., Dill, D. L., Julian, K., & Kochenderfer, M. J. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-63387-9_5 · Cited by 1,706 · cited 2×
“measurement artifacts or adversarial perturbations [182]; 2) resilience to drifting data distributions [49]; 3) ability to accurately assess the confidence of its own predictions [135], [139]; 4) safety and security aspects [21], [26], [84], [193]; 5) legal requirements or adherence to social norms [54], [60]; 6) abil…”
BackgroundIntelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685 · cited 3×
“Technically, explanation methods have been applied to a broad range of models ranging from simple bag-ofwords-type classifiers or logistic regression [13], [28] to feedforward or recurrent deep neural networks [9], [11], [13], [171] and, more recently, also to unsupervised learning models [85], [86].”
BackgroundMethodSmoothGrad: removing noise by adding noise
Smilkov, D., Thorat, N., Kim, B., Viégas, F., & Wattenberg, M. (2017) · arXiv · doi:10.48550/arxiv.1706.03825 · Cited by 1,467 · cited 8×
“In Section IV, we experiment with a combination of Integrated Gradients and SmoothGrad [176], similar to Expected Gradients (cf. [ 181]), where relevance scores obtained from Integrated Gradients are averaged over several integration paths that are drawn from some random distribution.”
BackgroundMethodInfluentialModeling of strength of high-performance concrete using artificial neural networks
Yeh, I.-C. (1998) · Cement and Concrete Research · doi:10.1016/s0008-8846(98)00165-3 · Cited by 1,447 · cited 4×
“Concrete Compressive Strength Data set [197], displayed at”
BackgroundMethodInfluentialQuantum-chemical insights from deep tensor neural networks
Schütt, K. T., Arbabzadah, F., Chmiela, S., Müller, K. R., & Tkatchenko, A. (2017) · Nature Communications · doi:10.1038/ncomms13890 · Cited by 1,445 · cited 5×
“At the same time, these methods were able to handle different types of data, including images [13], speech [20], text [10], [38], and structured data, such as molecules [162], [165] or genetic sequences [191].”
BackgroundMethodInfluentialSelf-citationOn the interpretation of weight vectors of linear models in multivariate neuroimaging
Haufe, S., Meinecke, F., Görgen, K., Dähne, S., Haynes, J.-D., Blankertz, B., & Bießmann, F. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2013.10.067 · Cited by 1,389 · cited 4×
“, scientists had to resort to the use of linear models (see [63] and [117]), even if this came at the expense of predictivity.”
BackgroundInfluentialExplainable AI: Interpreting, Explaining and Visualizing Deep Learning
(2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28954-6 · Cited by 1,377 · cited 31×
“Orthogonal to the quest for better and more holistic machine learning models, Explainable AI (XAI) [159], [73], [112], [128], [18] has developed as a subfield of machine learning that seeks to augment the training process, the learned representations and the decisions with human-interpretable explanations.”
BackgroundInfluentialExplaining nonlinear classification decisions with deep taylor decomposition
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K.-R. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.11.008 · Cited by 1,357 · cited 9×
“An alternate way of formalizing the problem of attribution of a function onto input features is offered by the recent framework of deep Taylor decomposition (DTD) [127].”
BackgroundMethodInfluentialSelf-citationLeCun, Y., Bottou, L., Orr, G., & Müller, K. (1998) · neural information processing systems · Cited by 1,333
LeCun, Y. A., Bottou, L., Orr, G. B., & Müller, K.-R. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_3 · Cited by 1,264 · cited 3×
“First, test results after training using stochastic gradient descent (SGD) [108] report reasonable performance, with exact and one-off [44], [168] prediction accuracy(3) of 56.”
InfluentialKim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., & Sayres, R. (2017) · arXiv · doi:10.48550/arxiv.1711.11279 · Cited by 1,240 · cited 2×
“For example, the TCAV method [89] considers directional derivatives in the space of activations (where the directions correspond to higher level human-interpretable concepts) in place of the input gradient.”
MethodKernel k-means: spectral clustering and normalized cuts
Dhillon, I. S., Guan, Y., & Kulis, B. (2004) · Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1014052.1014118 · Cited by 1,193
Evaluating the Visualization of What a Deep Neural Network Has Learned
Samek, W., Binder, A., Montavon, G., Lapuschkin, S., & Muller, K.-R. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2599820 · Cited by 1,164
Self-citationInterpretable Explanations of Black Boxes by Meaningful Perturbation
Fong, R. C., & Vedaldi, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.371 · Cited by 1,097 · cited 1×
“A further extension of occlusion analysis is meaningful perturbation [47], where an occluding pattern is synthesized, subject to a sparsity constraint, in order to engender the maximum drop of the function value f .”
BackgroundUnmasking Clever Hans predictors and assessing what machines really learn
Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., & Müller, K.-R. (2019) · Nature Communications · doi:10.1038/s41467-019-08987-4 · Cited by 1,047 · cited 10×
“In ML, the term “Clever Hans” can be used to designate strategies that mimic the expected behavior but are based on unexpected correlations or artifacts in the data [104].”
BackgroundMethodInfluentialSelf-citationExplainable machine learning for scientific insights and discoveries
Roscher, R., Bohn, B., Duarte, M. F., & Garcke, J. (2020) · IEEE Access · doi:10.1109/access.2020.2976199 · Cited by 1,040 · cited 1×
“Only recently, the technologies have become available to extract scientific insights from complex nonlinear models [121], [150], [166].”
Background(2014) · arXiv (Cornell University) · Cited by 942
A Unifying Review of Deep and Shallow Anomaly Detection
Ruff, L., Kauffmann, J. R., Vandermeulen, R. A., Montavon, G., Samek, W., Kloft, M., Dietterich, T. G., & Muller, K.-R. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3052449 · Cited by 914 · cited 1×
“However, for unsupervised problems, such as anomaly detection or clustering, although deep models have also reached successes [27], [155], [156], shallow models (e.”
BackgroundSelf-citationGNNExplainer: Generating Explanations for Graph Neural Networks
Ying, R., Bourgeois, D., You, J., Zitnik, M., & Leskovec, J. (2019) · arXiv · doi:10.48550/arxiv.1903.03894 · Cited by 850
On the Number of Linear Regions of Deep Neural Networks
Montúfar, G., Pascanu, R., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1402.1869 · Cited by 840 · cited 1×
“In particular, it can be shown that, for deep rectifier networks, the number of discontinuities of the gradient can grow in the worst case exponentially with depth [129].”
BackgroundAge and Gender Estimation of Unfiltered Faces
Eidinger, E., Enbar, R., & Hassner, T. (2014) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2014.2359646 · Cited by 809 · cited 3×
“First, test results after training using stochastic gradient descent (SGD) [108] report reasonable performance, with exact and one-off [44], [168] prediction accuracy(3) of 56.”
MethodToward Brain-Computer Interfacing
(2007) · The MIT Press eBooks · doi:10.7551/mitpress/7493.001.0001 · Cited by 785
Kernel PCA for novelty detection
Hoffmann, H. (2007) · Pattern Recognition · doi:10.1016/j.patcog.2006.07.009 · Cited by 762 · cited 1×
“, centroid-based [119], PCA-based [64], kernelbased [141], or combinations of them [37], [72], [164])”
Layer-Wise Relevance Propagation: An Overview
Montavon, G., Binder, A., Lapuschkin, S., Samek, W., & Müller, K.-R. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28954-6_10 · Cited by 750 · cited 10×
“A similar procedure with another choice of reference point gives LRP- (see [126]).”
BackgroundMethodInfluentialSelf-citationDEX: Deep EXpectation of Apparent Age from a Single Image
Rothe, R., Timofte, R., & Gool, L. V. (2015) · 2015 IEEE International Conference on Computer Vision Workshop (ICCVW) · doi:10.1109/iccvw.2015.41 · Cited by 688 · cited 8×
“The IMDB-WIKI dataset consists of 523,051 images from 20,284 celebrities (460,723 images from the Internet Movie Data Base (Imdb) and 62,328 images from Wikipedia) at different ages, labelled with 101 labels (0-100 years, one label per year).”
MethodInfluentialTop-Down Neural Attention by Excitation Backprop
Zhang, J., Bargal, S. A., Lin, Z., Brandt, J., Shen, X., & Sclaroff, S. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-017-1059-x · Cited by 680 · cited 1×
“Furthermore, a number of other propagation techniques have been proposed [100], [170], [171], [202] with some of the rules overlapping with LRP for certain choices of parameters.”
BackgroundA Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · cited 1×
“P R A C T I C A L M E T H O D S F O R E X P L A I N I N G D N N s In view of the multiple challenges posed by analyzing deep neural network functions, building robust and practical methods to explain their decisions has developed into an own research area [59], [128], [159], and an abundance of methods has been propos…”
BackgroundPolynomial calculation of the Shapley value based on sampling
Castro, J., Gómez, D., & Tejada, J. (2009) · Computers & Operations Research · doi:10.1016/j.cor.2008.04.004 · Cited by 601 · cited 1×
“, the occlusion method [201] and Shapley value resampling [29].”
MethodUnmasking Clever Hans predictors and assessing what machines really learn
Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., & Müller, K. (2019) · RePEc: Research Papers in Economics · Cited by 580
Self-citationInceptionism: Going Deeper into Neural Networks
Mordvintsev, A., Olah, C., & Tyka, M. (2015) · Cited by 574
Making the Black Box More Transparent: Understanding the Physical Implications of Machine Learning
McGovern, A., Lagerquist, R., John Gagne, D., Jergensen, G. E., Elmore, K. L., Homeyer, C. R., & Smith, T. (2019) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-18-0195.1 · Cited by 558
An explainable deep machine vision framework for plant stress phenotyping
Ghosal, S., Blystone, D., Singh, A. K., Ganapathysubramanian, B., Singh, A., & Sarkar, S. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1716999115 · Cited by 553 · cited 1×
“[50] also rely on training and explaining DNN image classifiers to get novel insights, this time in the area of plant stress classification.”
BackgroundSequential regulatory activity prediction across chromosomes with convolutional neural networks
Kelley, D. R., Reshef, Y. A., Bileschi, M., Belanger, D., McLean, C. Y., & Snoek, J. (2018) · Genome Research · doi:10.1101/gr.227819.117 · Cited by 537 · cited 1×
“[87] uses a gradient-based explanation technique to understand the activity prediction across chromosomes, whereas Chong et al.”
MethodGenerating Visual Explanations
Hendricks, L. A., Akata, Z., Rohrbach, M., Donahue, J., Schiele, B., & Darrell, T. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_1 · Cited by 534
Deep image mining for diabetic retinopathy screening
Quellec, G., Charrière, K., Boudi, Y., Cochener, B., & Lamard, M. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.04.012 · Cited by 520 · cited 1×
“Further applications of DNN explanation techniques for medical images include the detection of lesions in diabetic retinopathy data [145], pixelwise analysis of microscopy images from global image annotations [98], the validation of predictions in dermatology [199], and analysis of X-ray images [48].”
BackgroundRight for the Right Reasons: Training Differentiable Models by Constraining their Explanations
Ross, A. S., Hughes, M. C., & Doshi-Velez, F. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/371 · Cited by 515
Actionable Recourse in Linear Classification
Ustun, B., Spangher, A., & Liu, Y. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287566 · Cited by 501 · cited 1×
“Here, an explanation could prove equally useful as it could hint at actions to be taken on the sample to improve its predicted score [189], e.”
BackgroundGAN Dissection: Visualizing and Understanding Generative Adversarial NetworksPreprint
Bau, D., Zhu, J.-Y., Strobelt, H., Zhou, B., Tenenbaum, J. B., Freeman, W. T., & Torralba, A. B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.10597 · Cited by 499 · cited 1×
“Further methods do not seek to explain in terms of input features but in terms of the latent space, where the directions in the latent space code for higher level concepts, such as color, material, object part, or object [17], [18], [205].”
Not Just a Black Box: Learning Important Features Through Propagating Activation Differences
Shrikumar, A., Greenside, P., Shcherbina, A., & Kundaje, A. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.01713 · Cited by 496 · cited 4×
“T H E O R E T I C A L F O U N D AT I O N S O F E X P L A N AT I O N M E T H O D S In parallel to developing explanation methods that address application requirements, such as faithfulness, interpretability, usability, and runtime, some works have focused on building theoretical foundations for the problem of explanati…”
BackgroundMethodInfluentialVisualizing Deep Neural Network Decisions: Prediction Difference Analysis
Zintgraf, L. M., Cohen, T. S., Adel, T., & Welling, M. (2017) · arXiv · doi:10.48550/arxiv.1702.04595 · Cited by 487
The Shattered Gradients Problem: If resnets are the answer, then what is the question?Preprint
Balduzzi, D., Frean, M. R., Leary, L., Lewis, J. P., Ma, W.-D. K., & McWilliams, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.08591 · Cited by 472 · cited 1×
“of recent neural networks, where a ‘shattered gradient’ effect was observed [16], noting that the gradient locally resembles white noise.”
BackgroundLauritsen, S. M., Kristensen, M., Olsen, M. V., Larsen, M. S., Lauritsen, K. M., Jørgensen, M. J., Lange, J., & Thiesson, B. (2020) · Nature Communications · doi:10.1038/s41467-020-17431-x · Cited by 448 · cited 1×
“In another work [106], the input time series is formed by EHRs sampled at a much lower rate, from which a DNN model can decide among a set of medical conditions.”
BackgroundLearning to combine foveal glimpses with a third-order Boltzmann machine
Larochelle, H., & Hinton, G. E. (2010) · neural information processing systems · Cited by 436
Binding in models of perception and brain function
Malsburg, C. V. D. (1995) · Current Opinion in Neurobiology · doi:10.1016/0959-4388(95)80014-x · Cited by 417 · cited 2×
“In the neurosciences, von der Malsburg has coined the concept of ‘binding’ for neural strategies that allow sets of variables (neurons) to synchronize collectively by learning [192].”
BackgroundClassifying and segmenting microscopy images with deep multiple instance learning
Kraus, O. Z., Ba, J. L., & Frey, B. J. (2016) · Bioinformatics · doi:10.1093/bioinformatics/btw252 · Cited by 411 · cited 1×
“Further applications of DNN explanation techniques for medical images include the detection of lesions in diabetic retinopathy data [145], pixelwise analysis of microscopy images from global image annotations [98], the validation of predictions in dermatology [199], and analysis of X-ray images [48].”
BackgroundiNNvestigate neural networks!Preprint
Alber, M., Lapuschkin, S., Seegerer, P., Hägele, M., Schütt, K. T., Montavon, G., Samek, W., Müller, K., Dähne, S., & Kindermans, P.-J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.04260 · Cited by 400 · cited 2×
“Because highly predictive models are becoming increasingly complex both in terms of the number of parameters and structure of the model, XAI software must keep up with this increasing complexity [3], and at the same time, the human must also deal with explanations of increasingly subtler predictions [14].”
BackgroundMethodSelf-citationAn Efficient Explanation of Individual Classifications using Game Theory
StrumbeljErik, & KononenkoIgor (2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1756007 · Cited by 380 · cited 3×
“When transposing the method to the task of explaining an ML model [116], [179], the players of the cooperating game become the input features, and the payoff function becomes related to the DNN output.”
MethodUnderstanding the role of individual units in a deep neural network
Bau, D., Zhu, J.-Y., Strobelt, H., Lapedriza, A., Zhou, B., & Torralba, A. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1907375117 · Cited by 375 · cited 29×
“Orthogonal to the quest for better and more holistic machine learning models, Explainable AI (XAI) [159], [73], [112], [128], [18] has developed as a subfield of machine learning that seeks to augment the training process, the learned representations and the decisions with human-interpretable explanations.”
BackgroundInfluentialInterpretable deep neural networks for single-trial EEG classification
Sturm, I., Lapuschkin, S., Samek, W., & Müller, K.-R. (2016) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2016.10.008 · Cited by 374
Self-citationNguyen, A. M., Yosinski, J., & Clune, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.03616 · Cited by 371 · cited 1×
“Certain approaches, such as activationmaximization [134], [136], [174], aim at a global interpretation of the model, by identifying prototypical cases for the output quantity”
BackgroundTowards better understanding of gradient-based attribution methods for Deep Neural Networks
Ancona, M., Ceolini, E., Öztireli, C., & Groß, M. H. (2017) · Zurich Open Repository and Archive (University of Zurich) · doi:10.3929/ethz-b-000249929 · Cited by 368 · cited 4×
“Both the DeepExplain [4] and iNNvestigate [3] toolboxes built on top of the popular Keras [30] package for Python with TensorFlow backend for explaining Deep Neural Network models, and thus provide support for both CPU and GPU hardware and convenient access for users of Keras models.”
MethodInfluentialA Benchmark for Interpretability Methods in Deep Neural Networks
Hooker, S., Erhan, D., Kindermans, P.-J., & Kim, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.10758 · Cited by 365
Explaining Recurrent Neural Network Predictions in Sentiment Analysis
Arras, L., Montavon, G., Müller, K.-R., & Samek, W. (2017) · Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis · doi:10.18653/v1/w17-5221 · Cited by 361 · cited 3×
“Other neural network architectures have a more conventional structure but still require a nontrivial adaptation of existing explanation methods, for example, extensions of LRP have been proposed to deal with the special LSTM blocks in recurrent neural networks [9], [11] or to handle attention units in the context of n…”
BackgroundMethodSelf-citationSu, D., Zhang, H., Chen, H., Yi, J., Chen, P.-Y., & Gao, Y. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01258-8_39 · Cited by 342
Interpreting Deep Visual Representations via Network Dissection
Zhou, B., Bau, D., Oliva, A., & Torralba, A. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2858759 · Cited by 330 · cited 1×
“Further methods do not seek to explain in terms of input features but in terms of the latent space, where the directions in the latent space code for higher level concepts, such as color, material, object part, or object [17], [18], [205].”
Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
Brendel, W., & Bethge, M. (2019) · arXiv · doi:10.48550/arxiv.1904.00760 · Cited by 323 · cited 1×
“neural networks with an explicit top-level sum-pooling structure [143], [111], [28], [206], [25].”
BackgroundPrediction of mortality from 12-lead electrocardiogram voltage data using a deep neural network
Raghunath, S., Ulloa Cerna, A. E., Jing, L., vanMaanen, D. P., Stough, J., Hartzel, D. N., Leader, J. B., Kirchner, H. L., Stumpe, M. C., Hafez, A., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0870-z · Cited by 314
Plan and operation of the NHANES I Epidemiologic Followup Study, 1986.
Cox, C. S., Rothwell, S. T., Madans, J. H., Finucane, F. F., Freid, V. M., Kleinman, J. C., Barbano, H. E., & Feldman, J. J. (1987) · PubMed · Cited by 311
Pathologist-level interpretable whole-slide cancer diagnosis with deep learning
Zhang, Z., Chen, P., McGough, M., Xing, F., Wang, C., Bui, M., Xie, Y., Sapkota, M., Cui, L., Dhillon, J., et al. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0052-1 · Cited by 311 · cited 1×
“[204] also generate a free-text pathology report to clarify the decision of the classifier.”
BackgroundExplaining Deep Neural Networks with a Polynomial Time Algorithm for Shapley Values Approximation
Ancona, M., Öztireli, C., & Groß, M. H. (2019) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000342737 · Cited by 310
From Machine Learning to Explainable AI
Holzinger, A. (2018) · 2018 World Symposium on Digital Intelligence for Systems and Machines (DISA) · doi:10.1109/disa.2018.8490530 · Cited by 306 · cited 1×
“Orthogonal to the quest for better and more holistic ML models, explainable artificial intelligence (XAI) [18], [73], [112], [128], [159] has developed as a subfield of ML that seeks to augment the training process, the learned representations, and the decisions with human-interpretable explanations.”
BackgroundGraying the black box: Understanding DQNsPreprint
Zahavy, T., Ben-Zrihem, N., & Mannor, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.02658 · Cited by 304 · cited 1×
“In games such as the arcade game Atari Breakout, the computer player would progressively learn strategies commonly employed by human players, such as “tunneldigging” [124], [200].”
BackgroundAn axiomatic approach to the concept of interaction among players in cooperative games
Grabisch, M., & Roubens, M. (1999) · International Journal of Game Theory · doi:10.1007/s001820050125 · Cited by 299 · cited 1×
“Finally, we note that the Shapley framework also offers the possibility to quantify the joint contribution of two interacting players through the Shapley interaction index [55].”
MethodSafe Model-based Reinforcement Learning with Stability Guarantees
Berkenkamp, F., Turchetta, M., Schoellig, A. P., & Krause, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.08551 · Cited by 299 · cited 1×
“Instead, in practical engineering of systems, critical features that are typically considered beyond excellent prediction itself are (a) robustness of the system to measurement artefacts or adversarial perturbations [182], (b) resilience to drifting data distributions [49], (c) ability to accurately assess the confide…”
BackgroundLearning how to explain neural networks: PatternNet and PatternAttribution
Kindermans, P.-J., Schütt, K. T., Alber, M., Müller, K. R., Erhan, D., Kim, B., & Dähne, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.05598 · Cited by 264
Higher-Order Explanations of Graph Neural Networks via Relevant Walks
Schnake, T., Eberle, O., Lederer, J., Nakajima, S., Schutt, K. T., Muller, K.-R., & Montavon, G. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3115452 · Cited by 259 · cited 5×
“At the same time, these methods were able to handle different types of data, including images [13], speech [20], text [10], [38], and structured data, such as molecules [162], [165] or genetic sequences [191].”
BackgroundMethodInfluentialSelf-citation"What is relevant in a text document?": An interpretable machine learning approach
Arras, L., Horn, F., Montavon, G., Müller, K.-R., & Samek, W. (2017) · PLOS ONE · doi:10.1371/journal.pone.0181142 · Cited by 257 · cited 1×
“At the same time, these methods were able to handle different types of data, including images [13], speech [20], text [10], [38], and structured data, such as molecules [162], [165] or genetic sequences [191].”
MethodSelf-citationExplaining the unique nature of individual gait patterns with deep learning
Horst, F., Lapuschkin, S., Samek, W., Müller, K.-R., & Schöllhorn, W. I. (2019) · Scientific Reports · doi:10.1038/s41598-019-38748-8 · Cited by 256 · cited 1×
“[77] introduce explanation techniques to the domain of human gait recognition and show that nonlinear learning models are not only the better predictors but also they can, at the same time, learn physiologically meaningful features for subject prediction, which aligns with expected features used by linear models.”
BackgroundSelf-citationSupervised group Lasso with applications to microarray data analysis
Ma, S., Song, X., & Huang, J. (2007) · BMC Bioinformatics · doi:10.1186/1471-2105-8-60 · Cited by 238
Synthesizing the preferred inputs for neurons in neural networks via deep generator networks
Nguyen, A., Dosovitskiy, A., Yosinski, J., Brox, T., & Clune, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.09304 · Cited by 230
Towards Best Practice in Explaining Neural Network Decisions with LRP
Kohlbrenner, M., Bauer, A., Nakajima, S., Binder, A., Samek, W., & Lapuschkin, S. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206975 · Cited by 228 · cited 1×
“Specifically, we will now try an alternate LRP preset called “LRP-CMP” that applies a composite strategy [94], [103], [126] where different rules are applied at different layers.”
MethodSelf-citationExplaining Explanations: Axiomatic Feature Interactions for Deep NetworksPreprint
Janizek, J. D., Sturmfels, P., & Lee, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.04138 · Cited by 228 · cited 2×
“Other higher order methods include GNN-LRP that explains graph neural networks in terms of collections of edges [162] or integrated Hessians [81], which can be seen as the second-order extension of integrated gradients.”
MethodFooling Neural Network Interpretations via Adversarial Model ManipulationPreprint
Heo, J., Joo, S., & Moon, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.02041 · Cited by 225
Invited Commentary: Causal Diagrams and Measurement Bias
Hernan, M. A., & Cole, S. R. (2009) · American Journal of Epidemiology · doi:10.1093/aje/kwp293 · Cited by 220 · cited 1×
“, deriving mortality risk factors from epidemiological data, remains difficult due to unobserved effects, such as collider or measurement bias [69].”
The frequency range of the voice fundamental in the speech of male and female adults
Traunmüller, H., & Eriksson, A. (1993) · Cited by 211
Confidence estimation methods for neural networks: a practical comparison
Papadopoulos, G., Edwards, P. J., & Murray, A. F. (2001) · IEEE Transactions on Neural Networks · doi:10.1109/72.963764 · Cited by 206 · cited 1×
“measurement artifacts or adversarial perturbations [182]; 2) resilience to drifting data distributions [49]; 3) ability to accurately assess the confidence of its own predictions [135], [139]; 4) safety and security aspects [21], [26], [84], [193]; 5) legal requirements or adherence to social norms [54], [60]; 6) abil…”
BackgroundNeuralHydrology – Interpreting LSTMs in Hydrology
Kratzert, F., Herrnegger, M., Klotz, D., Hochreiter, S., & Klambauer, G. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_19 · Cited by 203 · cited 1×
“Another work [97] applies long short-term memory (LSTM) networks to the field of hydrology to predict the river discharge from meteorological observations.”
BackgroundAnalyzing Classifiers: Fisher Vectors and Deep Neural Networks
Lapuschkin, S., Binder, A., Montavon, G., Muller, K.-R., & Samek, W. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.318 · Cited by 201
Self-citationVisualizing the Impact of Feature Attribution Baselines
Sturmfels, P., Lundberg, S., & Lee, S.-I. (2020) · Distill · doi:10.23915/distill.00022 · Cited by 200 · cited 1×
“In Section IV, we experiment with a combination of integrated gradients and SmoothGrad [176], similar to expected gradients (see [181]), where relevance scores obtained from integrated gradients are averaged over several integration paths that are drawn from some random distribution.”
MethodExplainable and Interpretable Models in Computer Vision and Machine Learning
(2018) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-319-98131-4 · Cited by 199 · cited 1×
“[165], [22], [71], [186], [45], [159], [140]) ultimately contributing to further our sci-”
RISE: Randomized Input Sampling for Explanation of Black-box Models
Petsiuk, V., Das, A., & Saenko, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07421 · Cited by 196
Finding and removing Clever Hans: Using explanation methods to debug and improve deep models
Anders, C. J., Weber, L., Neumann, D., Samek, W., Müller, K., & Lapuschkin, S. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.07.015 · Cited by 189 · cited 3×
“7Or in the case of [7] deteriorate model performance, as the identified confounding feature is exclusive to the training data.”
BackgroundMethodInfluentialSelf-citationDetecting Statistical Interactions from Neural Network WeightsPreprint
Tsang, M., Cheng, D., & Liu, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.04977 · Cited by 188 · cited 1×
“While steps have already been taken in XAI to identify interacting variables [34], [42], [115], [162], [188], it will be necessary for a broader usage to generalize the approach, e.”
BackgroundLocal Rule-Based Explanations of Black Box Decision Systems
Guidotti, R., Monreale, A., Ruggieri, S., Pedreschi, D., Turini, F., & Giannotti, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.10820 · Cited by 179 · cited 1×
“Other methods that explain by building a local surrogate include LORE [58] and Anchors [148].”
MethodCLEVR-XAI: A benchmark dataset for the ground truth evaluation of neural network explanations
Arras, L., Osman, A., & Samek, W. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.11.008 · Cited by 179
Self-citationExplanations can be manipulated and geometry is to blame
Dombrowski, A.-K., Alber, M., Anders, C. J., Ackermann, M., Müller, K.-R., & Kessel, P. (2019) · arXiv · doi:10.48550/arxiv.1906.07983 · Cited by 171 · cited 1×
“Relatedly, even when the model remains unchanged, inputs could be perturbed imperceptibly to produce arbitrary explanations [39].”
BackgroundSelf-citationEvaluating Explanation Methods for Deep Learning in Security
Warnecke, A., Arp, D., Wressnegger, C., & Rieck, K. (2020) · IEEE European Symposium on Security and Privacy · doi:10.1109/eurosp48549.2020.00018 · Cited by 170 · cited 1×
“measurement artifacts or adversarial perturbations [182]; 2) resilience to drifting data distributions [49]; 3) ability to accurately assess the confidence of its own predictions [135], [139]; 4) safety and security aspects [21], [26], [84], [193]; 5) legal requirements or adherence to social norms [54], [60]; 6) abil…”
BackgroundVisualizing and Understanding Neural Machine Translation
Ding, Y., Liu, Y., Luan, H., & Sun, M. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1106 · Cited by 168 · cited 2×
“At the same time, these methods were able to handle different types of data, including images [13], speech [20], text [10], [38], and structured data, such as molecules [162], [165] or genetic sequences [191].”
MethodScoring of tumor-infiltrating lymphocytes: From visual estimation to machine learning
Klauschen, F., Müller, K.-R., Binder, A., Bockmayr, M., Hägele, M., Seegerer, P., Wienert, S., Pruneri, G., de Maria, S., Badve, S., et al. (2018) · Seminars in Cancer Biology · doi:10.1016/j.semcancer.2018.07.001 · Cited by 168 · cited 1×
“, lymphocytes, can be performed on these explanations (see [93]).”
BackgroundUnderstanding and Comparing Deep Neural Networks for Age and Gender Classification
Samek, W., Binder, A., Lapuschkin, S., & Müller, K. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.191 · Cited by 168 · cited 1×
“Specifically, we will now try an alternate LRP preset called ‘LRP-CMP’ that applies a composite strategy [103], [126], [94] where different rules are applied at different layers.”
MethodSelf-citationHägele, M., Seegerer, P., Lapuschkin, S., Bockmayr, M., Samek, W., Klauschen, F., Müller, K.-R., & Binder, A. (2020) · Scientific Reports · doi:10.1038/s41598-020-62724-2 · Cited by 165 · cited 1×
“Recently, deeper models have also been used for predicting and explaining tissue type [61].”
MethodSelf-citationFrom Clustering to Cluster Explanations via Neural Networks
Kauffmann, J., Esders, M., Ruff, L., Montavon, G., Samek, W., & Müller, K. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3185901 · Cited by 161
Self-citationExplanation in Second Generation Expert Systems
Swartout, W. R., & Moore, J. D. (1993) · Second Generation Expert Systems · doi:10.1007/978-3-642-77927-5_24 · Cited by 160 · cited 2×
“general assessment of the overall practical value of an explanation, which further includes human factors, such as understandability, manageability, and overall utility of the XAI system [133], [147], [184].”
BackgroundTowards Explainable NLP: A Generative Explanation Framework for Text Classification
Liu, H., Yin, Q., & Wang, W. Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1560 · Cited by 160
Morphological and molecular breast cancer profiling through explainable machine learning
Binder, A., Bockmayr, M., Hägele, M., Wienert, S., Heim, D., Hellweg, K., Ishii, M., Stenzinger, A., Hocke, A., Denkert, C., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00303-4 · Cited by 156 · cited 2×
“, [22], [45], [71], [140], [159], [165], [186]) ultimately contributing to further our scientific knowledge.”
BackgroundSelf-citationNarayanan, M., Chen, E., He, J., Kim, B., Gershman, S., & Doshi-Velez, F. (2018) · arXiv · doi:10.48550/arxiv.1802.00682 · Cited by 153 · cited 2×
“general assessment of the overall practical value of an explanation, which further includes human factors, such as understandability, manageability, and overall utility of the XAI system [133], [147], [184].”
BackgroundLocal Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter ValuesPreprint
Adebayo, J., Gilmer, J., Goodfellow, I., & Kim, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.03307 · Cited by 151
RUDDER: Return Decomposition for Delayed RewardsPreprint
Arjona-Medina, J. A., Gillhofer, M., Widrich, M., Unterthiner, T., Brandstetter, J., & Hochreiter, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07857 · Cited by 146 · cited 1×
“Furthermore, initial steps have been taken to use XAI beyond validation to arrive at better and more predictive models e.g. [152], [6], [8], [7].”
MethodPark, Y., Kwon, B., Heo, J., Hu, X., Liu, Y., & Moon, T. (2019) · Environmental Pollution · doi:10.1016/j.envpol.2019.113395 · Cited by 143 · cited 1×
“, [22], [45], [71], [140], [159], [165], [186]) ultimately contributing to further our scientific knowledge.”
BackgroundInterpretable Deep Learning in Drug Discovery
Preuer, K., Klambauer, G., Rippmann, F., Hochreiter, S., & Unterthiner, T. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_18 · Cited by 142
Galaxy morphology classification with deep convolutional neural networks
Zhu, X., Dai, J.-M., Bian, C., Chen, Y., Chen, S., & Hu, C. (2019) · Astrophysics and Space Science · doi:10.1007/s10509-019-3540-1 · Cited by 141 · cited 1×
“20(c) shows an ML-based analysis of galaxy morphologies [207].”
BackgroundExplaining and Interpreting LSTMs
Arras, L., Arjona-Medina, J., Widrich, M., Montavon, G., Gillhofer, M., Müller, K.-R., Hochreiter, S., & Samek, W. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_11 · Cited by 135 · cited 3×
“Other neural network architectures have a more conventional structure but still require a nontrivial adaptation of existing explanation methods, for example, extensions of LRP have been proposed to deal with the special LSTM blocks in recurrent neural networks [9], [11] or to handle attention units in the context of n…”
BackgroundMethodSelf-citationVisual explanation of evidence in additive classifiers
Poulin, B., Eisner, R., Szafron, D., Lu, P., Greiner, R., Wishart, D. S., Fyshe, A., Pearcy, B., Macdonell, C., & Anvik, J. (2006) · NPARC · Cited by 131
Understanding Individual Decisions of CNNs via Contrastive Backpropagation
Gu, J., Yang, Y., & Volker Tresp, C. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-20893-6_8 · Cited by 130
Ebert‐Uphoff, I., & Hilburn, K. (2020) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-20-0097.1 · Cited by 130
How Important Is a Neuron?Preprint
Dhamdhere, K., Sundararajan, M., & Yan, Q. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.12233 · Cited by 127
Interpreting and Explaining Deep Neural Networks for Classification of Audio SignalsPreprint
Becker, S., Ackermann, M. R., Lapuschkin, S., Müller, K., & Samek, W. (2018) · arXiv (Cornell University) · Cited by 124
Self-citationExplaining Convolutional Neural Networks using Softmax Gradient Layer-wise Relevance Propagation
Iwana, B. K., Kuroki, R., & Uchida, S. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00513 · Cited by 123 · cited 1×
“, in [57], [80], [126] along with practical solutions to overcome this effect.”
BackgroundA Roadmap for a Rigorous Science of Interpretability.Preprint
Doshi‐Velez, F., & Kim, B. (2017) · arXiv (Cornell University) · Cited by 119
NP-completeness for calculating power indices of weighted majority games
Matsui, Y., & Matsui, T. (2001) · Theoretical Computer Science · doi:10.1016/s0304-3975(00)00251-6 · Cited by 118 · cited 1×
“However, although theoretically valid, these Shapley interaction terms are, in most cases, infeasible to compute for complex ML models, such as deep neural networks, and approximations are, therefore, required [120].”
BackgroundMemisevic, R. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.53 · Cited by 114
Towards explaining anomalies: A deep Taylor decomposition of one-class models
Kauffmann, J., Müller, K., & Montavon, G. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107198 · Cited by 113 · cited 4×
“LRP assumes that the model is structured as (or can be converted to [85] and [86]) a neural network with a”
BackgroundMethodInfluentialSelf-citationAnalyzing Neuroimaging Data Through Recurrent Deep Learning Models
Thomas, A. W., Heekeren, H. R., Müller, K., & Samek, W. (2019) · Frontiers in Neuroscience · doi:10.3389/fnins.2019.01321 · Cited by 113 · cited 5×
“Another application of DNN explanation techniques in cognitive neuroscience is presented in [186], where a deep model is learned to predict whole-brain fMRI data.”
BackgroundMethodInfluentialSelf-citationThe Detection of Errors in Multivariate Data Using Principal Components
Hawkins, D. M. (1974) · Journal of the American Statistical Association · doi:10.1080/01621459.1974.10482950 · Cited by 108 · cited 2×
“However for unsupervised problems such as anomaly detection or clustering, although deep models have also reached successes [27], [156], [155], shallow models (e.g. centroid-based [119], PCA-based [64], kernel-based [141], or combinations of them [164], [37], [72]) remain popular workhorses.”
Deep Neural Network or Dermatologist?
Young, K. C., Booth, G., Simpson, B., Dutton, R., & Shrapnel, S. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-33850-3_6 · Cited by 99
Building and Interpreting Deep Similarity Models
Eberle, O., Büttner, J., Kräutli, F., Müller, K., Valleriani, M., & Montavon, G. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3020738 · Cited by 97
Self-citationVisualizing convolutional neural network protein-ligand scoring
Hochuli, J. E., Helbling, A., Skaist, T., Ragoza, M., & Koes, D. R. (2018) · Journal of Molecular Graphics and Modelling · doi:10.1016/j.jmgm.2018.06.005 · Cited by 94
Explaining Therapy Predictions with Layer-Wise Relevance Propagation in Neural Networks
Yang, Y., Volker Tresp, C., Wunderle, M., & Fasching, P. A. (2018) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi.2018.00025 · Cited by 88 · cited 1×
“Yet, another work [196] applies LSTM predictors together with LRP for transparent therapy prediction on patients suffering from metastatic breast cancer.”
BackgroundBatch Effect Confounding Leads to Strong Bias in Performance Estimates Obtained by Cross-Validation
Soneson, C., Gerster, S., & Delorenzi, M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0100335 · Cited by 84
Influence-Directed Explanations for Deep Convolutional Networks
Leino, K. R. M., Sen, S., Datta, A., Fredrikson, M., & Li, L. (2018) · IEEE International Test Conference (ITC) · doi:10.1109/test.2018.8624792 · Cited by 70
Explaining Image Classifiers by Removing Input Features Using Generative Models
Agarwal, C., & Nguyen, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69544-6_7 · Cited by 70 · cited 1×
“, using a generative model [2]) rather than setting them to gray was proposed as an enhancement.”
MethodInterpreting individual classifications of hierarchical networks
Landecker, W., Thomure, M. D., Bettencourt, L. M. A., Mitchell, M., Kenyon, G. T., & Brumby, S. P. (2013) · IEEE Symposium on Computational Intelligence and Data Mining (CIDM) · doi:10.1109/cidm.2013.6597214 · Cited by 64 · cited 1×
“Furthermore, a number of other propagation techniques have been proposed [100], [170], [171], [202] with some of the rules overlapping with LRP for certain choices of parameters.”
BackgroundProducing Radiologist-Quality Reports for Interpretable Deep Learning
Gale, W., Oakden‐Rayner, L., Carneiro, G., Palmer, L. J., & Bradley, A. P. (2019) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2019.8759236 · Cited by 64 · cited 1×
“Further applications of DNN explanation techniques for medical images include the detection of lesions in diabetic retinopathy data [145], pixelwise analysis of microscopy images from global image annotations [98], the validation of predictions in dermatology [199], and analysis of X-ray images [48].”
BackgroundVisualization of neural networks using saliency maps
Mørch, N. J. S., Kjems, U., Hansen, L. K., Svarer, C., Law, I. G., Lautrup, B. E., Strother, S. C., & Rehm, K. (2002) · IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural Networks · doi:10.1109/icnn.1995.488997 · Cited by 58 · cited 1×
“Like the method’s name suggests, the averaging process “smooths” the explanation and, in turn, also addresses the shattered gradient problem described in Section II-B (see also [14], [130], and [174] for earlier gradient-based explanation techniques).”
MethodLee, Y., Kummerow, C. D., & Ebert‐Uphoff, I. (2021) · Atmospheric measurement techniques · doi:10.5194/amt-14-2699-2021 · Cited by 58
Learning local features for age estimation on real-life faces
Shan, C. (2010) · doi:10.1145/1878039.1878045 · Cited by 55 · cited 1×
“First, test results after training using stochastic gradient descent (SGD) [108] report reasonable performance, with exact and one-off [44], [168] prediction accuracy(3) of 56.”
Visual Interpretation of Kernel‐Based Prediction Models
Hansen, K., Baehrens, D., Schroeter, T. S., Rupp, M., & Müller, K. (2011) · Molecular Informatics · doi:10.1002/minf.201100059 · Cited by 55 · cited 1×
“Transposing this concept of maximizing utility to the domain of explanation, quantifying the utility of the explanation would first require to define what is the ultimate target task (the explanation being only an intermediate step) and then assessing by how much the use of explanation by the human increases its perfo…”
Background
Research works printed but not linked: 13
- Arras, 2020, Ground truth evaluation of neural network explanations with CLEVR-XAI, arXiv:2003.07258 · arXiv
- 10.1146/annurev.neuro.26.041002.131047 · DOI · Paper page
- Chollet, 2015, Keras
- 10.1109/ICCV.2019.00304 · DOI
- Kokhlikyan, 2019, Pytorch Captum
- Lapuschkin, 2016, The layer-wise relevance propagation toolbox for artificial neural networks, J. Mach. Learn. Res., 17, 1
- 10.48550/arXiv.1312.4400 · DOI · Paper page
- 10.5555/2969033.2969153 · doi:10.5555/2969033.2969153
- 10.5555/3157382.3157477 · doi:10.5555/3157382.3157477
- Okuta, CuPy: A numpy-compatible library for NVIDIA GPU calculations, 151
- Oliphant, 2006, A Guide to NumPy, 1
- 10.48550/ARXIV.1404.7828 · DOI
- 10.5555/3045118.3045336 · doi:10.5555/3045118.3045336
Cited by
Combining Machine Learning and Computational Chemistry for Predictive Insights Into Chemical Systems
Keith, J. A., Vassilev-Galindo, V., Cheng, B., Chmiela, S., Gastegger, M., Müller, K.-R., & Tkatchenko, A. (2021) · Chemical Reviews · doi:10.1021/acs.chemrev.1c00107 · Cited by 849
Self-citationEdge artificial intelligence for 6g: vision, enabling technologies, and applications
Letaief, K. B., Shi, Y., Lu, J., & Lu, J. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3126076 · Cited by 829
“Specifically, the developed AI models and algorithms must be privacy-preserving, adversarial-resilient, robust, fair, optimal and interpretable [95].”
BackgroundExplainable AI (XAI): A systematic meta-survey of current challenges and future opportunities
Saeed, W., & Omlin, C. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110273 · Cited by 783
Albahri, A. S., Duhaim, A. M., Fadhel, M. A., Alnoor, A., Baqer, N. S., Alzubaidi, L., Albahri, O. S., Alamoodi, A. H., Bai, J., Salhi, A., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.03.008 · Cited by 716
Longo, L., Brcic, M., Cabitza, F., Choi, J., Confalonieri, R., Ser, J. D., Guidotti, R., Hayashi, Y., Herrera, F., Holzinger, A., et al. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102301 · Cited by 626
“A lot of work exists on explaining the decisions of a classifier with attribution methods [44].”
BackgroundSelf-citation
Metadata
- Authors
- Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R.
- Affiliations
- Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (DE)
- BIFOLD✔Berlin Institute for the Foundations of Learning and Data, Berlin, Germany
- Berlin Institute for the Foundations of Learning and Data (DE)
- ORCID
- Venue
- Proceedings of the IEEE
- Published
- 2021-03
- Volume
- 109
- Issue
- 3
- Pages
- 247-278
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W3132191748
- Cited by
- 1,477
- References
- 230
- Citations by year
- 2026: 305 · 2025: 343 · 2024: 326 · 2023: 273 · 2022: 152 · 2021: 74
- FWCI
- 72.25
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Proceedings of the IEEE (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion, other-oa)
- Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korea Government (2017-0-00451, 2019-0-00079)
- German Ministry for Education and Research (01IS14013A-E, 01GQ1115, 01GQ0850, 01IS18025A, 01IS18037A)
- German Research Foundation (Math+, EXC 2046/1, 390685689)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- explainable AI, interpretability, deep neural network, post-hoc explanation, interpretability evaluation, nonlinear machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.