The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Shortcomings of Top-Down Randomization-Based Sanity Checks for Evaluations of Deep Neural Network Explanations
Open accessAbstract
While the evaluation of explanations is an important step towards trustworthy models, it needs to be done carefully, and the employed metrics need to be well-understood. Specifically model randomization testing can be overinterpreted if regarded as a primary criterion for selecting or discarding explanation methods. To address shortcomings of this test, we start by observing an experimental gap in the ranking of explanation methods between randomization-based sanity checks [1] and model output faithfulness measures (e.g. [20]). We identify limitations of model-randomization-based sanity checks for the purpose of evaluating explanations. Firstly, we show that uninformative attribution maps created with zero pixel-wise covariance easily achieve high scores in this type of checks. Secondly, we show that top-down model randomization preserves scales of forward pass activations with high probability. That is, channels with large activations have a high probility to contribute strongly to the output, even after randomization of the network on top of them. Hence, explanations after randomization can only be expected to differ to a certain extent. This explains the observed experimental gap. In summary, these results demonstrate the inadequacy of model-randomization-based sanity checks as a criterion to rank attribution methods.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
For existing methods, discussion of what a good or reliable explanation constitutes is ongoing.
“Also, for existing methods, the discussion of what a good or reliable explanation constitutes is ongoing (91, inter alia).”
From Orchestrating explainable artificial intelligence for multimodal and longitudinal data in medical imaging (2024) · cites this paper as [91]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524
Self-citationAxiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087
Striving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447
Methods for interpreting and understanding deep neural networks
Montavon, G., Samek, W., & Müller, K.-R. (2018) · Digital Signal Processing · doi:10.1016/j.dsp.2017.10.011 · Cited by 2,665
Self-citationExplaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3060483 · Cited by 1,477
Self-citationEvaluating the Visualization of What a Deep Neural Network Has Learned
Samek, W., Binder, A., Montavon, G., Lapuschkin, S., & Muller, K.-R. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2599820 · Cited by 1,164
Self-citationUnmasking Clever Hans predictors and assessing what machines really learn
Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., & Müller, K.-R. (2019) · Nature Communications · doi:10.1038/s41467-019-08987-4 · Cited by 1,047
Self-citationTop-Down Neural Attention by Excitation Backprop
Zhang, J., Bargal, S. A., Lin, Z., Brandt, J., Shen, X., & Sclaroff, S. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-017-1059-x · Cited by 808
The (Un)reliability of Saliency Methods
Kindermans, P.-J., Hooker, S., Adebayo, J., Alber, M., Schütt, K. T., Dähne, S., Erhan, D., & Kim, B. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_14 · Cited by 741
Huang, X., Kroening, D., Ruan, W., Sharp, J. J., Sun, Y., Thamo, E., Wu, M., & Yi, X. (2020) · Computer Science Review · doi:10.1016/j.cosrev.2020.100270 · Cited by 632
Towards Robust Interpretability with Self-Explaining Neural NetworksPreprint
Alvarez-Melis, D., & Jaakkola, T. S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07538 · Cited by 591
Evaluating and Aggregating Feature-based Model Explanations
Bhatt, U., Weller, A., & Moura, J. M. F. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/417 · Cited by 267
CLEVR-XAI: A benchmark dataset for the ground truth evaluation of neural network explanations
Arras, L., Osman, A., & Samek, W. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.11.008 · Cited by 223
Self-citationWhen Explanations Lie: Why Many Modified BP Attributions FailPreprint
Sixt, L., Granz, M., & Landgraf, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.09818 · Cited by 74
Focus! Rating XAI Methods and Finding Biases
Arias-Duart, A., Parés, F., García-Gasulla, D., & Giménez-Ábalos, V. (2022) · IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) · doi:10.1109/fuzz-ieee55066.2022.9882821 · Cited by 59
Explaining Image Classifiers by Removing Input Features Using Generative Models
Agarwal, C., & Nguyen, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69544-6_7 · Cited by 44
Gradient-Based Vs. Propagation-Based Explanations: An Axiomatic Comparison
Montavon, G. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_13 · Cited by 36
Self-citationConcise Explanations of Neural Networks using Adversarial Training
Chalasani, P., Chen, J., Chowdhury, A. R., et al. (2020) · International Conference on Machine Learning · Cited by 30
Revisiting Sanity Checks for Saliency MapsPreprint
Yona, G., & Greenfeld, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.14297 · Cited by 14
Sundararajan, M., & Taly, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.04205 · Cited by 10
Interpretable Deep Learning: Interpretations, Interpretability, Trustworthiness, and Beyond.Preprint
Li, X., Xiong, H., Li, X., Wu, X., Zhang, X., Liu, J., Bian, J., & Dou, D. (2021) · arXiv (Cornell University) · Cited by 5
Investigating sanity checks for saliency maps with image and text classificationPreprint
Kokhlikyan, N., Miglani, V., Alsallakh, B., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.07475 · Cited by 3
Flagged first, then by citations
Research works printed but not linked: 5
- adebayo, 2018, Sanity checks for saliency maps, Advances in neural information processing systems, 31, 9525
- nilsson, 2020, Understanding ssim, ArXiv Preprint, 13846
- nguyen, 2020, On quantitative aspects of model interpretability, ArXiv Preprint
- smilkov, 2017, Smoothgrad: removing noise by adding noise, ArXiv Preprint
- balduzzi, 2017, The shattered gradients problem: If resnets are the answer, then what is the question?, International Conference on Machine Learning (ICML), 70, 342
Cited by
Sorted by citations
Pahud de Mortanges, A., Luo, H., Shu, S. Z., Kamath, A., Suter, Y., Shelan, M., Pöllinger, A., & Reyes, M. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01190-w · Cited by 93
“Also, for existing methods, the discussion of what a good or reliable explanation constitutes is ongoing ( 91 , inter alia ).”
Metadata
- Authors
- Binder, A., Weber, L., Lapuschkin, S., Montavon, G., Müller, K.-R., & Samek, W.
- Affiliations
- Singapore University of Technology and Design (SG)
- Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (DE)
- Berlin Institute for the Foundations of Learning and Data (DE)
- Korea University (KR)
- ORCID
- Venue
- 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2023-06
- Pages
- 16143-16152
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4386076467
- Cited by
- 21
- References
- 25
- Citations by year
- 2026: 2 · 2025: 8 · 2024: 7 · 2023: 4
- FWCI
- 3.73
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Figshare (submittedVersion, cc-by-nc-nd)
- Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
- arXiv (Cornell University)
- Funders
- European Commission (965221)
- Science Foundation Ireland
- Bundesministerium für Bildung und Forschung (01 IS 14013 A, Patho234, 01IS14013A-E, 01GQ0850, 01IS18037A, 01GQ1115, 01IS18025A, 01IS18037I, 031L0207C)
- Korea University (2019-0-00079, 2022-0-00984)
- Norges Forskningsråd (309439)
- Ministry of Science and ICT, South Korea (2022-0- 00984, 01GQ1115, 01GQ0850, 01IS18037A, 01IS18025A, 2019-0-00079)
- Institute for Information and Communications Technology Promotion (2019–0–00079, 2022-, 2022-0-00984)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- explainable AI, explanation evaluation, sanity checks, attribution maps
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
