Sanity Checks for Saliency Metrics
Open accessAbstract
Saliency maps are a popular approach to creating post-hoc explanations of image classifier outputs. These methods produce estimates of the relevance of each pixel to the classification output score, which can be displayed as a saliency map that highlights important pixels. Despite a proliferation of such methods, little effort has been made to quantify how good these saliency maps are at capturing the true relevance of the pixels to the classifier output (i.e. their “fidelity”). We therefore investigate existing metrics for evaluating the fidelity of saliency methods (i.e. saliency metrics). We find that there is little consistency in the literature in how such metrics are calculated, and show that such inconsistencies can have a significant effect on the measured fidelity. Further, we apply measures of reliability developed in the psychometric testing literature to assess the consistency of saliency metrics when applied to individual saliency maps. Our results show that saliency metrics can be statistically unreliable and inconsistent, indicating that comparative rankings between saliency methods generated using such metrics can be untrustworthy.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Visual XAI methods are generally regarded as intuitive and easy to interpret, but studies have identified significant limitations.
“Even though visual XAI methods are considered easy to interpret and intuitive, some studies pointed out significant limitations.”
From Explainable AI in medical imaging: An overview for clinical practitioners – Beyond saliency-based XAI approaches (2023) · cites this paper as [16]
Sanity-check studies of visual explanations lack consistency in evaluation metrics, making full evaluation of visual explanations challenging.
“Tomsett et al. [16] pointed out a lack of consistency concerning evaluation metrics in sanity-check studies concluding that despite the increasing effort, it remains challenging to evaluate visual explanations fully.”
From Explainable AI in medical imaging: An overview for clinical practitioners – Beyond saliency-based XAI approaches (2023) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513 · see citing works
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,305 · see citing works
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,890 · see citing works
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · see citing works
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · see citing works
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682 · see citing works
The mythos of model interpretability
Lipton, Z. C. (2018) · Communications of the ACM · doi:10.1145/3233231 · Cited by 5,427 · see citing works
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · see citing works
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524 · see citing works
Axiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087 · see citing works
Learning Important Features Through Propagating Activation Differences
Shrikumar, A., Greenside, P., & Kundaje, A. (2017) · arXiv · doi:10.48550/arxiv.1704.02685 · Cited by 2,761 · see citing works
Methods for interpreting and understanding deep neural networks
Montavon, G., Samek, W., & Müller, K.-R. (2018) · Digital Signal Processing · doi:10.1016/j.dsp.2017.10.011 · Cited by 2,665 · see citing works
Reliability: A Review of Psychometric Basics and Recent Marketing Practices
Peter, J. P. (1979) · Journal of Marketing Research · doi:10.2307/3150868 · Cited by 1,577 · see citing works
Explaining nonlinear classification decisions with deep taylor decomposition
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K.-R. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.11.008 · Cited by 1,357 · see citing works
Sanity Checks for Saliency Maps
Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018) · arXiv · doi:10.48550/arxiv.1810.03292 · Cited by 1,294 · see citing works
Psychological Testing: Principles and Applications
Murphy, K. R., & Davidshofer, C. O. (1988) · Cited by 1,174 · see citing works
Evaluating the Visualization of What a Deep Neural Network Has Learned
Samek, W., Binder, A., Montavon, G., Lapuschkin, S., & Muller, K.-R. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2599820 · Cited by 1,164 · see citing works
The (Un)reliability of Saliency Methods
Kindermans, P.-J., Hooker, S., Adebayo, J., Alber, M., Schütt, K. T., Dähne, S., Erhan, D., & Kim, B. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_14 · Cited by 652 · see citing works
Towards better understanding of gradient-based attribution methods for Deep Neural NetworksPreprint
Ancona, M., Ceolini, E., Öztireli, C., & Groß, M. H. (2017) · Zurich Open Repository and Archive (University of Zurich) · doi:10.3929/ethz-b-000249929 · Cited by 634 · see citing works
Towards Robust Interpretability with Self-Explaining Neural NetworksPreprint
Alvarez-Melis, D., & Alvarez-Melis, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07538 · Cited by 545 · see citing works
A Benchmark for Interpretability Methods in Deep Neural NetworksPreprint
Hooker, S., Erhan, D., Kindermans, P.-J., & Kim, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.10758 · Cited by 438 · see citing works
Explaining Recurrent Neural Network Predictions in Sentiment Analysis
Arras, L., Montavon, G., Müller, K.-R., & Samek, W. (2017) · Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis · doi:10.18653/v1/w17-5221 · Cited by 361 · see citing works
The (Un)reliability of saliency methodsPreprint
Kindermans, P.-J., Hooker, S., Adebayo, J., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00867 · Cited by 160 · see citing works
A unified view of gradient-based attribution methods for Deep Neural NetworksPreprint
Ancona, M., Ceolini, E., Öztireli, C., & Groß, M. H. (2017) · arXiv (Cornell University) · doi:10.3929/ethz-b-000237705 · Cited by 120 · see citing works
Investigating the influence of noise and distractors on the interpretation of neural networksPreprint
Kindermans, P.-J., Schütt, K. T., Müller, K., & Dähne, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.07270 · Cited by 104 · see citing works
Gradient-Based Vs. Propagation-Based Explanations: An Axiomatic Comparison
Montavon, G. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_13 · Cited by 36 · see citing works
Bias Also Matters: Bias Attribution for Deep Neural Network Explanation
Wang, S., Zhou, T., & Bilmes, J. (2019) · International Conference on Machine Learning · Cited by 24 · see citing works
Flagged first, then by citations
1 of the 30 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Towards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3331846 · Cited by 278 · see citing works
Borys, K., Schmitt, Y. A., Nauta, M., Seifert, C., Krämer, N., Friedrich, C. M., & Nensa, F. (2023) · European Journal of Radiology · doi:10.1016/j.ejrad.2023.110786 · Cited by 233 · see citing works
Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI
Tomsett, R., Preece, A., Braines, D., Cerutti, F., Chakraborty, S., Srivastava, M., Pearson, G., & Kaplan, L. (2020) · Patterns · doi:10.1016/j.patter.2020.100049 · Cited by 186 · see citing works
Self-citationRegularizing Black-box Models for Improved InterpretabilityPreprint
Plumb, G., Al-Shedivat, M., Cabrera, Á. A., Perer, A., Xing, E. P., & Talwalkar, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.06787 · Cited by 46 · see citing works
Explainable deep reinforcement learning for portfolio management: an empirical approach
Guan, M., & Liu, X.-Y. (2021) · Proceedings of the Second ACM International Conference on AI in Finance · doi:10.1145/3490354.3494415 · Cited by 28 · see citing works
Validating Automatic Concept-Based Explanations for AI-Based Digital Histopathology
Sauter, D., Lodde, G., Nensa, F., Schadendorf, D., Livingstone, E., & Kukuk, M. (2022) · Sensors · doi:10.3390/s22145346 · Cited by 26 · see citing works
Fresz, B., Dubovitskaya, E., Brajovic, D., Huber, M. F., & Horz, C. (2024) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1609/aies.v7i1.31648 · Cited by 17 · see citing works
“It should be noted that there is currently no consensus on the best way to evaluate the properties discussed, as different evaluation methods (with many listed in [27]) may contradict each other [40].”
BackgroundPe, S., Famiglini, L., Gallazzi, E., Bortolotto, C., Carone, L., Cisarri, A., Salina, A., Preda, L., Bellazzi, R., Cabitza, F., et al. (2024) · Methods of Information in Medicine · doi:10.1055/a-2562-2163 · Cited by 5 · see citing works
Data Representing Ground-Truth Explanations to Evaluate XAI MethodsPreprint
Amiri, S. S., Weber, R. O., Goel, P., Brooks, O., Gandley, A., Kitchell, B., & Zehm, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.09892 · Cited by 5 · see citing works
- Authors
- Tomsett, R., Harborne, D., Chakraborty, S., Gurram, P., & Preece, A.
- Affiliations
- IBM (United States) (US)
- Booz Allen Hamilton (United States) (US)
- Cardiff University (GB)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2020-04-03
- Volume
- 34
- Issue
- 04
- Pages
- 6021-6029
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W2992167130
- Cited by
- 3
- References
- 30
- Citations by year
- 2024: 1 · 2020: 1 · 2019: 1
- FWCI
- 0.10
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- www.aaai.org/ (unspecified)
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- ORCA Online Research @Cardiff (Cardiff University) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Government of the United Kingdom
- Army Research Laboratory (W911NF‐16‐3‐0001, W911NF)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- saliency maps, post-hoc explanation, sanity checks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
