Explaining by Removing: A Unified Framework for Model Explanation
Open accessThe DOI matches its OpenAlex record; 29 of the 29 references it cites were found in a registry or an index.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Self-citation"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 27×
“Introduction The proliferation of black-box models has made machine learning (ML) explainability increasingly important, and researchers have now proposed a variety of model explanation approaches (Zeiler and Fergus, 2014; Ribeiro et al., 2016; Lundberg and Lee, 2017).”
BackgroundMethodInfluentialRudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 2×
“…2013; Montavon et al., 2017; Sundararajan et al., 2017; Selvaraju et al., 2017), more general propagation-based methods (e.g., Springenberg et al., 2014; Bach et al., 2015; Kindermans et al., 2017; Zhang et al., 2018), and inherently inter- pretable models (e.g., Zhou et al., 2016; Rudin, 2019).”
BackgroundMethodFrom local explanations to global understanding with explainable AI for trees
Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., & Lee, S.-I. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-019-0138-9 · Cited by 10,551 · cited 17×
“…while accounting for complex feature interactions, such as correlations, redundancy, and complementary behavior (Lipovetsky and Conklin, 2001; Štrumbelj et al., 2009; Owen, 2014; Datta et al., 2016; Lundberg and Lee, 2017; Lundberg et al., 2020; Covert et al., 2020; Williamson and Feng, 2020).”
BackgroundMethodInfluentialSelf-citationPeeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
Strictly Proper Scoring Rules, Prediction, and Estimation
Gneiting, T., & Raftery, A. E. (2007) · Journal of the American Statistical Association · doi:10.1198/016214506000001437 · Cited by 5,793
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 3×
“Certain methods consider infinitesimal perturbations by calculating gradients2 (Simonyan et al., 2013; Sundararajan et al., 2017; Smilkov et al., 2017; Erion et al., 2019; Xu et al., 2020), but there are many possible input perturbations (Zeiler and Fergus, 2014; Ribeiro et al., 2016; Fong and…”
MethodExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524 · cited 3×
“, 2017), more general propagation-based methods (e.g., Springenberg et al., 2014; Bach et al., 2015; Kindermans et al., 2017; Zhang et al., 2018), and inherently inter-”
BackgroundMethodFast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
Clevert, D.-A., Unterthiner, T., & Hochreiter, S. (2015) · arXiv · doi:10.48550/arxiv.1511.07289 · Cited by 4,513 · cited 1×
“• For MNIST, we trained a 14-layer CNN consisting of convolutional layers with kernel size 3, max pooling layers, and ELU activations (Clevert et al., 2015).”
MethodStriving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447
Generative Image Inpainting with Contextual Attention
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., & Huang, T. S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00577 · Cited by 2,527
SmoothGrad: removing noise by adding noise
Smilkov, D., Thorat, N., Kim, B., Viégas, F., & Wattenberg, M. (2017) · arXiv · doi:10.48550/arxiv.1706.03825 · Cited by 1,467 · cited 2×
“Certain methods consider infinitesimal perturbations by calculating gradients2 (Simonyan et al., 2013; Sundararajan et al., 2017; Smilkov et al., 2017; Erion et al., 2019; Xu et al., 2020), but there are many possible input perturbations (Zeiler and Fergus, 2014; Ribeiro et al., 2016; Fong and…”
MethodSanity Checks for Saliency Maps
Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018) · arXiv · doi:10.48550/arxiv.1810.03292 · Cited by 1,294
Distribution-Free Predictive Inference for Regression
Lei, J., G’Sell, M., Rinaldo, A., Tibshirani, R. J., & Wasserman, L. (2018) · Journal of the American Statistical Association · doi:10.1080/01621459.2017.1307116 · Cited by 1,049 · cited 18×
“…to how Occlusion, PredDiff and CXPlain calculate local feature importance and how permutation tests and feature ablation (LOCO) summarize global feature importance (Breiman, 2001; Strobl et al., 2008; Zeiler and Fergus, 2014; Zintgraf et al., 2017; Lei et al., 2018; Schwab and Karlen, 2019).”
BackgroundMethodInfluentialObject Detectors Emerge in Deep Scene CNNs
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2014) · arXiv · doi:10.48550/arxiv.1412.6856 · Cited by 971
Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods
Slack, D., Hilgard, S., Jia, E., Singh, S., & Lakkaraju, H. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375830 · Cited by 900
Datta, A., Sen, S., & Zick, Y. (2016) · 2016 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2016.42 · Cited by 728 · cited 12×
“…has been considered explicitly by only a few works (Lundberg et al., 2020; Covert et al., 2020), as has the choice of how to summarize each feature’s influence based on a cooperative game (Štrumbelj et al., 2009; Datta et al., 2016; Lundberg and Lee, 2017; Frye et al., 2019; Covert et al., 2020).”
BackgroundMethodInfluentialExplainable machine learning in deployment
Bhatt, U., Xiang, A., Sharma, S., Weller, A., Taly, A., Jia, Y., Ghosh, J., Puri, R., Moura, J. M. F., & Eckersley, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3375624 · Cited by 633
Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376219 · Cited by 564 · cited 2×
“Perhaps due to their greater complexity, a recent user study showed that users were less likely to understand SHAP visualizations, experiencing a higher cognitive load than users who viewed simpler explanations (Kaur et al., 2020).”
BackgroundVisualizing Deep Neural Network Decisions: Prediction Difference Analysis
Zintgraf, L. M., Cohen, T. S., Adel, T., & Welling, M. (2017) · arXiv · doi:10.48550/arxiv.1702.04595 · Cited by 487 · cited 10×
“…to how Occlusion, PredDiff and CXPlain calculate local feature importance and how permutation tests and feature ablation (LOCO) summarize global feature importance (Breiman, 2001; Strobl et al., 2008; Zeiler and Fergus, 2014; Zintgraf et al., 2017; Lei et al., 2018; Schwab and Karlen, 2019).”
BackgroundMethodResultInfluentialConversational processes and causal explanation.
Hilton, D. J. (1990) · Psychological Bulletin · doi:10.1037/0033-2909.107.1.65 · Cited by 456
The explanation game: Explaining machine learning models using shapley values
Merrick, L., & Taly, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-57321-8_2 · Cited by 277
Hooker, G., Mentch, L., & Zhou, S. (2021) · Statistics and Computing · doi:10.1007/s11222-021-10057-z · Cited by 264
Shapley Effects for Global Sensitivity Analysis: Theory and Computation
Song, E., Nelson, B. L., & Staum, J. (2016) · SIAM/ASA Journal on Uncertainty Quantification · doi:10.1137/15m1048070 · Cited by 244 · cited 9×
“Shapley Effects analyzes a variance-based measure of a function’s sensitivity to its inputs, with the goal of discovering which features are responsible for the greatest variance reduction in the model output (Owen, 2014).”
MethodInfluentialProblems with Shapley-value-based explanations as feature importance measures
Kumar, I. E., Venkatasubramanian, S., Scheidegger, C., & Friedler, S. (2020) · arXiv · doi:10.48550/arxiv.2002.11097 · Cited by 228 · cited 39×
“Figure 13 shows more examples of MNIST explanations using the combination of the conditional distribution and Shapley value (i.e., LossSHAP).”
BackgroundInfluentialMiller, T., Howe, P., & Sonenberg, L. (2017) · arXiv · doi:10.48550/arxiv.1712.00547 · Cited by 218 · cited 2×
“This makes the interpretability literature difficult to navigate and raises questions about whether existing methods relate to human processes for explaining complex decisions (Miller et al., 2017; Miller, 2019).”
Background
Cited by
Shapley values for feature selection: the good, the bad, and the axioms
Fryer, D., Strumke, I., & Nguyen, H. (2021) · IEEE Access · doi:10.1109/access.2021.3119110 · Cited by 347
Co-design of human-centered, explainable AI for clinical decision support
Panigutti, C., Beretta, A., Fadda, D., Giannotti, F., Pedreschi, D., Perotti, A., & Rinzivillo, S. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3587271 · Cited by 127
Post-Hoc Explanations Fail to Achieve their Purpose in Adversarial Contexts
Bordt, S., Finck, M., Raidl, E., & von Luxburg, U. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533153 · Cited by 89
Explaining Deep Learning for ECG Analysis: Building Blocks for Auditing and Knowledge Discovery
Wagner, P., Mehari, T., Haverkamp, W., & Strodthoff, N. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108525 · Cited by 48
“This led to the emergence of explainable AI (XAI) as a subfield of machine learning, which tries to shed light on the decision process implemented by neural networks, see [15, 16, 17, 18] for general reviews.”
BackgroundContextCite: Attributing Model Generation to Context
Cohen-Wang, B., Shah, H., Georgiev, K., & Madry, A. (2024) · arXiv · doi:10.48550/arxiv.2409.00729 · Cited by 16
Metadata
- Authors
- Covert, I., Lundberg, S., & Lee, S.
- Venue
- arXiv (Cornell University)
- Publisher
- Cornell University
- Published
- 2020-11-21
- Volume
- 22
- Issue
- 209
- Pages
- 1-90
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2011.14878
- OpenAlex
- W3107600318
- Cited by
- 116
- References
- 29
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.