Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations
Open accessAbstract
Expressive classifiers such as neural networks are among the most accurate supervised learning methods in use today, but their opaque decision boundaries make them difficult to trust in critical applications. We propose a method to explain the predictions of any differentiable model via the gradient of the class label with respect to the input (which provides a normal to the decision boundary). Not only is this approach orders of magnitude faster at identifying input dimensions of high sensitivity than sample-based perturbation methods (e.g. LIME), but it also lends itself to efficiently discovering multiple qualitatively different decision boundaries as well as decision boundaries that are consistent with expert annotation. On multiple datasets, we show our approach generalizes much better when test conditions differ from those in training.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Some papers already optimize for interpretability by using a regularization term or objective function during training of the predictive model or by relating rewards to explainability and presentation quality in reinforcement learning.
“Some papers already optimize for interpretability by using a regularization term or objective function during training of the predictive model (e.g., [33,137,191,211,244,276,277,289]) or by relating rewards to explainability and presentation quality in reinforcement learning (e.g., [268]).”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [211]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
UCI Machine Learning Repository
Asuncion, A. (2007) · Medical Entomology and Zoology · Cited by 24,190 · see citing works
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · see citing works
The mythos of model interpretability
Lipton, Z. C. (2018) · Communications of the ACM · doi:10.1145/3233231 · Cited by 5,427 · see citing works
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · see citing works
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524 · see citing works
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · see citing works
Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks
Papernot, N., McDaniel, P. D., Wu, X., Jha, S., & Swami, A. (2016) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2016.41 · Cited by 3,123 · see citing works
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · see citing works
Self-citationLearning Important Features Through Propagating Activation Differences
Shrikumar, A., Greenside, P., & Kundaje, A. (2017) · arXiv · doi:10.48550/arxiv.1704.02685 · Cited by 2,761 · see citing works
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685 · see citing works
Explaining nonlinear classification decisions with deep taylor decomposition
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K.-R. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.11.008 · Cited by 1,357 · see citing works
Keil, F. C. (2006) · Annual Review of Psychology · doi:10.1146/annurev.psych.57.102904.190100 · Cited by 1,350 · see citing works
Rationalizing Neural Predictions
Lei, T., Barzilay, R., & Jaakkola, T. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1011 · Cited by 777 · see citing works
Not Just a Black Box: Learning Important Features Through Propagating Activation DifferencesPreprint
Shrikumar, A., Greenside, P., Shcherbina, A., & Kundaje, A. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.01713 · Cited by 704 · see citing works
Visualizing and Understanding Neural Models in NLP
Li, J., Chen, X., Hovy, E. H., & Jurafsky, D. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1082 · Cited by 673 · see citing works
Grad-CAM: Why did you say that?Preprint
Selvaraju, R. R., Das, A., Vedantam, R., Cogswell, M., Parikh, D., & Batra, D. (2016) · arXiv.org · doi:10.48550/arxiv.1611.07450 · Cited by 646 · see citing works
Zafar, M. B., Valera, I., Gomez Rodriguez, M., & Gummadi, K. P. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052660 · Cited by 632 · see citing works
Understanding Neural Networks through Representation ErasurePreprint
Li, J., Monroe, W., & Jurafsky, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.08220 · Cited by 493 · see citing works
Auditing black-box models for indirect influence
Adler, P., Falk, C., Friedler, S. A., Nix, T., Rybeck, G., Scheidegger, C., Smith, B., & Venkatasubramanian, S. (2018) · Knowledge and Information Systems · doi:10.1007/s10115-017-1116-3 · Cited by 280 · see citing works
Using “annotator rationales” to improve machine learning for text categorization
Zaidan, O. F., Eisner, J. M., & Piatko, C. (2007) · Cited by 255 · see citing works
Rationale-Augmented Convolutional Neural Networks for Text Classification
Zhang, Y., Marshall, I., & Wallace, B. C. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1076 · Cited by 249 · see citing works
Improving generalization performance using double backpropagation
Drucker, H., & LeCun, Y. (1992) · IEEE Transactions on Neural Networks · doi:10.1109/72.165600 · Cited by 238 · see citing works
Visualizing and Understanding Neural Models in NLPPreprint
Li, J., Chen, X., Hovy, E. H., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.01066 · Cited by 165 · see citing works
Gradients of CounterfactualsPreprint
Sundararajan, M., Taly, A., & Yan, Q. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.02639 · Cited by 106 · see citing works
An unexpected unity among methods for interpreting model predictionsPreprint
Lundberg, S., & Lee, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.07478 · Cited by 101 · see citing works
Annotator rationales for visual recognition
Donahue, J., & Grauman, K. L. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126394 · Cited by 66 · see citing works
Interpretation of Prediction Models Using the Input GradientPreprint
Hechtlinger, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.07634 · Cited by 60 · see citing works
Programs as Black-Box ExplanationsPreprint
Singh, S., Ribeiro, M. T., & Guestrin, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.07579 · Cited by 36 · see citing works
Rationale-Augmented Convolutional Neural Networks for Text ClassificationPreprint
Zhang, Y., Marshall, I. J., & Wallace, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.04469 · Cited by 23 · see citing works
Auditing Black-box Models by Obscuring FeaturesPreprint
Adler, P., Falk, C., Friedler, S. A., Rybeck, G., Scheidegger, C., Smith, B., & Venkatasubramanian, S. (2016) · arXiv (Cornell University) · Cited by 9 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller. How to explain individual classification decisions. Journal of Machine Learning Research, 11(Jun):1803-1831, 2010.
- Fong and Vedaldi, 2017] Ruth Fong and Andrea Vedaldi. Inter- pretable explanations of black boxes by meaningful perturbation. arXiv preprint arXiv:1704.03296, 2017. · arXiv
- Tao Lei, Regina Barzilay, and Tommi Jaakkola. Ra- tionalizing neural predictions. arXiv preprint arXiv:1606.04155, 2016. · arXiv
Cited by
Sorted by citations
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423 · see citing works
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · see citing works
“While there are several different techniques for obtaining local-explanations [96]–[100], recent work by Lundberg and Lee [101] have shown that there are equivalences among these techniques.”
MethodExplaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · see citing works
Definitions, methods, and applications in interpretable machine learning
Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., & Yu, B. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1900654116 · Cited by 2,167 · see citing works
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762 · see citing works
Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3060483 · Cited by 1,477 · see citing works
COVID-19 Image Data Collection: Prospective Predictions are the Future
Cohen, J., Morrison, P., Dao, L., Roth, K., Duong, T. Q., & Ghassemi, M. (2020) · The Journal of Machine Learning for Biomedical Imaging · doi:10.59275/j.melba.2020-48g7 · Cited by 1,026 · see citing works
Attention is not not Explanation
Wiegreffe, S., & Pinter, Y. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1002 · Cited by 966 · see citing works
Jain, S., & Wallace, B. C. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-1357 · Cited by 2,287 · see citing works
Interpretable Convolutional Neural Networks
Zhang, Q., Wu, Y. N., & Zhu, S.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00920 · Cited by 833 · see citing works
Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems
Yu, J., Lu, L., Meng, X., & Karniadakis, G. E. (2022) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2022.114823 · Cited by 737 · see citing works
Serrano, S., & Smith, N. A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1282 · Cited by 690 · see citing works
Towards Explainable Artificial Intelligence
Samek, W., & Müller, K. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_1 · Cited by 650 · see citing works
Ross, A. S., & Doshi‐Velez, F. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11504 · Cited by 621 · see citing works
Self-citationExplainable machine learning in deployment
Bhatt, U., Xiang, A., Sharma, S., Weller, A., Taly, A., Jia, Y., Ghosh, J., Puri, R., Moura, J. M. F., & Eckersley, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3375624 · Cited by 633 · see citing works
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2021) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3387166 · Cited by 785 · see citing works
On Interpretability of Artificial Neural Networks: A Survey
Fan, F., Xiong, J., Li, M., & Wang, G. (2021) · IEEE Transactions on Radiation and Plasma Medical Sciences · doi:10.1109/trpms.2021.3066428 · Cited by 573 · see citing works
Nauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023) · ACM Computing Surveys · doi:10.1145/3583558 · Cited by 590 · see citing works
Attention is not ExplanationPreprint
Jain, S., & Wallace, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.10186 · Cited by 468 · see citing works
Interpretability of deep learning models: A survey of results
Chakraborty, S., Tomsett, R., Raghavendra, R., Harborne, D., Alzantot, M., Cerutti, F., Srivastava, M. B., Preece, A. D., Julier, S., Rao, R., et al. (2017) · IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI) · doi:10.1109/uic-atc.2017.8397411 · Cited by 469 · see citing works
Accountability of AI Under the Law: The Role of Explanation
Doshi-Velez, F., Kortz, M., Budish, R., Bavitz, C., Gershman, S. J., O'Brien, D., Scott, K., Shieber, S., Waldo, J., Weinberger, D., et al. (2017) · doi:10.2139/ssrn.3064761 · Cited by 457 · see citing works
Self-citationAkata, Z., Balliet, D., de Rijke, M., Dignum, F., Dignum, V., Eiben, G., Fokkens, A., Grossi, D., Hindriks, K., Hoos, H., et al. (2020) · Computer · doi:10.1109/mc.2020.2996587 · Cited by 415 · see citing works
Women Also Snowboard: Overcoming Bias in Captioning Models
Hendricks, L. A., Burns, K., Saenko, K., Darrell, T., & Rohrbach, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_47 · Cited by 404 · see citing works
Deterministic Local Interpretable Model-Agnostic Explanations for Stable Explainability
Zafar, M. R., & Khan, N. (2021) · Machine Learning and Knowledge Extraction · doi:10.3390/make3030027 · Cited by 408 · see citing works
Counterfactual Samples Synthesizing for Robust Visual Question Answering
Chen, L., Yan, X., Xiao, J., Zhang, H., Pu, S., & Zhuang, Y. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01081 · Cited by 357 · see citing works
Hsieh, C.-Y., Li, C.-L., Yeh, C.-K., Nakhost, H., Fujii, Y., Ratner, A., Krishna, R., Lee, C.-Y., & Pfister, T. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.507 · Cited by 316 · see citing works
Binder 2.0 - Reproducible, interactive, sharable environments for science at scale
Project Jupyter, Bussonnier, M., Forde, J. Z., Freeman, J., Granger, B., Head, T., Holdgraf, C., Kelley, K., Nalvarte, G., Osheroff, A., et al. (2018) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-4af1f417-011 · Cited by 303 · see citing works
Erion, G., Janizek, J. D., Sturmfels, P., Lundberg, S. M., & Lee, S.-I. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00343-w · Cited by 300 · see citing works
Large pre-trained language models contain human-like biases of what is right and wrong to do
Schramowski, P., Turan, C., Andersen, N., Rothkopf, C. A., & Kersting, K. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00458-8 · Cited by 291 · see citing works
Neural Prototype Trees for Interpretable Fine-grained Image Recognition
Nauta, M., van Bree, R., & Seifert, C. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01469 · Cited by 269 · see citing works
Predicting COVID-19 Pneumonia Severity on Chest X-ray With Deep Learning
Cohen, J., Dao, L., Roth, K., Morrison, P., Bengio, Y., Abbasi, A., Shen, B., Mahsa, H. K., Ghassemi, M., Li, H., et al. (2020) · Cureus · doi:10.7759/cureus.9448 · Cited by 259 · see citing works
A Survey of the State of Explainable AI for Natural Language Processing
Danilevsky, M., Qian, K., Aharonov, R., Katsis, Y., Kawas, B., & Sen, P. (2020) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2020.aacl-main.46 · Cited by 380 · see citing works
Seq2seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Models
Strobelt, H., Gehrmann, S., Behrisch, M., Perer, A., Pfister, H., & Rush, A. M. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2865044 · Cited by 244 · see citing works
Fairness in Deep Learning: A Computational Perspective
Du, M., Yang, F., Zou, N., & Hu, X. (2020) · IEEE Intelligent Systems · doi:10.1109/mis.2020.3000681 · Cited by 218 · see citing works
Mueller, S. T., Hoffman, R. R., Clancey, W., Emrey, A., & Klein, G. (2019) · arXiv · doi:10.48550/arxiv.1902.01876 · Cited by 215 · see citing works
Beyond Sparsity: Tree Regularization of Deep Models for Interpretability
Wu, M., Hughes, M., Parbhoo, S., Zazzi, M., Roth, V., & Doshi-Velez, F. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11501 · Cited by 213 · see citing works
Self-citationExplanatory Interactive Machine Learning
Teso, S., & Kersting, K. (2019) · AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314293 · Cited by 276 · see citing works
Schramowski, P., Stammer, W., Teso, S., Brugger, A., Herbert, F., Shao, X., Luigs, H.-G., Mahlein, A.-K., & Kersting, K. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-0212-3 · Cited by 207 · see citing works
A Review on Explainability in Multimodal Deep Neural Nets
Joshi, G., Walambe, R., & Kotecha, K. (2021) · IEEE Access · doi:10.1109/access.2021.3070212 · Cited by 204 · see citing works
Visualizing the Impact of Feature Attribution Baselines
Sturmfels, P., Lundberg, S., & Lee, S. (2020) · Distill · doi:10.23915/distill.00022 · Cited by 206 · see citing works
Detecting and interpreting myocardial infarction using fully convolutional neural networks
Strodthoff, N., & Strodthoff, C. (2018) · Physiological Measurement · doi:10.1088/1361-6579/aaf34d · Cited by 206 · see citing works
CLEVR-XAI: A benchmark dataset for the ground truth evaluation of neural network explanations
Arras, L., Osman, A., & Samek, W. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.11.008 · Cited by 189 · see citing works
Human-Centered Artificial Intelligence for Designing Accessible Cultural Heritage
Pisoni, G., Díaz-Rodríguez, N., Gijlers, H., & Tonolli, L. (2021) · Applied Sciences · doi:10.3390/app11020870 · Cited by 183 · see citing works
Considerations for Evaluation and Generalization in Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2018) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-319-98131-4_1 · Cited by 176 · see citing works
Self-citationInterpretable and explainable machine learning: A methods‐centric overview with concrete examples
Marcinkevičs, R., & Vogt, J. E. (2023) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1493 · Cited by 180 · see citing works
Explainable Generative AI (GenXAI): a survey, conceptualization, and research agenda
Schneider, J. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10916-x · Cited by 184 · see citing works
Asking ‘Why’ in AI: Explainability of intelligent systems – perspectives and challenges
Preece, A. D. (2018) · Intelligent systems in accounting, finance and management/Intelligent systems in accounting, finance & management · doi:10.1002/isaf.1422 · Cited by 153 · see citing works
Beyond explaining: Opportunities and challenges of XAI-based model improvement
Weber, L., Lapuschkin, S., Binder, A., & Samek, W. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.11.013 · Cited by 148 · see citing works
Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems
Tomsett, R., Braines, D., Harborne, D., Preece, A. D., & Chakraborty, S. (2018) · ORCA Online Research @Cardiff (Cardiff University) · doi:10.48550/arxiv.1806.07552 · Cited by 127 · see citing works
Basak, C., Qin, S., & O'Connell, M. A. (2020) · Psychology and Aging · doi:10.1037/pag0000442 · Cited by 129 · see citing works
EDITS: Modeling and Mitigating Data Bias for Graph Neural Networks
Dong, Y., Liu, N., Jalaian, B. A., & Li, J. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3512173 · Cited by 125 · see citing works
Invariant Grounding for Video Question Answering
Li, Y., Wang, X., Xiao, J., Ji, W., & Chua, T. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00294 · Cited by 123 · see citing works
Finding and removing Clever Hans: Using explanation methods to debug and improve deep models
Anders, C. J., Weber, L., Neumann, D., Samek, W., Müller, K., & Lapuschkin, S. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.07.015 · Cited by 122 · see citing works
An Evaluation of the Human-Interpretability of Explanation
Lage, I., Chen, E., He, J., Narayanan, M., Kim, B., Gershman, S., & Doshi-Velez, F. (2019) · arXiv · doi:10.48550/arxiv.1902.00006 · Cited by 125 · see citing works
Self-citationWang, S., Wang, Q., & Zhao, J. (2020) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2020.102701 · Cited by 117 · see citing works
Dhanorkar, S., Wolf, C. T., Qian, K., Xu, A., Popa, L., & Li, Y. (2021) · Designing Interactive Systems Conference 2021 · doi:10.1145/3461778.3462131 · Cited by 119 · see citing works
Women also Snowboard: Overcoming Bias in Captioning ModelsPreprint
Burns, K., Hendricks, L. A., Saenko, K., Darrell, T., & Rohrbach, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.09797 · Cited by 113 · see citing works
A Perspective on Explanations of Molecular Prediction Models
Wellawatte, G. P., Gandhi, H. A., Seshadri, A., & White, A. D. (2023) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.2c01235 · Cited by 116 · see citing works
A Survey of the State of Explainable AI for Natural Language ProcessingPreprint
Danilevsky, M., Qian, K., Aharonov, R., Katsis, Y., Kawas, B., & Sen, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.00711 · Cited by 108 · see citing works
RelEx: A Model-Agnostic Relational Model Explainer
Zhang, Y., DeFazio, D., & Ramesh, A. (2020) · AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462562 · Cited by 141 · see citing works
Detecting Statistical Interactions from Neural Network WeightsPreprint
Tsang, M., Cheng, D., & Liu, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.04977 · Cited by 133 · see citing works
Clinical Predictive Models for COVID-19: Systematic Study
Schwab, P., Schütte, A. D., Dietz, B., & Bauer, S. (2020) · Journal of Medical Internet Research · doi:10.2196/21439 · Cited by 110 · see citing works
Anik, A. I., & Bunt, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445736 · Cited by 103 · see citing works
Kortylewski, A., Liu, Q., Wang, A., Sun, Y., & Yuille, A. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-020-01401-3 · Cited by 98 · see citing works
Incorporating Priors with Feature Attribution on Text Classification
Liu, F., & Avci, B. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1631 · Cited by 104 · see citing works
Examining CNN Representations With Respect to Dataset Bias
Zhang, Q., Wang, W., & Zhu, S. C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11833 · Cited by 95 · see citing works
EnD: Entangling and Disentangling deep representations for bias correction
Tartaglione, E., Barbano, C. A., & Grangetto, M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01330 · Cited by 85 · see citing works
Explainable AI: A review of applications to neuroimaging data
Farahani, F. V., Fiok, K., Lahijanian, B., Karwowski, W., & Douglas, P. K. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.906290 · Cited by 85 · see citing works
A survey on artificial intelligence in histopathology image analysis
Abdelsamea, M. M., Zidan, U., Senousy, Z., Gaber, M. M., Rakha, E. A., & Ilyas, M. (2022) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1474 · Cited by 80 · see citing works
Yang, Y., & Chui, T. F. M. (2021) · Hydrology and earth system sciences · doi:10.5194/hess-25-5839-2021 · Cited by 76 · see citing works
The Role of Explainability in Assuring Safety of Machine Learning in Healthcare
Jia, Y., McDermid, J. A., Lawton, T., & Habli, I. (2022) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2022.3171314 · Cited by 82 · see citing works
Evaluating Attribution for Graph Neural Networks
Sánchez-Lengeling, B., Wei, J. N., Lee, B. K., Reif, E., Wang, P. Y., Qian, W. W., McCloskey, K., Colwell, L. J., & Wiltschko, A. B. (2020) · neural information processing systems · Cited by 74 · see citing works
Debugging Tests for Model Explanations
Adebayo, J., Muelly, M., Liccardi, I., & Kim, B. (2020) · arXiv · doi:10.48550/arxiv.2011.05429 · Cited by 88 · see citing works
Interpretable CNNs for Object Classification
Zhang, Q., Wang, X., Wu, Y. N., Zhou, H., & Zhu, S. C. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2982882 · Cited by 71 · see citing works
Right for the Right Concept: Revising Neuro-Symbolic Concepts by Interacting with their Explanations
Stammer, W., Schramowski, P., & Kersting, K. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00362 · Cited by 73 · see citing works
The Road to Explainability is Paved with Bias: Measuring the Fairness of Explanations
Balagopalan, A., Zhang, H., Hamidieh, K., Hartvigsen, T., Rudzicz, F., & Ghassemi, M. (2022) · Conference on Fairness, Accountability and Transparency · doi:10.1145/3531146.3533179 · Cited by 115 · see citing works
Going Beyond XAI: A Systematic Survey for Explanation-Guided Learning
Gao, Y., Gu, S., Jiang, J., Hong, S.-S., Yu, D., & Zhao, L. (2024) · ACM Computing Surveys · doi:10.1145/3644073 · Cited by 71 · see citing works
Kiyasseh, D., Laca, J., Haque, T. F., Miles, B. J., Wagner, C., Donoho, D. A., Anandkumar, A., & Hung, A. J. (2023) · Communications Medicine · doi:10.1038/s43856-023-00263-3 · Cited by 69 · see citing works
Explaining A Black-box By Using A Deep Variational Information Bottleneck Approach
Bang, S., Xie, P., Lee, H., Wu, W., & Xing, E. P. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i13.17358 · Cited by 69 · see citing works
Problems in the deployment of machine-learned models in health care
Cohen, J., Cao, T., Viviano, J. D., Huang, C.-W., Fralick, M., Ghassemi, M., Mamdani, M., Greiner, R., & Bengio, Y. (2021) · Canadian Medical Association Journal · doi:10.1503/cmaj.202066 · Cited by 70 · see citing works
A systematic review of generalization research in medical image classification
Matta, S. C., Lamard, M., Zhang, P., Guilcher, A. L., Borderie, L., Cochener, B., & Quellec, G. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109256 · Cited by 75 · see citing works
Toward Interpretable Machine Learning: Transparent Deep Neural Networks and BeyondPreprint
Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Müller, K. (2020) · arXiv (Cornell University) · Cited by 70 · see citing works
A Survey of Evaluation Methods and Measures for Interpretable Machine LearningPreprint
Mohseni, S., Zarei, N., & Ragan, E. D. (2018) · arXiv (Cornell University) · Cited by 69 · see citing works
Bassi, P. R. A. S., Dertkigil, S. S. J., & Cavalli, A. (2024) · Nature Communications · doi:10.1038/s41467-023-44371-z · Cited by 71 · see citing works
Uncovering and Correcting Shortcut Learning in Machine Learning Models for Skin Cancer Diagnosis
Nauta, M., Walsh, R., Dubowski, A., & Seifert, C. (2021) · Diagnostics · doi:10.3390/diagnostics12010040 · Cited by 67 · see citing works
Towards relatable explainable AI with the perceptual process
Zhang, W., & Lim, B. Y. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501826 · Cited by 67 · see citing works
Transformer based contextualization of pre-trained word embeddings for irony detection in Twitter
González, J. Á., Hurtado, L.-F., & Plà, F. (2020) · Information Processing & Management · doi:10.1016/j.ipm.2020.102262 · Cited by 66 · see citing works
Dissonance Between Human and Machine Understanding
Zhang, Z., Singh, J., Gadiraju, U., & Anand, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359158 · Cited by 59 · see citing works
A review of data-driven modelling in drinking water treatment
Aliashrafi, A., Zhang, Y., Groenewegen, H., & Peleato, N. M. (2021) · Reviews in Environmental Science and Bio/Technology · doi:10.1007/s11157-021-09592-y · Cited by 59 · see citing works
Zhang, J., & Li, J. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106296 · Cited by 58 · see citing works
Assessing Heuristic Machine Learning Explanations with Model Counting
Narodytska, N., Shrotri, A. A., Meel, K. S., Ignatiev, A., & Marques-Silva, J. (2019) · International Conference on Theory and Applications of Satisfiability Testing · doi:10.1007/978-3-030-24258-9_19 · Cited by 68 · see citing works
Evaluating Feature Importance EstimatesPreprint
Hooker, S., Erhan, D., Kindermans, P.-J., & Kim, B. (2018) · arXiv (Cornell University) · Cited by 61 · see citing works
Learning Explainable Models Using Attribution Priors
Erion, G., Janizek, J. D., Sturmfels, P., Lundberg, S., & Lee, S. (2019) · Cited by 53 · see citing works
Counterfactual Samples Synthesizing and Training for Robust Visual Question Answering
Chen, L., Zheng, Y., Niu, Y., Zhang, H., & Xiao, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3290012 · Cited by 51 · see citing works
Leveraging explanations in interactive machine learning: An overview
Teso, S., Alkan, Ö., Stammer, W., & Daly, E. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1066049 · Cited by 55 · see citing works
Learning Variational Word Masks to Improve the Interpretability of Neural Text Classifiers
Chen, H., & Ji, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.347 · Cited by 52 · see citing works
Evaluating and Mitigating Bias in Image Classifiers: A Causal Perspective Using Counterfactuals
Dash, S., Balasubramanian, V. N., & Sharma, A. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00393 · Cited by 50 · see citing works
Zafar, M. R., & Khan, N. (2023) · doi:10.32920/22734359 · Cited by 48 · see citing works
Ortiz-Jiménez, G., Modas, A., Moosavi-Dezfooli, S.-M., & Pascal Frossard, B. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3050042 · Cited by 45 · see citing works
Explain and Predict, and then Predict Again
Zhang, Z., Rudra, K., & Anand, A. (2021) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3437963.3441758 · Cited by 48 · see citing works
- Authors
- Ross, A. S., Hughes, M. C., & Doshi-Velez, F.
- Affiliations
- Harvard University (US)
- Harvard University
- Venue
- Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence
- Published
- 2017-08
- Pages
- 2662-2670
- OpenAlex
- W2595017202
- Cited by
- 515
- References
- 31
- Citations by year
- 2026: 34 · 2025: 46 · 2024: 58 · 2023: 73 · 2022: 64 · 2021: 93 · 2020: 67 · 2019: 43 · 2018: 31 · 2017: 5 · 2012: 1
- FWCI
- 18.21
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Joint Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency (W911NF-16-1-0561)
- Air Force Office of Scientific Research (FA9550-17, FA9550-, FA9550-17-1, -16-1-)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- explainable AI, gradient-based explanation, decision boundaries, neural network interpretability, perturbation method, lime
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
