CoChat
Sign Up
PreprintCited by 22Explainable Artificial Intelligence (XAI)Ethics and Social Impacts of AISoftware Testing and Debugging Techniques

The DOI matches its DataCite record; 3 of the 3 references it cites were found in a registry or an index.

Interpretability Needs a New Paradigm

Open access
Madsen, Andreas, Lakkaraju, Himabindu, Reddy, Siva, Chandar, Sarath·2024·arXiv
DOI 10.48550/arxiv.2405.05386arXiv 2405.05386
Open this paper in CoChat — free accountOpen PDF

Abstract

Interpretability is the study of explaining models in understandable terms to humans. At present, interpretability is divided into two paradigms: the intrinsic paradigm, which believes that only models designed to be explained can be explained, and the post-hoc paradigm, which believes that black-box models can be explained. At the core of this debate is how each paradigm ensures its explanations are faithful, i.e., true to the model's behavior. This is important, as false but convincing explanations lead to unsupported confidence in artificial intelligence (AI), which can be dangerous. This paper's position is that we should think about new paradigms while staying vigilant regarding faithfulness. First, by examining the history of paradigms in science, we see that paradigms are constantly evolving. Then, by examining the current paradigms, we can understand their underlying beliefs, the value they bring, and their limitations. Finally, this paper presents 3 emerging paradigms for interpretability. The first paradigm designs models such that faithfulness can be easily measured. Another optimizes models such that explanations become faithful. The last paradigm proposes to develop models that produce both a prediction and an explanation.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2405.05386

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Conflating hypotheses with conclusions has been commonplace in mechanistic interpretability research, making validation an important area for the field to improve.

    “Conflating hypotheses with conclusions has regrettably been commonplace in mechanistic interpretability research, making validation an important area for the field to improve (Madsen et al., 2024;Stander et al., 2025).”

    From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Madsen et al., 2024

    Section 1.1 says that validating whether explanations faithfully reflect a model is challenging and warns against taking faithfulness for granted; the Conclusion calls for measurable definitions of faithfulness. But Section 2.2.2 merely describes mechanistic interpretability. The paper does not say that conflating hypotheses with conclusions is commonplace in mechanistic interpretability research. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 101

  1. Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., and Kim, B. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, vol- ume 2018-Decem, pp. 9505-9515.
  2. Curran Associates, Inc., 10 2018. URL http://arxiv.org/abs/ 1810.03292.
  3. Adebayo, J., Muelly, M., Abelson, H., and Kim, B. Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlation. In International Conference on Learning Representations, pp. 1-13, 2021. URL https: //openreview.net/forum?id=xNOVfCCvDpM.
  4. Agarwal, C., Tanneru, S. H., and Lakkaraju, H. Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Models. arXiv, 2024. URL http: //arxiv.org/abs/2402.04614.
  5. Ahern, I., Noack, A., Guzmán-Nateras, L., Dou, D., Li, B., and Huan, J. Normlime: A new feature importance metric for explaining deep neural networks. arXiv, 9 2019. ISSN 23318422. URL http://arxiv.org/ abs/1909.04200. · arXiv
  6. Amer, M. and Maul, T. A review of modulariza- tion techniques in artificial neural networks. Ar- tificial Intelligence Review, 52(1):527-561, 6 2019. ISSN 0269-2821. doi: 10.1007/s10462-019-09706-7. URL http://link.springer.com/10.1007/ s10462-019-09706-7. · DOI
  7. Andreas, J., Rohrbach, M., Darrell, T., and Klein, D. Neu- ral Module Networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 39-48. IEEE, 6 2016. ISBN 978-1-4673-8851-1. doi: 10.1109/CVPR.2016.12. URL http://ieeexplore. ieee.org/document/7780381/. · DOI
  8. Andrus, M., Spitzer, E., Brown, J., and Xiang, A. What we can't measure, We can't understand: Challenges to demographic data procurement in the pursuit of fairness. FAccT 2021 -Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 249- 260, 2021. doi: 10.1145/3442188.3445888. · DOI · Paper page
  9. Arya, V., Bellamy, R. K. E., Chen, P.-Y., Dhurandhar, A., Hind, M., Hoffman, S. C., Houde, S., Liao, Q. V., Luss, R., Mojsilović, A., Mourad, S., Pedemonte, P., Raghaven- dra, R., Richards, J., Sattigeri, P., Shanmugam, K., Singh, M., Varshney, K. R., Wei, D., and Zhang, Y. One Explana- tion Does Not Fit All: A Toolkit and Taxonomy of AI Ex- plainability Techniques. arXiv, 9 2019. ISSN 23318422. URL http://arxiv.org/abs/1909.03012. · arXiv
  10. Baehrens, D., Schroeter, T., Harmeling, S., Kawanabe, M., Hansen, K., and Müller, K. R. How to explain individual classification decisions. Journal of Machine Learning Research, 11:1803-1831, 12 2010. ISSN 15324435. URL http://arxiv.org/abs/0912.1128. · arXiv
  11. Bahdanau, D., Cho, K. H., and Bengio, Y. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015 -Conference Track Proceedings, pp. 1-15. International Conference on Learning Representations, ICLR, 9 2015. URL https://arxiv.org/abs/ 1409.0473.
  12. Bastings, J. and Filippova, K. The elephant in the in- terpretability room: Why use attention as explana- tion when we have saliency methods? In Proceed- ings of the Third BlackboxNLP Workshop on Analyz- ing and Interpreting Neural Networks for NLP, pp. 149-155, Stroudsburg, PA, USA, 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020. blackboxnlp-1.14. URL https://www.aclweb. org/anthology/2020.blackboxnlp-1. · DOI
  13. Bastings, J., Ebert, S., Zablotskaia, P., Sandholm, A., and Filippova, K. "Will You Find These Shortcuts?" A Pro- tocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 976-991, Stroudsburg, PA, USA, 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.emnlp-main.64. URL https: //aclanthology.org/2022.emnlp-main.64. Belinkov, Y. Probing Classifiers: Promises, Shortcomings, and Advances. arXiv, pp. 1-12, 2 2021. URL http: //arxiv.org/abs/2102.12452. · DOI
  14. Belinkov, Y. and Glass, J. Analysis Methods in Neural Language Processing: A Survey. Transactions of the Association for Computational Linguistics, 7:49-72, 4 2019. ISSN 2307-387X. doi: 10.1162/tacl{\ }a{\ } 00254. URL https://doi.org/10.1162/tacl_ a_00254. · DOI
  15. Belinkov, Y., Gehrmann, S., and Pavlick, E. Inter- pretability and Analysis in Neural NLP. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pp. 1-5, Stroudsburg, PA, USA, 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020. acl-tutorials.1. URL https://www.aclweb.org/ anthology/2020.acl-tutorials. · DOI
  16. Bhalla, U., Srinivas, S., and Lakkaraju, H. Verifiable Feature Attributions: A Bridge between Post Hoc Explainability and Inherent Interpretability. arXiv, pp. 1-14, 2023. URL http://arxiv.org/abs/2307.15007. · arXiv
  17. Bhatt, U., Xiang, A., Sharma, S., Weller, A., Taly, A., Jia, Y., Ghosh, J., Puri, R., Moura, J. M. F., and Eckersley, P. Explainable Machine Learning in Deployment. Pro- ceedings of the 2020 Conference on Fairness, Account- ability, and Transparency, pp. 648-657, 9 2019. doi: 10.1145/3351095.3375624. URL https://dl.acm. org/doi/10.1145/3351095.3375624. · DOI · Paper page
  18. Bien, J. and Tibshirani, R. Classification by Set Cover: The Prototype Vector Machine. arXiv, pp. 1-24, 2009. URL http://arxiv.org/abs/0908.2284. · arXiv
  19. Bilodeau, B., Jaques, N., Koh, P. W., and Kim, B. Impossibility theorems for feature attribution. Pro- ceedings of the National Academy of Sciences, 121(2):1-38, 1 2024. ISSN 0027-8424. doi: 10.1073/pnas.2304406120. URL https://pnas. org/doi/10.1073/pnas.2304406120http: //arxiv.org/abs/2212.11870. · DOI
  20. Binder, A., Montavon, G., Lapuschkin, S., Müller, K.- R., and Samek, W. Layer-Wise Relevance Propa- gation for Neural Networks with Local Renormaliza- tion Layers. In Artificial Neural Networks and Ma- chine Learning -ICANN 2016, volume 9887 LNCS, pp. 63-71, 2016. doi: 10.1007/978-3-319-44781-0{\ }8. URL http://link.springer.com/10.1007/ 978-3-319-44781-0_8. · DOI
  21. Borowski, J., Zimmermann, R. S., Schepers, J., Geirhos, R., Wallis, T. S., Bethge, M., and Brendel, W. Exemplary Natural Images Explain Cnn Activations Better Than State-of-the-Art Feature Visualization. ICLR 2021 -9th International Conference on Learning Representations, pp. 1-41, 2021.
  22. Cammarata, N., Carter, S., Goh, G., Olah, C., Petrov, M., and Schubert, L. Thread: Circuits. Distill, 5(3), 3 2020. ISSN 2476-0757. doi: 10.23915/distill.00024. URL https://distill.pub/2020/circuits. · DOI
  23. Chen, C., Li, O., Tao, C., Barnett, A. J., Su, J., and Rudin, C. This looks like that: Deep learning for interpretable image recognition. Advances in Neural Information Processing Systems, 32, 6 2019. ISSN 10495258. URL http: //arxiv.org/abs/1806.10574.
  24. Chen, J., Song, L., Wainwright, M. J., and Jordan, M. I. Learning to explain: An information-theoretic perspective on model interpretation. 35th International Conference on Machine Learning, ICML 2018, 2:1386-1418, 2 2018. URL http://arxiv.org/abs/1802.07814. · arXiv
  25. Chen, Y., Zhong, R., Ri, N., Zhao, C., He, H., Stein- hardt, J., Yu, Z., and McKeown, K. Do Models Ex- plain Themselves? Counterfactual Simulatability of Natural Language Explanations. arXiv, 2023. URL http://arxiv.org/abs/2307.08678. · arXiv
  26. Clark, K., Khandelwal, U., Levy, O., and Manning, C. D. What Does BERT Look at? An Analysis of BERT's Attention. In Proceedings of the 2019 ACL Work- shop BlackboxNLP: Analyzing and Interpreting Neu- ral Networks for NLP, pp. 276-286, Stroudsburg, PA, USA, 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-4828. URL https://www. aclweb.org/anthology/W19-4828. · DOI
  27. Clouatre, L., Parthasarathi, P., Zouaq, A., and Chandar, S. Local Structure Matters Most: Perturbation Study in NLU. In Findings of the Association for Computa- tional Linguistics: ACL 2022, pp. 3712-3731, Strouds- burg, PA, USA, 7 2022. Association for Computa- tional Linguistics. doi: 10.18653/v1/2022.findings-acl. 293. URL https://aclanthology.org/2022. findings-acl.293. · DOI
  28. Coenen, A., Reif, E., Yuan, A., Kim, B., Pearce, A., Viégas, F., Wattenberg, M., Kim, A. Y. B., Pearce, A., Viégas, F., Wattenberg, M., Yuan, A., Wattenberg, M., Viegas, F. B., Coenen, A., Pearce, A., and Kim, B. Visualizing and Measuring the Geometry of BERT. In Wallach, H., Larochelle, H., Beygelzimer, A., d\textquotesingle Alché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32, pp. 8594-8603.
  29. Curran Associates, Inc., 6 2019. URL https://proceedings. neurips.cc/paper/2019/file/ 159c1ffe5b61b41b3c4d8f4c2150f6c4-Paper. pdf.
  30. Conneau, A., Kruszewski, G., Lample, G., Barrault, L., and Baroni, M. What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic prop- erties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2126-2136, Stroudsburg, PA, USA, 2018. Association for Computational Linguistics. ISBN 9781948087322. doi: 10.18653/v1/P18-1198. URL http://aclweb.org/anthology/P18-1198. · DOI · Paper page
  31. Covert, I. C., Kim, C., and Lee, S.-I. Learning to Estimate Shapley Values with Vision Transform- ers. The Eleventh International Conference on Learning Representations, pp. 1-48, 2022. URL http://arxiv.org/abs/2206.05282https: //openreview.net/forum?id=5ktFNz_pJLK.
  32. Dara, S., Dhamercherla, S., Jadav, S. S., Babu, C. H. M., and Ahsan, M. J. Machine learning in drug discovery: a review. Artificial Intelligence Review, 55(3):1947-1999, 2022.
  33. Dauben, J. W. Georg Cantor and Pope Leo XIII: Math- ematics, Theology, and the Infinite. Journal of the History of Ideas, 38(1):85, 1 1977. ISSN 00225037. doi: 10.2307/2708842. URL https://www.jstor. org/stable/2708842?origin=crossref. · DOI
  34. Doshi-Velez, F. and Kim, B. Towards A Rigorous Science of Interpretable Machine Learning. arXiv, 2 2017. URL http://arxiv.org/abs/1702.08608. · arXiv
  35. ElShawi, R., Sherif, Y., Al-Mallah, M., and Sakr, S. ILIME: Local and Global Interpretable Model- Agnostic Explainer of Black-Box Decision. In Welzer, T., Eder, J., Podgorelec, V., and Kamišalić Lat- ifić, A. (eds.), Advances in Databases and In- formation Systems, pp. 53-68. Springer Interna- tional Publishing, Cham, 2019. ISBN 978-3-030- 28730-6. doi: 10.1007/978-3-030-28730-6{\ }4. URL http://link.springer.com/10.1007/ 978-3-030-28730-6_4. · DOI
  36. Elton, D. C. Self-explaining AI as an Alterna- tive to Interpretable AI. Lecture Notes in Com- puter Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioin- formatics), 12177 LNAI:95-106, 2 2020. ISSN 16113349. doi: 10.1007/978-3-030-52152-3{\ }10. URL http://link.springer.com/10.1007/ 978-3-030-52152-3_10. · DOI
  37. Fashandi, H. Neural module networks: A review. Neu- rocomputing, 552:126518, 2023. ISSN 18728286. doi: 10.1016/j.neucom.2023.126518. URL https://doi. org/10.1016/j.neucom.2023.126518. · DOI
  38. Flora, M., Potvin, C., McGovern, A., and Handler, S. Com- paring Explanation Methods for Traditional Machine Learning Models Part 1: An Overview of Current Meth- ods and Quantifying Their Disagreement. arXiv, pp. 1-22, 2022. URL http://arxiv.org/abs/2211. 08943.
  39. Geirhos, R., Zimmermann, R. S., Bilodeau, B., Brendel, W., and Kim, B. Don't trust your eyes: on the (un)reliability of feature visualizations. arXiv, 2023. URL http:// arxiv.org/abs/2306.04719. · arXiv
  40. Goodman, B. and Flaxman, S. European union regulations on algorithmic decision making and a "right to explana- tion". AI Magazine, 38(3):50-57, 2017. ISSN 07384602. doi: 10.1609/aimag.v38i3.2741. · DOI · Paper page
  41. Gray, J. Did poincaré say "set theory is a dis- ease"? The Mathematical Intelligencer, 13(1): 19-22, 12 1991. ISSN 0343-6993. doi: 10.1007/ BF03024067. URL https://maa.org/press/ periodicals/convergence/quotations/ poincare-jules-henri-1854-1912-0http: //link.springer.com/10.1007/ BF03024067. · DOI
  42. Gupta, N., Lin, K., Roth, D., Singh, S., and Gardner, M. Neural Module Networks for Reasoning over Text. In International Conference on Learning Representa- tions (ICLR), 12 2020. URL https://openreview. net/forum?id=SygWvAVFPr.
  43. Han, T., Srinivas, S., and Lakkaraju, H. Which Explanation Should I Choose? A Function Approximation Perspective to Characterizing Post Hoc Explanations. Advances in Neural Information Processing Systems, 35(NeurIPS), 2022. ISSN 10495258. URL http://arxiv.org/ abs/2206.01254. · arXiv
  44. Hase, P., Xie, H., and Bansal, M. The Out-of-Distribution Problem in Explainability and Search Methods for Fea- ture Importance Explanations. Advances in Neural In- formation Processing Systems, 5(NeurIPS):3650-3666, 2021. ISSN 10495258.
  45. Hewitt, J. and Liang, P. Designing and Interpreting Probes with Control Tasks. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2733-2743, Stroudsburg, PA, USA, 2019. Association for Computational Linguistics. ISBN 9781950737901. doi: 10.18653/v1/D19-1275. URL https://www. aclweb.org/anthology/D19-1275. · DOI · Paper page
  46. Hooker, S., Erhan, D., Kindermans, P.-J. J., and Kim, B. A benchmark for interpretability methods in deep neural networks. In Advances in Neural Information Processing Systems, volume 32, 6 2019. URL http://arxiv. org/abs/1806.10758.
  47. Huang, S., Mamidanna, S., Jangam, S., Zhou, Y., and Gilpin, L. H. Can Large Language Models Explain Them- selves? A Study of LLM-Generated Self-Explanations. arXiv, 2023. URL http://arxiv.org/abs/2310. 11207.
  48. Jacovi, A. and Goldberg, Y. Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pp. 4198-4205, Stroudsburg, PA, USA, 4 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020. acl-main.386. URL https://www.aclweb.org/ anthology/2020.acl-main.386. · DOI · Paper page
  49. Jain, S. and Wallace, B. C. Attention is not Explanation. In Proceedings of the 2019 Conference of the North, volume 1, pp. 3543-3556, Stroudsburg, PA, USA, 2 2019. Association for Computational Linguistics. ISBN 9781950737130. doi: 10.18653/v1/N19-1357. URL http://aclweb.org/anthology/N19-1357. · DOI
  50. Jethani, N., Sudarshan, M., Aphinyanaphongs, Y., and Ran- ganath, R. Have We Learned to Explain?: How Inter- pretability Methods Can Learn to Encode Predictions in their Interpretations. Proceedings of International Con- ference on Artificial Intelligence and Statistics (AISTATS), 130:1459-1467, 2021. ISSN 26403498.
  51. Jethani, N., Sudarshan, M., Covert, I., Lee, S. I., and Ran- ganath, R. Fastshap: Real-Time Shapley Value Estima- tion. ICLR 2022 -10th International Conference on Learning Representations, pp. 1-23, 2022.
  52. Jiménez-Luna, J., Grisoni, F., and Schneider, G. Drug discovery with explainable artificial intelligence. Nature Machine Intelligence, 2(10):573-584, 2020.
  53. Karpathy, A., Johnson, J., and Fei-Fei, L. Visualizing and Understanding Recurrent Networks. arXiv, pp. 1- 12, 6 2015. URL http://arxiv.org/abs/1506. 02078.
  54. Kim, B. Beyond interpretability: developing a lan- guage to shape our relationships with AI. In The In- ternational Conference on Learning Representations, 2022. URL https://iclr.cc/Conferences/ 2022/Schedule?showEvent=7237.
  55. Kim, B., Rudin, C., and Shah, J. The Bayesian case model: A generative approach for case-based reasoning and pro- totype classification. Advances in Neural Information Processing Systems, 3(January):1952-1960, 2014. ISSN 10495258.
  56. Kindermans, P.-J., Hooker, S., Adebayo, J., Alber, M., Schütt, K. T., Dähne, S., Erhan, D., and Kim, B. The (Un)reliability of Saliency Methods. In Lecture Notes in Computer Science (including subseries Lec- ture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 11700 LNCS, pp. 267-280. Springer, 11 2019. doi: 10.1007/978-3-030-28954-6{\ }14. URL http://link.springer.com/10. 1007/978-3-030-28954-6_14. · DOI
  57. Kodiyan, A. A. An overview of ethical issues in using AI systems in hiring with a case study of Amazon's AI based hiring tool. Researchgate Preprint, pp. 1-19, 2019.
  58. Krishna, S., Han, T., Gu, A., Pombra, J., Jabbari, S., Wu, S., and Lakkaraju, H. The Disagreement Problem in Ex- plainable Machine Learning: A Practitioner's Perspective. arXiv, 2022. URL http://arxiv.org/abs/2202. 01602.
  59. Kuhn, T. S. The Structure of Scientific Revolutions. Uni- versity of Chicago Press, 3rd editio edition, 1996. ISBN 978-0-226-45807-6.
  60. Lanham, T., Chen, A., Radhakrishnan, A., Steiner, B., Denison, C., Hernandez, D., Li, D., Durmus, E., Hub- inger, E., Kernion, J., Lukošiūtė, K., Nguyen, K., Cheng, N., Joseph, N., Schiefer, N., Rausch, O., Larson, R., McCandlish, S., Kundu, S., Kadavath, S., Yang, S., Henighan, T., Maxwell, T., Telleen-Lawton, T., Hume, T., Hatfield-Dodds, Z., Kaplan, J., Brauner, J., Bow- man, S. R., and Perez, E. Measuring Faithfulness in Chain-of-Thought Reasoning. arXiv, 2023. URL http://arxiv.org/abs/2307.13702. · arXiv
  61. Li, J., Monroe, W., and Jurafsky, D. Understanding Neural Networks through Representation Erasure. arXiv, 2016. URL http://arxiv.org/abs/1612.08220. · arXiv
  62. Lipton, Z. C. The mythos of model interpretability. Com- munications of the ACM, 61(10):36-43, 9 2018. ISSN 0001-0782. doi: 10.1145/3233231. URL https: //dl.acm.org/doi/10.1145/3233231. · DOI · Paper page
  63. Lyu, Q., Apidianaki, M., and Callison-Burch, C. Towards Faithful Model Explanation in NLP: A Survey. Computational Linguistics, 50(2): 657-723, 6 2024. ISSN 0891-2017. doi: 10.1162/coli{\ }a{\ }00511. URL http://arxiv. org/abs/2209.11326https://direct.mit. edu/coli/article/50/2/657/119158/ Towards-Faithful-Model-Explanation-in-NLP-A-Survey. · DOI
  64. Madsen, A., Meade, N., Adlakha, V., and Reddy, S. Evalu- ating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and Re- training. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 1731-1751, Abu Dhabi, United Arab Emirates, 12 2022a. Association for Compu- tational Linguistics. URL https://aclanthology. org/2022.findings-emnlp.125.
  65. Madsen, A., Reddy, S., and Chandar, S. Post-hoc In- terpretability for Neural NLP: A Survey. ACM Com- puting Surveys, 55(8):1-42, 8 2022b. ISSN 0360- 0300. doi: 10.1145/3546577. URL https://dl.acm. org/doi/10.1145/3546577. · DOI
  66. Madsen, A., Chandar, S., and Reddy, S. Are self- explanations from Large Language Models faithful? The 62nd Annual Meeting of the Association for Com- putational Linguistics, 1 2024a. URL https:// openreview.net/forum?id=0fB5OROAIq.
  67. Madsen, A., Reddy, S., Chandar, S., and Anonymous. Faithfulness Measurable Masked Language Models. In Forty-first International Conference on Machine Learning, 2024b. URL https://openreview. net/forum?id=tw1PwpuAuNhttp://arxiv. org/abs/2310.07819.
  68. McCoy, T., Pavlick, E., and Linzen, T. Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Lan- guage Inference. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pp. 3428-3448, Stroudsburg, PA, USA, 2019. Association for Computational Linguistics. ISBN 9781950737482. doi: 10.18653/v1/P19-1334. URL https://www. aclweb.org/anthology/P19-1334. · DOI · Paper page
  69. Meister, C., Lazov, S., Augenstein, I., and Cotterell, R. Is Sparse Attention more Interpretable? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 122-129, Stroudsburg, PA, USA, 8 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-short.17. URL http://arxiv.org/abs/2106.01087https: //aclanthology.org/2021.acl-short. · DOI
  70. Messiah, A. Quantum Mechanics. North Holland, John Wiley & Sons, 1966. ISBN 0486409244. Meta. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv, 2023. URL http://arxiv.org/ abs/2307.09288. · arXiv
  71. Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., and Yu, B. Definitions, methods, and appli- cations in interpretable machine learning. Proceed- ings of the National Academy of Sciences of the United States of America, 116(44):22071-22080, 10 2019. ISSN 10916490. doi: 10.1073/pnas.1900654116. URL http://www.pnas.org/lookup/doi/10. 1073/pnas.1900654116. · DOI · Paper page
  72. Nguyen, A., Yosinski, J., and Clune, J. Multifaceted Feature Visualization: Uncovering the Different Types of Fea- tures Learned By Each Neuron in Deep Neural Networks. Visualization for Deep Learning workshop at ICML, 2016. URL http://arxiv.org/abs/1602.03616. · arXiv
  73. Olah, C., Mordvintsev, A., and Schubert, L. Feature Vi- sualization. Distill, 2(11), 11 2017. ISSN 2476-0757. doi: 10.23915/distill.00007. URL https://distill. pub/2017/feature-visualization. OpenAI. GPT-4 Technical Report. OpenAI, 4:1-100, 3 2023. URL http://arxiv.org/abs/2303.08774. · DOI · Paper page
  74. Parcalabescu, L. and Frank, A. On Measuring Faithfulness of Natural Language Explanations. arXiv, 2023. URL http://arxiv.org/abs/2311.07466. · arXiv
  75. Preuer, K., Klambauer, G., Rippmann, F., Hochreiter, S., and Unterthiner, T. Interpretable deep learning in drug discovery. Explainable AI: interpreting, explaining and visualizing deep learning, pp. 331-345, 2019.
  76. Ribeiro, M. T., Singh, S., and Guestrin, C. "Why should i trust you?" Explaining the predictions of any clas- sifier. In Proceedings of the ACM SIGKDD Interna- tional Conference on Knowledge Discovery and Data Mining, volume 13-17-Augu, pp. 1135-1144, New York, NY, USA, 8 2016. ACM. ISBN 9781450342322. doi: 10.1145/2939672.2939778. URL https://dl.acm. org/doi/10.1145/2939672.2939778. · DOI · Paper page
  77. Robnik-Šikonja, M. and Bohanec, M. Perturbation-Based Explanations of Prediction Models. Springer Interna- tional Publishing, 2018. ISBN 9783319904030. doi: 10.1007/978-3-319-90403-0{\ }9. URL http://dx. doi.org/10.1007/978-3-319-90403-0_9. · DOI · Paper page
  78. Rogers, A., Kovaleva, O., and Rumshisky, A. A Primer in BERTology: What We Know About How BERT Works. Transactions of the Association for Computa- tional Linguistics, 8:842-866, 12 2020. ISSN 2307- 387X. doi: 10.1162/tacl{\ }a{\ }00349. URL https: //direct.mit.edu/tacl/article/96482. · DOI
  79. Rudin, C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5): 206-215, 2019. ISSN 2522-5839. doi: 10.1038/ s42256-019-0048-x. URL http://www.nature. com/articles/s42256-019-0048-x. · DOI · Paper page
  80. Samek, W., Binder, A., Montavon, G., Lapuschkin, S., and Muller, K.-R. Evaluating the Visualization of What a Deep Neural Network Has Learned. IEEE Transactions on Neural Networks and Learning Systems, 28(11):2660- 2673, 11 2017. ISSN 2162-237X. doi: 10.1109/TNNLS. 2016.2599820. URL https://ieeexplore.ieee. org/document/7552539/. · DOI · Paper page
  81. Sangroya, A., Rastogi, M., Anantaram, C., and Vig, L. Guided-LIME: Structured sampling based hybrid ap- proach towards explaining blackbox machine learning models. In CEUR Workshop Proceedings, volume 2699, 2020.
  82. Schut, L., Tomasev, N., McGrath, T., Hassabis, D., Paquet, U., and Kim, B. Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZero. arXiv, pp. 1-61, 10 2023. URL http://arxiv.org/abs/ 2310.16410.
  83. Seo, J., Choe, J., Koo, J., Jeon, S., Kim, B., and Jeon, T. Noise-adding Methods of Saliency Map as Series of Higher Order Partial Derivative. In 2018 ICML Workshop on Human Interpretability in Machine Learning, 6 2018. URL http://arxiv.org/abs/1806.03000. · arXiv
  84. Serrano, S. and Smith, N. A. Is Attention Inter- pretable? In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pp. 2931-2951, Stroudsburg, PA, USA, 6 2019. Asso- ciation for Computational Linguistics. doi: 10.18653/ v1/P19-1282. URL https://www.aclweb.org/ anthology/P19-1282. · DOI · Paper page
  85. Shrikumar, A., Greenside, P., and Kundaje, A. Learning important features through propagating activation differ- ences. In 34th International Conference on Machine Learning, ICML 2017, volume 7, pp. 4844-4866, 2017. ISBN 9781510855144. URL https://arxiv.org/.
  86. Slack, D., Hilgard, S., Jia, E., Singh, S., and Lakkaraju, H. Fooling LIME and SHAP. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and So- ciety, pp. 180-186, New York, NY, USA, 2 2020. ACM. ISBN 9781450371100. doi: 10.1145/3375627. 3375830. URL https://dl.acm.org/doi/10. 1145/3375627.3375830. · DOI · Paper page
  87. Smilkov, D., Thorat, N., Kim, B., Viégas, F., and Watten- berg, M. SmoothGrad: removing noise by adding noise. ICML workshop on visualization for deep learning, 2017. ISSN 23318422. URL https://goo.gl/EfVzEE.
  88. Smith, J. David Hilbert's Radio Ad- dress. Convergence, 2014. doi: 10.4169/ convergence20140202. URL https://old.maa. org/press/periodicals/convergence/ david-hilberts-radio-address-hilbert-and-mathemat Subramanian, S., Bogin, B., Gupta, N., Wolfson, T., Singh, S., Berant, J., and Gardner, M. Obtaining faithful interpretations from compositional neural net- works. Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 5594- 5608, 2020. ISSN 0736587X. doi: 10.18653/v1/2020. acl-main.495. URL https://www.aclweb.org/ anthology/2020.acl-main.495. · DOI
  89. Sundararajan, M., Taly, A., and Yan, Q. Axiomatic attri- bution for deep networks. In 34th International Con- ference on Machine Learning, ICML 2017, volume 7, pp. 5109-5118, 3 2017. ISBN 9781510855144. URL http://arxiv.org/abs/1703.01365. · arXiv
  90. Tenney, I., Das, D., and Pavlick, E. BERT Rediscovers the Classical NLP Pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4593-4601, Stroudsburg, PA, USA, 5 2019. Association for Computational Linguistics. doi: 10. 18653/v1/P19-1452. URL https://www.aclweb. org/anthology/P19-1452. · DOI · Paper page
  91. Thorne, J., Vlachos, A., Christodoulopoulos, C., and Mit- tal, A. Generating Token-Level Explanations for Nat- ural Language Inference. In Proceedings of the 2019 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short Pa- pers), volume 1, pp. 963-969, Stroudsburg, PA, USA, 2019. Association for Computational Linguistics. ISBN 9781950737130. doi: 10.18653/v1/N19-1101. URL http://aclweb.org/anthology/N19-1101. · DOI
  92. Turpin, M., Michael, J., Perez, E., and Bowman, S. R. Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. In Thirty-seventh Conference on Neural Information Processing Systems, pp. 1-32, 5 2023. URL http://arxiv.org/abs/2305.04388https: //openreview.net/forum?id=bzs4uPLXvi.
  93. Vafa, K., Deng, Y., Blei, D., and Rush, A. Rationales for Sequential Predictions. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10314-10332, Stroudsburg, PA, USA, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.807. URL https:// aclanthology.org/2021.emnlp-main.807. · DOI · Paper page
  94. Vashishth, S., Upadhyay, S., Tomar, G. S., and Faruqui, M. Attention Interpretability Across NLP Tasks. arXiv, 9 2019. URL http://arxiv.org/abs/1909. 11218.
  95. Voita, E. and Titov, I. Information-Theoretic Prob- ing with Minimum Description Length. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 183- 196, Stroudsburg, PA, USA, 3 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020. emnlp-main.14. URL https://www.aclweb.org/ anthology/2020.emnlp-main.14. Xiang, A. and Raji, I. D. On the Legal Compatibility of Fairness Definitions. Workshop on Human-Centric Machine Learning at the 33rd Conference on Neural Information Processing Systems, 2019. URL http: //arxiv.org/abs/1912.00761. · DOI · Paper page
  96. Yeh, C.-K., Hsieh, C.-Y., Suggala, A., Inouye, D. I., Raviku- mar, P. K., Sai Suggala, A., Inouye, D. I., and Raviku- mar, P. K. On the (In)fidelity and Sensitivity of Expla- nations. In Wallach, H., Larochelle, H., Beygelzimer, A., d\textquotesingle Alché-Buc, F., Fox, E., and Gar- nett, R. (eds.), Advances in Neural Information Pro- cessing Systems 32, pp. 10967-10978.
  97. Yoon, J., Jordon, J., and Van Der Schaar, M. Invase: Instance-wise variable selection using neural networks. 7th International Conference on Learning Representa- tions, ICLR 2019, pp. 1-24, 2019.
  98. Yosinski, J., Clune, J., Nguyen, A., Fuchs, T., and Lip- son, H. Understanding Neural Networks Through Deep Visualization. In Deep Learning Workshop at 31st Inter- national Conference on Machine Learning, 2015. URL http://arxiv.org/abs/1506.06579. · arXiv
  99. Zhang, K. and Bowman, S. Language Modeling Teaches You More than Translation Does: Lessons Learned Through Auxiliary Syntactic Task Analysis. In Proceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 359-361, Stroudsburg, PA, USA, 2018. Association for Computational Linguistics. doi: 10. 18653/v1/W18-5448. URL http://aclweb.org/ anthology/W18-5448. · DOI
  100. Zhou, Y. and Shah, J. The Solvability of Interpretability Evaluation Metrics. In Findings of the Association for Computational Linguistics: EACL, 2023. URL http: //arxiv.org/abs/2205.08696.
  101. Zimmermann, R. S., Borowski, J., Geirhos, R., Bethge, M., Wallis, T. S., and Brendel, W. How Well do Feature Visualizations Support Causal Understanding of CNN Activations? Advances in Neural Information Process- ing Systems, 14(NeurIPS):11730-11744, 2021. ISSN 10495258.

Cited by

Sorted by citations

Metadata

Authors
Madsen, A., Lakkaraju, H., Reddy, S., & Chandar, S.
Affiliations
  • Polytechnique Montréal (CA)
  • Harvard University (US)
  • Canadian Institute for Advanced Research (CA)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2405.05386
arXiv
2405.05386
OpenAlex
W4396821993
Cited by
22
References
3
Citations by year
2026: 7 · 2025: 9 · 2024: 5
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Funders
  • Canadian Institute for Advanced Research
  • Natural Sciences and Engineering Research Council of Canada
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
model interpretability, explainable AI, post-hoc explanation, black-box models, paradigm evolution

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review