Towards Faithful Model Explanation in NLP: A Survey
The DOI matches its Crossref record; 102 of the 102 references it cites were found in a registry or an index.
Abstract
End-to-end neural Natural Language Processing (NLP) models are notoriously difficult to understand. This has given rise to numerous efforts towards model explainability in recent years. One desideratum of model explanation is faithfulness, that is, an explanation should accurately represent the reasoning process behind the model’s prediction. In this survey, we review over 110 model explanation methods in NLP through the lens of faithfulness. We first discuss the definition and evaluation of faithfulness, as well as its significance for explainability. We then introduce recent advances in faithful explanation, grouping existing approaches into five categories: similarity-based methods, analysis of model-internal structures, backpropagation-based methods, counterfactual intervention, and self-explanatory models. For each category, we synthesize its representative studies, strengths, and weaknesses. Finally, we summarize their common virtues and remaining challenges, and reflect on future work directions towards faithful explainability in NLP.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Explainable AI literature has long emphasized how post hoc explanations may reflect alignment objectives or stylistic conventions rather than underlying mechanisms 48,49 .”
From Structured reasoning failures compromise LLM interpretation of clinical oncology notes (2026) · cites this paper as [49]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
ASPECTS OF THE THEORY OF SYNTAX
Chomsky, N. (1964) · doi:10.21236/ad0616323 · Cited by 13,587
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
17. A Value for n-Person Games
Shapley, L. S. (1953) · Contributions to the Theory of Games (AM-28), Volume II · doi:10.1515/9781400881970-018 · Cited by 6,950
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524 · cited 4×
“This type of evaluation has been widely adopted in both vision (Bach et al. 2015; Shrikumar, Greenside, and Kundaje 2017; Chen et al. 2018) and language (Arras et al. 2016; Chen et al. 2018; Serrano and Smith 2019; Jain and Wallace 2019; DeYoung et al. 2020; Atanasova et al. 2020).”
MethodInfluentialChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
Striving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 2×
“When experimenting with this model on the SNLI dataset (Bowman et al. 2015), Camburu et al. (2018) discover a trade-off between the task accuracy and the Plausibility of the explanation.”
MethodTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
Learning Important Features Through Propagating Activation Differences
Shrikumar, A., Greenside, P., & Kundaje, A. (2017) · arXiv · doi:10.48550/arxiv.1704.02685 · Cited by 2,761
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Definitions, methods, and applications in interpretable machine learning
Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., & Yu, B. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1900654116 · Cited by 2,167 · cited 4×
“the extent to which the internal mechanics of a model can be presented in understandable terms to a human (Lipton 2017; Murdoch et al. 2019; Barredo Arrieta et al. 2020).”
BackgroundInfluentialAnchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
Large Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · cited 2×
“(2019) apply NMN to HotpotQA (Yang et al. 2018), a QA dataset involving reasoning across multiple documents.”
MethodCLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781
Understanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514
SmoothGrad: removing noise by adding noise
Smilkov, D., Thorat, N., Kim, B., Viégas, F., & Wattenberg, M. (2017) · arXiv · doi:10.48550/arxiv.1706.03825 · Cited by 1,467 · cited 4×
“Figure 8: A visualization of different gradient methods on qualitative image classification examples (figure adapted from (Smilkov et al. 2017)).”
BackgroundMethodInfluentialExplaining nonlinear classification decisions with deep taylor decomposition
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K.-R. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.11.008 · Cited by 1,357
Sanity Checks for Saliency Maps
Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018) · arXiv · doi:10.48550/arxiv.1810.03292 · Cited by 1,294
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., & Sayres, R. (2017) · arXiv · doi:10.48550/arxiv.1711.11279 · Cited by 1,240
Evaluating the Visualization of What a Deep Neural Network Has Learned
Samek, W., Binder, A., Montavon, G., Lapuschkin, S., & Muller, K.-R. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2599820 · Cited by 1,164
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
Voita, E., Talbot, D., Moiseev, F., Sennrich, R., & Titov, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1580 · Cited by 1,146 · cited 3×
“Similar to previous findings in single sequence tasks, attention heads in MT also capture semantic and syntactic dependencies (Voita et al. 2018; Raganato and Tiedemann 2018; Voita et al. 2019).”
BackgroundMethodResultSuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051
Quantifying Attention Flow in Transformers
Abnar, S., & Zuidema, W. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.385 · Cited by 995
Attention is not not Explanation
Wiegreffe, S., & Pinter, Y. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1002 · Cited by 966
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
McCoy, R. T., Pavlick, E., & Linzen, T. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1334 · Cited by 944
Transformer Interpretability Beyond Attention Visualization
Chefer, H., Gur, S., & Wolf, L. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00084 · Cited by 903
Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods
Slack, D., Hilgard, S., Jia, E., Singh, S., & Lakkaraju, H. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375830 · Cited by 900
THE ESTIMATION OF CAUSAL EFFECTS FROM OBSERVATIONAL DATA
Winship, C., & Morgan, S. L. (1999) · Annual Review of Sociology · doi:10.1146/annurev.soc.25.1.659 · Cited by 890
Visualizing and Understanding Recurrent Networks
Karpathy, A., Johnson, J., & Fei-Fei, L. (2015) · arXiv · doi:10.48550/arxiv.1506.02078 · Cited by 845
Ethayarajh, K. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1006 · Cited by 841
HellaSwag: Can a Machine Really Finish Your Sentence?
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1472 · Cited by 830
Rationalizing Neural Predictions
Lei, T., Barzilay, R., & Jaakkola, T. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1011 · Cited by 777
Interpretation of Neural Networks Is Fragile
Ghorbani, A., Abid, A., & Zou, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013681 · Cited by 765
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757 · cited 1×
“users may still trust it simply because the explanation looks convincing (Bansal et al. 2021).”
BackgroundDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Serrano, S., & Smith, N. A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1282 · Cited by 690 · cited 1×
“This type of evaluation has been widely adopted in both vision (Bach et al. 2015; Shrikumar, Greenside, and Kundaje 2017; Chen et al. 2018) and language (Arras et al. 2016; Chen et al. 2018; Serrano and Smith 2019; Jain and Wallace 2019; DeYoung et al. 2020; Atanasova et al. 2020).”
MethodExplainable Prediction of Medical Codes from Clinical Text
Mullenbach, J., Wiegreffe, S., Duke, J., Sun, J., & Eisenstein, J. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1100 · Cited by 597
WinoGrande: An Adversarial Winograd Schema Challenge at Scale
Sakaguchi, K., Le Bras, R., Bhagavatula, C., & Choi, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6399 · Cited by 579
Training Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575
ERASER: A Benchmark to Evaluate Rationalized NLP Models
DeYoung, J., Jain, S., Rajani, N. F., Lehman, E., Xiong, C., Socher, R., & Wallace, B. C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.408 · Cited by 561
Generating Visual Explanations
Hendricks, L. A., Akata, Z., Rohrbach, M., Donahue, J., Schiele, B., & Darrell, T. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_1 · Cited by 534
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness?
Jacovi, A., & Goldberg, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.386 · Cited by 503
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018) · arXiv · doi:10.48550/arxiv.1803.05457 · Cited by 462
Mao, J., Gan, C., Kohli, P., Tenenbaum, J. B., & Wu, J. (2019) · arXiv · doi:10.48550/arxiv.1904.12584 · Cited by 444
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., & Gardner, M. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1246 · Cited by 393 · cited 1×
“Their model is capable of answering questions involving discrete reasoning in the DROP dataset (Dua et al. 2019), for example, Who kicked the longest field goal in the second quarter?, given a long description of the match.”
BackgroundDesigning and Interpreting Probes with Control Tasks
Hewitt, J., & Liang, P. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1275 · Cited by 393
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.10625 · Cited by 380
e-SNLI: Natural language inference with natural language explanations
Camburu, O.-M., Rocktäschel, T., Lukasiewicz, T., & Blunsom, P. (2018) · arXiv · doi:10.48550/arxiv.1812.01193 · Cited by 367 · cited 3×
“2016), language (Camburu et al. 2018), multimodal tasks (Park et al.”
BackgroundMethodExplaining Recurrent Neural Network Predictions in Sentiment Analysis
Arras, L., Montavon, G., Müller, K.-R., & Samek, W. (2017) · Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis · doi:10.18653/v1/w17-5221 · Cited by 361 · cited 1×
“In NLP, LRP has been applied/extended to CNNs and RNNs on sentence classification tasks, including topic and sentiment classification (Arras et al. 2016, 2017), to explain which tokens are most important to the prediction.”
MethodSolving Quantitative Reasoning Problems with Language Models
Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.14858 · Cited by 358 · cited 1×
“2022b), Minerva (Lewkowycz et al. 2022), and DIVERSE (Li et al.”
Background(2022) · doi:10.52202/068431-0278 · Cited by 358
Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems
Ling, W., Yogatama, D., Dyer, C., & Blunsom, P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1015 · Cited by 325
Brains and algorithms partially converge in natural language processing
Caucheteux, C., & King, J.-R. (2022) · Communications Biology · doi:10.1038/s42003-022-03036-1 · Cited by 318
On the Robustness of Interpretability Methods
Alvarez-Melis, D., & Jaakkola, T. S. (2018) · arXiv · doi:10.48550/arxiv.1806.08049 · Cited by 310
Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection
Ravfogel, S., Elazar, Y., Gonen, H., Twiton, M., & Goldberg, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.647 · Cited by 287
Geva, M., Goldberg, Y., & Berant, J. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1107 · Cited by 284
Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?
Hase, P., & Bansal, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.491 · Cited by 254
Information-Theoretic Probing with Minimum Description Length
Voita, E., & Titov, I. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.14 · Cited by 251
Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond
Feder, A., Keith, K. A., Manzoor, E., Pryzant, R., Sridhar, D., Wood-Doughty, Z., Eisenstein, J., Grimmer, J., Reichart, R., Roberts, M. E., et al. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00511 · Cited by 220
Learning to Deceive with Attention-Based Explanations
Pruthi, D., Gupta, M., Dhingra, B., Neubig, G., & Lipton, Z. C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.432 · Cited by 203
Faithful Chain-of-Thought Reasoning
Lyu, Q., Havaldar, S., Stein, A., Zhang, L., Rao, D., Wong, E., Apidianaki, M., & Callison-Burch, C. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.20 · Cited by 201
Self-citationLet Me Explain: Impact of Personal and Impersonal Explanations on Trust in Recommender Systems
Kunkel, J., Donkers, T., Michael, L., Barbu, C.-M., & Ziegler, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300717 · Cited by 156
PAL: Program-aided Language Models
Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., & Neubig, G. (2022) · arXiv · doi:10.48550/arxiv.2211.10435 · Cited by 155
Can language models learn from explanations in context?
Lampinen, A., Dasgupta, I., Chan, S., Mathewson, K., Tessler, M., Creswell, A., McClelland, J., Wang, J., & Hill, F. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.38 · Cited by 142
Tenney, I., Wexler, J., Bastings, J., Bolukbasi, T., Coenen, A., Gehrmann, S., Jiang, E., Pushkarna, M., Radebaugh, C., Reif, E., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.15 · Cited by 138 · cited 4×
“The following tools implement some type(s) of counterfactual intervention: Captum31, LIT32 (Tenney et al. 2020), LIME33 (Ribeiro, Singh, and Guestrin 2016), SHAP34 (Lundberg and Lee 2017), Anchors35 (Ribeiro, Singh, and Guestrin 2018), Seq2Seq-Vis36 (Strobelt et al. 2019), and the What-if Tool37…”
MethodInfluentialProbing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?
Ravichander, A., Belinkov, Y., & Hovy, E. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.295 · Cited by 132
Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals
Elazar, Y., Ravfogel, S., Jacovi, A., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00359 · Cited by 126 · cited 6×
“Two pieces of concurrent work, Amnesic Probing (Elazar et al. 2021) and CausalLM (Feder et al. 2021), are representative examples.”
BackgroundMethodInfluentialChen, W., Ma, X., Wang, X., & Cohen, W. W. (2022) · arXiv · doi:10.48550/arxiv.2211.12588 · Cited by 125
ProofWriter: Generating implications, proofs, and abductive statements over natural language
Tafjord, O., Dalvi, B., & Clark, P. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.317 · Cited by 122
ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language
Tafjord, O., Mishra, B. D., & Clark, P. (2020) · arXiv · doi:10.48550/arxiv.2012.13048 · Cited by 122
Measuring Association Between Labels and Free-Text Rationales
Wiegreffe, S., Marasović, A., & Smith, N. A. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.804 · Cited by 119
AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models
Wallace, E., Tuyls, J., Wang, J., Subramanian, S., Gardner, M., & Singh, S. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations · doi:10.18653/v1/d19-3002 · Cited by 116 · cited 5×
“…et al. 2020; Schuster, Chen, and Degen 2020); • World and Commonsense knowledge: Factual knowledge (Petroni et al. 2019; Kassner and Schütze 2020), Numeracy (Wallace et al. 2019c), Time (Zhou et al. 2019; Thukral, Kukreja, and Kavouras 2021), Space (Mirzaee et al. 2021; Masis and Anderson 2021),…”
BackgroundMethodInfluentialMeasuring Association Between Labels and Free-Text Rationales
Wiegreffe, S., Marasović, A., & Smith, N. A. (2020) · arXiv · doi:10.48550/arxiv.2010.12762 · Cited by 106
Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations
Jung, J., Qin, L., Welleck, S., Brahman, F., Bhagavatula, C., Bras, R. L., & Choi, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11822 · Cited by 98
Debugging Tests for Model Explanations
Adebayo, J., Muelly, M., Liccardi, I., & Kim, B. (2020) · arXiv · doi:10.48550/arxiv.2011.05429 · Cited by 88
Contrastive Explanations for Model Interpretability
Jacovi, A., Swayamdipta, S., Ravfogel, S., Elazar, Y., Choi, Y., & Goldberg, Y. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.120 · Cited by 83 · cited 2×
“3 Prior work has different definitions of “concepts”, including but not limited to phrases (Rajagopal et al. 2021) and high-level features (Jacovi et al. 2021).”
BackgroundMethodHow do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking
De Cao, N., Schlichtkrull, M. S., Aziz, W., & Titov, I. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.262 · Cited by 75
Making Language Models Better Reasoners with Step-Aware Verifier
Li, Y., Lin, Z., Zhang, S., Fu, Q., Chen, B., Lou, J.-G., & Chen, W. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.291 · Cited by 69
Hase, P., Zhang, S., Xie, H., & Bansal, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.390 · Cited by 62
How to Explain Individual Classification Decisions
(2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1859912 · Cited by 54 · cited 2×
“The most straightforward idea is to take the gradient itself (referred to as Simple Gradients or Vanilla Gradients), ∂M(x)∂xi , as the feature relevance (Baehrens et al. 2010; Simonyan, Vedaldi, and Zisserman 2014).”
Background(2021) · doi:10.18653/v1/2021.conll-1.15 · Cited by 51
On Measuring Faithfulness or Self-consistency of Natural Language Explanations
Parcalabescu, L., & Frank, A. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.329 · Cited by 38
Faithful Chain-of-Thought Reasoning
Lyu, Q., Havaldar, S., Stein, A., Zhang, L., Rao, D., Wong, E., Apidianaki, M., & Callison-Burch, C. (2023) · arXiv · doi:10.48550/arxiv.2301.13379 · Cited by 23
Self-citationWhat if This Modified That? Syntactic Interventions with Counterfactual Embeddings
Tucker, M., Qian, P., & Levy, R. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.76 · Cited by 16
(2022) · doi:10.18653/v1/2022.conll-1.8 · Cited by 5
Research works printed but not linked: 152
- Abraham, 2022, CEBaB: Estimating the causal effects of real-world concepts on NLP model behavior, Advances in Neural Information Processing Systems, 35, 17582
- Adi, 2017, Fine-grained analysis of sentence embeddings using auxiliary prediction tasks
- Alvarez-Melis, 2018, Towards robust interpretability with self-explaining neural networks, 7786
- Amini, 2022, Naturalistic causal probing for morpho-syntax, Transactions of the Association for Computational Linguistics, 11, 384, 10.1162/tacl_a_00554 · DOI
- Andreas, 2016, Learning to compose neural networks for question answering, 1545, 10.18653/v1/N16-1181 · DOI · Paper page
- Andreas, 2016, Neural module networks, 39, 10.1109/CVPR.2016.12 · DOI · Paper page
- Arras, 2016, Explaining predictions of non-linear classifiers in NLP, 1, 10.18653/v1/W16-1601 · DOI · Paper page
- Atanasova, 2020, A diagnostic study of explainability techniques for text classification, 3256, 10.18653/v1/2020.emnlp-main.263 · DOI · Paper page
- Bastings, 2019, Interpretable neural predictions with differentiable binary variables, 2963, 10.18653/v1/P19-1284 · DOI
- Bastings, 2022, “Will you find these shortcuts?” A protocol for evaluating the faithfulness of input salience methods for text classification, 976, 10.18653/v1/2022.emnlp-main.64 · DOI
- Bastings, 2020, The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?, 149, 10.18653/v1/2020.blackboxnlp-1.14 · DOI · Paper page
- Basu, 2021, Influence functions in deep learning are fragile
- Bau, 2019, Identifying and controlling important neurons in neural machine translation
- Belinkov, 2020, On the linguistic representational power of neural machine translation models, Computational Linguistics, 46, 1, 10.1162/coli_a_00367 · DOI · Paper page
- Belinkov, 2019, Analysis methods in neural language processing: A survey, Transactions of the Association for Computational Linguistics, 7, 49, 10.1162/tacl_a_00254 · DOI · Paper page
- Bogin, 2021, Latent compositional representations improve systematic generalization in grounded question answering, Transactions of the Association for Computational Linguistics, 9, 195, 10.1162/tacl_a_00361 · DOI
- Brunner, 2018, Natural language multitasking: Analyzing and improving syntactic saliency of hidden representations, arXiv preprint arXiv:1801.06024 · arXiv
- Calderon, 2022, DoCoGen: Domain counterfactual generation for low resource domain adaptation, 7727, 10.18653/v1/2022.acl-long.533 · DOI
- Camburu, 2020, Make up your mind! Adversarial generation of inconsistent natural language explanations, 4157, 10.18653/v1/2020.acl-main.382 · DOI
- Caruana, 1999, Case-based explanation of non-case-based learning methods., Proceedings of the AMIA Symposium, 212
- Cashman, 2018, RNNbow: Visualizing learning via backpropagation gradients in RNNs, IEEE Computer Graphics and Applications, 38, 39, 10.1109/MCG.2018.2878902 · DOI
- Chan, 2022, A comparative study of faithfulness metrics for model interpretability methods, 5029, 10.18653/v1/2022.acl-long.345 · DOI · Paper page
- Chen, 2022, REV: Information-theoretic evaluation of free-text rationales, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2007, 10.18653/v1/2023.acl-long.112 · DOI
- Chen, 2018, Learning to explain: An information-theoretic perspective on model interpretation, 882
- Choenni, 2021, Stepmothers are mean and academics are pretentious: What do pretrained language models learn about you?, 1477, 10.18653/v1/2021.emnlp-main.111 · DOI
- Clark, 2020, ELECTRA: Pre-training text encoders as discriminators rather than generators
- Clinciu, 2021, A study of automatic metrics for the evaluation of natural language explanations, 2376, 10.18653/v1/2021.eacl-main.202 · DOI
- Conneau, 2018, What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties, 2126, 10.18653/v1/P18-1198 · DOI · Paper page
- Creswell, 2022, Faithful reasoning using large language models, ArXiv preprint
- Dalvi, 2021, Explaining answers with entailment trees, 7358, 10.18653/v1/2021.emnlp-main.585 · DOI · Paper page
- Dalvi, 2022, Discovering latent concepts learned in BERT
- Danilevsky, 2020, A survey of the state of explainable AI for natural language processing, 447
- De Cao, 2022, Sparse interventions in language models with differentiable masking, 16, 10.18653/v1/2022.blackboxnlp-1.2 · DOI
- Denil, 2015, Extraction of salient sentences from labelled documents, ArXiv preprint, arXiv:1412.6815 [cs] · arXiv
- Deutsch, 2019, A general-purpose algorithm for constrained sequential inference, 482, 10.18653/v1/K19-1045 · DOI
- Ding, 2021, Evaluating saliency methods for neural language models, 5034, 10.18653/v1/2021.naacl-main.399 · DOI
- Eberle, 2022, Do transformer models show similar attention patterns to task-specific human gaze?, 4295, 10.18653/v1/2022.acl-long.296 · DOI
- Ebrahimi, 2018, HotFlip: White-box adversarial examples for text classification, 31, 10.18653/v1/P18-2006 · DOI · Paper page
- Ethayarajh, 2021, Attention flows are Shapley Value explanations, 49, 10.18653/v1/2021.acl-short.8 · DOI
- Feder, 2021, CausaLM: Causal model explanation through counterfactual language models, Computational Linguistics, 47, 333, 10.1162/coli_a_00404 · DOI · Paper page
- Feng, 2018, Pathologies of neural models make interpretations difficult, 3719, 10.18653/v1/D18-1407 · DOI · Paper page
- Finlayson, 2021, Causal analysis of syntactic agreement mechanisms in neural language models, 1828, 10.18653/v1/2021.acl-long.144 · DOI · Paper page
- Gardner, 2020, Evaluating models’ local decision boundaries via contrast sets, 1307, 10.18653/v1/2020.findings-emnlp.117 · DOI · Paper page
- Gupta, 2020, Neural module networks for reasoning over text
- Haghighatkhah, 2022, Better hit the nail on the head than beat around the bush: Removing protected attributes with a single projection, 8395, 10.18653/v1/2022.emnlp-main.575 · DOI
- Halpern, 2005, Causes and explanations: A structural-model approach. Part I: Causes, The British Journal for the Philosophy of Science, 56, 843, 10.1093/bjps/axi147 · DOI · Paper page
- Hamilton, 2022, Is neuro-symbolic AI meeting its promises in natural language processing? A structured review, 1, 10.3233/SW-223228 · DOI · Paper page
- Han, 2020, Explaining black box predictions and unveiling data artifacts through influence functions, 5553, 10.18653/v1/2020.acl-main.492 · DOI · Paper page
- Hao, 2021, Self-attention attribution: Interpreting information interactions inside Transformer, 12963, 10.1609/aaai.v35i14.17533 · DOI · Paper page
- Harrington, 1985, Harvey Friedman’s Research on the Foundations of Mathematics
- Hase, 2022, When can models learn from explanations? A formal framework for understanding the roles of explanation data, 29, 10.18653/v1/2022.lnls-1.4 · DOI · Paper page
- Hendrycks, 2021, Measuring massive multitask language understanding
- Herman, 2017, The promise and peril of human evaluation for model interpretability, ArXiv preprint
- Hiebert, 2018, Interpreting word-level hidden state behaviour of character-level LSTM language models, 258, 10.18653/v1/W18-5428 · DOI
- Hong, 2022, METGEN: A module-based entailment tree generation framework for answer explanation, 1887, 10.18653/v1/2022.findings-naacl.145 · DOI
- Hooker, 2019, A benchmark for interpretability methods in deep neural networks, 9734
- Hu, 2017, Learning to reason: End-to-end module networks for visual question answering, 804, 10.1109/ICCV.2017.93 · DOI · Paper page
- Jacovi, 2021, Aligning faithful interpretations with their social attribution, Transactions of the Association for Computational Linguistics, 9, 294, 10.1162/tacl_a_00367 · DOI · Paper page
- Jain, 2019, Attention is not explanation, 3543
- Jain, 2020, Learning to faithfully rationalize by construction, 4459, 10.18653/v1/2020.acl-main.409 · DOI · Paper page
- Janizek, 2021, Explaining explanations: Axiomatic feature interactions for deep networks, Journal of Machine Learning Research, 22, 104:1
- Jiang, 2020, Cold-start and interpretability: Turning regular expressions into trainable recurrent neural networks, 3193, 10.18653/v1/2020.emnlp-main.258 · DOI
- Jiang, 2019, Explore, propose, and assemble: An interpretable model for multi-hop reading comprehension, 2714, 10.18653/v1/P19-1261 · DOI · Paper page
- Ju, 2022, Logic traps in evaluating attribution scores, 5911, 10.18653/v1/2022.acl-long.407 · DOI
- Kádár, 2017, Representation of linguistic form and function in recurrent neural networks, Computational Linguistics, 43, 761, 10.1162/COLI_a_00300 · DOI · Paper page
- Karidi, 2021, Putting words in BERT’s mouth: Navigating contextualized vector spaces with pseudowords, 10300, 10.18653/v1/2021.emnlp-main.806 · DOI
- Kassner, 2021, BeliefBank: Adding memory to a pre-trained language model for a systematic notion of belief, 8849, 10.18653/v1/2021.emnlp-main.697 · DOI
- Kaushik, 2020, Learning the difference that makes a difference with counterfactually-augmented data
- Kaushik, 2018, How much reading does reading comprehension require? A critical investigation of popular benchmarks, 5010, 10.18653/v1/D18-1546 · DOI · Paper page
- Kindermans, 2019, The (un)reliability of saliency methods, 267, 10.1007/978-3-030-28954-6_14 · DOI · Paper page
- Kindermans, 2018, Learning how to explain neural networks: PatternNet and PatternAttribution
- Kokhlikyan, 2020, Captum: A unified and generic model interpretability library for PyTorch, ArXiv preprint
- Krishnamurthy, 2013, Jointly learning to parse and perceive: Connecting natural language to the physical world, Transactions of the Association for Computational Linguistics, 1, 193, 10.1162/tacl_a_00220 · DOI · Paper page
- Kumar, 2022, Probing classifiers are unreliable for concept removal and detection, Advances in Neural Information Processing Systems, 35, 17994
- Kumar, 2020, NILE: Natural language inference with faithful natural language explanations, 8730, 10.18653/v1/2020.acl-main.771 · DOI · Paper page
- Lakkaraju, 2020, “How do I fool you?”: Manipulating user trust via misleading black box explanations, 79, 10.1145/3375627.3375833 · DOI · Paper page
- Laugel, 2018, Defining locality for surrogates in post-hoc interpretablity
- Levesque, 2012, The Winograd Schema Challenge, 10
- Li, 2020, Evaluating explanation methods for neural machine translation, 365, 10.18653/v1/2020.acl-main.35 · DOI
- Li, 2016, Visualizing and understanding neural models in NLP, 681, 10.18653/v1/N16-1082 · DOI · Paper page
- Li, 2016, Understanding neural networks through representation erasure, ArXiv preprint
- Lipton, 2016, The Mythos of Model Interpretability, ArXiv preprint
- Liu, 2022, Rethinking attention-model explainability through faithfulness violation test, 13807
- Lovering, 2020, Information-theoretic probing explains reliance on spurious features
- Lu, 2021, Influence patterns for explaining information flow in BERT, 4461
- Madaan, 2021, Improving neural model performance through natural language feedback on their explanations, ArXiv preprint
- Marasovic, 2022, Few-shot self-rationalization with natural language prompts, 410, 10.18653/v1/2022.findings-naacl.31 · DOI · Paper page
- Martins, 2016, From softmax to sparsemax: A sparse model of attention and multi-label classification, 1614
- Ming, 2017, Understanding hidden memories of recurrent neural networks, 13, 10.1109/VAST.2017.8585721 · DOI · Paper page
- Montavon, 2019, Layer-wise relevance propagation: An overview, 193, 10.1007/978-3-030-28954-6_10 · DOI · Paper page
- Mosca, 2022, SHAP-based explanation methods: A review for NLP interpretability, 4593
- Mylonas, 2022, An attention matrix for every decision: Faithfulness-based arbitration among multiple attention-based interpretations of transformers in text classification, Data Mining and Knowledge Discovery, 38, 128, 10.1007/s10618-023-00962-4 · DOI
- Narang, 2020, WT5?! Training Text-to-Text Models to Explain their predictions, ArXiv preprint
- Nie, 2018, A theoretical explanation for perplexing behaviors of backpropagation-based visualizations, 3806
- Nye, 2021, Show your work: Scratchpads for intermediate computation with language models, Deep Learning for Code Workshop
- OpenAI. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774. · arXiv
- Park, 2018, Multimodal explanations: Justifying decisions and pointing to the evidence, 8779, 10.1109/CVPR.2018.00915 · DOI · Paper page
- Pascual, 2021, Telling BERT’s full story: From local attention to global aggregation, 105, 10.18653/v1/2021.eacl-main.9 · DOI
- Pezeshkpour, 2021, An empirical comparison of instance attribution methods for NLP, 967, 10.18653/v1/2021.naacl-main.75 · DOI
- Poerner, 2018, Interpretable textual neuron representations for NLP, 325, 10.18653/v1/W18-5437 · DOI
- Poerner, 2018, Evaluating neural network explanation methods using hybrid documents and morphosyntactic agreement, 340, 10.18653/v1/P18-1032 · DOI · Paper page
- Poliak, 2018, Hypothesis only baselines in natural language inference, 180, 10.18653/v1/S18-2023 · DOI · Paper page
- Pruthi, 2022, Evaluating explanations: How much do explanations from the teacher aid students?, Transactions of the Association for Computational Linguistics, 10, 359, 10.1162/tacl_a_00465 · DOI · Paper page
- Qian, 2022, Limitations of language models in arithmetic and symbolic induction, 10.18653/v1/2023.acl-long.516 · DOI
- Qian, 2016, Analyzing linguistic knowledge in sequential model of sentence, 826, 10.18653/v1/D16-1079 · DOI · Paper page
- Raganato, 2020, Fixed encoder self-attention patterns in transformer-based machine translation, 556, 10.18653/v1/2020.findings-emnlp.49 · DOI · Paper page
- Rajagopal, 2021, SELFEXPLAIN: A self-explaining architecture for neural text classifiers, 836, 10.18653/v1/2021.emnlp-main.64 · DOI · Paper page
- Rajani, 2019, Explain yourself! Leveraging language models for commonsense reasoning, 4932, 10.18653/v1/P19-1487 · DOI · Paper page
- Ramamurthy, 2020, Model agnostic multilevel explanations, 5968
- Ravfogel, 2022, Log-linear guardedness and its implications, 9413, 10.18653/v1/2023.acl-long.523 · DOI
- Reif, 2019, Visualizing and measuring the geometry of BERT, 8592
- Roese, 1995, Counterfactual thinking: A critical overview, 1
- Sajjad, 2022, Neuron-level interpretation of deep NLP models: A survey, Transactions of the Association for Computational Linguistics, 10, 1285, 10.1162/tacl_a_00519 · DOI
- Schwartz, 2018, Bridging CNNs, RNNs, and weighted finite-state machines, 295, 10.18653/v1/P18-1028 · DOI
- Shrikumar, 2017, Not just a black box: Learning important features through propagating activation differences, ArXiv preprint, arXiv:1605.01713 [cs] · arXiv
- Sia, 2022, Logical satisfiability of counterfactuals for faithful explanations in NLI, 9837, 10.1609/aaai.v37i8.26174 · DOI
- Strobelt, 2019, Seq2seq-Vis: A visual debugging tool for sequence-to-sequence models, IEEE Transactions on Visualization and Computer Graphics, 25, 353, 10.1109/TVCG.2018.2865044 · DOI · Paper page
- Strobelt, 2018, LSTMVis: A tool for visual analysis of hidden state dynamics in recurrent neural networks, IEEE Transactions on Visualization and Computer Graphics, 24, 667, 10.1109/TVCG.2017.2744158 · DOI · Paper page
- Subramanian, 2020, Obtaining faithful interpretations from compositional neural networks, 5594, 10.18653/v1/2020.acl-main.495 · DOI · Paper page
- Sundararajan, 2017, Axiomatic attribution for deep networks, 3319
- Sushil, 2018, Patient representation learning and interpretable evaluation using clinical notes, Journal of Biomedical Informatics, 84, 103, 10.1016/j.jbi.2018.06.016 · DOI · Paper page
- Tsang, 2020, How does this interaction affect me? Interpretable attribution for feature interactions, 6147
- Tutek, 2020, Staying true to your word: (How) can attention become explanation?, 131, 10.18653/v1/2020.repl4nlp-1.17 · DOI
- Vashishth, 2019, Attention interpretability across NLP tasks, ArXiv preprint
- Vaswani, 2017, Attention is all you need, 5998
- Veldhoen, 2016, Diagnostic classifiers: Revealing how neural networks process hierarchical structure, CoCo@ NIPS, 69
- Vig, 2019, Visualizing attention in transformer-based language representation models, ArXiv preprint
- Vig, 2020, Investigating gender bias in language models using causal mediation analysis, 12388
- Wallace, 2018, Interpreting neural networks with nearest neighbors, 136, 10.18653/v1/W18-5416 · DOI · Paper page
- Wallace, 2019, Universal adversarial triggers for attacking and analyzing NLP, 2153, 10.18653/v1/D19-1221 · DOI · Paper page
- Wallace, 2020, Interpreting predictions of NLP models, 20, 10.18653/v1/2020.emnlp-tutorials.3 · DOI
- Wang, 2020, Gradient-based analysis of NLP models is manipulable, 247, 10.18653/v1/2020.findings-emnlp.24 · DOI · Paper page
- Wang, 2022, A fine-grained interpretability evaluation benchmark for neural NLP, 70, 10.18653/v1/2022.conll-1.6 · DOI
- Wexler, 2020, The What-If Tool: Interactive probing of machine learning models, IEEE Transactions on Visualization and Computer Graphics, 26, 56, 10.1109/TVCG.2019.2934619 · DOI · Paper page
- Wiegreffe, 2022, Reframing human-AI collaboration for generating free-text explanations, 632, 10.18653/v1/2022.naacl-main.47 · DOI · Paper page
- Wu, 2021, Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models, 6707, 10.18653/v1/2021.acl-long.523 · DOI · Paper page
- Xie, 2017, An interpretable knowledge transfer model for knowledge base completion, 950, 10.18653/v1/P17-1088 · DOI · Paper page
- Yang, 2019, Benchmarking attribution methods with relative feature importance, arXiv preprint arXiv:1907.09701 · arXiv
- Ye, 2023, Explanation Selection Using Unlabeled Data for In-Context Learning, 10.18653/v1/2023.emnlp-main.41 · DOI
- Ye, 2022, Complementary explanations for effective in-context learning, 4469, 10.18653/v1/2023.findings-acl.273 · DOI · Paper page
- Ye, 2021, Connecting attributions and QA model behavior on realistic counterfactuals, 5496, 10.18653/v1/2021.emnlp-main.447 · DOI
- Yeh, 2019, On the (in)fidelity and sensitivity of explanations, 10965
- Yeh, 2020, On completeness-aware concept-based explanations in deep neural networks, 20554
- Yi, 2018, Neural-symbolic VQA: Disentangling reasoning from vision and language understanding, 1039
- Yin, 2022, On the sensitivity and stability of model interpretations in NLP, 2631, 10.18653/v1/2022.acl-long.188 · DOI
- Yin, 2022, Interpreting language models with contrastive explanations, 184, 10.18653/v1/2022.emnlp-main.14 · DOI · Paper page
- Zaidan, 2007, Using “Annotator Rationales” to Improve Machine Learning for Text Categorization, 260
- Zheng, 2022, The irrationality of neural rationale models, 64, 10.18653/v1/2022.trustnlp-1.6 · DOI
- Zhou, 2022, Do feature attribution methods correctly attribute features?, 9623, 10.1609/aaai.v36i9.21196 · DOI · Paper page
- Zhou, 2022, ExSum: From local explanations to model understanding, 5359, 10.18653/v1/2022.naacl-main.392 · DOI
- Zhou, 2023, The Solvability of Interpretability Evaluation Metrics, 2399, 10.18653/v1/2023.findings-eacl.182 · DOI
- Zmigrod, 2019, Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology, 1651, 10.18653/v1/P19-1161 · DOI · Paper page
Cited by
LLMs Will Always Hallucinate, and We Need to Live with This
Banerjee, S., Agarwal, A., & Singla, S. (2025) · Lecture Notes in Networks and Systems · doi:10.1007/978-3-031-99965-9_39 · Cited by 93
Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2305.04388 · Cited by 262
“Evaluating Faithfulness of CoT Explanations Evaluating the faithfulness of explanations has a long history (Jacovi and Goldberg, 2020; Lyu et al., 2022).”
BackgroundOn Measuring Faithfulness or Self-consistency of Natural Language Explanations
Parcalabescu, L., & Frank, A. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.329 · Cited by 38
“Also not subject to this study about faithfulness of NLEs , is work that aims to increase the faithfulness of post-hoc inter-pretability methods (see Lyu et al., 2022a for overview). deceiving inner workings of “sleeper agents” (Hub-inger et al., 2024) hiding under surface-level self-consistency.”
InfluentialQureshi, M. A., Noor, A. A., Manzoor, A., Mazhar Qureshi, M. D., Younus, A., & Rashwan, W. (2026) · PLOS One · doi:10.1371/journal.pone.0351473 · Cited by 7
Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations
Chen, Y., Zhong, R., Ri, N., Zhao, C., He, H., Steinhardt, J., Yu, Z., & McKeown, K. (2023) · arXiv · doi:10.48550/arxiv.2307.08678 · Cited by 5
Metadata
- Authors
- Lyu, Q., Apidianaki, M., & Callison-Burch, C.
- Affiliations
- University of Pennsylvania, Department of Computer and Information Science. lyuqing@sas.upenn.edu
- University of Pennsylvania, Department of Computer and Information Science. marapi@seas.upenn.edu
- University of Pennsylvania, Department of Computer and Information Science. ccb@seas.upenn.edu
- Venue
- Computational Linguistics
- Publisher
- MIT Press
- Published
- 2024
- Volume
- 50
- Issue
- 2
- Pages
- 657-723
- ISSN
- 0891-2017
- 1530-9312
- Cited by
- 120
- References
- 102
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.