The DOI matches its Crossref record; 75 of the 75 references it cites were found in a registry or an index.
Towards Faithful Model Explanation in NLP: A Survey
Abstract
End-to-end neural Natural Language Processing (NLP) models are notoriously difficult to understand. This has given rise to numerous efforts towards model explainability in recent years. One desideratum of model explanation is faithfulness, that is, an explanation should accurately represent the reasoning process behind the model’s prediction. In this survey, we review over 110 model explanation methods in NLP through the lens of faithfulness. We first discuss the definition and evaluation of faithfulness, as well as its significance for explainability. We then introduce recent advances in faithful explanation, grouping existing approaches into five categories: similarity-based methods, analysis of model-internal structures, backpropagation-based methods, counterfactual intervention, and self-explanatory models. For each category, we synthesize its representative studies, strengths, and weaknesses. Finally, we summarize their common virtues and remaining challenges, and reflect on future work directions towards faithful explainability in NLP.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Explainable AI literature has long emphasized how post hoc explanations may reflect alignment objectives or stylistic conventions rather than underlying mechanisms 48,49 .”
From Structured reasoning failures compromise LLM interpretation of clinical oncology notes (2026) · cites this paper as [49]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
ASPECTS OF THE THEORY OF SYNTAX
Chomsky, N. (1964) · doi:10.21236/ad0616323 · Cited by 13,587
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
17. A Value for n-Person Games
Shapley, L. S. (1953) · Contributions to the Theory of Games (AM-28), Volume II · doi:10.1515/9781400881970-018 · Cited by 6,950
Research works printed but not linked: 175
- Abraham, 2022, CEBaB: Estimating the causal effects of real-world concepts on NLP model behavior, Advances in Neural Information Processing Systems, 35, 17582
- Adebayo, 2018, Sanity checks for saliency maps, 9525
- Adebayo, 2020, Debugging tests for model explanations, Advances in Neural Information Processing Systems, 33, 700
- Adi, 2017, Fine-grained analysis of sentence embeddings using auxiliary prediction tasks
- Alvarez-Melis, 2018, Towards robust interpretability with self-explaining neural networks, 7786
- Amini, 2022, Naturalistic causal probing for morpho-syntax, Transactions of the Association for Computational Linguistics, 11, 384, 10.1162/tacl_a_00554 · DOI
- Andreas, 2016, Learning to compose neural networks for question answering, 1545, 10.18653/v1/N16-1181 · DOI · Paper page
- Andreas, 2016, Neural module networks, 39, 10.1109/CVPR.2016.12 · DOI · Paper page
- Arras, 2016, Explaining predictions of non-linear classifiers in NLP, 1, 10.18653/v1/W16-1601 · DOI
- Atanasova, 2020, A diagnostic study of explainability techniques for text classification, 3256, 10.18653/v1/2020.emnlp-main.263 · DOI · Paper page
- Bahdanau, 2015, Neural machine translation by jointly learning to align and translate
- Bastings, 2019, Interpretable neural predictions with differentiable binary variables, 2963, 10.18653/v1/P19-1284 · DOI
- Bastings, 2022, “Will you find these shortcuts?” A protocol for evaluating the faithfulness of input salience methods for text classification, 976, 10.18653/v1/2022.emnlp-main.64 · DOI
- Bastings, 2020, The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?, 149, 10.18653/v1/2020.blackboxnlp-1.14 · DOI
- Basu, 2021, Influence functions in deep learning are fragile
- Bau, 2019, Identifying and controlling important neurons in neural machine translation
- Belinkov, 2020, On the linguistic representational power of neural machine translation models, Computational Linguistics, 46, 1, 10.1162/coli_a_00367 · DOI
- Belinkov, 2019, Analysis methods in neural language processing: A survey, Transactions of the Association for Computational Linguistics, 7, 49, 10.1162/tacl_a_00254 · DOI · Paper page
- Bogin, 2021, Latent compositional representations improve systematic generalization in grounded question answering, Transactions of the Association for Computational Linguistics, 9, 195, 10.1162/tacl_a_00361 · DOI
- Bowman, 2015, A large annotated corpus for learning natural language inference, 632, 10.18653/v1/D15-1075 · DOI · Paper page
- Brunner, 2018, Natural language multitasking: Analyzing and improving syntactic saliency of hidden representations, arXiv preprint arXiv:1801.06024 · arXiv
- Calderon, 2022, DoCoGen: Domain counterfactual generation for low resource domain adaptation, 7727, 10.18653/v1/2022.acl-long.533 · DOI
- Camburu, 2020, Make up your mind! Adversarial generation of inconsistent natural language explanations, 4157, 10.18653/v1/2020.acl-main.382 · DOI
- Caruana, 1999, Case-based explanation of non-case-based learning methods., Proceedings of the AMIA Symposium, 212
- Cashman, 2018, RNNbow: Visualizing learning via backpropagation gradients in RNNs, IEEE Computer Graphics and Applications, 38, 39, 10.1109/MCG.2018.2878902 · DOI
- Chan, 2022, A comparative study of faithfulness metrics for model interpretability methods, 5029, 10.18653/v1/2022.acl-long.345 · DOI
- Chen, 2022, REV: Information-theoretic evaluation of free-text rationales, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2007, 10.18653/v1/2023.acl-long.112 · DOI
- Chen, 2018, Learning to explain: An information-theoretic perspective on model interpretation, 882
- Chen, 2022, Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks, Transactions on Machine Learning Research
- Choenni, 2021, Stepmothers are mean and academics are pretentious: What do pretrained language models learn about you?, 1477, 10.18653/v1/2021.emnlp-main.111 · DOI
- Clark, 2020, ELECTRA: Pre-training text encoders as discriminators rather than generators
- Clinciu, 2021, A study of automatic metrics for the evaluation of natural language explanations, 2376, 10.18653/v1/2021.eacl-main.202 · DOI
- Conneau, 2018, What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties, 2126, 10.18653/v1/P18-1198 · DOI · Paper page
- Creswell, 2022, Faithful reasoning using large language models, ArXiv preprint
- Dalvi, 2021, Explaining answers with entailment trees, 7358, 10.18653/v1/2021.emnlp-main.585 · DOI
- Dalvi, 2022, Discovering latent concepts learned in BERT
- Danilevsky, 2020, A survey of the state of explainable AI for natural language processing, 447
- De Cao, 2022, Sparse interventions in language models with differentiable masking, 16, 10.18653/v1/2022.blackboxnlp-1.2 · DOI
- Denil, 2015, Extraction of salient sentences from labelled documents, ArXiv preprint, arXiv:1412.6815 [cs] · arXiv
- Deutsch, 2019, A general-purpose algorithm for constrained sequential inference, 482, 10.18653/v1/K19-1045 · DOI
- Ding, 2021, Evaluating saliency methods for neural language models, 5034, 10.18653/v1/2021.naacl-main.399 · DOI
- Eberle, 2022, Do transformer models show similar attention patterns to task-specific human gaze?, 4295, 10.18653/v1/2022.acl-long.296 · DOI
- Ebrahimi, 2018, HotFlip: White-box adversarial examples for text classification, 31, 10.18653/v1/P18-2006 · DOI · Paper page
- Ethayarajh, 2021, Attention flows are Shapley Value explanations, 49, 10.18653/v1/2021.acl-short.8 · DOI
- Feder, 2021, CausaLM: Causal model explanation through counterfactual language models, Computational Linguistics, 47, 333, 10.1162/coli_a_00404 · DOI
- Feng, 2018, Pathologies of neural models make interpretations difficult, 3719, 10.18653/v1/D18-1407 · DOI · Paper page
- Finlayson, 2021, Causal analysis of syntactic agreement mechanisms in neural language models, 1828, 10.18653/v1/2021.acl-long.144 · DOI
- Gardner, 2020, Evaluating models’ local decision boundaries via contrast sets, 1307, 10.18653/v1/2020.findings-emnlp.117 · DOI · Paper page
- Gupta, 2020, Neural module networks for reasoning over text
- Haghighatkhah, 2022, Better hit the nail on the head than beat around the bush: Removing protected attributes with a single projection, 8395, 10.18653/v1/2022.emnlp-main.575 · DOI
- Halpern, 2005, Causes and explanations: A structural-model approach. Part I: Causes, The British Journal for the Philosophy of Science, 56, 843, 10.1093/bjps/axi147 · DOI · Paper page
- Hamilton, 2022, Is neuro-symbolic AI meeting its promises in natural language processing? A structured review, 1, 10.3233/SW-223228 · DOI
- Han, 2020, Explaining black box predictions and unveiling data artifacts through influence functions, 5553, 10.18653/v1/2020.acl-main.492 · DOI
- Hao, 2021, Self-attention attribution: Interpreting information interactions inside Transformer, 12963, 10.1609/aaai.v35i14.17533 · DOI · Paper page
- Harrington, 1985, Harvey Friedman’s Research on the Foundations of Mathematics
- Hase, 2022, When can models learn from explanations? A formal framework for understanding the roles of explanation data, 29, 10.18653/v1/2022.lnls-1.4 · DOI
- Hendrycks, 2021, Measuring massive multitask language understanding
- Herman, 2017, The promise and peril of human evaluation for model interpretability, ArXiv preprint
- Hiebert, 2018, Interpreting word-level hidden state behaviour of character-level LSTM language models, 258, 10.18653/v1/W18-5428 · DOI
- Hong, 2022, METGEN: A module-based entailment tree generation framework for answer explanation, 1887, 10.18653/v1/2022.findings-naacl.145 · DOI
- Hooker, 2019, A benchmark for interpretability methods in deep neural networks, 9734
- Hu, 2017, Learning to reason: End-to-end module networks for visual question answering, 804, 10.1109/ICCV.2017.93 · DOI · Paper page
- Jacovi, 2020, Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?, 4198, 10.18653/v1/2020.acl-main.386 · DOI · Paper page
- Jacovi, 2021, Aligning faithful interpretations with their social attribution, Transactions of the Association for Computational Linguistics, 9, 294, 10.1162/tacl_a_00367 · DOI
- Jain, 2019, Attention is not explanation, 3543
- Jain, 2020, Learning to faithfully rationalize by construction, 4459, 10.18653/v1/2020.acl-main.409 · DOI
- Janizek, 2021, Explaining explanations: Axiomatic feature interactions for deep networks, Journal of Machine Learning Research, 22, 104:1
- Jiang, 2020, Cold-start and interpretability: Turning regular expressions into trainable recurrent neural networks, 3193, 10.18653/v1/2020.emnlp-main.258 · DOI
- Jiang, 2019, Explore, propose, and assemble: An interpretable model for multi-hop reading comprehension, 2714, 10.18653/v1/P19-1261 · DOI
- Ju, 2022, Logic traps in evaluating attribution scores, 5911, 10.18653/v1/2022.acl-long.407 · DOI
- Kádár, 2017, Representation of linguistic form and function in recurrent neural networks, Computational Linguistics, 43, 761, 10.1162/COLI_a_00300 · DOI
- Karidi, 2021, Putting words in BERT’s mouth: Navigating contextualized vector spaces with pseudowords, 10300, 10.18653/v1/2021.emnlp-main.806 · DOI
- Kassner, 2021, BeliefBank: Adding memory to a pre-trained language model for a systematic notion of belief, 8849, 10.18653/v1/2021.emnlp-main.697 · DOI
- Kaushik, 2020, Learning the difference that makes a difference with counterfactually-augmented data
- Kaushik, 2018, How much reading does reading comprehension require? A critical investigation of popular benchmarks, 5010, 10.18653/v1/D18-1546 · DOI · Paper page
- Kindermans, 2019, The (un)reliability of saliency methods, 267, 10.1007/978-3-030-28954-6_14 · DOI · Paper page
- Kindermans, 2018, Learning how to explain neural networks: PatternNet and PatternAttribution
- Kokhlikyan, 2020, Captum: A unified and generic model interpretability library for PyTorch, ArXiv preprint
- Krishnamurthy, 2013, Jointly learning to parse and perceive: Connecting natural language to the physical world, Transactions of the Association for Computational Linguistics, 1, 193, 10.1162/tacl_a_00220 · DOI
- Kumar, 2022, Probing classifiers are unreliable for concept removal and detection, Advances in Neural Information Processing Systems, 35, 17994
- Kumar, 2020, NILE: Natural language inference with faithful natural language explanations, 8730, 10.18653/v1/2020.acl-main.771 · DOI
- Lakkaraju, 2020, “How do I fool you?”: Manipulating user trust via misleading black box explanations, 79, 10.1145/3375627.3375833 · DOI · Paper page
- Laugel, 2018, Defining locality for surrogates in post-hoc interpretablity
- Levesque, 2012, The Winograd Schema Challenge, 10
- Li, 2020, Evaluating explanation methods for neural machine translation, 365, 10.18653/v1/2020.acl-main.35 · DOI
- Li, 2016, Visualizing and understanding neural models in NLP, 681, 10.18653/v1/N16-1082 · DOI · Paper page
- Li, 2016, Understanding neural networks through representation erasure, ArXiv preprint
- Lipton, 2016, The Mythos of Model Interpretability, ArXiv preprint
- Liu, 2022, Rethinking attention-model explainability through faithfulness violation test, 13807
- Lovering, 2020, Information-theoretic probing explains reliance on spurious features
- Lu, 2021, Influence patterns for explaining information flow in BERT, 4461
- Madaan, 2021, Improving neural model performance through natural language feedback on their explanations, ArXiv preprint
- Mao, 2019, The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision
- Marasovic, 2022, Few-shot self-rationalization with natural language prompts, 410, 10.18653/v1/2022.findings-naacl.31 · DOI
- Martins, 2016, From softmax to sparsemax: A sparse model of attention and multi-label classification, 1614
- McCoy, 2019, Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference, 3428, 10.18653/v1/P19-1334 · DOI · Paper page
- Ming, 2017, Understanding hidden memories of recurrent neural networks, 13, 10.1109/VAST.2017.8585721 · DOI · Paper page
- Montavon, 2019, Layer-wise relevance propagation: An overview, 193, 10.1007/978-3-030-28954-6_10 · DOI · Paper page
- Mosca, 2022, SHAP-based explanation methods: A review for NLP interpretability, 4593
- Mueller, 2022, Causal analysis of syntactic agreement neurons in multilingual language models, 95, 10.18653/v1/2022.conll-1.8 · DOI
- Mullenbach, 2018, Explainable prediction of medical codes from clinical text, 1101, 10.18653/v1/N18-1100 · DOI · Paper page
- Mylonas, 2022, An attention matrix for every decision: Faithfulness-based arbitration among multiple attention-based interpretations of transformers in text classification, Data Mining and Knowledge Discovery, 38, 128, 10.1007/s10618-023-00962-4 · DOI
- Narang, 2020, WT5?! Training Text-to-Text Models to Explain their predictions, ArXiv preprint
- Nie, 2018, A theoretical explanation for perplexing behaviors of backpropagation-based visualizations, 3806
- Nye, 2021, Show your work: Scratchpads for intermediate computation with language models, Deep Learning for Code Workshop
- OpenAI. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774. · arXiv
- Parcalabescu, 2024, On measuring faithfulness or self-consistency of natural language explanations, arXiv
- Park, 2018, Multimodal explanations: Justifying decisions and pointing to the evidence, 8779, 10.1109/CVPR.2018.00915 · DOI · Paper page
- Pascual, 2021, Telling BERT’s full story: From local attention to global aggregation, 105, 10.18653/v1/2021.eacl-main.9 · DOI
- Petroni, 2019, Language models as knowledge bases?, 2463, 10.18653/v1/D19-1250 · DOI · Paper page
- Pezeshkpour, 2021, An empirical comparison of instance attribution methods for NLP, 967, 10.18653/v1/2021.naacl-main.75 · DOI
- Poerner, 2018, Interpretable textual neuron representations for NLP, 325, 10.18653/v1/W18-5437 · DOI
- Poerner, 2018, Evaluating neural network explanation methods using hybrid documents and morphosyntactic agreement, 340, 10.18653/v1/P18-1032 · DOI
- Poliak, 2018, Hypothesis only baselines in natural language inference, 180, 10.18653/v1/S18-2023 · DOI · Paper page
- Pruthi, 2022, Evaluating explanations: How much do explanations from the teacher aid students?, Transactions of the Association for Computational Linguistics, 10, 359, 10.1162/tacl_a_00465 · DOI
- Qian, 2022, Limitations of language models in arithmetic and symbolic induction, 10.18653/v1/2023.acl-long.516 · DOI
- Qian, 2016, Analyzing linguistic knowledge in sequential model of sentence, 826, 10.18653/v1/D16-1079 · DOI
- Raganato, 2020, Fixed encoder self-attention patterns in transformer-based machine translation, 556, 10.18653/v1/2020.findings-emnlp.49 · DOI
- Rajagopal, 2021, SELFEXPLAIN: A self-explaining architecture for neural text classifiers, 836, 10.18653/v1/2021.emnlp-main.64 · DOI
- Rajani, 2019, Explain yourself! Leveraging language models for commonsense reasoning, 4932, 10.18653/v1/P19-1487 · DOI · Paper page
- Ramamurthy, 2020, Model agnostic multilevel explanations, 5968
- Ravfogel, 2022, Log-linear guardedness and its implications, 9413, 10.18653/v1/2023.acl-long.523 · DOI
- Ravichander, 2021, Probing the probing paradigm: Does probing accuracy entail task relevance?, 3363, 10.18653/v1/2021.eacl-main.295 · DOI · Paper page
- Reif, 2019, Visualizing and measuring the geometry of BERT, 8592
- Ribeiro, 2016, “Why should I trust you?”: Explaining the predictions of any classifier, 1135, 10.1145/2939672.2939778 · DOI · Paper page
- Roese, 1995, Counterfactual thinking: A critical overview, 1
- Sajjad, 2022, Neuron-level interpretation of deep NLP models: A survey, Transactions of the Association for Computational Linguistics, 10, 1285, 10.1162/tacl_a_00519 · DOI
- Schwartz, 2018, Bridging CNNs, RNNs, and weighted finite-state machines, 295, 10.18653/v1/P18-1028 · DOI
- Shrikumar, 2017, Learning important features through propagating activation differences, 3145
- Shrikumar, 2017, Not just a black box: Learning important features through propagating activation differences, ArXiv preprint, arXiv:1605.01713 [cs] · arXiv
- Sia, 2022, Logical satisfiability of counterfactuals for faithful explanations in NLI, 9837, 10.1609/aaai.v37i8.26174 · DOI
- Slack, 2020, Fooling LIME and SHAP: Adversarial attacks on post hoc explanation methods, 180, 10.1145/3375627.3375830 · DOI · Paper page
- Smilkov, 2017, SmoothGrad: Removing noise by adding noise, ArXiv preprint
- Strobelt, 2019, Seq2seq-Vis: A visual debugging tool for sequence-to-sequence models, IEEE Transactions on Visualization and Computer Graphics, 25, 353, 10.1109/TVCG.2018.2865044 · DOI · Paper page
- Strobelt, 2018, LSTMVis: A tool for visual analysis of hidden state dynamics in recurrent neural networks, IEEE Transactions on Visualization and Computer Graphics, 24, 667, 10.1109/TVCG.2017.2744158 · DOI · Paper page
- Subramanian, 2020, Obtaining faithful interpretations from compositional neural networks, 5594, 10.18653/v1/2020.acl-main.495 · DOI
- Sundararajan, 2017, Axiomatic attribution for deep networks, 3319
- Sushil, 2018, Patient representation learning and interpretable evaluation using clinical notes, Journal of Biomedical Informatics, 84, 103, 10.1016/j.jbi.2018.06.016 · DOI
- Tsang, 2020, How does this interaction affect me? Interpretable attribution for feature interactions, 6147
- Tutek, 2020, Staying true to your word: (How) can attention become explanation?, 131, 10.18653/v1/2020.repl4nlp-1.17 · DOI
- Vashishth, 2019, Attention interpretability across NLP tasks, ArXiv preprint
- Vaswani, 2017, Attention is all you need, 5998
- Veldhoen, 2016, Diagnostic classifiers: Revealing how neural networks process hierarchical structure, CoCo@ NIPS, 69
- Vig, 2019, Visualizing attention in transformer-based language representation models, ArXiv preprint
- Vig, 2020, Investigating gender bias in language models using causal mediation analysis, 12388
- Voita, 2020, Information-theoretic probing with minimum description length, 183, 10.18653/v1/2020.emnlp-main.14 · DOI · Paper page
- Wallace, 2018, Interpreting neural networks with nearest neighbors, 136, 10.18653/v1/W18-5416 · DOI
- Wallace, 2019, Universal adversarial triggers for attacking and analyzing NLP, 2153, 10.18653/v1/D19-1221 · DOI · Paper page
- Wallace, 2020, Interpreting predictions of NLP models, 20, 10.18653/v1/2020.emnlp-tutorials.3 · DOI
- Wang, 2019, SuperGLUE: A stickier benchmark for general-purpose language understanding systems, 3261
- Wang, 2019, GLUE: A multi-task benchmark and analysis platform for natural language understanding, 10.18653/v1/W18-5446 · DOI · Paper page
- Wang, 2020, Gradient-based analysis of NLP models is manipulable, 247, 10.18653/v1/2020.findings-emnlp.24 · DOI
- Wang, 2022, A fine-grained interpretability evaluation benchmark for neural NLP, 70, 10.18653/v1/2022.conll-1.6 · DOI
- Wang, 2022, Self-consistency improves chain of thought reasoning in language models
- Wexler, 2020, The What-If Tool: Interactive probing of machine learning models, IEEE Transactions on Visualization and Computer Graphics, 26, 56, 10.1109/TVCG.2019.2934619 · DOI · Paper page
- Wiegreffe, 2022, Reframing human-AI collaboration for generating free-text explanations, 632, 10.18653/v1/2022.naacl-main.47 · DOI
- Wiegreffe, 2019, Attention is not not explanation, 11, 10.18653/v1/D19-1002 · DOI · Paper page
- Wu, 2021, Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models, 6707, 10.18653/v1/2021.acl-long.523 · DOI · Paper page
- Xie, 2017, An interpretable knowledge transfer model for knowledge base completion, 950, 10.18653/v1/P17-1088 · DOI
- Yang, 2019, Benchmarking attribution methods with relative feature importance, arXiv preprint arXiv:1907.09701 · arXiv
- Ye, 2023, Explanation Selection Using Unlabeled Data for In-Context Learning, 10.18653/v1/2023.emnlp-main.41 · DOI
- Ye, 2022, Complementary explanations for effective in-context learning, 4469, 10.18653/v1/2023.findings-acl.273 · DOI
- Ye, 2021, Connecting attributions and QA model behavior on realistic counterfactuals, 5496, 10.18653/v1/2021.emnlp-main.447 · DOI
- Yeh, 2019, On the (in)fidelity and sensitivity of explanations, 10965
- Yeh, 2020, On completeness-aware concept-based explanations in deep neural networks, 20554
- Yi, 2018, Neural-symbolic VQA: Disentangling reasoning from vision and language understanding, 1039
- Yin, 2022, On the sensitivity and stability of model interpretations in NLP, 2631, 10.18653/v1/2022.acl-long.188 · DOI
- Yin, 2022, Interpreting language models with contrastive explanations, 184, 10.18653/v1/2022.emnlp-main.14 · DOI
- Zaidan, 2007, Using “Annotator Rationales” to Improve Machine Learning for Text Categorization, 260
- Zheng, 2022, The irrationality of neural rationale models, 64, 10.18653/v1/2022.trustnlp-1.6 · DOI
- Zhou, 2022, Least-to-most prompting enables complex reasoning in large language models
- Zhou, 2022, Do feature attribution methods correctly attribute features?, 9623, 10.1609/aaai.v36i9.21196 · DOI
- Zhou, 2022, ExSum: From local explanations to model understanding, 5359, 10.18653/v1/2022.naacl-main.392 · DOI
- Zhou, 2023, The Solvability of Interpretability Evaluation Metrics, 2399, 10.18653/v1/2023.findings-eacl.182 · DOI
- Zmigrod, 2019, Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology, 1651, 10.18653/v1/P19-1161 · DOI · Paper page
Cited by
LLMs Will Always Hallucinate, and We Need to Live with This
Banerjee, S., Agarwal, A., & Singla, S. (2025) · Lecture Notes in Networks and Systems · doi:10.1007/978-3-031-99965-9_39 · Cited by 93
Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2305.04388 · Cited by 262
“Evaluating Faithfulness of CoT Explanations Evaluating the faithfulness of explanations has a long history (Jacovi and Goldberg, 2020; Lyu et al., 2022).”
BackgroundQureshi, M. A., Noor, A. A., Manzoor, A., Mazhar Qureshi, M. D., Younus, A., & Rashwan, W. (2026) · PLOS One · doi:10.1371/journal.pone.0351473 · Cited by 7
Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations
Chen, Y., Zhong, R., Ri, N., Zhao, C., He, H., Steinhardt, J., Yu, Z., & McKeown, K. (2023) · arXiv · doi:10.48550/arxiv.2307.08678 · Cited by 5
Lext: Towards evaluating trustworthiness of natural language explanations
Shailya, K., Rajpal, S., Krishnan, G. S., & Ravindran, B. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732104 · Cited by 5
Metadata
- Authors
- Lyu, Q., Apidianaki, M., & Callison-Burch, C.
- Affiliations
- University of Pennsylvania, Department of Computer and Information Science. lyuqing@sas.upenn.edu
- University of Pennsylvania, Department of Computer and Information Science. marapi@seas.upenn.edu
- University of Pennsylvania, Department of Computer and Information Science. ccb@seas.upenn.edu
- Venue
- Computational Linguistics
- Publisher
- MIT Press
- Published
- 2024
- Volume
- 50
- Issue
- 2
- Pages
- 657-723
- ISSN
- 0891-2017
- 1530-9312
- Cited by
- 120
- References
- 75
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Towards Faithful Model Explanation in NLP: A Survey. https://app.cochat.ai/papers/10.1162/coli_a_00511
- The paper
- Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards Faithful Model Explanation in NLP: A Survey. Computational Linguistics. https://doi.org/10.1162/coli_a_00511
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.