On Measuring Faithfulness or Self-consistency of Natural Language Explanations
Open accessAbstract
Large language models (LLMs) can explain their predictions through post-hoc or Chainof-Thought (CoT) explanations.But an LLM could make up reasonably sounding explanations that are unfaithful to its underlying reasoning.Recent work has designed tests that aim to judge the faithfulness of post-hoc or CoT explanations.In this work we argue that these faithfulness tests do not measure faithfulness to the models' inner workings -but rather their self-consistency at output level.Our contributions are three-fold: i) We clarify the status of faithfulness tests in view of model explainability, characterising them as self-consistency tests instead.This assessment we underline by ii) constructing a Comparative Consistency Bank for self-consistency tests that for the first time compares existing tests on a common suite of 11 open LLMs and 5 tasksincluding iii) our new self-consistency measure CC-SHAP.CC-SHAP is a fine-grained measure (not a test) of LLM self-consistency.It compares how a model's input contributes to the predicted answer and to generating the explanation.Our fine-grained CC-SHAP metric allows us iii) to compare LLM behaviour when making predictions and to analyse the effect of other consistency tests at a deeper level, which takes us one step further towards measuring faithfulness by bringing us closer to the internals of the model than strictly surface outputoriented tests.Our code is available at https: //github.com/Heidelberg-NLP/CC-SHAP Method Example InstanceTest Instance for Unfaithful Model Unfaithfulness Case Autom.Eval.Annot.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Self-explanations are a prevailing approach for understanding LLM-generated responses and are widely used in analyzing language model behavior.
“Self-explanations are a prevailing approach on par with traditional model explanations like LIME saliency maps [71] for understanding LLM-generated responses [25, 32, 53] and are widely used in analyzing language model behavior [66, 99].”
From Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (2025) · cites this paper as [66]
The Introduction describes LLM-generated explanations as a way to understand the reasons behind an answer, and §2.2 discusses research testing those explanations. This supports their use in analyzing model behavior, but our copy does not establish that self-explanations are a prevailing approach or are on par with LIME saliency maps. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 45
- Chirag Agarwal, Sree Harsha Tanneru, and Himabindu Lakkaraju. 2024. Faithfulness vs. plausibility: On the (un) reliability of explanations from large language models. arXiv preprint arXiv:2402.04614. · arXiv
- Pepa Atanasova, Oana-Maria Camburu, Christina Li- oma, Thomas Lukasiewicz, Jakob Grue Simonsen, and Isabelle Augenstein. 2023. Faithfulness tests for natural language explanations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 283-294, Toronto, Canada. Association for Computational Linguistics. · DOI
- Pepa Atanasova, Jakob Grue Simonsen, Christina Li- oma, and Isabelle Augenstein. 2022. Fact checking with insufficient evidence. Transactions of the Asso- ciation for Computational Linguistics, 10:746-763. · DOI
- Yonatan Belinkov, Adam Poliak, Stuart Shieber, Ben- jamin Van Durme, and Alexander Rush. 2019. Don't take the premise for granted: Mitigating artifacts in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics, pages 877-891, Florence, Italy. Association for Computational Linguistics. · DOI · Paper page
- Steven Bills, Nick Cammarata, Dan Moss- ing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. 2023. Language mod- els can explain neurons in language models. https://openaipublic.blob.core.windows. net/neuron-explainer/paper/index.html.
- Alexander Binder, Grégoire Montavon, Sebastian La- puschkin, Klaus-Robert Müller, and Wojciech Samek. 2016. Layer-wise relevance propagation for neural networks with local renormalization layers. In Inter- national Conference on Artificial Neural Networks, pages 63-71. Springer.
- Marc Braun and Jenny Kunz. 2024. A hypothesis-driven framework for the analysis of self-rationalising mod- els. arXiv preprint arXiv:2402.04787. · arXiv
- Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-snli: Natu- ral language inference with natural language expla- nations. Advances in Neural Information Processing Systems, 31.
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. · arXiv
- Yu-Neng Chuang, Guanchu Wang, Chia-Yuan Chang, Ruixiang Tang, Fan Yang, Mengnan Du, Xuant- ing Cai, and Xia Hu. 2024. Large language models as faithful explainers. arXiv preprint arXiv:2402.04678. · arXiv
- Antonia Creswell, Murray Shanahan, and Irina Higgins. 2022. Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09712. · arXiv
- Noa Garcia, Yusuke Hirota, Yankun Wu, and Yuta Nakashima. 2023. Uncurated image-text datasets: Shedding light on demographic bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6957-6966.
- Yair Gat, Nitay Calderon, Amir Feder, Alexander Chapanin, Amit Sharma, and Roi Reichart. 2023. Faithful explanations of black-box nlp models us- ing llm-generated counterfactuals. arXiv preprint arXiv:2310.00603. · arXiv
- Leo A Harrington, Michael D Morley, A Šcedrov, and Stephen G Simpson. 1985. Harvey Friedman's re- search on the foundations of mathematics. Elsevier.
- Shiyuan Huang, Siddarth Mamidanna, Shreedhar Jangam, Yilun Zhou, and Leilani H Gilpin. 2023. Can large language models explain themselves? a study of llm-generated self-explanations. arXiv preprint arXiv:2310.11207. · arXiv
- Evan Hubinger, Carson Denison, Jesse Mu, Mike Lam- bert, Meg Tong, Monte MacDiarmid, Tamera Lan- ham, Daniel M Ziegler, Tim Maxwell, Newton Cheng, et al. 2024. Sleeper agents: Training decep- tive llms that persist through safety training. arXiv preprint arXiv:2401.05566. · arXiv
- Alon Jacovi and Yoav Goldberg. 2020. Towards faith- fully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198-4205, On- line. Association for Computational Linguistics. · DOI · Paper page
- Sarthak Jain and Byron C. Wallace. 2019. Attention is not Explanation. In Proceedings of the 2019 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short Pa- pers), pages 3543-3556, Minneapolis, Minnesota. Association for Computational Linguistics. · DOI
- Albert Q Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. · arXiv
- Nicholas Kroeger, Dan Ley, Satyapriya Krishna, Chirag Agarwal, and Himabindu Lakkaraju. 2023. Are large language models post hoc explainers? arXiv preprint arXiv:2310.05797. · arXiv
- Jenny Kunz and Marco Kuhlmann. 2024. Properties and challenges of llm-generated explanations. arXiv preprint arXiv:2402.10532. · arXiv
- Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernan- dez, Dustin Li, Esin Durmus, Evan Hubinger, Jack- son Kernion, et al. 2023. Measuring faithful- ness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702. · arXiv
- Scott M Lundberg and Su-In Lee. 2017. A unified ap- proach to interpreting model predictions. Advances in neural information processing systems, 30.
- Qing Lyu, Marianna Apidianaki, and Chris Callison- Burch. 2022a. Towards faithful model explanation in nlp: A survey. arXiv preprint arXiv:2209.11326. · arXiv
- Qing Lyu, Marianna Apidianaki, and Chris Callison- Burch. 2022b. Towards faithful model explanation in nlp: A survey. arXiv preprint arXiv:2209.11326. · arXiv
- Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. Faithful chain-of- thought reasoning. arXiv preprint arXiv:2301.13379. · arXiv
- Andreas Madsen, Sarath Chandar, and Siva Reddy. 2024. Can large language models explain them- selves? arXiv preprint arXiv:2401.07927. · arXiv
- Katie Matton, Robert Ness, and Emre Kiciman. 2024. Walk the talk? measuring the faithfulness of large lan- guage model explanations. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models.
- Nicole Meister, Dora Zhao, Angelina Wang, Vikram V Ramaswamy, Ruth Fong, and Olga Russakovsky. 2022. Gender artifacts in visual datasets. arXiv preprint arXiv:2206.09191. · arXiv
- Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048-11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. · DOI · Paper page
- Christoph Molnar. 2022. Interpretable Machine Learn- ing, 2 edition. Lulu. com.
- Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. 2020. Wt5?! training text-to-text models to explain their predictions. arXiv preprint arXiv:2004.14546. · arXiv
- Letitia Parcalabescu and Anette Frank. 2024. Do vision & language decoders use images and text equally? how self-consistent are their explanations? arXiv preprint arXiv:2404.18624. for faithful explanations in nli. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 37, pages 9837-9845. · arXiv
- Noah Y Siegel, Oana-Maria Camburu, Nicolas Heess, and Maria Perez-Ortiz. 2024. The probabilities also matter: A more faithful metric for faithfulness of free- text explanations in large language models. arXiv preprint arXiv:2404.03189. · arXiv
- Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Pro- ceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Ma- chine Learning Research, pages 3319-3328. PMLR.
- Mirac Suzgun, Nathan Scales, Nathanael Schärli, Se- bastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261. · arXiv
- Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085. · arXiv
- Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. · arXiv
- Miles Turpin, Julian Michael, Ethan Perez, and Samuel R Bowman. 2023. Language models don't always say what they think: Unfaithful explana- tions in chain-of-thought prompting. arXiv preprint arXiv:2305.04388. · arXiv
- Cunxiang Wang, Shuailong Liang, Yili Jin, Yilong Wang, Xiaodan Zhu, and Yue Zhang. 2020. SemEval- 2020 task 4: Commonsense validation and explana- tion. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 307-321, Barcelona (online). International Committee for Computational Linguistics. · DOI
- Albert Webson and Ellie Pavlick. 2022. Do prompt- based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2300-2344, Seattle, United States. Association for Computational Linguistics. · DOI · Paper page
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits rea- soning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837.
- Sarah Wiegreffe, Ana Marasović, and Noah A. Smith. 2021. Measuring association between labels and free-text rationales. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 10266-10284, Online and Punta Cana, Dominican Republic. Association for Compu- tational Linguistics. · DOI · Paper page
- Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not not explanation. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 11-20, Hong Kong, China. Association for Computational Linguistics. · DOI · Paper page
- Mo Yu, Shiyu Chang, Yang Zhang, and Tommi Jaakkola. 2019. Rethinking cooperative rationalization: In- trospective extraction and complement control. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 4094- 4103, Hong Kong, China. Association for Computa- tional Linguistics. · DOI
Cited by
Sorted by citations
Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces
Yang, J., Yang, S., Gupta, A. W., Han, R., Fei-Fei, L., & Xie, S. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.00994 · Cited by 120 · see citing works
“Self-explanations are a prevailing approach on par with traditional model explanations like LIME saliency maps [71] for understanding LLM-generated responses [25, 32, 53] and are widely used in analyzing language model behavior [66, 99].”
Pahud de Mortanges, A., Luo, H., Shu, S. Z., Kamath, A., Suter, Y., Shelan, M., Pöllinger, A., & Reyes, M. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01190-w · Cited by 93 · see citing works
Siegel, N. Y., Camburu, O.-M., Heess, N., & Perez-Ortiz, M. (2024) · arXiv · doi:10.48550/arxiv.2404.03189 · Cited by 5
“Parcalabescu and Frank (2023) argue that “a test that is able to interrogate a model’s inner workings would be akin to a lie detector that uses more internal cues that cannot be easily suppressed”.”
Background
- Authors
- Parcalabescu, L., & Frank, A.
- Affiliations
- Heidelberg University (DE)
- ORCID
- Frank, Anette
- Venue
- Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2024
- Pages
- 6048-6089
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4402671681
- Cited by
- 38
- References
- 0
- Citations by year
- 2026: 21 · 2025: 12 · 2024: 5
- FWCI
- 9.16
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Deutsche Forschungsgemeinschaft (35/1597‐1, INST 35/1597-1, INST35/1597‐1FUGG, 35/1597-1 FUGG)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- self-consistency, large language models, post-hoc explanation, explainability, self-consistency test, SHAP
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
