Faithful Chain-of-Thought Reasoning
Open accessAbstract
While Chain-of-Thought (CoT) prompting boosts Language Models' (LM) performance on a gamut of complex reasoning tasks, the generated reasoning chain does not necessarily reflect how the model arrives at the answer (aka. faithfulness). We propose Faithful CoT, a reasoning framework involving two stages: Translation (Natural Language query $\rightarrow$ symbolic reasoning chain) and Problem Solving (reasoning chain $\rightarrow$ answer), using an LM and a deterministic solver respectively. This guarantees that the reasoning chain provides a faithful explanation of the final answer. Aside from interpretability, Faithful CoT also improves empirical performance: it outperforms standard CoT on 9 of 10 benchmarks from 4 diverse domains, with a relative accuracy gain of 6.3% on Math Word Problems (MWP), 3.4% on Planning, 5.5% on Multi-hop Question Answering (QA), and 21.4% on Relational Inference. Furthermore, with GPT-4 and Codex, it sets the new state-of-the-art few-shot performance on 7 datasets (with 95.0+ accuracy on 6 of them), showing a strong synergy between faithfulness and accuracy.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The grounding bottleneck has historically limited symbolic reasoning pipelines to relatively narrow and well-formalized domains.
“More broadly, this bottleneck has historically limited symbolic reasoning pipelines to relatively narrow and well-formalized domains Olausson et al. [2023], Lyu et al. [2023].”
From Reason Popper-ly: Patching In-Context Reasoning with Inductive Logic Programming (2026) · cites this paper as Lyu et al. [2023]
The paper’s Related Work discussion (page unavailable) says two concurrent symbolic-language approaches tackled only math and simple symbolic reasoning tasks; Appendix B.3 repeats that narrower scope. It does not establish that a grounding bottleneck historically limited symbolic reasoning pipelines generally to well-formalized domains. (LLM, conf=0.82)
Experimental — accuracy not yet measuredLLMs occasionally make unfaithful reasoning, i.e., the derived conclusion does not follow the previously generated reasoning chain.
“Recent studies (Golovneva et al., 2023;Ribeiro et al., 2023b;Lyu et al., 2023) found that LLMs occasionally make unfaithful reasoning, i.e., the derived conclusion does not follow the previously generated reasoning chain.”
From Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning (2023) · cites this paper as Lyu et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Solving General Arithmetic Word Problems
Roy, S., & Roth, D. (2016) · arXiv · doi:10.48550/arxiv.1608.01413 · Cited by 275 · cited 4×
“We follow Wei et al. (2022) and consider the same five MWP benchmarks: GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), MultiArith (Roy and Roth, 2015), ASDiv (Miao et al., 2020), and AQuA (Ling et al., 2017).”
MethodInfluential
Flagged first, then by citations
Research works printed but not linked: 29
- Michael Ahn, Anthony Brohan, Noah Brown, Yev- gen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jes- month, Nikhil J. Joshi, Ryan Julian, Dmitry Kalash- nikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Pe- ter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nico- las Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. 2022. Do As I Can, Not As I Say: Grounding Language in Robotic Affordances. ArXiv:2204.01691 [cs]. · DOI · Paper page
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901.
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas- try, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cum- mings, Matthias Plappert, Fotios Chantzis, Eliza- beth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welin- der, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Eval- uating Large Language Models Trained on Code. ArXiv:2107.03374 · arXiv
- Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2022. Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks. ArXiv:2211.12588 [cs. · DOI · Paper page
- Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Prob- lems. ArXiv:2110.14168 [cs. · DOI · Paper page
- Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Gra- ham Neubig. 2022. PAL: Program-aided Language Models. ArXiv:2211.10435 [cs. · DOI · Paper page
- Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Transactions of the Association for Computational Linguistics, 9:346- 361. · DOI · Paper page
- Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Ba- jwa, Michael Specter, and Lalana Kagal. 2018. Ex- plaining Explanations: An Overview of Interpretabil- ity of Machine Learning. In 2018 IEEE 5th Inter- national Conference on Data Science and Advanced Analytics (DSAA), pages 80-89. · DOI · Paper page
- L. A. Harrington, M. D. Morley, A. Šcedrov, and S. G. Simpson. 1985. Harvey Friedman's Research on the Foundations of Mathematics. Elsevier. Google- Books-ID: 2plPRR4LDxIC.
- Bernease Herman. 2019. The Promise and Peril of Human Evaluation for Model Interpretability. arXiv:1711.07414 [cs, stat]. ArXiv: 1711.07414. · arXiv
- Alon Jacovi and Yoav Goldberg. 2020. Towards faith- fully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198-4205. Asso- ciation for Computational Linguistics. · DOI · Paper page
- Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brah- man, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. 2022. Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations. ArXiv:2205.11822 [cs. · arXiv
- Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. Solving Quanti- tative Reasoning Problems with Language Models. ArXiv:2206.14858 [cs]. · arXiv
- Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2022. On the Ad- vance of Making Language Models Better Reasoners. ArXiv:2206.02336 [cs. · DOI
- Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain alge- braic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 158-167. Association for Computational Linguistics. ArXiv:1705.04146 [cs. · DOI · Paper page
- Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2020. A diverse corpus for evaluating and developing English math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975-984, Online. Association for Computational Linguistics. · DOI · Paper page
- Maxwell Nye, Anders Johan Andreassen, Guy Gur- Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show Your Work: Scratchpads for Intermediate Computation with Language Models. ArXiv:2112.00114 [cs]. OpenAI. 2023. Gpt-4 technical report. · DOI · Paper page
- Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080-2094, Online. Association for Computational Linguistics. · DOI · Paper page
- Silviu Pitis, Michael R Zhang, Andrew Wang, and Jimmy Ba. 2023. Boosted prompt ensem- bles for large language models. arXiv preprint arXiv:2304.05970. · arXiv
- Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, and Zachary C. Lipton. 2020. Learning to deceive with attention-based explanations. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4782- 4793, Online. Association for Computational Lin- guistics. · DOI
- Jing Qian, Hong Wang, Zekun Li, Shiyang Li, and Xifeng Yan. 2022. Limitations of Language Models in Arithmetic and Symbolic Induction. ArXiv:2208.05051 · arXiv
- Marco Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "why should I trust you?": Explaining the pre- dictions of any classifier. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demon- strations, KDD '16, pages 97-101, New York, NY, USA. Association for Computational Linguistics. · DOI · Paper page
- Gabriele Sarti, Nils Feldhus, Ludwig Sickert, and Os- kar van der Wal. 2023. Inseq: An interpretabil- ity toolkit for sequence generation models. arXiv preprint arXiv:2302.13942. · arXiv
- Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. 2019. CLUTRR: A diagnostic benchmark for inductive reasoning from text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 4506-4515, Hong Kong, China. Association for Com- putational Linguistics. · DOI · Paper page
- Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. 2020. Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods. In Proceedings of the AAAI/ACM Con- ference on AI, Ethics, and Society, pages 180-186. Association for Computing Machinery, New York, NY, USA. · DOI · Paper page
- Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-Consistency Improves Chain of Thought Reasoning in Language Models. ArXiv:2203.11171 [cs. · DOI · Paper page
- Kayo Yin and Graham Neubig. 2022. Interpreting lan- guage models with contrastive explanations. arXiv preprint arXiv:2202.10419. · arXiv
- Hanlin Zhang, Ziyang Li, Jiani Huang, Mayur Naik, and Eric Xing. 2022. Improved logical reasoning of language models via differentiable symbolic pro- gramming. In First Workshop on Pre-training: Per- spectives, Pitfalls, and Paths Forward at ICML 2022.
- Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. 2022. Least- to-Most Prompting Enables Complex Reasoning in Large Language Models. ArXiv:2205.10625 [cs]. · arXiv
Cited by
Sorted by citations
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972 · see citing works
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516 · see citing works
“…tasks through Chain-of-Thought prompting (Wei et al., 2022), they occasionally exhibit unfaithful reasoning characterized by inconsistencies within the reasoning steps or conclusions that do not logically follow the reasoning chain (Golovneva et al., 2022; Ribeiro et al., 2023; Lyu et al., 2023).”
Reasoning with Language Model is Planning with World Model
Hao, S., Gu, Y., Ma, H., Hong, J., Wang, Z., Wang, D., & Hu, Z. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.507 · Cited by 168 · see citing works
Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
Pan, L., Albalak, A., Wang, X., & Wang, W. Y. (2023) · arXiv · doi:10.48550/arxiv.2305.12295 · Cited by 144 · see citing works
“Recent studies (Golovneva et al., 2023; Ribeiro et al., 2023b; Lyu et al., 2023) found that LLMs occasionally make unfaithful reasoning, i.e. , the derived conclusion does not follow the previously generated reasoning chain.”
BackgroundKim, S. S. Y., Vaughan, J. W., Liao, Q. V., Lombrozo, T., & Russakovsky, O. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714020 · Cited by 58 · see citing works
Token-Budget-Aware LLM Reasoning
Han, T., Wang, Z., Fang, C., Zhao, S., Ma, S., & Chen, Z. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.1274 · Cited by 31 · see citing works
Calibrating Large Language Models with Sample Consistency
Lyu, Q., Shridhar, K., Malaviya, C., Zhang, L., Elazar, Y., Tandon, N., Apidianaki, M., Sachan, M., & Callison-Burch, C. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i18.34120 · Cited by 21 · see citing works
Self-citationWang, K., Duan, F., Wang, S., Li, P., Xian, Y., Yin, C., Rong, W., & Xiong, Z. (2023) · arXiv · doi:10.48550/arxiv.2308.13259 · Cited by 16 · see citing works
Leveraging Structured Information for Explainable Multi-hop Question Answering and Reasoning
Li, R., & Du, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.452 · Cited by 12 · see citing works
Reasoning Models Don't Always Say What They Think
Chen, Y., Benton, J., Radhakrishnan, A., Uesato, J., Denison, C., Schulman, J., Somani, A., Hase, P., Wagner, M., Roger, F., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.05410 · Cited by 18 · see citing works
“To improve faithfulness, researchers have developed test-time methods using question decomposition (Radhakrishnan et al., 2023), symbolic reasoning (Lyu et al., 2023), and causal mediation analysis on counterfactuals (Paul et al., 2024 (Clymer et al., 2024).”
Token-Budget-Aware LLM Reasoning
Han, T., Wang, Z., Fang, C., Zhao, S., Ma, S., & Chen, Z. (2024) · arXiv · Cited by 31 · see citing works
“Among them, Chain-of-Thought (CoT) prompting has emerged as a key technique for improving LLM reasoning by breaking problems into intermediate steps, enabling better performance on multiple tasks (Wei et al., 2022; Lyu et al., 2023; Li et al., 2023; Feng et al., 2024).”
On Measuring Faithfulness or Self-consistency of Natural Language Explanations
Parcalabescu, L., & Frank, A. (2023) · arXiv · doi:10.48550/arxiv.2311.07466 · Cited by 27 · see citing works
“One line of work – i.a., Sanchez et al., 2023; Creswell et al., 2022; Radhakrishnan et al., 2023; Lyu et al., 2023; Gat et al., 2023 – aims to increase the faithfulness of LLMs by changing the way in which the model generates its final prediction, e.g., using a Python interpreter (Lyu et al., 2023).”
Reason Popper-ly: Patching In-Context Reasoning with Inductive Logic Programming
Chen, Z., & Ma, M. (2026) · arXiv · doi:10.48550/arxiv.2607.23019
“…either translate natural-language problems into formal representations and delegate reasoning to external solvers Pan et al. [2023], Olausson et al. [2023], or inject symbolic structure into the reasoning process to improve faithfulness and controllability Xu et al. [2024], Lyu et al. [2023].”
Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Models
Sel, B., Al-Tawaha, A., Khattar, V., Jia, R., & Jin, M. (2023) · arXiv · doi:10.48550/arxiv.2308.10379 · Cited by 15 · see citing works
SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning
Miao, N., Teh, Y. W., & Rainforth, T. (2023) · arXiv · doi:10.48550/arxiv.2308.00436 · Cited by 18 · see citing works
“We omit results from Faithful-CoT (Lyu et al., 2023), because it has already been shown to decrease the accuracies on GSM8K and MATH by 11.”
Method
- Authors
- Lyu, Q., Havaldar, S., Stein, A., Zhang, L., Rao, D., Wong, E., Apidianaki, M., & Callison-Burch, C.
- Affiliations
- University of Pennsylvania (US)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2301.13379
- OpenAlex
- W4318903783
- Cited by
- 23
- References
- 1
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 2 · 2023: 16
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1928631)
- Defense Advanced Research Projects Agency (FA8750-19-2-1004, FA8750)
- Intelligence Advanced Research Projects Activity
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- chain-of-thought prompting, faithfulness, explainable AI, symbolic reasoning, mathematical word problems, multi-hop question answering, relational inference, planning, few-shot learning, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
