Faithful Logical Reasoning via Symbolic Chain-of-Thought
Open accessAbstract
While the recent Chain-of-Thought (CoT) technique enhances the reasoning ability of large language models (LLMs) with the theory of mind, it might still struggle in handling logical reasoning that relies much on symbolic expressions and rigid deducing rules.To strengthen the logical reasoning capability of LLMs, we propose a novel Symbolic Chain-of-Thought, namely SymbCoT, a fully LLM-based framework that integrates symbolic expressions and logic rules with CoT prompting.Technically, building upon an LLM, SymbCoT 1) first translates the natural language context into the symbolic format, and then 2) derives a step-by-step plan to solve the problem with symbolic logical rules, 3) followed by a verifier to check the translation and reasoning chain.Via thorough evaluations on 5 standard datasets with both First-Order Logic and Constraint Optimization symbolic expressions, SymbCoT shows striking improvements over the CoT method consistently, meanwhile refreshing the current stateof-the-art performances.We further demonstrate that our system advances in more faithful, flexible, and explainable logical reasoning.To our knowledge, this is the first to combine symbolic expressions and rules into CoT for logical reasoning with LLMs.Code is open at https://github.com/Aiden0526/SymbCoT.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In most cases, CoT enhancement can improve model performance.
“In most cases, this enhancement can improve model performance39,40.”
From Enhancing diagnostic capability with multi-agents conversational large language models (2025) · cites this paper as [40]
Section 4.2 (Main Result; page unavailable) reports that SymbCoT outperformed vanilla CoT on the paper’s logical-reasoning tests. This supports performance gains from the specific enhancement studied, but not the broader claim that CoT enhancements improve model performance in most cases generally. (LLM, conf=0.82)
Experimental — accuracy not yet measuredChain-of-Thought (CoT) techniques enable models to generate more extensive outputs detailing their reasoning processes.
“Conversely, techniques such as Chain-of-Thought (CoT) enable models to generate more extensive outputs detailing their reasoning processes.”
From Enhancing diagnostic capability with multi-agents conversational large language models (2025) · cites this paper as [40]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2305.10601 · Cited by 1,158 · see citing works
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
Self-citationNExT-GPT: Any-to-Any Multimodal LLMPreprint
Wu, S., Fei, H., Qu, L., Ji, W., & Chua, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.05519 · Cited by 132 · see citing works
Self-citationFOLIO: Natural Language Reasoning with First-Order Logic
Han, S., Schoelkopf, H., Zhao, Y., Qi, Z., Riddell, M., Zhou, W., Coady, J., Peng, D., Qiao, Y., Benson, L., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.00840 · Cited by 35 · see citing works
Code Prompting: a Neural Symbolic Method for Complex Reasoning in Large Language ModelsPreprint
Hu, Y., Yang, H.-T., Lin, Z.-C., & Zhang, M. (2023) · arXiv.org · doi:10.48550/arxiv.2305.18507 · Cited by 20 · see citing works
Formal Specifications from Natural LanguagePreprint
Hahn, C., Schmitt, F., Tillman, J., Metzger, N., Siber, J., & Finkbeiner, B. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.01962 · Cited by 19 · see citing works
Cumulative Reasoning with Large Language ModelsPreprint
Zhang, Y., Yang, J., Yang, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.04371 · Cited by 15 · see citing works
Dunivin, Z. O. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.15170 · Cited by 11 · see citing works
Sun, H., Xu, W., Liu, W., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2310.18659 · Cited by 8
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Flagged first, then by citations
Research works printed but not linked: 26
- Alfonso Amayuelas, Shuai Zhang, Susie Xi Rao, and Ce Zhang. 2022. Neural methods for logical reason- ing over knowledge graphs. In Proceedings of the International Conference on Learning Representa- tions.
- Maciej Besta, Nils Blach, Ales Kubicek, Robert Ger- stenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadom- ski, Piotr Nyczyk, and Torsten Hoefler. 2023. Graph of thoughts: Solving elaborate problems with large language models. CoRR, abs/2308.09687. · arXiv
- Herbert B. Enderton. 1972. A mathematical introduc- tion to logic. Academic Press.
- Hao Fei, Bobo Li, Qian Liu, Lidong Bing, Fei Li, and Tat-Seng Chua. 2023. Reasoning implicit sentiment with chain-of-thought prompting. In Proceedings of the Annual Meeting of the Association for Computa- tional Linguistics, pages 1171-1182.
- Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Meishan Zhang, Mong Li Lee, and Wynne Hsu. 2024. Video-of-thought: Step-by-step video reasoning from perception to cognition. In Proceedings of the Inter- national Conference on Machine Learning, ICML.
- Hao Fei, Shengqiong Wu, Yafeng Ren, and Meishan Zhang. 2022. Matching structure for dual learning. In Proceedings of the International Conference on Machine Learning, ICML, pages 6373-6391.
- Vedant Gaur and Nikunj Saunshi. 2023. Reasoning in large language models through symbolic math word problems. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 5889-5903.
- Olga Gerasimova, Nikita Severin, and Ilya Makarov. 2023. Comparative analysis of logic reasoning and graph neural networks for ontology-mediated query answering with a covering axiom. IEEE Access, 11:88074-88086.
- Jie Huang and Kevin Chen-Chuan Chang. 2023. To- wards reasoning in large language models: A survey. In Proceedings of the Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1049- 1065.
- Shima Imani, Liang Du, and Harsh Shrivastava. 2023. MathPrompter: Mathematical reasoning using large language models. In Proceedings of the Annual Meet- ing of the Association for Computational Linguistics, pages 37-42.
- Tatsuro Inaba, Hirokazu Kiyomaru, Fei Cheng, and Sadao Kurohashi. 2023. MultiTool-CoT: GPT-3 can use multiple external tools with chain of thought prompting. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1522-1532.
- Henry Markovits and Robert Vachon. 1989. Reason- ing with contrary-to-fact propositions. Journal of Experimental Child Psychology, 47:398-412.
- Theo Olausson, Alex Gu, Ben Lipkin, Cedegao Zhang, Armando Solar-Lezama, Joshua Tenenbaum, and Roger Levy. 2023. LINC: A neurosymbolic approach for logical reasoning by combining language mod- els with first-order logic provers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 5153-5176.
- Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welin- der, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instruc- tions with human feedback. In Proceedings of the Annual Conference on Neural Information Process- ing Systems, pages 27730-27744.
- Liangming Pan, Alon Albalak, Xinyi Wang, and William Wang. 2023. Logic-LM: Empowering large language models with symbolic solvers for faithful logical reasoning. In Proceedings of the Conference on Empirical Methods in Natural Language Process- ing, pages 3806-3824.
- Ajay Patel, Bryan Li, Mohammad Sadegh Rasooli, Noah Constant, Colin Raffel, and Chris Callison- Burch. 2023. Bidirectional language models are also few-shot learners. In Proceedings of the Interna- tional Conference on Learning Representations.
- John Alan Robinson. 1965. A machine-oriented logic based on the resolution principle. J. ACM, 12(1):23- 41.
- Abulhair Saparov and He He. 2023. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought. In Proceedings of the International Conference on Learning Representations.
- Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. 2021. ProofWriter: Generating implications, proofs, and abductive statements over natural language. In Pro- ceedings of the Annual Meeting of the Association for Computational Linguistics, pages 3621-3634.
- Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023a. Plan-and-solve prompting: Improving zero- shot chain-of-thought reasoning by large language models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 2609-2634.
- Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowd- hery, and Denny Zhou. 2023b. Self-consistency im- proves chain of thought reasoning in language mod- els. In Proceedings of the International Conference on Learning Representations.
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. In Proceedings of the Annual Conference on Neural In- formation Processing Systems, pages 24824-24837.
- Jiacheng Ye, Chengzu Li, Lingpeng Kong, and Tao Yu. 2023. Generating data for symbolic language with large language models. In Proceedings of the Con- ference on Empirical Methods in Natural Language Processing, pages 8418-8443.
- Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023b. Automatic chain of thought prompt- ing in large language models. In Proceedings of the International Conference on Learning Representa- tions.
- Li Zheng, Hao Fei, Fei Li, Bobo Li, Lizi Liao, Donghong Ji, and Chong Teng. 2024. Reverse multi- choice dialogue commonsense inference with graph- of-thought. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19688- 19696.
- Wanjun Zhong, Siyuan Wang, Duyu Tang, Zenan Xu, Daya Guo, Yining Chen, Jiahai Wang, Jian Yin, Ming Zhou, and Nan Duan. 2022. Analytical reasoning of text. In Proceedings of the North American Chap- ter of the Association for Computational Linguistics, pages 2306-2319.
Cited by
Sorted by citations
Enhancing diagnostic capability with multi-agents conversational large language models
Chen, X., Yi, H., You, M., Liu, W., Wang, L., Li, H., Zhang, X., Guo, Y., Fan, L., Chen, G., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01550-0 · Cited by 117 · see citing works
Yager, K. G. (2024) · Digital Discovery · doi:10.1039/d4dd00178h · Cited by 12 · see citing works
Li, Y., Fu, X., Verma, G., Buitelaar, P., & Liu, M. (2025) · arXiv · doi:10.48550/arxiv.2510.24476 · Cited by 4 · see citing works
- Authors
- Xu, J., Fei, H., Pan, L., Liu, Q., Lee, M.-L., & Hsu, W.
- Affiliations
- National University of Singapore (SG)
- University of California, Santa Barbara (US)
- University of Auckland (NZ)
- Venue
- Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2024
- Pages
- 13326-13365
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4402671211
- Cited by
- 53
- References
- 10
- Citations by year
- 2026: 23 · 2025: 21 · 2024: 9
- FWCI
- 15.64
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- ResearchSpace (University of Auckland) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Funders
- Baidu
- Ministry of Education - Singapore (MOE-MOET32022-0001)
- Ministry of Education, India
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- logical reasoning, large language models, symbolic expression, first-order logic, constraint optimization, reasoning verification, explainable reasoning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
