Solving Math Word Problems by Combining Language Models With Symbolic Solvers
Open accessAbstract
Automatically generating high-quality step-by-step solutions to math word problems has many applications in education. Recently, combining large language models (LLMs) with external tools to perform complex reasoning and calculation has emerged as a promising direction for solving math word problems, but prior approaches such as Program-Aided Language model (PAL) are biased towards simple procedural problems and less effective for problems that require declarative reasoning. We propose an approach that combines an LLM that can incrementally formalize word problems as a set of variables and equations with an external symbolic solver that can solve the equations. Our approach achieves comparable accuracy to the original PAL on the GSM8K benchmark of math word problems and outperforms PAL by an absolute 20% on ALGEBRA, a new dataset of more challenging word problems extracted from Algebra textbooks. Our work highlights the benefits of using declarative and incremental representations when interfacing with an external tool for solving complex math word problems. Our data and prompts are publicly available at https://github.com/joyheyueya/declarative-math-word-problem.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs have an impressive ability to convert text into formal structures like math equations and programming languages.
“In our experience with advanced LLMs such as Llama2 and GPT-4, we've noticed their impressive ability to convert text into formal structures like math equations [21] and programming languages [22].”
From ChatLogic: Integrating Logic Programming with Large Language Models for Multi-Step Reasoning (2024) · cites this paper as [21]
LLMs have demonstrated a notable ability to comprehend textual inputs and translate them into formal programs.
“However, they have demonstrated a notable ability to comprehend textual inputs and translate them into formal programs, such as mathematical equations (He-Yueya et al., 2023) or Python codes (Gao et al., 2023).”
From Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning (2023) · cites this paper as He-Yueya et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 14
- T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
- M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. · arXiv
- A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. · arXiv
- K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. · arXiv
- D. D. Cummins. Children's interpretations of arithmetic word problems. Cognition and instruction, 8(3):261-289, 1991.
- J. del Olmo-Muñoz, J. A. González-Calero, P. D. Diago, D. Arnau, and M. Arevalillo-Herráez. Intelligent tutoring systems for word problem solving in covid-19 days: could they have been (part of) the solution? ZDM-Mathematics Education, pages 1-14, 2022.
- L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig. Pal: Program- aided language models. arXiv preprint arXiv:2211.10435, 2022. · arXiv
- D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Stein- hardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. · arXiv
- A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14858, 2022. · arXiv
- O. Polozov, E. O'Rourke, A. M. Smith, L. Zettlemoyer, S. Gulwani, and Z. Popović. Personal- ized mathematical word problem generation. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
- S. Ritter, J. R. Anderson, K. R. Koedinger, and A. Corbett. Cognitive tutor: Applied research in mathematics education. Psychonomic bulletin & review, 14:249-255, 2007.
- J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022. · arXiv
- I. G. Zaigralin. Basic Algebra with Applications. Ivan G. Zaigralin, 6 edition, 2018.
- D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, O. Bousquet, Q. Le, and E. Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625, 2022. · arXiv
Cited by
Sorted by citations
Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
Pan, L., Albalak, A., Wang, X., & Wang, W. Y. (2023) · arXiv · doi:10.48550/arxiv.2305.12295 · Cited by 144 · see citing works
“However, they have demonstrated a notable ability to comprehend textual inputs and translate them into formal programs, such as mathematical equations (He-Yueya et al., 2023) or Python codes (Gao et al., 2023).”
BackgroundZhang, C., Liu, X., Ziska, K., Jeon, S., Yu, C.-L., & Xu, Y. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642647 · Cited by 96 · see citing works
ChatLogic: Integrating Logic Programming with Large Language Models for Multi-Step Reasoning
Wang, Z., Liu, J., Bao, Q., Rong, H., & Zhang, J. (2024) · 2024 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650138 · Cited by 8 · see citing works
Puech, R., Macina, J., Chatain, J., Sachan, M., & Kapur, M. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.1348 · Cited by 11 · see citing works
- Authors
- He-Yueya, J., Poesia, G., Wang, R. E., & Goodman, N. D.
- Affiliations
- Stanford University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2304.09102
- OpenAlex
- W4366460267
- Cited by
- 16
- References
- 0
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 5 · 2023: 6
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Mathematics, Computing, and Information Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- mathematical word problems, large language models, symbolic solvers, declarative reasoning, equation solving, algebra word problems, GSM8K
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
