The DOI matches its DataCite record; 37 of the 37 references it cites were found in a registry or an index.
PAL: Program-aided Language Models
Open accessAbstract
Large language models (LLMs) have recently demonstrated an impressive ability to perform arithmetic and symbolic reasoning tasks, when provided with a few examples at test time ("few-shot prompting"). Much of this success can be attributed to prompting methods such as "chain-of-thought'', which employ LLMs for both understanding the problem description by decomposing it into steps, as well as solving each step of the problem. While LLMs seem to be adept at this sort of step-by-step decomposition, LLMs often make logical and arithmetic mistakes in the solution part, even when the problem is decomposed correctly. In this paper, we present Program-Aided Language models (PAL): a novel approach that uses the LLM to read natural language problems and generate programs as the intermediate reasoning steps, but offloads the solution step to a runtime such as a Python interpreter. With PAL, decomposing the natural language problem into runnable steps remains the only learning task for the LLM, while solving is delegated to the interpreter. We demonstrate this synergy between a neural LLM and a symbolic interpreter across 13 mathematical, symbolic, and algorithmic reasoning tasks from BIG-Bench Hard and other benchmarks. In all these natural language reasoning tasks, generating code using an LLM and reasoning using a Python interpreter leads to more accurate results than much larger models. For example, PAL using Codex achieves state-of-the-art few-shot accuracy on the GSM8K benchmark of math word problems, surpassing PaLM-540B which uses chain-of-thought by absolute 15% top-1. Our code and data are publicly available at http://reasonwithpal.com/ .
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“or limit tool use to task-specific settings only (e.g., Gao et al., 2022;Parisi et al., 2022)”
From Toolformer: Language Models Can Teach Themselves to Use Tools (2023) · cites this paper as Gao et al., 2022
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A promising design paradigm for AI agents [65] is to combine an LLM with tools that interact with a broader environment [14, 35, 40, 47, 51, 53, 55, 69].”
From AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (2024) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Programs are increasingly used as the medium through which LLM agents reason, act, and model their environments.”
From Code as Agent Harness (2026) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Early works [66, 7] demonstrate that delegating computation to programs substantially improves reliability by moving intermediate reasoning into structured, verifiable execution traces.”
From Code as Agent Harness (2026) · cites this paper as [7]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Program-aided reasoning methods externalize intermediate computation into executable code [6, 7, 8]”
From Code as Agent Harness (2026) · cites this paper as [7]
The cited source says what the claim attributes to it.
In the citing paper
“On the other hand, large language models have been shown to be capable of describing their generated problem solutions in both text (Wei et al., 2022;Kojima et al., 2022;Zhou et al., 2023) and code (Gao et al., 2022;Chen et al., 2022) formats.”
From Teaching Large Language Models to Self-Debug (2023) · cites this paper as Gao et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“While language models are often effective at proposing reasoning steps, they remain unreliable at faithfully carrying out symbolic, logical, or arithmetic computation [7].”
From Code as Agent Harness (2026) · cites this paper as [7]
The cited source says what the claim attributes to it.
In the citing paper
“For example, some subsample instances from existing benchmarks that cannot be solved by current models [93, 82], or augment them to construct more challenging evaluations [21, 44, 49].”
From SciCode: A Research Coding Benchmark Curated by Scientists (2024) · cites this paper as [21]
The cited source says what the claim attributes to it.
In the citing paper
“This motivates an established alternative-the tool-use paradigm [9, 10]-in which, rather than perform every reasoning step internally, a model delegates well-specified tasks to external tools.”
From PrologMCP: A Standardized Prolog Tool Interface for LLM Agents (2026) · cites this paper as [10]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 7
- Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language Models are Few-Shot Learners. In NeurIPS , 2020.
- Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021b. · arXiv
- Demeter, D. and Downey, D. Just add functions: A neuralsymbolic language model. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pp. 7634-7642, 2020.
- Gellenbeck, E. M. and Cook, C. R. An investigation of procedure and variable names as beacons during program comprehension. In Empirical studies of programmers: Fourth workshop , pp. 65-81. Ablex Publishing, Norwood, NJ, 1991.
- Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the MATH dataset, 2021. URL https://openreview.net/forum? id=7Bywt2mQsCe .
- Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The Curious Case of Neural Text Degeneration. In ICLR , 2019.
- Takang, A. A., Grubb, P. A., and Macredie, R. D. The effects of comments and identifier names on program comprehensibility: an experimental investigation. J. Prog. Lang. , 4(3):143-167, 1996.
Cited by
Sorted by citations
Items 101–192 of 192
An LLM Compiler for Parallel Function Calling
Kim, S., Moon, S., Tabrizi, R., Lee, N., Mahoney, M. W., Keutzer, K., & Gholami, A. (2023) · arXiv · doi:10.48550/arxiv.2312.04511 · Cited by 12
“…of Large Language Models (LLMs) have expanded the applicability of LLMs beyond content generation to solving complex problems [51, 23, 57, 5, 50, 63, 8, 55, 12]; and recent works have shown how this reasoning capability could be helpful in improving accuracy for solving complex and logical tasks.”
BackgroundDas, A., Chen, S., Shyu, M., & Sadiq, S. B. (2023) · IEEE International Conference on Collaboration and Internet Computing (CIC) · doi:10.1109/cic58953.2023.00021 · Cited by 11
Knowledge-Centered Dual-Process Reasoning for Math Word Problems With Large Language Models
Liu, J., Huang, Z., Liu, Q., Ma, Z., Zhai, C., & Chen, E. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3556367 · Cited by 13
Plan, Verify and Switch: Integrated Reasoning with Diverse X-of-Thoughts
Liu, T., Guo, Q., Yang, Y., Hu, X., Zhang, Y., Qiu, X., & Zhang, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.169 · Cited by 23
Self-citationAgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases
Chen, Z., Xiang, Z., Xiao, C., Song, D., & Li, B. (2024) · arXiv · doi:10.48550/arxiv.2407.12784 · Cited by 246
“The LLM agent will execute the generated action by calling build-in tools [9] or external APIs.”
A Review on Neuro-symbolic AI Improvements to Natural Language Processing
Keber, M., Grubišić, I., Barešić, A., & Jović, A. (2024) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro60963.2024.10569741 · Cited by 12
Prompt Engineering for Automatic Short Answer Grading in Brazilian Portuguese
Mello, R. F., Rodrigues, L., Cabral, L., Pereira, F. D., Júnior, C. P., Gašević, D., & Ramalho, G. L. (2024) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/sbie.2024.242424 · Cited by 9
Exploring formal defeasible reasoning of large language models: A Chain-of-Thought approach
Li, Z., Chen, C., Li, M., & Liao, B. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113564 · Cited by 9
The Magic of IF: Investigating Causal Reasoning Abilities in Large Language Models of Code
Liu, X., Yin, D., Zhang, C., Feng, Y., & Zhao, D. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.574 · Cited by 24
Automated Design of Agentic Systems
Hu, S., Lu, C., & Clune, J. (2024) · arXiv · doi:10.48550/arxiv.2408.08435 · Cited by 10
“We evaluate the top 3 agents from MGSM by transferring them to (1) popular math domains: GSM8K (Cobbe et al., 2021), GSM-Hard (Gao et al., 2023), and (2) non-math domains: MMLU (Multi-task) and DROP (Reading Comprehension), as detailed in Section 4.2.”
InfluentialAPI-Assisted Code Generation for Question Answering on Varied Table Structures
Cao, Y., Chen, S., Liu, R., Wang, Z., & Fried, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.897 · Cited by 13
MURMUR: Modular Multi-Step Reasoning for Semi-Structured Data-to-Text Generation
Saha, S., Yu, X., Bansal, M., Pasunuru, R., & Çelikyılmaz, A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.704 · Cited by 12
Three Questions Concerning the Use of Large Language Models to Facilitate Mathematics Learning
Yen, A.-Z., & Hsu, W.-L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.201 · Cited by 15
Leveraging Training Data in Few-Shot Prompting for Numerical Reasoning
Jie, Z., & Lu, W. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.668 · Cited by 15
Question Answering as Programming for Solving Time-Sensitive Questions
Zhu, X., Yang, C., Chen, B., Li, S., Lou, J., & Yang, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.787 · Cited by 11
Self-citationWhen Do Program-of-Thought Works for Reasoning?
Bi, Z., Zhang, N., Jiang, Y., Deng, S., Zheng, G., & Chen, H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29721 · Cited by 31
AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
Debenedetti, E., Zhang, J., Balunović, M., Beurer-Kellner, L., Fischer, M., & Tramèr, F. (2024) · arXiv · doi:10.48550/arxiv.2406.13352 · Cited by 616
“A promising design paradigm for AI agents [60] is to combine an LLM with tools that interact with a broader environment [12, 33, 37, 43, 47, 48, 50, 64].”
BackgroundAdvanced System Integration: Analyzing OpenAPI Chunking for Retrieval-Augmented Generation
Pesl, R. D., Mathew, J. G., Mecella, M., & Aiello, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-94571-7_8 · Cited by 12
Explanation Selection Using Unlabeled Data for Chain-of-Thought Prompting
Ye, X., & Durrett, G. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.41 · Cited by 11
Building Multi-Agent Copilot towards Autonomous Agricultural Data Management and Analysis
Pan, Y., Sun, J., Yu, H., Luck, J. D., Bai, G., Chamara, N., Ge, Y., & Awada, T. N. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10826038 · Cited by 9
Prompt Engineering Through the Lens of Optimal Control
Luo, Y., Tang, Y., Shen, C., Zhou, Z., & Dong, B. (2023) · Journal of Machine Learning · doi:10.4208/jml.231023 · Cited by 28
Relational Programming with Foundational Models
Li, Z., Huang, J., Liu, J., Zhu, F., Zhao, E., Dodds, W., Velingker, N., Alur, R., & Naik, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i9.28934 · Cited by 14
Complex QA and language models hybrid architectures, SurveyPreprint
Daull, X., Bellot, P., Bruno, E., Martin, V., & Murisasco, É. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.09051 · Cited by 32
CRT-QA: A Dataset of Complex Reasoning Question Answering over Tabular Data
Zhang, Z., Li, X., Gao, Y., & Lou, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.132 · Cited by 10
GCoT: Chain-of-Thought Prompt Learning for Graphs
Yu, X., Zhou, C., Kuai, Z., Zhang, X., & Fang, Y. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3736974 · Cited by 11
Time-CoT for Enhancing Time Reasoning Factual Question Answering in Large Language Models
Yin, B., & Hu, N. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650885 · Cited by 5
Graph of Logic: Enhancing LLM Reasoning with Graphs and Symbolic Logic
Alotaibi, F., Kulkarni, A., & Zhou, D. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825450 · Cited by 5
SEER : A Knapsack approach to Exemplar Selection for In-Context HybridQA
Tonglet, J., Reusens, M., Borchert, P., & Baesens, B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.837 · Cited by 8
Sensation in Gamification: A Qualitative Investigation of Background Music in Gamified Learning
de Freitas, J. A., Oliveira, M. K., Martinelli, C., Amorim, F., Toda, A. M., Palomino, P. T., Klock, A. C. T., Guerino, G. C., Avila-Santos, A. P., & Rodrigues, L. (2024) · Journal on Interactive Systems · doi:10.5753/jis.2024.4501 · Cited by 5
A Comprehensive Evaluation of Multimodal Large Language Models in Hydrological ApplicationsPreprint
Kadiyala, L., Mermer, O., Samuel, R. D. J., Sermet, Y., & Demir, İ. (2024) · doi:10.31223/x5tq37 · Cited by 5
Vemula, S. (2024) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10893561 · Cited by 7
Kadlčík, M., Štefánik, M., Sotolář, O., & Martinek, V. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.742 · Cited by 6
Enhancing Quantitative Reasoning Skills of Large Language Models through Dimension Perception
Huang, Y., He, Q., Liang, J., Jiang, S., Xiao, Y., & Chen, Y. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00066 · Cited by 6
Pilade, F., Licata, M. C. G., Vasiliev, I., Fubelli, G., & Gennari, R. (2025) · Computers & Geosciences · doi:10.1016/j.cageo.2025.105914 · Cited by 4
Exploring the Curious Case of Code Prompts
Zhang, L., Dugan, L. D., Xu, H., & Callison-Burch, C. (2023) · doi:10.18653/v1/2023.nlrse-1.2 · Cited by 6
Mind the Labels: Describing Relations in Knowledge Graphs With Pretrained Models
Kasner, Z., Konstas, I., & Dušek, O. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.176 · Cited by 4
Resources and Few-shot Learners for In-context Learning in Slavic Languages
Štefánik, M., Kadlčík, M., Gramacki, P., & Sojka, P. (2023) · Workshop on Slavic Natural Language Processing (SlavicNLP) · doi:10.18653/v1/2023.bsnlp-1.12 · Cited by 8
Automated Theorem Provers Help Improve Large Language Model Reasoning
McGinness, L. P., & Baumgartner, P. (2024) · EPiC series in computing · doi:10.29007/2n9m · Cited by 9
XU, G., Huang, W., & Jia, W. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4893335 · Cited by 5
SciCode: A Research Coding Benchmark Curated by Scientists
Tian, M., Gao, L., Zhang, S. D., Chen, X., Fan, C., Guo, X., Haas, R., Ji, P., Krongchon, K., Li, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.13168 · Cited by 15
“For example, some subsample instances from existing benchmarks that cannot be solved by current models [93, 82], or augment them to construct more challenging evaluations [21, 44, 49].”
BackgroundSelf-citationReliable Reasoning Beyond Natural Language
Borazjanizadeh, N., & Piantadosi, S. T. (2024) · arXiv · doi:10.48550/arxiv.2407.11373 · Cited by 3
“If Bob is in the 7th person in the line, counting left to right, what is the number of Alex?”
Liao, H., Tian, J., Hu, S., Zhu, Z., He, H., & Jin, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887828 · Cited by 6
Abdelkarim, S., Lu, D., Flores, D., Jaeggi, S. M., & Baldi, P. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100170 · Cited by 6
Dynamic Voting for Efficient Reasoning in Large Language Models
Xue, M., Liu, D., Lei, W., Ren, X., Yang, B., Xie, J., Zhang, Y., Peng, D., & Lv, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.203 · Cited by 10
ToolWriter: Question Specific Tool Synthesis for Tabular Data
Gemmell, C., & Dalton, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1003 · Cited by 13
It Ain’t Over: A Multi-aspect Diverse Math Word Problem Dataset
Kim, J., Kim, Y., Baek, I., Bak, J., & Lee, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.927 · Cited by 7
A Comprehensive Evaluation of Tool-Assisted Generation Strategies
Jacovi, A., Caciularu, A., Herzig, J., Aharoni, R., Bohnet, B., & Geva, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.926 · Cited by 7
Towards Generative Search and Recommendation: A keynote at RecSys 2023
Chua, T. (2023) · ACM SIGIR Forum · doi:10.1145/3642979.3642986 · Cited by 3
Foundational Design Principles and Patterns for Building Robust and Adaptive GenAI-Native Systems
Vandeputte, F. (2025) · ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) · doi:10.1145/3759429.3762620 · Cited by 8
Frugal LMs Trained to Invoke Symbolic Solvers Achieve Parameter-Efficient Arithmetic Reasoning
Dutta, S., Pandey, I., Singh, J., Manchanda, S., Chakrabarti, S., & Chakraborty, T. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29750 · Cited by 4
Nissen, D., Yu, T., Babtista, R., & Shang, Y. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825517 · Cited by 5
Learning Multi-Step Reasoning by Solving Arithmetic Tasks
Wang, T., & Lü, W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.106 · Cited by 4
Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation
Zelikman, E., Lorch, E., Mackey, L., & Kalai, A. T. (2023) · arXiv · doi:10.48550/arxiv.2310.02304 · Cited by 4
“…on letting models reason with access to an interpreter such as Program of Thoughts prompting (Chen et al., 2022), Program-aided Language Models (Gao et al., 2023), Re-flexion (Shinn et al., 2023), or ReAct (Yao et al., 2022), while yet others abstracted this scaffolding structure such as…”
BackgroundComplex Reasoning in Natural Languag
Zhao, W., Geva, M., Lin, B. Y., Yasunaga, M., Madaan, A., & Yu, T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-tutorials.2 · Cited by 5
Self-citationArithmeticGPT: empowering small-size large language models with advanced arithmetic skills
Liu, Z., Zheng, Y., Yin, Z., Chen, J., Liu, T., Mi, T., & Luo, W. (2025) · Machine Learning · doi:10.1007/s10994-024-06681-1 · Cited by 2
Simple Action Model: Enabling LLM to Sequential Function Calling Tool Chain
Sen, R. S., Amalkrishna, M., Prithviraj, R., Jose, S. P., Joseph, N., & Thomas, R. (2024) · International Conference on Advancement in Renewable Energy and Intelligent Systems (AREIS) · doi:10.1109/areis62559.2024.10893677 · Cited by 2
Improving Mathematics Tutoring With A Code Scratchpad
Upadhyay, S., Ginsberg, E., & Callison-Burch, C. (2023) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2023.bea-1.2 · Cited by 4
Watson, W., Cho, N., Balch, T. R., & Veloso, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.442 · Cited by 5
Iterative improvements from feedback for language modelsPreprint
Li, Y. (2023) · doi:10.14293/pr2199.000220.v1 · Cited by 3
PrologMCP: A Standardized Prolog Tool Interface for LLM Agents
Mensfelt, A., Prabhakaran, A., Haret, A., Trencsenyi, V., & Stathis, K. (2026) · arXiv · doi:10.48550/arxiv.2606.14935 · Cited by 1
Naushad, R., Gupta, R., Bhutiyal, T., & Prajapati, V. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-65668-2_17 · Cited by 1
A Combinatorial Strategy for API Completion: Deep Learning and Heuristics
Liu, Y., Yin, Y., Deng, J., Li, W., & Peng, Z. (2024) · Electronics · doi:10.3390/electronics13183669 · Cited by 1
Research Report: AI Security is a LangSec Problem
von Hippel, M., & Miyazono, E. (2025) · IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw67851.2025.00011 · Cited by 1
CountNet: Utilising Repetition Counts in Sequential Recommendation
Petrov, A. V., Taniskidou, E. K., & Murphy, S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88714-7_4 · Cited by 1
Artificial Intelligence in Math Education of Engineers
Schönle, M. (2025) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon62633.2025.11016373 · Cited by 1
SparseX: Efficient Segment-Level KV Cache Sharing for Interleaved LLM Serving
Zhang, Q., Chen, K., Liao, N., Lin, Z., Tang, B., Xiong, F., Li, Z., & Wang, X. (2026) · arXiv · doi:10.48550/arxiv.2606.01751
“The downstream tasks cover a broad range of domains to verify applicability in collaborative reasoning systems, including MATH [25], GSM-Hard [26], and AQUA-RAT [27] for mathematical reasoning; SciBench [28] and GPQA [29] for scientific reasoning; MMLU-Pro [30] for general knowledge understanding;…”
ReaComp: Compiling LLM Reasoning into Symbolic Solvers for Efficient Program Synthesis
Naik, A., Mathur, Y., Prakam, Rose, C., & Mortensen, D. (2026) · arXiv · doi:10.48550/arxiv.2605.05485
“PAL [Gao et al., 2023] delegates intermediate computations to executable programs, which can reduce arithmetic and logical errors.”
When Does Restricting a Coding Agent to execute_code Help? A Regime $\times$ Agent-Design Ablation
Yang, H., Yu, Q., & Desell, T. (2026) · arXiv · doi:10.48550/arxiv.2607.10569
On Effectiveness and Efficiency of Agentic Tool-calling and RL Training
Liu, T., Qian, C., Cief, M., He, Y., Dan, D., Aletras, N., & Kazai, G. (2026) · arXiv
PAACE: A Plan-Aware Automated Agent Context Engineering Framework
Yuksel, K. A. (2025) · arXiv · doi:10.48550/arxiv.2512.16970
Ning, X., Tieu, K., Fu, D., Wei, T., Li, Z., Bei, Y., Zou, J., Ai, M., Liu, Z., Li, T.-W., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.18747
SORTIE: Dependency-Aware Symbolic Reasoning for Logical Data-to-text Generation
Zhao, X., Fu, T., Liu, L., Kong, L., Shi, S., & Yan, R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.715 · Cited by 1
Reasoning arithmetic word problems entailing implicit relations based on the chain-of-thought model
Meng, H., Yue, L., Sun, G., & Shen, J. (2023) · STEM Education · doi:10.3934/steme.2023015
Symbolic Planning and Code Generation for Grounded Dialogue
Chiu, J., Zhao, W., Chen, D., Vaduguru, S., Rush, A. M., & Fried, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.460 · Cited by 3
Symbolic Planning and Code Generation for Grounded Dialogue
Chiu, J., Zhao, W., Chen, D., Vaduguru, S., Rush, A. M., & Fried, D. (2023) · doi:10.18653/v1/2023.pandl-1.5
Gupta, D. (2024) · doi:10.59350/g7bme-msw06
Kravchenko, Y. A., Валерьевич, К. Э., Юрьевич, К. Д., Rodzin, S. I., Pavlov, N., & Rodzina, L. (2024) · International Conference on Information Technologies in Engineering Education (Inforino) · doi:10.1109/inforino60363.2024.10552028
ANSWERED: Adaptive Tool-Augmented LLMs with Strategic Error Feedback for Compositional Reasoning
Pan, Y., Zhang, X., Zhang, H., Liu, Y., Wang, H., Huang, Q., & Wang, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5669-8_22
Assessing the Potential of Large Language Models for Chemical Engineering Applications
Xuan, L., Haoxiang, Z., Baozheng, J., & You, L. (2024) · doi:10.1145/3698385.3699878
Complex Claim Verification via Human Fact-Checking Imitation with Large Language Models
Xu, B., Sun, F., Liu, X., Wu, P., Wang, X., & Pan, L. (2024) · International Joint Symposium on Artificial Intelligence and Natural Language Processing (iSAI-NLP) · doi:10.1109/isai-nlp64410.2024.10799276 · Cited by 1
Liu, X., Yin, D., Zhang, C., Zhao, D., & Feng, Y. (2025) · Computational Linguistics · doi:10.1162/coli_a_00548 · Cited by 4
Exploring performance and consistency in mathematical reasoning across problem types
Hwang, Y., Lim, J., Lee, Y., & Choi, H. (2025) · IEEE International Conference on Big Data and Smart Computing (BigComp) · doi:10.1109/bigcomp64353.2025.00031
Wu, O. T., Chan, F. K. S., & Law, Y. N. (2024) · International Conference on Computers and Artificial Intelligence Technology (CAIT) · doi:10.1109/cait64506.2024.10963264
Uluçam, E., & Alptekin, S. E. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-032-00071-2_9
Transformers and State-Space Models: Fine-Tuning Techniques for Solving Differential Equations
Ignatenko, V., Surkov, A., Zakharov, V., & Koltcov, S. (2025) · Sci · doi:10.3390/sci7030130
Gupta, D. (2024) · doi:10.59350/j804h-9b851
An electronic product carbon footprint dataset for question answering
Zhao, K., Chathoth, A. K., Balaji, B., & Lee, S. (2026) · Scientific Data · doi:10.1038/s41597-026-06544-5
Videla, A. (2026) · Queue · doi:10.1145/3831357
Catching Developers in the Flow: Low-Latency Agentic Program Repair at Google Scale
Ziftci, C., Greene, S., Liu, R., Dalloro, L., & Dini, L. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834511
MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems
Ye, R., Tang, S., Ge, R., Du, Y., Yin, Z., Chen, S., & Shao, J. (2025) · arXiv · doi:10.48550/arxiv.2503.03686
Metadata
- Authors
- Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., & Neubig, G.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2211.10435
- OpenAlex
- W4309591663
- Cited by
- 155
- References
- 0
- Citations by year
- 2026: 3 · 2025: 28 · 2024: 48 · 2023: 76
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (public-domain)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, code generation, Python interpreter, symbolic reasoning, mathematical reasoning, few-shot prompting, chain-of-thought, mathematical word problems, GSM8K
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
