PAL: Program-aided Language Models
Open accessAbstract
Large language models (LLMs) have recently demonstrated an impressive ability to perform arithmetic and symbolic reasoning tasks, when provided with a few examples at test time ("few-shot prompting"). Much of this success can be attributed to prompting methods such as "chain-of-thought'', which employ LLMs for both understanding the problem description by decomposing it into steps, as well as solving each step of the problem. While LLMs seem to be adept at this sort of step-by-step decomposition, LLMs often make logical and arithmetic mistakes in the solution part, even when the problem is decomposed correctly. In this paper, we present Program-Aided Language models (PAL): a novel approach that uses the LLM to read natural language problems and generate programs as the intermediate reasoning steps, but offloads the solution step to a runtime such as a Python interpreter. With PAL, decomposing the natural language problem into runnable steps remains the only learning task for the LLM, while solving is delegated to the interpreter. We demonstrate this synergy between a neural LLM and a symbolic interpreter across 13 mathematical, symbolic, and algorithmic reasoning tasks from BIG-Bench Hard and other benchmarks. In all these natural language reasoning tasks, generating code using an LLM and reasoning using a Python interpreter leads to more accurate results than much larger models. For example, PAL using Codex achieves state-of-the-art few-shot accuracy on the GSM8K benchmark of math word problems, surpassing PaLM-540B which uses chain-of-thought by absolute 15% top-1. Our code and data are publicly available at http://reasonwithpal.com/ .
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A promising design paradigm for AI agents is to combine an LLM with tools that interact with a broader environment.
“A promising design paradigm for AI agents [65] is to combine an LLM with tools that interact with a broader environment [14, 35, 40, 47, 51, 53, 55, 69].”
From AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (2024) · cites this paper as [14]
The Introduction describes PAL combining an LLM with an external Python interpreter, supporting the LLM-plus-tool design. The paper uses that interpreter to solve reasoning problems; it does not show tools interacting with a broader environment or evaluate AI agents generally. (LLM, conf=0.82)
Experimental — accuracy not yet measuredPrograms are increasingly used as the medium through which LLM agents reason, act, and model their environments.
“Programs are increasingly used as the medium through which LLM agents reason, act, and model their environments.”
From Code as Agent Harness (2026) · cites this paper as [7]
The Introduction and Section 3 describe PAL using programs as intermediate steps for language-model reasoning, with execution delegated to an interpreter. They do not establish that programs are increasingly used across LLM agents to act or model their environments. Page numbers are unavailable in our copy. (LLM, conf=0.88)
Experimental — accuracy not yet measuredDelegating computation to programs substantially improves reliability by moving intermediate reasoning into structured, verifiable execution traces.
“Early works [66, 7] demonstrate that delegating computation to programs substantially improves reliability by moving intermediate reasoning into structured, verifiable execution traces.”
From Code as Agent Harness (2026) · cites this paper as [7]
The Introduction and Conclusion describe PAL generating programs as intermediate reasoning steps and delegating execution to a Python interpreter; the Abstract reports improved accuracy. But the paper describes executable programs, not intermediate reasoning presented as verifiable execution traces, so that characterization is not fully supported. (LLM, conf=0.81)
Experimental — accuracy not yet measuredProgram-aided reasoning methods externalize intermediate computation into executable code.
“Program-aided reasoning methods externalize intermediate computation into executable code [6, 7, 8]”
From Code as Agent Harness (2026) · cites this paper as [7]
Large language models have been shown to be capable of describing their generated problem solutions in both text and code formats.
“On the other hand, large language models have been shown to be capable of describing their generated problem solutions in both text (Wei et al., 2022;Kojima et al., 2022;Zhou et al., 2023) and code (Gao et al., 2022;Chen et al., 2022) formats.”
From Teaching Large Language Models to Self-Debug (2023) · cites this paper as Gao et al., 2022
Language models are often effective at proposing reasoning steps but remain unreliable at faithfully carrying out symbolic, logical, or arithmetic computation.
“While language models are often effective at proposing reasoning steps, they remain unreliable at faithfully carrying out symbolic, logical, or arithmetic computation [7].”
From Code as Agent Harness (2026) · cites this paper as [7]
Researchers are developing synthetic challenging benchmarks, often involving models in the construction of evaluation instances, such as subsampling instances from existing benchmarks that cannot be solved by current models or augmenting them to construct more challenging evaluations.
“For example, some subsample instances from existing benchmarks that cannot be solved by current models [93, 82], or augment them to construct more challenging evaluations [21, 44, 49].”
From SciCode: A Research Coding Benchmark Curated by Scientists (2024) · cites this paper as [21]
The tool-use paradigm is an established alternative in which a model delegates well-specified tasks to external tools.
“This motivates an established alternative-the tool-use paradigm [9, 10]-in which, rather than perform every reasoning step internally, a model delegates well-specified tasks to external tools.”
From PrologMCP: A Standardized Prolog Tool Interface for LLM Agents (2026) · cites this paper as [10]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 7
- Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language Models are Few-Shot Learners. In NeurIPS , 2020.
- Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021b. · arXiv
- Demeter, D. and Downey, D. Just add functions: A neuralsymbolic language model. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pp. 7634-7642, 2020.
- Gellenbeck, E. M. and Cook, C. R. An investigation of procedure and variable names as beacons during program comprehension. In Empirical studies of programmers: Fourth workshop , pp. 65-81. Ablex Publishing, Norwood, NJ, 1991.
- Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the MATH dataset, 2021. URL https://openreview.net/forum? id=7Bywt2mQsCe .
- Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The Curious Case of Neural Text Degeneration. In ICLR , 2019.
- Takang, A. A., Grubb, P. A., and Macredie, R. D. The effects of comments and identifier names on program comprehensibility: an experimental investigation. J. Prog. Lang. , 4(3):143-167, 1996.
Cited by
Sorted by citations
Self-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2019 · Cited by 755 · see citing works
Self-citationUnleashing the potential of prompt engineering for large language models
CHEN, B. ‐., Zhang, Z., Langrené, N., & Zhu, S. (2025) · Patterns · doi:10.1016/j.patter.2025.101260 · Cited by 764 · see citing works
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505 · see citing works
Towards Reasoning in Large Language Models: A Survey
Huang, J., & Chang, K. C.-C. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.67 · Cited by 428 · see citing works
When large language models meet personalization: perspectives of challenges and opportunities
Chen, J., Liu, Z., Huang, X., Wu, C., Liu, Q., Jiang, G., Pu, Y., Lei, Y., Chen, X., Wang, X., et al. (2024) · World Wide Web · doi:10.1007/s11280-024-01276-1 · Cited by 367 · see citing works
Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health
Tian, S., Jin, Q., Yeganova, L., Lai, P.-T., Zhu, Q., Chen, X., Yang, Y., Chen, Q., Kim, W., Comeau, D. C., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad493 · Cited by 359 · see citing works
Self-citation(2023) · doi:10.18653/v1/2023.emnlp-main.574 · Cited by 249 · see citing works
ViperGPT: Visual Inference via Python Execution for Reasoning
Surís, D., Menon, S., & Vondrick, C. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01092 · Cited by 273 · see citing works
Self-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 219 · see citing works
“We select two examples from the training set on which CODEX fails when prompted with PaL-styled prompts, and manually write the correct solution (yi+1) and reasoning (fbi) for REFINE and FEEDBACK.”
MethodSelf-citationEnhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy
Shao, Z., Gong, Y., Shen, Y., Huang, M., Duan, N., & Chen, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.620 · Cited by 200 · see citing works
Wang, M., Wang, M., Xu, X., Yang, L., Cai, D., & Yin, M. (2023) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2023.3324714 · Cited by 176 · see citing works
(2025) · doi:10.1109/icse55347.2025.00157 · Cited by 157 · see citing works
Hallucination Mitigation for Retrieval-Augmented Large Language Models: A Review
Zhang, W., & Zhang, J. (2025) · Mathematics · doi:10.3390/math13050856 · Cited by 145 · see citing works
Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
Pan, L., Albalak, A., Wang, X., & Wang, W. Y. (2023) · arXiv · doi:10.48550/arxiv.2305.12295 · Cited by 144 · see citing works
“, 2023) or a Python interpreter (Gao et al., 2022; Chen et al., 2022) into LLMs can greatly enhance performance by offloading numerical calculations.”
BackgroundReasoning with Language Model Prompting: A Survey
Qiao, S., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S., Tan, C., Huang, F., & Chen, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.294 · Cited by 218 · see citing works
Large Language Models Can Be Easily Distracted by Irrelevant Context
Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E., Schärli, N., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2302.00093 · Cited by 134 · see citing works
“Following prior work on solving GSM8K problems with code (Chowdhery et al., 2022; Gao et al., 2022; Chen et al., 2022), we include a Python program as the problem solution in the prompt, and execute the generated Python code using an external Python interpreter to obtain the final answer.”
MethodResultTowards Faithful Model Explanation in NLP: A Survey
Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024) · Computational Linguistics · doi:10.1162/coli_a_00511 · Cited by 143 · see citing works
Cui, C., Ma, Y., Cao, X., Ye, W., & Wang, Z. (2024) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00101 · Cited by 132 · see citing works
Faithful Chain-of-Thought Reasoning
Lyu, Q., Havaldar, S., Stein, A., Zhang, L., Rao, D., Wong, E., Apidianaki, M., & Callison-Burch, C. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.20 · Cited by 201 · see citing works
Dibia, V. C. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.11 · Cited by 125 · see citing works
Prompt Engineering for Healthcare: Methodologies and Applications
Wang, J., Shi, E., Yu, S., Wu, Z., Ma, C., Dai, H., Yang, Q., Kang, Y., Wu, J., Hu, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.14670 · Cited by 150 · see citing works
Receive, Reason, and React: Drive as You Say, With Large Language Models in Autonomous Vehicles
Cui, C., Ma, Y., Cao, X., Ye, W., & Wang, Z. (2024) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2024.3381793 · Cited by 121 · see citing works
TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs
Liang, Y., Wu, C., Song, T., Wu, W., Xia, Y., Liu, Y., Ou, Y., Lu, S., Ji, L., Mao, S., et al. (2024) · Intelligent Computing · doi:10.34133/icomputing.0063 · Cited by 108 · see citing works
Recent Advances in Large Language Models for Healthcare
Nassiri, K., & Akhloufi, M. A. (2024) · BioMedInformatics · doi:10.3390/biomedinformatics4020062 · Cited by 105 · see citing works
Exploring the potential of using ChatGPT in physics education
Liang, Y., Zou, D., Xie, H., & Wang, F. L. (2023) · Smart Learning Environments · doi:10.1186/s40561-023-00273-7 · Cited by 98 · see citing works
An Overview on IEEE 802.11bf: WLAN Sensing
Du, R. Y., Hua, H., Xie, H., Song, X., Lyu, Z., Hu, M., Narengerile, Xin, Y., McCann, S., Montemurro, M., et al. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3408899 · Cited by 113 · see citing works
Zhang, Q., Xu, C., Li, J., Sun, Y., Bao, J., & Zhang, D. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125861 · Cited by 94 · see citing works
Gorilla: Large Language Model Connected with Massive APIs
Patil, S. G., Zhang, T., Wang, X., & Gonzalez, J. E. (2023) · arXiv · doi:10.48550/arxiv.2305.15334 · Cited by 226 · see citing works
Generalized Planning in PDDL Domains with Pretrained Large Language Models
Silver, T., Dan, S., Srinivas, K., Tenenbaum, J. B., Kaelbling, L. P., & Katz, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i18.30006 · Cited by 90 · see citing works
Teaching Large Language Models to Self-Debug
Chen, X., Lin, M., Schärli, N., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2304.05128 · Cited by 77 · see citing works
“Several prompting methods have been proposed to unlock latent abilities of large language models (Wei et al., 2022; Nye et al., 2021; Kojima et al., 2022; Zhou et al., 2023; Khot et al., 2022; Zhou et al., 2022; Gao et al., 2022; Chen et al., 2022).”
BackgroundAre Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond
Xu, F., Lin, Q., Han, J., Zhao, T., Liu, J., & Cambria, E. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3536008 · Cited by 76 · see citing works
Few-shot In-context Learning on Knowledge Base Question Answering
Li, T., Ma, X., Zhuang, A., Gu, Y., Su, Y., & Chen, W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.385 · Cited by 75 · see citing works
Assessing LLMs in malicious code deobfuscation of real-world malware campaigns
Patsakis, C., Casino, F., & Lykousas, N. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124912 · Cited by 83 · see citing works
A Survey of Robot Intelligence with Large Language Models
Jeong, H., Lee, H., Kim, C., & Shin, S. (2024) · Applied Sciences · doi:10.3390/app14198868 · Cited by 80 · see citing works
DePlot: One-shot visual language reasoning by plot-to-table translation
Liu, F., Eisenschlos, J. M., Piccinno, F., Krichene, S., Pang, C., Lee, K., Joshi, M., Chen, W., Collier, N., & Altün, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.660 · Cited by 78 · see citing works
UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning
Masry, A., Kavehzadeh, P., Long, X., Hoque, E., & Joty, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.906 · Cited by 97 · see citing works
LLM for Test Script Generation and Migration: Challenges, Capabilities, and Opportunities
Yu, S., Fang, C., Ling, Y., Wu, C., & Chen, Z. (2023) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs60937.2023.00029 · Cited by 77 · see citing works
Discuss Before Moving: Visual Language Navigation via Multi-expert Discussions
Long, Y., Li, X., Cai, W., & Dong, H. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611565 · Cited by 73 · see citing works
AIGC-empowered smart manufacturing: Prospects and challenges
Leng, J., Zheng, K., Li, R. C., Chen, C., Wang, B., Liu, Q., Chen, X., & Shen, W. (2025) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2025.103076 · Cited by 70 · see citing works
A Survey of Deep Learning for Mathematical Reasoning
Lu, P., Qiu, L., Yu, W., Welleck, S., & Chang, K.-W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.817 · Cited by 84 · see citing works
Retrieving Multimodal Information for Augmented Generation: A Survey
Zhao, R., Chen, H., Wang, W., Jiao, F., Long, X., Qin, C., Ding, B., Guo, X., Li, M., Li, X., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.314 · Cited by 66 · see citing works
Code4Struct: Code Generation for Few-Shot Event Structure Prediction
Wang, X., Li, S., & Ji, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.202 · Cited by 63 · see citing works
Extracting Financial Data from Unstructured Sources: Leveraging Large Language Models
Li, H., Gao, H., Wu, C., & Vasarhelyi, M. A. (2024) · Journal of Information Systems · doi:10.2308/isys-2023-047 · Cited by 71 · see citing works
DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., Vardhamanan, S., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.03714 · Cited by 60 · see citing works
“Task inconsistencies between text and evidence (Gao et al., 2023a).”
BackgroundMethodFoundation Models for Decision Making: Problems, Methods, and Opportunities
Yang, S., Nachum, O., Du, Y., Wei, J., Abbeel, P., & Schuurmans, D. (2023) · arXiv · doi:10.48550/arxiv.2303.04129 · Cited by 60 · see citing works
Fact-Checking Complex Claims with Program-Guided Reasoning
Pan, L., Wu, X., Lu, X., Luu, A. T., Wang, W. Y., Kan, M., & Nakov, P. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.386 · Cited by 78 · see citing works
Efficient Prompting for LLM-Based Generative Internet of Things
Xiao, B., Kantarcı, B., Kang, J., Niyato, D. T., & Guizani, M. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3470210 · Cited by 60 · see citing works
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Duan, N., & Chen, W. (2023) · arXiv · doi:10.48550/arxiv.2305.11738 · Cited by 57 · see citing works
“We can teach the LLMs to use tools by pre-training [102], fine-tuning [93, 103], or in-context learning [97, 101].”
BackgroundMethodSelf-citationOlausson, T., Gu, A., Lipkin, B., Zhang, C., Solar-Lezama, A., Tenenbaum, J., & Levy, R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.313 · Cited by 57 · see citing works
MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering
Liu, F., Piccinno, F., Krichene, S., Pang, C., Lee, K., Joshi, M., Altün, Y., Collier, N., & Eisenschlos, J. M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.714 · Cited by 79 · see citing works
Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization
Shen, C., Cheng, L., Nguyen, X.-P., You, Y., & Bing, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.278 · Cited by 71 · see citing works
Towards Autonomous Testing Agents via Conversational Large Language Models
Feldt, R., Kang, S., Yoon, J., & Yoo, S. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00148 · Cited by 49 · see citing works
Shi, W., Xu, R., Zhuang, Y., Yu, Y., Zhang, J., Wu, H., Zhu, Y., Ho, J. C., Yang, C., & Wang, M. D. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.1245 · Cited by 54 · see citing works
“Alternatively, an autonomous agent could facilitate clinicians to communicate with EHRs in nat-ural languages, translating clinical questions into machine-interpretable queries, planning a sequence of actions, and ultimately delivering the final responses.”
BackgroundMethodInfluentialNie, T., Sun, J., & Ma, W. (2025) · Artificial Intelligence for Transportation · doi:10.1016/j.ait.2025.100003 · Cited by 56 · see citing works
Batch Prompting: Efficient Inference with Large Language Model APIs
Cheng, Z., Kasai, J., & Yu, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.74 · Cited by 54 · see citing works
Romero, O. J., Zimmerman, J., Steinfeld, A. M., & Tomasic, A. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v2i1.27706 · Cited by 44 · see citing works
Oniani, D., Wu, X., Visweswaran, S., Kapoor, S., Kooragayalu, S., Polanska, K., & Wang, Y. (2024) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi61247.2024.00111 · Cited by 47 · see citing works
Explainable Claim Verification via Knowledge-Grounded Reasoning with Large Language Models
Wang, H., & Shu, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.416 · Cited by 49 · see citing works
Teaching Code LLMs to Use Autocompletion Tools in Repository-Level Code Generation
Wang, C., Zhang, J., Feng, Y., Li, T., Sun, W., Liu, Y., & Peng, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3714462 · Cited by 40 · see citing works
You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects
Bouzenia, I., & Pradel, M. (2025) · Proceedings of the ACM on Software Engineering · doi:10.1145/3728922 · Cited by 35 · see citing works
LLM4CAD: Multimodal Large Language Models for Three-Dimensional Computer-Aided Design Generation
Li, X., Sun, Y., & Sha, Z. (2024) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4067085 · Cited by 41 · see citing works
Fernandes, P., Deutsch, D., Finkelstein, M., Riley, P., Martins, A. F. T., Neubig, G., Garg, A., Clark, J. H., Freitag, M., & Fırat, O. (2023) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2023.wmt-1.100 · Cited by 46 · see citing works
Self-citationCross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages
Qin, L., Chen, Q., Wei, F., Huang, S., & Che, W. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.163 · Cited by 50 · see citing works
A comprehensive overview of Generative AI (GAI): Technologies, applications, and challenges
Zhang, Z., Zhang, J., Zhang, X., & Mai, W. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129645 · Cited by 37 · see citing works
TheoremQA: A Theorem-driven Question Answering Dataset
Chen, W., Yin, M., Ku, M., Lu, P., Wan, Y., Ma, X., Xu, J., Wang, X., & Xia, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.489 · Cited by 34 · see citing works
Liang, Y., Wang, J., Zhu, H., Wang, L., Qian, W., & Lan, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.263 · Cited by 37 · see citing works
Evaluating Diverse Large Language Models for Automatic and General Bug Reproduction
Kang, S., Yoon, J., Askarbekkyzy, N., & Yoo, S. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3450837 · Cited by 33 · see citing works
Domain-specific chatbots for science using embeddings
Yager, K. G. (2023) · Digital Discovery · doi:10.1039/d3dd00112a · Cited by 31 · see citing works
Large language models as tool makers
Cai, T., Wang, X., Ma, T., Chen, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2305.17126 · Cited by 28 · see citing works
Xavier, C., Rodrigues, L., da Costa, N. T., Neto, R., Alves, G., Falcão, T. P., Gašević, D., & Mello, R. F. (2025) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2025.3562379 · Cited by 30 · see citing works
ChatCoT: Tool-Augmented Chain-of-Thought Reasoning on Chat-based Large Language Models
Chen, Z., Zhou, K., Zhang, B., Gong, Z., Zhao, X., & Wen, J.-R. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.985 · Cited by 30 · see citing works
Reliable Natural Language Understanding with Large Language Models and Answer Set Programming
Rajasekharan, A., Zeng, Y., Padalkar, P., & Gupta, G. (2023) · Electronic Proceedings in Theoretical Computer Science · doi:10.4204/eptcs.385.27 · Cited by 26 · see citing works
Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Models
Sarch, G., Wu, Y., Tarr, M., & Fragkiadaki, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.226 · Cited by 25 · see citing works
Chern, I.-C., Chern, S., Chen, S., Yuan, W., Feng, K., Zhou, C., He, J., Neubig, G., & Liu, P. (2023) · arXiv · doi:10.48550/arxiv.2307.13528 · Cited by 29 · see citing works
“For response annotation, we utilize the target value provided in GSMHard (Gao et al., 2022b) to annotate the generated responses.”
BackgroundMethodInfluentialSelf-citationCREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models
Qian, C., Han, C., Fung, Y., Qin, Y., Liu, Z., & Ji, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.462 · Cited by 24 · see citing works
An Investigation of Applying Large Language Models to Spoken Language Learning
Gao, Y., Nuchged, B., Li, Y., & Peng, L. (2023) · Applied Sciences · doi:10.3390/app14010224 · Cited by 22 · see citing works
Natural Language to Code Generation in Interactive Data Science Notebooks
Yin, P., Li, W.-D., Xiao, K., Rao, A., Wen, Y., Shi, K., Howland, J., Bailey, P., Catasta, M., Michalewski, H., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.9 · Cited by 22 · see citing works
Tab-CoT: Zero-shot Tabular Chain of Thought
Jin, Z., & Lü, W. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.651 · Cited by 22 · see citing works
Is GPT-4 fair? An empirical analysis in automatic short answer grading
Rodrigues, L., Xavier, C., da Costa, N. T., Gašević, D., & Mello, R. F. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100428 · Cited by 26 · see citing works
Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communication
Yin, Z., Sun, Q., Chang, C., Guo, Q., Dai, J., Huang, X., & Qiu, X. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.936 · Cited by 26 · see citing works
GestureGPT: Toward Zero-Shot Free-Form Hand Gesture Understanding with Large Language Model Agents
Zeng, X., Wang, X., Zhang, T., Yu, C., Zhao, S., & Chen, Y. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3698145 · Cited by 37 · see citing works
L2CEval : Evaluating Language-to-Code Generation Capabilities of Large Language Models
Ni, A., Yin, P., Zhao, Y., Riddell, M., Feng, T., Shen, R., Yin, S., Ye, L., Yavuz, S., Xiong, C., et al. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00705 · Cited by 31 · see citing works
Adaptive-solver framework for dynamic strategy selection in large language model reasoning
Zhou, J., Zhong, W., Wang, Y., & Wang, J. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2024.104052 · Cited by 19 · see citing works
Confucius: Iterative Tool Learning from Introspection Feedback by Easy-to-Difficult Curriculum
Gao, S., Shi, Z., Zhu, M., Fang, B., Xin, X., Ren, P., Chen, Z., Ma, J., & Ren, Z. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29759 · Cited by 19 · see citing works
Let’s Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs
Aggarwal, P., Madaan, A., Yang, Y., & Mausam, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.761 · Cited by 29 · see citing works
Self-citationIntegration of cognitive tasks into artificial general intelligence test for large models
Qu, Y., Wei, C., Du, P., Che, W., Zhang, C., Ouyang, W., Bian, Y., Xu, F., Hu, B., Du, K., et al. (2024) · iScience · doi:10.1016/j.isci.2024.109550 · Cited by 18 · see citing works
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · doi:10.31219/osf.io/ac4sp · Cited by 51 · see citing works
OctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 17 · see citing works
“Model Execution A promising avenue for improving performance on HUMANEVALFIX is letting the model execute the given code or its own generated code and inspect its output (Chen et al., 2022; 2023c; Yasunaga & Liang, 2021; Li et al., 2022a; Gao et al., 2023; Dong et al., 2023; Zhang et al., 2023c; Madaan et al., 2023b;…”
BackgroundLogiCoT: Logical Chain-of-Thought Instruction Tuning
Liu, H., Teng, Z., Cui, L., Zhang, C., Zhou, Q., & Zhang, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.191 · Cited by 21 · see citing works
Retrieval-Augmented Generation and LLM Agents for Biomimicry Design Solutions
Toukmaji, C., & Tee, A. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v3i1.31210 · Cited by 17 · see citing works
Wu, H., Zhang, Y., Han, Z., Hou, Y., Wang, L., Liu, S., Gong, Q., & Ge, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113057 · Cited by 17 · see citing works
Automatic Model Selection with Large Language Models for Reasoning
Zhao, J., Xie, Y., Kawaguchi, K., He, J., & Xie, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.55 · Cited by 13 · see citing works
Generative AI as Economic Agents
Immorlica, N., Lucier, B., & Slivkins, A. (2024) · ACM SIGecom Exchanges · doi:10.1145/3699824.3699832 · Cited by 13 · see citing works
Arithmetic Reasoning with LLM: Prolog Generation & Permutation
Yang, X., Chen, B., & Tam, Y.-C. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers) · doi:10.18653/v1/2024.naacl-short.61 · Cited by 12 · see citing works
Yager, K. G. (2024) · Digital Discovery · doi:10.1039/d4dd00178h · Cited by 12 · see citing works
Bandara, E., Foytik, P. B., Shetty, S. S., Mukkamala, R., Rahman, A., Liang, X., Ng, W. K., & Zoysa, K. D. (2024) · IEEE Consumer Communications and Networking Conference · doi:10.1109/ccnc51664.2024.10454701 · Cited by 14 · see citing works
Das, A., Chen, S., Shyu, M., & Sadiq, S. B. (2023) · IEEE International Conference on Collaboration and Internet Computing (CIC) · doi:10.1109/cic58953.2023.00021 · Cited by 11 · see citing works
Knowledge-Centered Dual-Process Reasoning for Math Word Problems With Large Language Models
Liu, J., Huang, Z., Liu, Q., Ma, Z., Zhai, C., & Chen, E. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3556367 · Cited by 11 · see citing works
Plan, Verify and Switch: Integrated Reasoning with Diverse X-of-Thoughts
Liu, T., Guo, Q., Yang, Y., Hu, X., Zhang, Y., Qiu, X., & Zhang, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.169 · Cited by 12 · see citing works
Self-citationAgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases
Chen, Z., Xiang, Z., Xiao, C., Song, D., & Li, B. (2024) · arXiv · doi:10.48550/arxiv.2407.12784 · Cited by 10 · see citing works
“The LLM agent will execute the generated action by calling build-in tools [9] or external APIs.”
- Authors
- Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., & Neubig, G.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2211.10435
- OpenAlex
- W4309591663
- Cited by
- 155
- References
- 0
- Citations by year
- 2026: 3 · 2025: 28 · 2024: 48 · 2023: 76
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (public-domain)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, code generation, Python interpreter, symbolic reasoning, mathematical reasoning, few-shot prompting, chain-of-thought, mathematical word problems, GSM8K
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
