The DOI matches its DataCite record; 37 of the 37 references it cites were found in a registry or an index.
PAL: Program-aided Language Models
Open accessAbstract
Large language models (LLMs) have recently demonstrated an impressive ability to perform arithmetic and symbolic reasoning tasks, when provided with a few examples at test time ("few-shot prompting"). Much of this success can be attributed to prompting methods such as "chain-of-thought'', which employ LLMs for both understanding the problem description by decomposing it into steps, as well as solving each step of the problem. While LLMs seem to be adept at this sort of step-by-step decomposition, LLMs often make logical and arithmetic mistakes in the solution part, even when the problem is decomposed correctly. In this paper, we present Program-Aided Language models (PAL): a novel approach that uses the LLM to read natural language problems and generate programs as the intermediate reasoning steps, but offloads the solution step to a runtime such as a Python interpreter. With PAL, decomposing the natural language problem into runnable steps remains the only learning task for the LLM, while solving is delegated to the interpreter. We demonstrate this synergy between a neural LLM and a symbolic interpreter across 13 mathematical, symbolic, and algorithmic reasoning tasks from BIG-Bench Hard and other benchmarks. In all these natural language reasoning tasks, generating code using an LLM and reasoning using a Python interpreter leads to more accurate results than much larger models. For example, PAL using Codex achieves state-of-the-art few-shot accuracy on the GSM8K benchmark of math word problems, surpassing PaLM-540B which uses chain-of-thought by absolute 15% top-1. Our code and data are publicly available at http://reasonwithpal.com/ .
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“or limit tool use to task-specific settings only (e.g., Gao et al., 2022;Parisi et al., 2022)”
From Toolformer: Language Models Can Teach Themselves to Use Tools (2023) · cites this paper as Gao et al., 2022
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A promising design paradigm for AI agents [65] is to combine an LLM with tools that interact with a broader environment [14, 35, 40, 47, 51, 53, 55, 69].”
From AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (2024) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Programs are increasingly used as the medium through which LLM agents reason, act, and model their environments.”
From Code as Agent Harness (2026) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Early works [66, 7] demonstrate that delegating computation to programs substantially improves reliability by moving intermediate reasoning into structured, verifiable execution traces.”
From Code as Agent Harness (2026) · cites this paper as [7]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Program-aided reasoning methods externalize intermediate computation into executable code [6, 7, 8]”
From Code as Agent Harness (2026) · cites this paper as [7]
The cited source says what the claim attributes to it.
In the citing paper
“On the other hand, large language models have been shown to be capable of describing their generated problem solutions in both text (Wei et al., 2022;Kojima et al., 2022;Zhou et al., 2023) and code (Gao et al., 2022;Chen et al., 2022) formats.”
From Teaching Large Language Models to Self-Debug (2023) · cites this paper as Gao et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“While language models are often effective at proposing reasoning steps, they remain unreliable at faithfully carrying out symbolic, logical, or arithmetic computation [7].”
From Code as Agent Harness (2026) · cites this paper as [7]
The cited source says what the claim attributes to it.
In the citing paper
“For example, some subsample instances from existing benchmarks that cannot be solved by current models [93, 82], or augment them to construct more challenging evaluations [21, 44, 49].”
From SciCode: A Research Coding Benchmark Curated by Scientists (2024) · cites this paper as [21]
The cited source says what the claim attributes to it.
In the citing paper
“This motivates an established alternative-the tool-use paradigm [9, 10]-in which, rather than perform every reasoning step internally, a model delegates well-specified tasks to external tools.”
From PrologMCP: A Standardized Prolog Tool Interface for LLM Agents (2026) · cites this paper as [10]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 7
- Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language Models are Few-Shot Learners. In NeurIPS , 2020.
- Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021b. · arXiv
- Demeter, D. and Downey, D. Just add functions: A neuralsymbolic language model. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pp. 7634-7642, 2020.
- Gellenbeck, E. M. and Cook, C. R. An investigation of procedure and variable names as beacons during program comprehension. In Empirical studies of programmers: Fourth workshop , pp. 65-81. Ablex Publishing, Norwood, NJ, 1991.
- Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the MATH dataset, 2021. URL https://openreview.net/forum? id=7Bywt2mQsCe .
- Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The Curious Case of Neural Text Degeneration. In ICLR , 2019.
- Takang, A. A., Grubb, P. A., and Macredie, R. D. The effects of comments and identifier names on program comprehensibility: an experimental investigation. J. Prog. Lang. , 4(3):143-167, 1996.
Cited by
Sorted by citations
Toolformer: Language Models Can Teach Themselves to Use Tools
Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023) · arXiv · doi:10.48550/arxiv.2302.04761 · Cited by 1,150
Unleashing the potential of prompt engineering for large language models
CHEN, B. ‐., Zhang, Z., Langrené, N., & Zhu, S. (2025) · Patterns · doi:10.1016/j.patter.2025.101260 · Cited by 1,097
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505
Towards Reasoning in Large Language Models: A Survey
Huang, J., & Chang, K. C.-C. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.67 · Cited by 428
When large language models meet personalization: perspectives of challenges and opportunities
Chen, J., Liu, Z., Huang, X., Wu, C., Liu, Q., Jiang, G., Pu, Y., Lei, Y., Chen, X., Wang, X., et al. (2024) · World Wide Web · doi:10.1007/s11280-024-01276-1 · Cited by 367
Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health
Tian, S., Jin, Q., Yeganova, L., Lai, P.-T., Zhu, Q., Chen, X., Yang, Y., Chen, Q., Kim, W., Comeau, D. C., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad493 · Cited by 359
Self-citation(2023) · doi:10.18653/v1/2023.emnlp-main.574 · Cited by 249
ViperGPT: Visual Inference via Python Execution for Reasoning
Surís, D., Menon, S., & Vondrick, C. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01092 · Cited by 273
Self-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 2,314
“We select two examples from the training set on which CODEX fails when prompted with PaL-styled prompts, and manually write the correct solution (yi+1) and reasoning (fbi) for REFINE and FEEDBACK.”
MethodSelf-citationEnhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy
Shao, Z., Gong, Y., Shen, Y., Huang, M., Duan, N., & Chen, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.620 · Cited by 200
Wang, M., Wang, M., Xu, X., Yang, L., Cai, D., & Yin, M. (2023) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2023.3324714 · Cited by 199
(2025) · doi:10.1109/icse55347.2025.00157 · Cited by 157
Hallucination Mitigation for Retrieval-Augmented Large Language Models: A Review
Zhang, W., & Zhang, J. (2025) · Mathematics · doi:10.3390/math13050856 · Cited by 145
Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
Pan, L., Albalak, A., Wang, X., & Wang, W. Y. (2023) · arXiv · doi:10.48550/arxiv.2305.12295 · Cited by 144
“, 2023) or a Python interpreter (Gao et al., 2022; Chen et al., 2022) into LLMs can greatly enhance performance by offloading numerical calculations.”
BackgroundReasoning with Language Model Prompting: A Survey
Qiao, S., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S., Tan, C., Huang, F., & Chen, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.294 · Cited by 346
Large Language Models Can Be Easily Distracted by Irrelevant Context
Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E., Schärli, N., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2302.00093 · Cited by 134
“Following prior work on solving GSM8K problems with code (Chowdhery et al., 2022; Gao et al., 2022; Chen et al., 2022), we include a Python program as the problem solution in the prompt, and execute the generated Python code using an external Python interpreter to obtain the final answer.”
MethodResultTowards Faithful Model Explanation in NLP: A Survey
Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024) · Computational Linguistics · doi:10.1162/coli_a_00511 · Cited by 196
Cui, C., Ma, Y., Cao, X., Ye, W., & Wang, Z. (2024) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00101 · Cited by 156
Faithful Chain-of-Thought Reasoning
Lyu, Q., Havaldar, S., Stein, A., Zhang, L., Rao, D., Wong, E., Apidianaki, M., & Callison-Burch, C. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.20 · Cited by 201
Dibia, V. C. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.11 · Cited by 170
Prompt Engineering for Healthcare: Methodologies and Applications
Wang, J., Shi, E., Yu, S., Wu, Z., Ma, C., Dai, H., Yang, Q., Kang, Y., Wu, J., Hu, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.14670 · Cited by 150
Receive, Reason, and React: Drive as You Say, With Large Language Models in Autonomous Vehicles
Cui, C., Ma, Y., Cao, X., Ye, W., & Wang, Z. (2024) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2024.3381793 · Cited by 137
TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs
Liang, Y., Wu, C., Song, T., Wu, W., Xia, Y., Liu, Y., Ou, Y., Lu, S., Ji, L., Mao, S., et al. (2024) · Intelligent Computing · doi:10.34133/icomputing.0063 · Cited by 108
Recent Advances in Large Language Models for Healthcare
Nassiri, K., & Akhloufi, M. A. (2024) · BioMedInformatics · doi:10.3390/biomedinformatics4020062 · Cited by 108
Exploring the potential of using ChatGPT in physics education
Liang, Y., Zou, D., Xie, H., & Wang, F. L. (2023) · Smart Learning Environments · doi:10.1186/s40561-023-00273-7 · Cited by 98
An Overview on IEEE 802.11bf: WLAN Sensing
Du, R. Y., Hua, H., Xie, H., Song, X., Lyu, Z., Hu, M., Narengerile, Xin, Y., McCann, S., Montemurro, M., et al. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3408899 · Cited by 132
Zhang, Q., Xu, C., Li, J., Sun, Y., Bao, J., & Zhang, D. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125861 · Cited by 105
Gorilla: Large Language Model Connected with Massive APIs
Patil, S. G., Zhang, T., Wang, X., & Gonzalez, J. E. (2023) · arXiv · doi:10.48550/arxiv.2305.15334 · Cited by 226
Generalized Planning in PDDL Domains with Pretrained Large Language Models
Silver, T., Dan, S., Srinivas, K., Tenenbaum, J. B., Kaelbling, L. P., & Katz, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i18.30006 · Cited by 150
Teaching Large Language Models to Self-Debug
Chen, X., Lin, M., Schärli, N., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2304.05128 · Cited by 812
“Several prompting methods have been proposed to unlock latent abilities of large language models (Wei et al., 2022; Nye et al., 2021; Kojima et al., 2022; Zhou et al., 2023; Khot et al., 2022; Zhou et al., 2022; Gao et al., 2022; Chen et al., 2022).”
BackgroundAre Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond
Xu, F., Lin, Q., Han, J., Zhao, T., Liu, J., & Cambria, E. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3536008 · Cited by 164
Few-shot In-context Learning on Knowledge Base Question Answering
Li, T., Ma, X., Zhuang, A., Gu, Y., Su, Y., & Chen, W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.385 · Cited by 126
Assessing LLMs in malicious code deobfuscation of real-world malware campaigns
Patsakis, C., Casino, F., & Lykousas, N. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124912 · Cited by 92
A Survey of Robot Intelligence with Large Language Models
Jeong, H., Lee, H., Kim, C., & Shin, S. (2024) · Applied Sciences · doi:10.3390/app14198868 · Cited by 89
DePlot: One-shot visual language reasoning by plot-to-table translation
Liu, F., Eisenschlos, J. M., Piccinno, F., Krichene, S., Pang, C., Lee, K., Joshi, M., Chen, W., Collier, N., & Altün, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.660 · Cited by 101
UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning
Masry, A., Kavehzadeh, P., Long, X., Hoque, E., & Joty, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.906 · Cited by 186
LLM for Test Script Generation and Migration: Challenges, Capabilities, and Opportunities
Yu, S., Fang, C., Ling, Y., Wu, C., & Chen, Z. (2023) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs60937.2023.00029 · Cited by 100
Discuss Before Moving: Visual Language Navigation via Multi-expert Discussions
Long, Y., Li, X., Cai, W., & Dong, H. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611565 · Cited by 115
AIGC-empowered smart manufacturing: Prospects and challenges
Leng, J., Zheng, K., Li, R. C., Chen, C., Wang, B., Liu, Q., Chen, X., & Shen, W. (2025) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2025.103076 · Cited by 74
A Survey of Deep Learning for Mathematical Reasoning
Lu, P., Qiu, L., Yu, W., Welleck, S., & Chang, K.-W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.817 · Cited by 145
Retrieving Multimodal Information for Augmented Generation: A Survey
Zhao, R., Chen, H., Wang, W., Jiao, F., Long, X., Qin, C., Ding, B., Guo, X., Li, M., Li, X., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.314 · Cited by 75
Code4Struct: Code Generation for Few-Shot Event Structure Prediction
Wang, X., Li, S., & Ji, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.202 · Cited by 128
Extracting Financial Data from Unstructured Sources: Leveraging Large Language Models
Li, H., Gao, H., Wu, C., & Vasarhelyi, M. A. (2024) · Journal of Information Systems · doi:10.2308/isys-2023-047 · Cited by 76
DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., Vardhamanan, S., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.03714 · Cited by 911
“Task inconsistencies between text and evidence (Gao et al., 2023a).”
BackgroundMethodFoundation Models for Decision Making: Problems, Methods, and Opportunities
Yang, S., Nachum, O., Du, Y., Wei, J., Abbeel, P., & Schuurmans, D. (2023) · arXiv · doi:10.48550/arxiv.2303.04129 · Cited by 60
Fact-Checking Complex Claims with Program-Guided Reasoning
Pan, L., Wu, X., Lu, X., Luu, A. T., Wang, W. Y., Kan, M., & Nakov, P. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.386 · Cited by 130
Efficient Prompting for LLM-Based Generative Internet of Things
Xiao, B., Kantarcı, B., Kang, J., Niyato, D. T., & Guizani, M. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3470210 · Cited by 77
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Duan, N., & Chen, W. (2023) · arXiv · doi:10.48550/arxiv.2305.11738 · Cited by 561
“We can teach the LLMs to use tools by pre-training [102], fine-tuning [93, 103], or in-context learning [97, 101].”
BackgroundMethodSelf-citationOlausson, T., Gu, A., Lipkin, B., Zhang, C., Solar-Lezama, A., Tenenbaum, J., & Levy, R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.313 · Cited by 57
MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering
Liu, F., Piccinno, F., Krichene, S., Pang, C., Lee, K., Joshi, M., Altün, Y., Collier, N., & Eisenschlos, J. M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.714 · Cited by 79
Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization
Shen, C., Cheng, L., Nguyen, X.-P., You, Y., & Bing, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.278 · Cited by 112
Towards Autonomous Testing Agents via Conversational Large Language Models
Feldt, R., Kang, S., Yoon, J., & Yoo, S. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00148 · Cited by 49
Shi, W., Xu, R., Zhuang, Y., Yu, Y., Zhang, J., Wu, H., Zhu, Y., Ho, J. C., Yang, C., & Wang, M. D. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.1245 · Cited by 54
“Alternatively, an autonomous agent could facilitate clinicians to communicate with EHRs in nat-ural languages, translating clinical questions into machine-interpretable queries, planning a sequence of actions, and ultimately delivering the final responses.”
BackgroundMethodInfluentialNie, T., Sun, J., & Ma, W. (2025) · Artificial Intelligence for Transportation · doi:10.1016/j.ait.2025.100003 · Cited by 70
Batch Prompting: Efficient Inference with Large Language Model APIs
Cheng, Z., Kasai, J., & Yu, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.74 · Cited by 95
Romero, O. J., Zimmerman, J., Steinfeld, A. M., & Tomasic, A. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v2i1.27706 · Cited by 56
Oniani, D., Wu, X., Visweswaran, S., Kapoor, S., Kooragayalu, S., Polanska, K., & Wang, Y. (2024) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi61247.2024.00111 · Cited by 57
Explainable Claim Verification via Knowledge-Grounded Reasoning with Large Language Models
Wang, H., & Shu, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.416 · Cited by 83
Teaching Code LLMs to Use Autocompletion Tools in Repository-Level Code Generation
Wang, C., Zhang, J., Feng, Y., Li, T., Sun, W., Liu, Y., & Peng, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3714462 · Cited by 40
You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects
Bouzenia, I., & Pradel, M. (2025) · Proceedings of the ACM on Software Engineering · doi:10.1145/3728922 · Cited by 35
LLM4CAD: Multimodal Large Language Models for Three-Dimensional Computer-Aided Design Generation
Li, X., Sun, Y., & Sha, Z. (2024) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4067085 · Cited by 56
Fernandes, P., Deutsch, D., Finkelstein, M., Riley, P., Martins, A. F. T., Neubig, G., Garg, A., Clark, J. H., Freitag, M., & Fırat, O. (2023) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2023.wmt-1.100 · Cited by 92
Self-citationCross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages
Qin, L., Chen, Q., Wei, F., Huang, S., & Che, W. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.163 · Cited by 89
A comprehensive overview of Generative AI (GAI): Technologies, applications, and challenges
Zhang, Z., Zhang, J., Zhang, X., & Mai, W. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129645 · Cited by 45
TheoremQA: A Theorem-driven Question Answering Dataset
Chen, W., Yin, M., Ku, M., Lu, P., Wan, Y., Ma, X., Xu, J., Wang, X., & Xia, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.489 · Cited by 141
Liang, Y., Wang, J., Zhu, H., Wang, L., Qian, W., & Lan, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.263 · Cited by 46
Evaluating Diverse Large Language Models for Automatic and General Bug Reproduction
Kang, S., Yoon, J., Askarbekkyzy, N., & Yoo, S. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3450837 · Cited by 40
Domain-specific chatbots for science using embeddings
Yager, K. G. (2023) · Digital Discovery · doi:10.1039/d3dd00112a · Cited by 35
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.15877 · Cited by 29
“We believe that program-aided language models [20] for task completion and reasoning provide the possibility towards artificial general intelligence.”
BackgroundLarge language models as tool makers
Cai, T., Wang, X., Ma, T., Chen, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2305.17126 · Cited by 28
Xavier, C., Rodrigues, L., da Costa, N. T., Neto, R., Alves, G., Falcão, T. P., Gašević, D., & Mello, R. F. (2025) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2025.3562379 · Cited by 35
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
Snell, C., Lee, J., Xu, K., & Kumar, A. (2024) · arXiv · doi:10.48550/arxiv.2408.03314 · Cited by 27
ChatCoT: Tool-Augmented Chain-of-Thought Reasoning on Chat-based Large Language Models
Chen, Z., Zhou, K., Zhang, B., Gong, Z., Zhao, X., & Wen, J.-R. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.985 · Cited by 30
Reliable Natural Language Understanding with Large Language Models and Answer Set Programming
Rajasekharan, A., Zeng, Y., Padalkar, P., & Gupta, G. (2023) · Electronic Proceedings in Theoretical Computer Science · doi:10.4204/eptcs.385.27 · Cited by 26
Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communication
Yin, Z., Sun, Q., Chang, C., Guo, Q., Dai, J., Huang, X., & Qiu, X. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.936 · Cited by 26
Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Models
Sarch, G., Wu, Y., Tarr, M., & Fragkiadaki, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.226 · Cited by 25
Chern, I.-C., Chern, S., Chen, S., Yuan, W., Feng, K., Zhou, C., He, J., Neubig, G., & Liu, P. (2023) · arXiv · doi:10.48550/arxiv.2307.13528 · Cited by 29
“For response annotation, we utilize the target value provided in GSMHard (Gao et al., 2022b) to annotate the generated responses.”
BackgroundMethodInfluentialSelf-citationCREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models
Qian, C., Han, C., Fung, Y., Qin, Y., Liu, Z., & Ji, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.462 · Cited by 24
An Investigation of Applying Large Language Models to Spoken Language Learning
Gao, Y., Nuchged, B., Li, Y., & Peng, L. (2023) · Applied Sciences · doi:10.3390/app14010224 · Cited by 35
Natural Language to Code Generation in Interactive Data Science Notebooks
Yin, P., Li, W.-D., Xiao, K., Rao, A., Wen, Y., Shi, K., Howland, J., Bailey, P., Catasta, M., Michalewski, H., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.9 · Cited by 74
Tab-CoT: Zero-shot Tabular Chain of Thought
Jin, Z., & Lü, W. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.651 · Cited by 28
Is GPT-4 fair? An empirical analysis in automatic short answer grading
Rodrigues, L., Xavier, C., da Costa, N. T., Gašević, D., & Mello, R. F. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100428 · Cited by 26
GestureGPT: Toward Zero-Shot Free-Form Hand Gesture Understanding with Large Language Model Agents
Zeng, X., Wang, X., Zhang, T., Yu, C., Zhao, S., & Chen, Y. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3698145 · Cited by 41
L2CEval : Evaluating Language-to-Code Generation Capabilities of Large Language Models
Ni, A., Yin, P., Zhao, Y., Riddell, M., Feng, T., Shen, R., Yin, S., Ye, L., Yavuz, S., Xiong, C., et al. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00705 · Cited by 49
Adaptive-solver framework for dynamic strategy selection in large language model reasoning
Zhou, J., Zhong, W., Wang, Y., & Wang, J. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2024.104052 · Cited by 19
Confucius: Iterative Tool Learning from Introspection Feedback by Easy-to-Difficult Curriculum
Gao, S., Shi, Z., Zhu, M., Fang, B., Xin, X., Ren, P., Chen, Z., Ma, J., & Ren, Z. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29759 · Cited by 31
A Careful Examination of Large Language Model Performance on Grade School Arithmetic
Zhang, H., Da, J., Lee, D., Robinson, V., Wu, C., Song, W., Zhao, T., Raja, P., Zhuang, C., Slack, D., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.00332 · Cited by 19
Let’s Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs
Aggarwal, P., Madaan, A., Yang, Y., & Mausam, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.761 · Cited by 106
Self-citationIntegration of cognitive tasks into artificial general intelligence test for large models
Qu, Y., Wei, C., Du, P., Che, W., Zhang, C., Ouyang, W., Bian, Y., Xu, F., Hu, B., Du, K., et al. (2024) · iScience · doi:10.1016/j.isci.2024.109550 · Cited by 20
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ac4sp · Cited by 143
OctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 245
“Model Execution A promising avenue for improving performance on HUMANEVALFIX is letting the model execute the given code or its own generated code and inspect its output (Chen et al., 2022; 2023c; Yasunaga & Liang, 2021; Li et al., 2022a; Gao et al., 2023; Dong et al., 2023; Zhang et al., 2023c; Madaan et al., 2023b;…”
BackgroundLogiCoT: Logical Chain-of-Thought Instruction Tuning
Liu, H., Teng, Z., Cui, L., Zhang, C., Zhou, Q., & Zhang, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.191 · Cited by 37
Retrieval-Augmented Generation and LLM Agents for Biomimicry Design Solutions
Toukmaji, C., & Tee, A. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v3i1.31210 · Cited by 18
ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models
Xu, B., Peng, Z., Lei, B., Mukherjee, S., Liu, Y., & Xu, D. (2023) · arXiv · doi:10.48550/arxiv.2305.18323 · Cited by 17
Wu, H., Zhang, Y., Han, Z., Hou, Y., Wang, L., Liu, S., Gong, Q., & Ge, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113057 · Cited by 21
Automatic Model Selection with Large Language Models for Reasoning
Zhao, J., Xie, Y., Kawaguchi, K., He, J., & Xie, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.55 · Cited by 39
Generative AI as Economic Agents
Immorlica, N., Lucier, B., & Slivkins, A. (2024) · ACM SIGecom Exchanges · doi:10.1145/3699824.3699832 · Cited by 19
Arithmetic Reasoning with LLM: Prolog Generation & Permutation
Yang, X., Chen, B., & Tam, Y.-C. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers) · doi:10.18653/v1/2024.naacl-short.61 · Cited by 12
Yager, K. G. (2024) · Digital Discovery · doi:10.1039/d4dd00178h · Cited by 12
Bandara, E., Foytik, P. B., Shetty, S. S., Mukkamala, R., Rahman, A., Liang, X., Ng, W. K., & Zoysa, K. D. (2024) · IEEE Consumer Communications and Networking Conference · doi:10.1109/ccnc51664.2024.10454701 · Cited by 15
Metadata
- Authors
- Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., & Neubig, G.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2211.10435
- OpenAlex
- W4309591663
- Cited by
- 155
- References
- 0
- Citations by year
- 2026: 3 · 2025: 28 · 2024: 48 · 2023: 76
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (public-domain)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, code generation, Python interpreter, symbolic reasoning, mathematical reasoning, few-shot prompting, chain-of-thought, mathematical word problems, GSM8K
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
