The DOI matches its DataCite record; 92 of the 93 references it cites were found in a registry or an index, and 1 are not checked yet.
Program Synthesis with Large Language Models
Open accessAbstract
This paper explores the limits of the current generation of large language models for program synthesis in general purpose programming languages. We evaluate a collection of such models (with between 244M and 137B parameters) on two new benchmarks, MBPP and MathQA-Python, in both the few-shot and fine-tuning regimes. Our benchmarks are designed to measure the ability of these models to synthesize short Python programs from natural language descriptions. The Mostly Basic Programming Problems (MBPP) dataset contains 974 programming tasks, designed to be solvable by entry-level programmers. The MathQA-Python dataset, a Python version of the MathQA benchmark, contains 23914 problems that evaluate the ability of the models to synthesize code from more complex text. On both datasets, we find that synthesis performance scales log-linearly with model size. Our largest models, even without finetuning on a code dataset, can synthesize solutions to 59.6 percent of the problems from MBPP using few-shot learning with a well-designed prompt. Fine-tuning on a held-out portion of the dataset improves performance by about 10 percentage points across most model sizes. On the MathQA-Python dataset, the largest fine-tuned model achieves 83.8 percent accuracy. Going further, we study the model's ability to engage in dialog about code, incorporating human feedback to improve its solutions. We find that natural language feedback from a human halves the error rate compared to the model's initial prediction. Additionally, we conduct an error analysis to shed light on where these models fall short and what types of programs are most difficult to generate. Finally, we explore the semantic grounding of these models by fine-tuning them to predict the results of program execution. We find that even our best models are generally unable to predict the output of a program given a specific input.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“In practice, large pools of test cases, Dj , can either originate from human generated documentation (Nassif et al., 2021;Lemieux et al., 2023) or GPTgenerated unit tests (Austin et al., 2021).”
From Does Few-Shot Learning Help LLM Performance in Code Synthesis? (2024) · cites this paper as Austin et al., 2021
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Code data (e.g., from Stack Exchange and GitHub) substantially enhances program synthesis capabilities [37,38].”
From A Survey of Large Language Models (2026) · cites this paper as [38]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Neural code synthesis approaches to code completion generate suggestions by using a language model to predict what the user might type next (the completion) from the context of what they are working on at the moment (the prompt) [2].”
From Productivity Assessment of Neural Code Completion (2022) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Similar studies in the code space are limited to the number of few-shot examples in the prompt, and the effect of length of prompt on performance [9,15].”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [9]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, for the Performance category, existing works have used the Computational Accuracy metric to evaluate generative code models [9,15,33,88], but not other metrics we uncovered regarding the characteristics of the generated artifacts and run-time efficiency.”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [9]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large pre-trained language models perform remarkably well on tasks that can be done 'in one pass', such as generating realistic text (Brown et al., 2020) or synthesizing computer programs (Chen et al., 2021; Austin et al., 2021).”
From Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021) · cites this paper as Austin et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, they struggle with tasks that require unbounded multi-step computation, such as adding integers (Brown et al., 2020) or executing programs (Austin et al., 2021).”
From Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021) · cites this paper as Austin et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The 'MBPP-aug + CodeNet + single line' model achieved an overall synthesis accuracy of 54%, compared to the 62% accuracy of the original few-shot model in Austin et al. (2021).”
From Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021) · cites this paper as Austin et al. (2021)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“MBPP is another large, commonly used benchmark of Python problems.”
From MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current benchmarks for evaluating neural code models focus on only a small subset of programming languages, excluding many popular languages such as Go or Rust.”
From Measuring The Impact Of Programming Language Distribution (2023) · cites this paper as Austin et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“HumanEval [7]-developed by OpenAI for evaluating Codex-and other benchmarks [2,13,20] only consist of programming problems in a single language”
From CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X (2023) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent large language models (LLMs) have demonstrated strong capabilities in understanding and generating code, from competitive programming to repository-level software engineering.”
From Code as Agent Harness (2026) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In fact, existing techniques evaluate on predefined prompt templates with little to no modifications (Austin et al., 2021;Liu et al., 2024b).”
From Does Few-Shot Learning Help LLM Performance in Code Synthesis? (2024) · cites this paper as Austin et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The development of evaluations in tandem with language models (LMs) has substantially contributed to the rapid advancement of these models [29, 11, 7, 25, 81, 27, 72].”
From SciCode: A Research Coding Benchmark Curated by Scientists (2024) · cites this paper as [7]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“HumanEval [5], MBPP [3], and APPS [13] score function-level synthesis; CrossCodeEval [11] targets cross-file completion; CodeClash [33] and PinchBench [18] probe further axes of coding capability.”
From Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks (2026) · cites this paper as [3]
The cited source says what the claim attributes to it.
In the citing paper
“Researchers have already investigated the benefits of natural language feedback to guide program synthesis [2]”
From Productivity Assessment of Neural Code Completion (2022) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“Researchers have already investigated the benefits of natural-language feedback to guide program synthesis, 2 so the conversational framing of coding completions is not a radical proposal.”
From Measuring GitHub Copilot's Impact on Productivity (2024) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“Direct execution prediction has been shown to perform poorly on Python programs in Austin et al. (2021).”
From Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Similarly, large-scale language models struggle to predict the result of executing Python code, even code which is a solution to a programming task the model is able to solve (Austin et al., 2021).”
From Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“However, previous work has found that textual similarity metrics correlate only weakly with code correctness [1,2,27], highlighting the importance of benchmark suites with unit tests.”
From MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“MBPP dataset is proposed by [2], containing 974 problems in Python.”
From CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X (2023) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“Different from prior works on utilizing human feedback for code repair, where the feedback message explains the code errors and how to fix them (Chen et al., 2023a;Austin et al., 2021), SELF-DEBUGGING teaches the model to identify the implementation errors via investigating into execution results and explaining the code by itself.”
From Teaching Large Language Models to Self-Debug (2023) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“MBPP prompts follow a templated structure where the critical task specification appears mid-prompt:”
From Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression (2026) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“The Mostly Basic Python Problems (MBPP) benchmark (Austin et al., 2021) takes a complementary approach, providing 974 crowd-sourced programming problems intended to be solvable by entry-level programmers.”
From Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression (2026) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) evaluate isolated functions.”
From SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents (2026) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“MBPP, on the other hand, comprises 974 programming problems created by crowdsource contributors.”
From Qwen2.5-Coder Technical Report (2024) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“In particular, MBPP -verified is sourced from (Yang et al., 2024;Misu et al., 2024) and contains 78 programs from the original MBPP dataset (Austin et al., 2021).”
From AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinement (2024) · cites this paper as Austin et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Part of this paper’s reference list could not be split into single entries.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“The models we use in this paper are dense left-to-right decoder-only Transformer language models [Vaswani et al., 2017] trained on a combination of web documents, dialog data, and Wikipedia.”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“P L ] 1 6 A ug 2 text [Brown et al., 2020, Raffel et al., 2019] and to solve a rapidly expanding set of modeling and reasoning tasks [Devlin et al., 2019, big-bench collaboration, 2021].”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 1×
“…line of previous work on neural language models of natural language text [Mikolov et al., 2010, Sutskever et al., 2011, Józefowicz et al., 2016, Dai and Le, 2015, Peters et al., 2018, Howard and Ruder, 2018], especially recent large Transformer models [Radford et al., 2018, Brown et al., 2020].”
BackgroundOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · cited 1×
“Our work is inspired by the long line of previous work on neural language models of natural language text [Mikolov et al., 2010, Sutskever et al., 2011, Józefowicz et al., 2016, Dai and Le, 2015, Peters et al., 2018, Howard and Ruder, 2018], especially recent large Transformer models [Radford et…”
MethodKudo, T., & Richardson, J. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-2012 · Cited by 4,454 · cited 1×
“81T BPE tokens with a vocabulary of 32K tokens using the SentencePiece library [Kudo and Richardson, 2018].”
MethodUniversal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 1×
“…line of previous work on neural language models of natural language text [Mikolov et al., 2010, Sutskever et al., 2011, Józefowicz et al., 2016, Dai and Le, 2015, Peters et al., 2018, Howard and Ruder, 2018], especially recent large Transformer models [Radford et al., 2018, Brown et al., 2020].”
BackgroundThe Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · cited 1×
“These results suggest that methods such as prompt-tuning [Li and Liang, 2021, Lester et al., 2021] could yield substantial performance improvements in this domain.”
ResultThe Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092 · cited 1×
“These results suggest that methods such as prompt-tuning [Li and Liang, 2021, Lester et al., 2021] could yield substantial performance improvements in this domain.”
Result
Research works printed but not linked: 1
- [10]10. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, Will Guss, Alex Nichol, Igor Babuschkin, Suchir Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, July 2021. URL http://arxiv.org/abs/2107.03374 . · arXiv
Cited by
Sorted by citations
Items 401–500 of 529
Infinite-parameter Large Language ModelPreprint
Ding, F. (2024) · doi:10.33774/coe-2024-7khtw
Cao, J., & Lu, Y. (2025) · Qeios · doi:10.32388/mlaotg · Cited by 1
SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios
Le, T., Thai, M. V. T., Manh, D. N., Nhat, H. P., & Bui, N. D. Q. (2025) · arXiv · doi:10.48550/arxiv.2512.18470 · Cited by 79
“Building on this, MBPP (Austin et al., 2021) expanded the scope with roughly 1,000 crowd-sourced tasks aimed at entry-level programmers, while APPS (Hendrycks et al., 2021) introduced 10,000 diverse problems spanning simple to advanced algorithmic challenges.”
iCLP: Large Language Model Reasoning with Implicit Cognition Latent Planning
Chen, S., & Niu, D. (2025) · arXiv · doi:10.48550/arxiv.2512.24014
Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression
Johnson, W. (2026) · arXiv · Cited by 1
“The Mostly Basic Python Problems (MBPP) benchmark (Austin et al., 2021) takes a complementary approach, providing 974 crowd-sourced programming problems intended to be solvable by entry-level programmers.”
Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks
Zheng, M., Han, K., Li, B., Xu, H., Tian, Y., He, W., Zhou, H., Guo, J., Hu, H., Ma, L., et al. (2026) · arXiv · doi:10.48550/arxiv.2606.12344
“The scan evaluates resolve-rate, cost, and operational criteria under different sensitivity scenarios.”
OctoBench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Coding
Ding, D., Liu, S., Yang, E., Lin, J., Chen, Z., Dou, S., Guo, H., Cheng, W., Zhao, P., Xiao, C., et al. (2026) · arXiv · doi:10.48550/arxiv.2601.10343
A Unified Approach to Routing and Cascading for LLMs
Dekoninck, J., Baader, M., & Vechev, M. (2024) · arXiv · doi:10.48550/arxiv.2410.10347
“(Hendrycks et al., 2021), GSM8k (Cobbe et al., 2021), and MBPP (Austin et al., 2021), alongside answers from eleven different models ranging from GPT-4 (OpenAI, 2023) to Mistral-7B (Jiang et al., 2023).”
Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages
Ivanova, M., Zadorozhny, P., Levichev, R., Petrov, I., Pavel, A., Lopatin, I., Kutalev, A., & Babaev, D. (2026) · arXiv · doi:10.48550/arxiv.2606.20517 · Cited by 1
“To measure these abilities, benchmarks such as HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), and APPS (Hendrycks et al., 2021) have been widely adopted.”
Measuring The Impact Of Programming Language Distribution
Orlanski, G., Xiao, K., Garcia, X., Hui, J., Howland, J., Malmaud, J., Austin, J., Singh, R., & Catasta, M. (2023) · arXiv · doi:10.48550/arxiv.2302.01973 · Cited by 21
“Existing datasets are either mono-lingual (Chen et al., 2021; Austin et al., 2021; Lai et al., 2022) or limited to only a subset of popular programming languages (Roziere et al., 2020).”
BackgroundMethodInfluentialSelf-citationReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification
Lee, H., Oh, S., Kim, J., Shin, J., & Tack, J. (2025) · arXiv · doi:10.48550/arxiv.2502.14565 · Cited by 26
“Also, we verify that ReVISE effectively enhances reasoning in the coding domain (i.e., MBPP (Austin et al., 2021)).”
Jiang, J., Shen, J., Kim, S., Yoo, K. M., Kim, J., & Kim, S. (2026) · arXiv · doi:10.48550/arxiv.2603.05863 · Cited by 8
“…set of seven widely-used code generation benchmarks, ranging from foundational programming challenges such as HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), and EvalPlus (Liu et al., 2023), to significantly more complex and competitive programming problems found in BigCodeBench (Zhuo…”
InfluentialLive-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?
Xia, C. S., Wang, Z., Yang, Y., Wei, Y., & Zhang, L. (2025) · arXiv · doi:10.48550/arxiv.2511.13646 · Cited by 2
“1 Introduction Large Language Models (LLMs) have rapidly progressed from simple code autocompletion [7, 11, 21] to interactive agents capable of navigating repositories, running tests, and submitting patches end-to-end [40, 33, 45, 36, 9, 22].”
Zhou, C., Zhang, X., Song, D., Chen, X., Gu, W., Ma, H., Tian, Y., Zhang, M., & Hu, L. (2025) · arXiv · doi:10.48550/arxiv.2502.09183 · Cited by 18
“Concretely, data refinement improves the fundamental programming abilities of the RefineCoder series, as demonstrated by continuous performance gains over iterations on HumanEval, HumanEval+, and MBPP+. Notably, RefineCoder-QW-7B shows a remarkable 7.9 point increase ( Iter0 -> Iter3 ) in Pass@1 on MBPP+, highlighting…”
InfluentialHarnessOpt-Bench: Evaluating LLMs at Harness Optimization
Ursekar, V., Shanker, A., Maurya, Y., Yasser, S., Kalmath, V. S., Chatrath, V., & Xue, Y. (2026) · arXiv · doi:10.48550/arxiv.2608.06301
“Coding agent evaluation has grown from function-level synthesis [4, 2] to repository-scale tasks in real execution environments [9], and tasks requiring iterative optimization, such as machine learning engineering [3] and kernel optimization [19].”
Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneity
Zhang, H., Cui, Z., Chen, J., Wang, X., Zhang, Q., Wang, Z., Wu, D., & Hu, S. (2025) · arXiv · doi:10.48550/arxiv.2502.08788 · Cited by 84
Token Reduction Is Not Cost Reduction
Weinberger, S., & Hozez, A. (2026) · arXiv · doi:10.48550/arxiv.2607.12161 · Cited by 1
Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
Cheng, X., Zeng, W., Dai, D., Chen, Q., Wang, B., Xie, Z., Huang, K., Yu, X., Hao, Z., Zhang, H., et al. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.226 · Cited by 3
Ning, X., Tieu, K., Fu, D., Wei, T., Li, Z., Bei, Y., Zou, J., Ai, M., Liu, Z., Li, T.-W., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.18747
Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?
Chen, Z., Sun, Z., Shi, Y., Lo, D., & Jiang, L. (2026) · arXiv · doi:10.48550/arxiv.2607.01211 · Cited by 9
Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation
Heineman, D., Hofmann, V., Magnusson, I., Gu, Y., Smith, N. A., Hajishirzi, H., Lo, K., & Dodge, J. (2025) · arXiv · doi:10.48550/arxiv.2508.13144
“HumanEval [8], HumanEval+ [34], MBPP [1], MBPP+ [34] Using strong LLMs have become a tool for augmenting existing benchmarks with more difficult questions or answer choices [64] and re-evaluating benchmark quality [62], and may provide a cheap method for improving signal .”
Multi-agent Architecture Search via Agentic Supernet
Zhang, G., Niu, L., Fang, J., Wang, K., Bai, L., & Wang, X. (2025) · arXiv · doi:10.48550/arxiv.2502.04180
EvoFlow: Evolving Diverse Agentic Workflows On The Fly
Zhang, G., Chen, K., Wan, G., Chang, H., Cheng, H., Wang, K., Hu, S., & Bai, L. (2025) · arXiv · doi:10.48550/arxiv.2502.07373
“…on six public benchmarks covering four domains: (1) math reasoning , GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b), and MultiArith (Roy & Roth, 2016); (2) code generation , HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021)); (3) embodied , ALFWorld (Shridhar et al., 2021).”
FlowReasoner: Reinforcing Query-Level Meta-Agents
Gao, H., Liu, Y., He, Y., Dou, L., Du, C., Deng, Z., Hooi, B., Lin, M., & Pang, T. (2025) · arXiv · doi:10.48550/arxiv.2504.15257
“Among various benchmarks, we consider three representative datasets: BigCodeBench[65], which emphasizes engineering-oriented tasks, and two algorithmically focused benchmarks, HumanEval[6] and MBPP [4].”
Multi-Agent Design: Optimizing Agents with Better Prompts and Topologies
Zhou, H., Wan, X., Sun, R., Palangi, H., Iqbal, S., Vulić, I., Korhonen, A., & Arık, S. Ö. (2025) · arXiv · doi:10.48550/arxiv.2502.02533
ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimization
Wang, Y., Yang, L., Li, G., Wang, M., & Aragam, B. (2025) · arXiv · doi:10.48550/arxiv.2502.04306
Mendel Gödel Machine: Recursive Self-Improving Coding Agents via Comparative Evolution
Liu, C., Liu, Y., Yan, S., Tresp, V., & Ma, Y. (2026) · arXiv · doi:10.48550/arxiv.2608.07645
Nanbeige4.2-3B: Unlocking Agentic Capabilities in a Compact Model
Lab, N., :, Yang, C., Huang, C., Lan, F., Chen, H., Zhou, H., Song, H., Cao, J., Zhu, J., et al. (2026) · arXiv · doi:10.48550/arxiv.2607.22083
“We select a diverse set of benchmarks, including reasoning-oriented metrics (GSM8K [6], BBH [24], and MBPP [1]) and knowledge-oriented metrics (MMLU-Pro [27], SuperGPQA [9], and GPQA [21]). with a multi-stage RL recipe designed to improve response quality, reasoning efficiency, and long-horizon…”
Towards A Measure Of General Machine IntelligencePreprint
Venkatasubramanian, G., Kar, S., Singh, A., Mishra, S. K., Yadav, D., & Chandak, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.12075 · Cited by 2
(2022) · The MIT Press eBooks · doi:10.7551/mitpress/13770.003.0013
Peer Review #3 of "Code4ML: a large-scale dataset of annotated Machine Learning code (v0.1)"
(2023) · PeerJ Computer Science · doi:10.7287/peerj-cs.1230v0.1/reviews/3
Peer Review #2 of "Code4ML: a large-scale dataset of annotated Machine Learning code (v0.1)"
(2023) · PeerJ Computer Science · doi:10.7287/peerj-cs.1230v0.1/reviews/2
Peer Review #1 of "Code4ML: a large-scale dataset of annotated Machine Learning code (v0.1)"
(2023) · PeerJ Computer Science · doi:10.7287/peerj-cs.1230v0.1/reviews/1
Tourassi, G. D., Shankar, M. A., & Wang, F. (2023) · OECD eBooks · doi:10.1787/0e1b4c2f-en
Eroom’s Law and the decline in the productivity of biopharmaceutical R&D
Scannell, J. W. (2023) · OECD eBooks · doi:10.1787/6f256d17-en
The end of Moore’s Law? Innovation in computer systems continues at a high pace
Kressel, H. (2023) · OECD eBooks · doi:10.1787/63e48242-en · Cited by 1
Quantifying the “cognitive extent” of science and how it has changed over time and across countries
Milojević, S. (2023) · OECD eBooks · doi:10.1787/1f717652-en
Interpretability: Should – and can – we understand the reasoning of machine-learning systems?
Cartwright, H. (2023) · OECD eBooks · doi:10.1787/2d0478b9-en
OECD (2023) · OECD eBooks · doi:10.1787/f1262928-en
On Sample-Efficient Code Generation
Han, H., Kim, Y. J., Kim, B., Lee, Y., Lee, K., Lee, K. C., Lee, M., Bae, K., & Hwang, S.-W. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.73
Analyzing Code Text Strings for Code Evaluation
Wang, T., & Chen, Z. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386406 · Cited by 1
Synthetic APIs: Enabling Language Models to Act as Interlocutors Between Natural Language and Code
Mullins, R. S., & Terry, M. (2023) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2023.073
Self-citationSrivastava, N., & Kim, S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4790013
CodeSift: An LLM-Based Reference-Less Framework for Automatic Code Validation
Aggarwal, P., Chatterjee, O., Dai, T., Mohapatra, P., Paulovicks, B. D., Blancett, B., & Magalhaes, A. D. (2024) · IEEE ... International Conference on Cloud Computing · doi:10.1109/cloud62652.2024.00052
Xu, Y., Shang, Z., Zou, H., Yu, H., & Gao, S. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112206
Unveiling the Power of Neural Program Synthesis in High Level Code Generation
Sai, G. K., Harshith, K. S. S., Varma, K. A., Kiran, G. Y., & Swaminathan, J. (2024) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt61001.2024.10725949
Code Generation Based on Quality-Aware Reordering
Yuan, S., & Yu, L. (2024) · International Conference on Intelligent Computing and Signal Processing (ICSP) · doi:10.1109/icsp62122.2024.10743331
Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language ModelsPreprint
Gao, K., Xia, S., Xu, K., Torr, P. H. S., & Gu, J. (2024) · Qeios · doi:10.32388/758n37
On the Generalizability of Transformer Models to Code Completions of Different Lengths
Cooper, N., Tufano, R., Bavota, G., & Poshyvanyk, D. (2024) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58944.2024.00042
LLM-based method for generating vulnerable code equivalents
Chen, J., Liu, J., Wang, J., & Li, D. (2025) · doi:10.1117/12.3058052 · Cited by 1
Predicting ChatGPT’s Ability to Solve Complex Programming Challenges
Ho, N., May, J., Ngo, B., Formato, J., Ngo, L., Ho, V. L., & Bui, H. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825139
Intelligent Q&A System for Ventilators Based on Knowledge Graph-Enhanced LLM
Ma, C., Zheng, Z., Yao, Q., Li, W., Xu, R., He, J., & Duan, Y. (2025) · Lecture notes in electrical engineering · doi:10.1007/978-981-96-1395-3_17
Enhancing Code Completion with Position Embedding of Abstract Syntax Tree
Li, J., Lu, J., & Zhu, L. (2024) · IEEE International Conference on Frontiers Technology of Information and Computer (ICFTIC) · doi:10.1109/icftic64248.2024.10913308
Boosting self-repair workflow with brainstorming for code generation
Zhu, J., & Shi, Z. (2024) · Proceedings - Asian Test Symposium/Proceedings · doi:10.1109/ats64447.2024.10915518
Global Grounding in Flexible Robot Tasks With Object Number Variety
Sucker, S., & Henrich, D. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3560864
“LLMs allow the synthesis of code [23], programming of robots [24], and the open-vocab grounding of suitable physical objects [25], [26] througheasyprompting.”
Guetari, R., Kaabi, R. S., Hamzaoui, H., & Kraïem, N. (2025) · Research Square · doi:10.21203/rs.3.rs-6353409/v1
Wang, X. (2025) · Research Square · doi:10.21203/rs.3.rs-6551928/v1
Imran, A., & Gopalakrishnan, K. (2025) · Apress eBooks · doi:10.1007/979-8-8688-0989-7_4
Data-Driven Security Frameworks: AI-Infused Digital Twin Software Solutions for IoT
Babu, B. S., Joseph, A. S. K., Premalatha, J., Alisha, M., Chekuri, N., & Saravanan, V. (2025) · International Journal of Computational and Experimental Science and Engineering · doi:10.22399/ijcesen.1721
“More than 170 data centers in 36 countries work together to analyze the data stored by the Large Hadron Collider (LHC), which has the potential to reach storage capacities of 10 GB/s in future runs [5].”
Evaluating Language Models for Computer Graphics Code Completion
Kels, J., Dahou, A. H., & Mathiak, B. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00017
“We propose a zero shot function completion task, similar to existing benchmarks like HumanEval [3], MBPP [12] and others.”
SnipGen: A Mining Repository Framework for Evaluating LLMs for Code
Rodríguez-Cárdenas, D., Velasco, A., & Poshyvanyk, D. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00084 · Cited by 2
Khant, K. S., Lin, H. Y., & Thongtanunam, P. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00044
Augmenting the Generality and Performance of Large Language Models for Software Engineering
Peña, F. C. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion66252.2025.00059 · Cited by 1
Effective Code Generation Using Program Synthesis And CodeLLM
Sadasivam, G. S., & Dharun Kanna, N. (2025) · International Conference on Computational Innovations and Engineering Sustainability (ICCIES) · doi:10.1109/iccies63851.2025.11032972
“Austin et al. [2] investigate the application of large language models for program synthesis.”
Coft: Making Large Language Models Better Zero-Shot Learners for Code Generation
Li, W., Qian, Y., Gao, K., Chen, H., Wang, X., Tong, Y., Li, L., Wu, Y., & Zhao, C. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00059 · Cited by 1
“To evaluate performance, we measured each model’s results on three code generation bench-marks—HumanEval [18], MBPP [44], and BigCodeBench [45]—using pass@1 as the metric.”
Advancing Large Language Models in Code Generation: Usaco Benchmark and Bug Mitigation Insights
Trentini, J., Liu, V., Peng, Y., & Zong, Z. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00057 · Cited by 1
“MBPP includes 974 Python problems focusing on short problem descriptions and math concepts [64].”
Algorithmic Inversion: A Learnable Algorithm Representation for Code Generation
Shi, Z., Wu, F., Zeng, W., Kong, Y., Shen, S., & Wu, Y. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00054
“We select four open-source models that are widely used in the research community and have competitive performances on common coding benchmarks, such as HumanEval [4] and MBPP [32], to serve as foundation for our experiments.”
Rouabhia, D., & Hadjadj, I. (2025) · Research Square · doi:10.21203/rs.3.rs-6793290/v1
Exploring and Enhancing Advanced MoE Models: From Deepspeed-MoE to DeepSeek-V3
Zhang, Z., Gao, Y., & Chen, D. (2025) · International Seminar on Artificial Intelligence, Networking and Information Technology (AINIT) · doi:10.1109/ainit65432.2025.11035928
SuperCode: Sustainability PER AI-driven Co-design
Broekema, P. C., & van Nieuwpoort, R. V. (2025) · ACM International Conference on Computing Frontiers: Workshops and Special Sessions (CF Companion) · doi:10.1145/3706594.3727576 · Cited by 1
“AI-supported code generation has recently become possible [1, 58] and has seen only very limited application in high-performance computing.”
PTC: Prefix Tuning CodeT5 for High-Quality Secure Network Measurement Script Generation
Wang, Y., & Han, X. (2025) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn64029.2025.00024
Zhijie, Z., Deng, Y., Qi, H., & Wang, M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5351223
Reasoning Distillation and Structural Alignment for Improved Code GenerationPreprint
Jalilifard, A., Raimundo, M. M., & Rocha, A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5403116
Benchmarking Large Language Models for Code Generation
Nogueira, R. P. (2025) · European Dependable Computing Conference · doi:10.1109/edcc-c66476.2025.00032
“Five of the most commonly used benchmarks for evaluating LLMs in text-to-code generation are HumanEval [2], MBPP [12], APPS [3], CodeXGlue [13], and XCodeEval [14].”
Nehzati, M. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1662220
“The HumanEval-Extended dataset comprised 2,500 Python programming problems derived from OpenAI’s HumanEval benchmark following Austin et al. (2021) approach.”
MACEDON : Supporting Programmers with Real-Time Multi-Dimensional Code Evaluation and Optimization
Liu, X., You, Y., Qiu, X., Ma, T., & Zhao, J. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747655
“Similarly, MBPP [4] provides a set of crowd-sourced programming problems with corresponding test cases, allowing for a broader evaluation of LLMs’ code generation capabilities.”
Conformal Prediction in the Age of Multimodal Foundation Models: A Survey
Stavan, C., Tilaganji, K., Krishnan, S. M., Kancheti, S. S., & Balasubramanian, V. N. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-15120-9_13
TheBotCompany: Self-Organizing Multi-agent Systems for Continuous Software Development
Lyu, W., Xiao, Y., Zhang, Y., & Sun, Y. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834358 · Cited by 3
How Do Large Language Models Perform in Deep Learning Code Generation? An Empirical Study
Ma, X., Du, X., Li, C., Meng, J., Fang, X., Ding, W., & Zheng, Z. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3816024 · Cited by 1
Yuan, L., Liu, S., Wang, Y., Zhu, S., Wang, T., Mao, T., & Shao, S. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10838-y
“Experiments are conducted on the MBPP, APPS, and MBJP datasets using UniXcoder, CodeT5 +, and Text-Embedding-3-Large (see Fig.”
InfluentialKátai, Z., & Iclănzan, D. (2026) · Acta Universitatis Sapientiae Informatica · doi:10.1007/s44427-026-00033-8
“Foundational benchmarks such as HumanEval [3] and MBPP [23] are now saturated for top models, motivating newer efforts with stronger test suites and improved contamination con-trol [24].”
Sivapiran, S., & Uddin, G. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808123 · Cited by 1
“HumanEval+ extends the original HumanEval [5] with 164 instances and 80x more test cases, while MBPP+ enhances MBPP [4] with 378 instances and 35x more testing coverage.”
Compiling Code LLMs into Lightweight Executables
Shi, J., He, J., Yang, Z., Yang, C., Klymenko, M. V., Hoang, T., Xu, X., Xing, Z., & Lo, D. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808196
CertiCoder: Towards MISRA-Compliant C Code Generation with LLMs
Gou, M., Yao, Z., Ma, H., Zhang, E.-D., Zhou, J., & He, F. (2026) · Proc. ACM Softw. Eng. · doi:10.1145/3797120
Chen, X., Wang, Q., Wang, S. Q., Zhang, W., Liu, Y., Liu, B. F., & Jiang, Z. H. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808099
Unlocking LLM Code Correction with Iterative Feedback Loops
Zhang, L., & Kothari, S. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-24807-7_49 · Cited by 2
A Technology for Generating and Evaluating the Quality of Source Code Using Large Language Models
Pastukh, O., Yatsyshyn, V. V., & Zadvornyi, O. (2026) · Cybernetics and Systems Analysis · doi:10.1007/s10559-026-00876-1
Guidelines to Prompt Large Language Models for Code Generation: An Empirical Characterization
Midolo, A., Giagnorio, A., Zampetti, F., Tufano, R., Bavota, G., & Penta, M. D. D. (2026) · IEEE International Conference on Program Comprehension · doi:10.1145/3794763.3794819 · Cited by 6
“One is related to the choice of the three benchmarks used (BigCodeBench, HumanEval+, and MBPP+), and the other to the specific programming language considered (Python).”
InfluentialiBRAHIMOĞLU, N., Demirezen, M. U., & Yıldız, F. (2026) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-026-01490-x · Cited by 1
Graph-Aware Reinforcement Learning for Reusable Prompt Compression in Black-Box LLMsPreprint
Eskandarpour, M., Haddadnejad, P., & Jannati, M. J. (2026) · Research Square · doi:10.21203/rs.3.rs-10458568/v1
Efficiency Evaluation Framework for LLM-Generated Code
Avagian, D. A., & Ball, A. L. J. (2026) · Pattern Recognition and Image Analysis · doi:10.1134/s1054661826700306
“Benchmarks like HumanEval [5] and MBPP [1] (and their extensions HumanEval+ and MBPP+ [21]) that have become de facto standards for research on code correctness therefore prove inadequate for detailed efficiency analysis, since the problems composing them often lack the necessary asymptotic…”
LLM-based API Argument Completion with Knowledge-Augmented Prompts
Akram, W., Jiang, Y., Khan, H. A., Jalil, F., & Liu, H. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773158
AI Employees: A Role Programming Framework for Supervising Autonomous Systems
Zheng, S. (2026) · International Conference on Automation Control, Algorithm and Intelligent Bionics (ACAIB) · doi:10.1145/3828836.3828919
SWE-PDB: Teaching LLMs to Leverage Debugging Tools via Agentic Training
Liu, J., Hu, X., & Xia, X. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832117
“HumanEval-buggy , MBPP-buggy , and Syn-Debug are constructed by injecting bugs into correct solutions from HumanEval [10] and MBPP [5] using our data synthesis pipeline.”
Code Generation Beyond Mainstream Languages
Delgado, D. (2026) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1145/3837062.3839350
“Existing evaluations of code generation primarily target high-resource GPLs through functional unit testing on standard benchmarks like HumanEval [4] and MBPP [1].”
What Breaks When LLMs Code? Characterizing Operational Safety Failures of Agentic Code Assistants
Hasan, A. A., & Biswas, S. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834393 · Cited by 5
“Prior work focuses on code correctness [26], adversarial robustness [4, 39, 45], and security vulnerabilities [69, 72], primarily assessing safety under malicious use.”
Needle in the Repo: Diagnosing Maintainability Failures in AI-Generated Repository Edits
Zhu, H., Zhang, Q., Wang, J., Yang, Z., & Qiu, Y. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834401
Metadata
- Authors
- Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., & Sutton, C.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2108.07732
- OpenAlex
- W3195140297
- Cited by
- 388
- References
- 93
- Citations by year
- 2026: 44 · 2025: 129 · 2024: 92 · 2023: 85 · 2022: 21 · 2021: 15
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- TIB Data Manager
- Funders
- National Natural Science Foundation of China (62472339, 62302375)
- Hankyong National University
- Ministry of Industry and Information Technology of the People's Republic of China
- HORIZON EUROPE Framework Programme (101070305)
- Fundação para a Ciência e a Tecnologia (UIDB/00319, 00319, UIDB/00319/2020, UIDB/05757/2020)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- program synthesis, large language models, Python code generation, MBPP, few-shot learning, fine-tuning, model size scaling, human feedback, error analysis, semantic grounding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
