The DOI matches its DataCite record; 92 of the 93 references it cites were found in a registry or an index, and 1 are not checked yet.
Program Synthesis with Large Language Models
Open accessAbstract
This paper explores the limits of the current generation of large language models for program synthesis in general purpose programming languages. We evaluate a collection of such models (with between 244M and 137B parameters) on two new benchmarks, MBPP and MathQA-Python, in both the few-shot and fine-tuning regimes. Our benchmarks are designed to measure the ability of these models to synthesize short Python programs from natural language descriptions. The Mostly Basic Programming Problems (MBPP) dataset contains 974 programming tasks, designed to be solvable by entry-level programmers. The MathQA-Python dataset, a Python version of the MathQA benchmark, contains 23914 problems that evaluate the ability of the models to synthesize code from more complex text. On both datasets, we find that synthesis performance scales log-linearly with model size. Our largest models, even without finetuning on a code dataset, can synthesize solutions to 59.6 percent of the problems from MBPP using few-shot learning with a well-designed prompt. Fine-tuning on a held-out portion of the dataset improves performance by about 10 percentage points across most model sizes. On the MathQA-Python dataset, the largest fine-tuned model achieves 83.8 percent accuracy. Going further, we study the model's ability to engage in dialog about code, incorporating human feedback to improve its solutions. We find that natural language feedback from a human halves the error rate compared to the model's initial prediction. Additionally, we conduct an error analysis to shed light on where these models fall short and what types of programs are most difficult to generate. Finally, we explore the semantic grounding of these models by fine-tuning them to predict the results of program execution. We find that even our best models are generally unable to predict the output of a program given a specific input.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“In practice, large pools of test cases, Dj , can either originate from human generated documentation (Nassif et al., 2021;Lemieux et al., 2023) or GPTgenerated unit tests (Austin et al., 2021).”
From Does Few-Shot Learning Help LLM Performance in Code Synthesis? (2024) · cites this paper as Austin et al., 2021
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Code data (e.g., from Stack Exchange and GitHub) substantially enhances program synthesis capabilities [37,38].”
From A Survey of Large Language Models (2026) · cites this paper as [38]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Neural code synthesis approaches to code completion generate suggestions by using a language model to predict what the user might type next (the completion) from the context of what they are working on at the moment (the prompt) [2].”
From Productivity Assessment of Neural Code Completion (2022) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Similar studies in the code space are limited to the number of few-shot examples in the prompt, and the effect of length of prompt on performance [9,15].”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [9]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, for the Performance category, existing works have used the Computational Accuracy metric to evaluate generative code models [9,15,33,88], but not other metrics we uncovered regarding the characteristics of the generated artifacts and run-time efficiency.”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [9]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large pre-trained language models perform remarkably well on tasks that can be done 'in one pass', such as generating realistic text (Brown et al., 2020) or synthesizing computer programs (Chen et al., 2021; Austin et al., 2021).”
From Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021) · cites this paper as Austin et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, they struggle with tasks that require unbounded multi-step computation, such as adding integers (Brown et al., 2020) or executing programs (Austin et al., 2021).”
From Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021) · cites this paper as Austin et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The 'MBPP-aug + CodeNet + single line' model achieved an overall synthesis accuracy of 54%, compared to the 62% accuracy of the original few-shot model in Austin et al. (2021).”
From Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021) · cites this paper as Austin et al. (2021)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“MBPP is another large, commonly used benchmark of Python problems.”
From MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current benchmarks for evaluating neural code models focus on only a small subset of programming languages, excluding many popular languages such as Go or Rust.”
From Measuring The Impact Of Programming Language Distribution (2023) · cites this paper as Austin et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“HumanEval [7]-developed by OpenAI for evaluating Codex-and other benchmarks [2,13,20] only consist of programming problems in a single language”
From CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X (2023) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent large language models (LLMs) have demonstrated strong capabilities in understanding and generating code, from competitive programming to repository-level software engineering.”
From Code as Agent Harness (2026) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In fact, existing techniques evaluate on predefined prompt templates with little to no modifications (Austin et al., 2021;Liu et al., 2024b).”
From Does Few-Shot Learning Help LLM Performance in Code Synthesis? (2024) · cites this paper as Austin et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The development of evaluations in tandem with language models (LMs) has substantially contributed to the rapid advancement of these models [29, 11, 7, 25, 81, 27, 72].”
From SciCode: A Research Coding Benchmark Curated by Scientists (2024) · cites this paper as [7]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“HumanEval [5], MBPP [3], and APPS [13] score function-level synthesis; CrossCodeEval [11] targets cross-file completion; CodeClash [33] and PinchBench [18] probe further axes of coding capability.”
From Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks (2026) · cites this paper as [3]
The cited source says what the claim attributes to it.
In the citing paper
“Researchers have already investigated the benefits of natural language feedback to guide program synthesis [2]”
From Productivity Assessment of Neural Code Completion (2022) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“Researchers have already investigated the benefits of natural-language feedback to guide program synthesis, 2 so the conversational framing of coding completions is not a radical proposal.”
From Measuring GitHub Copilot's Impact on Productivity (2024) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“Direct execution prediction has been shown to perform poorly on Python programs in Austin et al. (2021).”
From Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Similarly, large-scale language models struggle to predict the result of executing Python code, even code which is a solution to a programming task the model is able to solve (Austin et al., 2021).”
From Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“However, previous work has found that textual similarity metrics correlate only weakly with code correctness [1,2,27], highlighting the importance of benchmark suites with unit tests.”
From MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“MBPP dataset is proposed by [2], containing 974 problems in Python.”
From CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X (2023) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“Different from prior works on utilizing human feedback for code repair, where the feedback message explains the code errors and how to fix them (Chen et al., 2023a;Austin et al., 2021), SELF-DEBUGGING teaches the model to identify the implementation errors via investigating into execution results and explaining the code by itself.”
From Teaching Large Language Models to Self-Debug (2023) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“MBPP prompts follow a templated structure where the critical task specification appears mid-prompt:”
From Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression (2026) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“The Mostly Basic Python Problems (MBPP) benchmark (Austin et al., 2021) takes a complementary approach, providing 974 crowd-sourced programming problems intended to be solvable by entry-level programmers.”
From Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression (2026) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) evaluate isolated functions.”
From SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents (2026) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“MBPP, on the other hand, comprises 974 programming problems created by crowdsource contributors.”
From Qwen2.5-Coder Technical Report (2024) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“In particular, MBPP -verified is sourced from (Yang et al., 2024;Misu et al., 2024) and contains 78 programs from the original MBPP dataset (Austin et al., 2021).”
From AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinement (2024) · cites this paper as Austin et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Part of this paper’s reference list could not be split into single entries.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“The models we use in this paper are dense left-to-right decoder-only Transformer language models [Vaswani et al., 2017] trained on a combination of web documents, dialog data, and Wikipedia.”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“P L ] 1 6 A ug 2 text [Brown et al., 2020, Raffel et al., 2019] and to solve a rapidly expanding set of modeling and reasoning tasks [Devlin et al., 2019, big-bench collaboration, 2021].”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 1×
“…line of previous work on neural language models of natural language text [Mikolov et al., 2010, Sutskever et al., 2011, Józefowicz et al., 2016, Dai and Le, 2015, Peters et al., 2018, Howard and Ruder, 2018], especially recent large Transformer models [Radford et al., 2018, Brown et al., 2020].”
BackgroundOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · cited 1×
“Our work is inspired by the long line of previous work on neural language models of natural language text [Mikolov et al., 2010, Sutskever et al., 2011, Józefowicz et al., 2016, Dai and Le, 2015, Peters et al., 2018, Howard and Ruder, 2018], especially recent large Transformer models [Radford et…”
MethodKudo, T., & Richardson, J. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-2012 · Cited by 4,454 · cited 1×
“81T BPE tokens with a vocabulary of 32K tokens using the SentencePiece library [Kudo and Richardson, 2018].”
MethodUniversal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 1×
“…line of previous work on neural language models of natural language text [Mikolov et al., 2010, Sutskever et al., 2011, Józefowicz et al., 2016, Dai and Le, 2015, Peters et al., 2018, Howard and Ruder, 2018], especially recent large Transformer models [Radford et al., 2018, Brown et al., 2020].”
BackgroundThe Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · cited 1×
“These results suggest that methods such as prompt-tuning [Li and Liang, 2021, Lester et al., 2021] could yield substantial performance improvements in this domain.”
ResultThe Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092 · cited 1×
“These results suggest that methods such as prompt-tuning [Li and Liang, 2021, Lester et al., 2021] could yield substantial performance improvements in this domain.”
Result
Research works printed but not linked: 1
- [10]10. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, Will Guss, Alex Nichol, Igor Babuschkin, Suchir Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, July 2021. URL http://arxiv.org/abs/2107.03374 . · arXiv
Cited by
Sorted by citations
Items 301–400 of 530
GPT-4 vs. GPT-3.5 AS CODING ASSISTANTSPreprint
Moussiades, L., Zografos, G., & Papakostas, G. A. (2024) · Research Square · doi:10.21203/rs.3.rs-3920214/v1 · Cited by 4
Benchmarking Large Language Models for Bio-Image Analysis Code GenerationPreprint
Haase, R., Tischer, C., Hèriché, J.-K., & Scherf, N. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.04.19.590278 · Cited by 8
SciCode: A Research Coding Benchmark Curated by Scientists
Tian, M., Gao, L., Zhang, S. D., Chen, X., Fan, C., Guo, X., Haas, R., Ji, P., Krongchon, K., Li, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.13168 · Cited by 15
“Soon after model capabilities improved, execution-based evaluation gained in popularity [28, 7, 11].”
BackgroundMethodZhou, X., Weyssow, M., Widyasari, R., Zhang, T., He, J., Lyu, Y., Chang, J., Zhang, B., Huang, D., & Lo, D. (2025) · arXiv · doi:10.48550/arxiv.2502.06215 · Cited by 3
Token-Budget-Aware LLM Reasoning
Han, T., Wang, Z., Fang, C., Zhao, S., Ma, S., & Chen, Z. (2024) · arXiv · Cited by 31
On the Impacts of Contexts on Repository-Level Code Generation
Hai, N. L., Nguyen, D. M., & Bui, N. D. Q. (2024) · arXiv · doi:10.48550/arxiv.2406.11927 · Cited by 3
Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study
Wang, Y., Pradel, M., & Liu, Z. (2026) · Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering · doi:10.1145/3744916.3764576 · Cited by 3
“We envision PatchDiff to be useful for building a sustainable and robust patch validation ecosystem for assessing issue-solving tools.”
VERINA: Benchmarking Verifiable Code Generation
Ye, Z., Yan, Z., He, J., Kasriel, T., Yang, K., & Song, D. (2025) · arXiv · doi:10.48550/arxiv.2505.23135 · Cited by 3
“We first consider MBPP-DFY-50 [18], which contains MBPP [24] coding problems paired with human-verified solutions in Dafny.”
InfluentialAugmented Intelligence for Instructional Systems in Simulation-Based Training
van Oijen, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-60609-0_7 · Cited by 3
Gamification of conceptual modeling education with UML class diagrams: an experimental analysis
Garaccione, G., Coppola, R., Ardito, L., & Torchiano, M. (2025) · Software & Systems Modeling · doi:10.1007/s10270-025-01282-5 · Cited by 3
Enhancing Code Intelligence with CodeT5: A Unified Approach to Code Analysis and Generation
Reddy, P. Y., Aarthi, N. S. S., Pooja, S., Veerendra, B., & Sumathi, D. (2025) · International Conference on Artificial Intelligence and Data Engineering (AIDE) · doi:10.1109/aide64228.2025.10987356 · Cited by 3
Eberhardinger, M., Maucher, J., & Maghsudi, S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment · doi:10.1609/aiide.v19i1.27516 · Cited by 4
It Ain’t Over: A Multi-aspect Diverse Math Word Problem Dataset
Kim, J., Kim, Y., Baek, I., Bak, J., & Lee, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.927 · Cited by 7
Emerging Trends in LLM Benchmarking
Desai, A. P., Prajapati, R., Ravi, T., Luqman, M., & Yadav, P. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825845 · Cited by 4
Generative AI in Software Engineering: Revolutionizing Code Generation and Debugging
Saravanan, V., Kavitha, S., Ravi, S., Seetha, A., Rambabu, C., & Kanth, T. V. R. (2025) · International Journal of Computational and Experimental Science and Engineering · doi:10.22399/ijcesen.1718 · Cited by 7
“These models, trained on vast corpora of open-source code, can learn complex programming patterns, syntax rules, and context-based reasoning, enabling them to mimic the behavior of expert developers [5].”
FormalSpecCpp: A Dataset of C++ Formal Specifications created using LLMs
Chakraborty, M., Pirkelbauer, P., & Yi, Q. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00113 · Cited by 3
SwiftEval: Developing a Language-Specific Benchmark for LLM-generated Code Evaluation
Petrukha, I., Kurliak, Y., & Stulova, N. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00015 · Cited by 3
“Recent approaches have primarily focused on evaluating English-to-Python code generation (e.g. [1], [9], [12], [13]).”
AI-Driven Document Analysis: Employing Streamlit, Faiss, Nvidia Nemo
Singh, N. T., Kaur, H., Dhiman, J., Aryan, A., Rani, J., & Wadhwa, M. (2025) · International Conference on Inventive Computing and Informatics (ICICI) · doi:10.1109/icici65870.2025.11069902 · Cited by 3
“It is a machine learning package that is built to solve one kind of problem called natural language processing (NLP) problems that include large language models and question answering [15].”
Benchmarking AI: Toward Inclusive Evaluation of Language Models
Jovanovic, M., & Campbell, M. (2025) · Computer · doi:10.1109/mc.2025.3575880 · Cited by 3
A Self-Iteration Code Generation Method Based on Large Language Models
Chang, T., Chen, S., Fan, G., & Feng, Z. (2023) · IEEE International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/icpads60453.2023.00049 · Cited by 3
Code Generation Using Self-Interactive Assistant
Zhao, Z., Sun, J., Cai, C., & Wei, Z. (2024) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00377 · Cited by 3
Self-citationE-code: Mastering efficient code generation through pretrained models and expert encoder group
Pan, Y., Lyu, C., Yang, Z., Li, L., Liu, Q., & Shao, X. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107602 · Cited by 4
An Empirical Study on Automatically Detecting AI-Generated Source Code: How Far Are We?Preprint
Suh, H., Tafreshipour, M., Li, J., Bhattiprolu, A., & Ahmed, I. (2025) · Qeios · doi:10.32388/sbl97o · Cited by 3
Assessing Small Language Models for Code Generation: An Empirical Study with BenchmarksPreprint
Hasan, M. M., Waseem, M., Kemell, K., Rasku, J., Ala-Rantala, J., & Abrahamsson, P. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5472927 · Cited by 3
Seed-Coder: Let the Code Model Curate Data for Itself
Seed, B., Zhang, Y., Su, J., Sun, Y., Xi, C., Xiao, X., Zheng, S., Zhang, A., Liu, K., Zan, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.03524 · Cited by 3
“MBPP [Austin et al., 2021] is created by crowd-sourcing participants to write 974 Python programming problems, each of which is comprised of a function signature with a docstring, as well as a few test cases.”
Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation
Zelikman, E., Lorch, E., Mackey, L., & Kalai, A. T. (2023) · arXiv · doi:10.48550/arxiv.2310.02304 · Cited by 4
“Thus, the goal is to find an improver program I with high expected utility when used on a task, For training, we assume we are given a collection of n downstream tasks D ∼ D n drawn independently from distribution D .”
R-U-SURE? Uncertainty-Aware Code Suggestions By Maximizing Utility Across Random User Intents
Johnson, D. D., Tarlow, D., & Walder, C. (2023) · arXiv · doi:10.48550/arxiv.2303.00732 · Cited by 3
FullStack Bench: Evaluating LLMs as Full Stack Coders
Bytedance-Seed-Foundation-Code-Team, :, Cheng, Y., Chen, J., Chen, J., Chen, L., Chen, L., Chen, W., Chen, Z., Geng, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.00535 · Cited by 2
NoviCode: Generating Programs from Natural Language Utterances by Novices
Mordechai, A. A., Goldberg, Y., & Tsarfaty, R. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00694 · Cited by 4
Collaboration with Generative AI to improve Requirements Change
Kong, Y., Zhang, N., Duan, Z., & Yu, B. (2025) · Computer Standards & Interfaces · doi:10.1016/j.csi.2025.104013 · Cited by 4
Evaluating Correct-Consistency and Robustness in Code-Generating LLMs
Honarvar, S. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10988971 · Cited by 2
“To evaluate the safety properties of LLMs for code, several works [3], [4], [9]–[12] have focused on assessing their ability to generate correct code.”
CoCoNUT: Structural Code Understanding does not fall out of a tree
Beger, C., & Dutta, S. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00021 · Cited by 3
Trustworthiness of Large Language Models for Code
Khati, D. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion66252.2025.00063 · Cited by 3
“These models are extensively applied to various SE tasks, including code completion [1]–[5], code summarization [6], [7], review [8], [9], traceability [10], translation [11], test case generation [12]–[14], clone detection [15], [16], and bug fixing [17]–[24].”
Retry and Refine: A Multi-agent Framework for LLM Based Reliable Unit Test Generation
Ghosh, T., & Abbas, N. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-11402-0_7 · Cited by 2
Li, J., Deng, H., Zhang, Y., Zhang, K., Shao, T., Zhao, T., Wang, W., Jin, Z. G., Li, G., Liu, Y., et al. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3797106 · Cited by 5
Retrofitting language models to operate over bytes
Minixhofer, B., Murray, T., Limisiewicz, T., Korhonen, A., Zettlemoyer, L., Smith, N. A., Ponti, E. D., Soldaini, L., & Hofmann, V. (2026) · Nature · doi:10.1038/s41586-026-11111-4 · Cited by 19
Better Modeling the Programming World with Code Concept Graphs-augmented Multi-modal Learning
Weyssow, M., Sahraoui, H. A., & Liu, B. (2022) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1145/3510455.3512771 · Cited by 7
Probing Semantic Grounding in Language Models of Code with Representational Similarity Analysis
Naik, S., Patil, R., Agarwal, S., & Baths, V. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-22137-8_29 · Cited by 2
Hey, T. (2023) · OECD eBooks · doi:10.1787/7b7b1bce-en · Cited by 4
Meta-requirements for LLM-Based Knowledge Exploration Tools in Information Systems Research
Sjöström, J., & Cronholm, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-61175-9_29 · Cited by 2
LLM-based Interactive Code Generation: Empirical Evaluation
Shaikhelislamov, D., Drobyshevskiy, M., & Belevantsev, A. A. (2024) · Ivannikov ISPRAS Open Conference (ISPRAS) · doi:10.1109/ispras64596.2024.10899123 · Cited by 2
Question Selection for Multimodal Code Search Synthesis Using Probabilistic Version Spaces
Wu, J., Jiang, Y., Wei, L., Xu, C., Cheung, S.-C., & Xu, C. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3565387 · Cited by 2
Deriving Coding-Specific Sub-Models from LLMs using Resource-Efficient Pruning
Puccioni, L., Farshin, A., Scazzariello, M., Wang, C., Chiesa, M., & Kostić, D. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00028 · Cited by 2
Piccolo, S., & Stevens, H. P. (2025) · GigaScience · doi:10.1093/gigascience/giaf149 · Cited by 1
“We gathered short-form, Python programming exercises from Austin et al. [ 29 ] ( n = 426) and Piccolo et al. ( n = 133) [ 30 ].”
The Case for a Single Model that can Both Generate Continuations and Fill-in-the-Blank
Ippolito, D., Dugan, L. D., Reif, E., Yuan, A., Coenen, A., & Callison-Burch, C. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.185 · Cited by 2
LFPS: Learned Formal Proof Strengthening for Efficient Hardware Verification
Kang, M., Nova, A., Singh, E., Bathini, G. S., & Viktorov, Y. (2023) · doi:10.1109/iccad57390.2023.10323681 · Cited by 2
DINGO: Towards Diverse and Fine-Grained Instruction-Following Evaluation
Gu, Z., Sun, X., Lian, F., Kang, Z., Xu, C., & Fan, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29768 · Cited by 3
A Survey of Natural Language-Based Editing of Low-Code Applications Using Large Language Models
Gorissen, S. C., Sauer, S., & Beckmann, W. G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-64576-1_15 · Cited by 2
Schuster, T., Kalyan, A., Polozov, O., & Kalai, A. T. (2021) · arXiv · doi:10.48550/arxiv.2106.05784 · Cited by 2
MiMo: Unlocking the Reasoning Potential of Language Model - From Pretraining to Posttraining
Xiaomi, L.-C., :, Xia, B., Shen, B., Cici, Zhu, D., Zhang, D., Wang, G., Zhang, H., Liu, H., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.07608 · Cited by 1
“Coding : LiveCodeBench (Jain et al., 2024), HumanEval (Chen et al., 2021), HumanEval+ (Liu et al., 2023), MBPP (Austin et al., 2021), MBPP+ (Liu et al., 2023), CRUXEval (Gu et al., 2024).”
Self-citationMeasuring AI Ability to Complete Long Software Tasks
Kwa, T., West, B., Becker, J., Deng, A., Garcia, K., Hasin, M., Jawhar, S., Kinniment, M., Rush, N., Von Arx, S., et al. (2025) · Advances in Neural Information Processing Systems 38 · doi:10.52202/085713-3086 · Cited by 21
“HumanEval [36] and MBPP [37] provide programming challenges of varying complexity, while more complex benchmarks like SWE-bench [28] and APPS [38] test more sophisticated programming abilities.”
Does Few-Shot Learning Help LLM Performance in Code Synthesis?
Xu, D., Xie, T., Xia, B., Li, H., Bai, Y., Sun, Y., & Wang, W. (2024) · arXiv · doi:10.48550/arxiv.2412.02906 · Cited by 3
“In fact, existing techniques evaluate on predefined prompt templates with little to no modifications (Austin et al., 2021; Liu et al., 2024b).”
SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers
Xiang, Y., Yan, H., Ouyang, S., Gui, L., & He, Y. (2025) · arXiv · doi:10.48550/arxiv.2504.00255 · Cited by 1
“…has emerged as a prominent application of LLMs, with benchmarks ranging from basic programming tasks (Chen et al., 2021; Jain et al., 2024; Austin et al., 2021; Hendrycks et al., 2021; Liu et al., 2022) to realistic software engineering challenges like SWE-bench (Jimenez et al., 2023),…”
McEval: Massively Multilingual Code Evaluation
Chai, L., Liu, S., Yang, J., Yin, Y., Jin, K., Liu, J., Sun, T., Zhang, G., Ren, C., Guo, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.07436 · Cited by 73
“Some works focus on the multilingual scenarios (Wang et al., 2023; Athiwaratkun et al., 2023; Peng et al., 2024; Zheng et al., 2023b) by extending the Python-only HumanEval/MBPP benchmark (e.g. MultiPL-E (Cassano et al., 2023)), which is challenged by the number of the languages and data leaking problem (Jain et al.,…”
BackgroundInfluentialBerges, V.-P., Oğuz, B., Haziza, D., Yih, W.-T., Zettlemoyer, L., & Ghosh, G. (2024) · arXiv · doi:10.48550/arxiv.2412.09764 · Cited by 46
“Our results (section 4) indicate that memory layers improve the factual accuracy of language models by over 100% as measured by factual QA benchmarks, while also improving significantly on coding (HumanEval, MBPP) and general knowledge (Hellaswag, MMLU).”
SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks
Li, X., Liu, Y., Chen, W., You, B., Di, Z., He, Y., Zheng, S., Choe, K. W., Sun, J., Wang, S., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.12670 · Cited by 1
“Other suites emphasize tool-mediated workflows, interactive execution feedback, or domain specialization (Yao et al., 2025; Trivedi et al., 2024; Yang et al., 2023; Chan et al., 2025; Zhang et al., 2024; Zhuo et al., 2025; Austin et al., 2021; Ye et al., 2025).”
SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents
Zhao, B., Srikanth, D., Wu, Y., & Jiang, Z. (2026) · arXiv · doi:10.48550/arxiv.2605.21384 · Cited by 33
“This directly instantiates the reward hacking framework from (Skalse et al., 2022), where optimizing a proxy reward ˆ 𝑅 diverges from the true objective 𝑅 ∗ ; here ˆ 𝑅 = 𝑠 val and ∗ = 𝑠 test .”
Context as a Tool: Context Management for Long-Horizon SWE-Agents
Liu, S., Yang, J., Jiang, B., Li, Y., Guo, J., Liu, X., & Dai, B. (2025) · arXiv · doi:10.48550/arxiv.2512.22087 · Cited by 35
“Large language models (LLMs) (Yang et al., 2025a; Touvron et al., 2023; Anthropic, 2025; Achiam et al., 2023) have achieved remarkable progress on tasks such as code generation and bug fixing (Chen, 2021; Austin et al., 2021; Liu et al., 2024b; Chai et al., 2024).”
EvoAgentX: An Automated Framework for Evolving Agentic Workflows
Wang, Y., Liu, S., Fang, J., & Meng, Z. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2025.emnlp-demos.47 · Cited by 1
Automatic Unit Test Data Generation and Actor-Critic Reinforcement Learning for Code Synthesis
Gorinski, P. J., Zimmer, M., Λάμπουρας, Γ., Deik, D. G. X., & Iacobacci, I. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.28 · Cited by 1
(2024) · Journal of System and Management Sciences · doi:10.33168/jsms.2024.1015 · Cited by 3
Forsthofer, N., & Kunc, O. (2025) · AIAA SciTech Forum · doi:10.2514/6.2025-1791 · Cited by 1
On the Applicability of Code Language Models to Scientific Computing Programs
Zhao, Q., Liu, F., Long, X., Wu, C., & Zhang, L. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3564599 · Cited by 2
“MultiPL-E [8] is a system for translating unit test-driven code generation benchmarks to new languages, which extends the HumanEval benchmark [11] and MBPP benchmark [4] to 18 languages that compass a range of programming paradigms and popularity.”
UniGenCoder: Merging SEQ2SEQ and SEQ2TREE Paradigms for Unified Code Generation
Shao, L., Yan, Y., Poshyvanyk, D., & Su, J. (2025) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1109/icse-nier66352.2025.00020 · Cited by 3
Toward Enhancing Reasoning Capabilities of LLMs: An Approach via Synthetic Logic Corpus
Morishita, T., Morio, G., Yamaguchi, A., & Sogawa, Y. (2025) · Journal of Natural Language Processing · doi:10.5715/jnlp.32.520 · Cited by 1
Random Perturbation Attack on LLMs for Code Generation
Peng, Q., Zhang, C., Mangal, R., Păsăreanu, C. S., & Jia, L. (2025) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain66642.2025.00052 · Cited by 1
“In evaluating LLMs for code generation, HumanEval[2] and MBPP (Mostly Basic Python Problems[1]) are widely used benchmark datasets[10].”
Code Ranking with Structure Awareness Contrastive Learning
Huang, H., Cao, L., Wang, J., Yu, T., & Cai, Y. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00053 · Cited by 1
“Tovalidate the effectiveness of SACL-CR, we perform experiments on two code generation benchmarks: HumanEval [6] and MBPP [7].”
An end-to-end approach for fixing concurrency bugs via SHB-based context extractor
Li, Z., Yi, Q., Xiao, K., Ji, Z., & Liang, H. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112888 · Cited by 1
Chen, L., Ahmed, N. K., Capotă, M., Willke, T., Hasabnis, N., & Jannesari, A. (2025) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps64566.2025.00055 · Cited by 2
“It leads to the development of several benchmarks, including HumanEval [8], MBPP [9], MBXP [19], CodeContests [20], and DS-1000 [21].”
Enhancing Active Learning in Education Through ChatGPT and Concept Maps: A Case Study
Schicchi, D., & Taibi, D. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-94002-6_1 · Cited by 1
Le, N., Ninh, Q., Le, T., & Nguyen, H. T. (2025) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2025.110607 · Cited by 1
Mao, K., Hu, B., Lin, R., Li, Z., Lu, G., & Zhang, Z. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1660912 · Cited by 4
SEER: Self-Enhancing Chain-of-Thought Compression for Reasoning Models
Huang, K., Liu, S., Hu, X., Xu, T., Bao, L., & Xia, X. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832120 · Cited by 14
Automata-Based Steering of Large Language Models for Diverse Structured Generation
Luan, X., Wei, Z., Zhang, Y., & Sun, M. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-4213-0_2 · Cited by 4
Kokate, R., & Onkar, P. S. (2026) · Design Science · doi:10.1017/dsj.2025.10044 · Cited by 4
“…performs across different tasks, ranging from language skills and reasoning – HellaSwag (Zellers et al. 2019), MMLU (Hendrycks et al. 2020), for mathematical tasks – GSM8K (Cobbe et al. 2021), MATH (Hendrycks et al. 2021) and MBPP (Austin et al. 2021) and for coding – HumanEval (Chen et al. 2021).”
A Systematic Literature Review of Parameter-Efficient Fine-Tuning for Large Code Models
Afrin, S., Haque, M. Z., & Mastropaolo, A. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3796522 · Cited by 12
“…Bug Repair [7, 33, 36, 57, 58, 61, 107, 155, 156, 161, 172], Code Summarization and Documentation [6, 8, 93, 94, 141, 169, 169], Code Completion [13, 23, 24, 41, 53, 80], Software Testing [95, 98, 118, 122, 134, 135], and Code Review [69, 75, 88, 137] have increasingly benefited from the…”
Regression Accumulation in Multi-turn LLM Programming Conversations
Huang, A., Ma, L., Tahir, A., Zhang, Q., Xiao, L., & Xiao, L. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834357 · Cited by 1
Think Before You Code: Dual Reasoning for the NLSafety–Utility Trade-Off in LLM Code Generation
Tan, H., Wang, H., & Tan, S. H. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834385
Sheng, C. Y., Xin, C. Y., & Jie, T. G. (2026) · Cureus Journal of Computer Science. · doi:10.7759/s44389-026-00047-5 · Cited by 1
Assessing the Sustainability of LLM Inference through Energy–Accuracy Analysis
Ali, S. B. M., Danish, S. M., Sattar, Z., & Asad, A. (2026) · ACM International Conference on Future Energy Systems (e-Energy) · doi:10.1145/3744255.3811741 · Cited by 1
“…We evaluate the models on four representative tasks, each using two widely adopted benchmarks: text summarization (XSum [20], CNN/DailyMail [13]), mathematical reasoning (GSM8K [8], MATH-500 [12]), question answering (TruthfulQA [17], Alpaca-52K [16]), and code generation (HumanEval [4], MBPP [7]).”
How much does an LLM know about my programming language?
Delgado, D., Burgueño, L., & Clarisó, R. (2026) · ACM SIGPLAN International Conference on Software Language Engineering (SLE) · doi:10.1145/3806383.3815518 · Cited by 1
“Widely used benchmarks such as HumanEval [Chen et al. 2021] and MBPP [Austin et al. 2021] have evolved mainly with respect to functional correctness since their introduction as gold standards for code generation.”
InfluentialAn Automated Methodology for Generating Labeled Datasets of Semantic Errors in Code
Kassem, M., Ribeiro, F., & Nadi, S. (2026) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1145/3786181.3788730 · Cited by 1
Foundation Models for Text Generation
Paaß, G., & Giesselbach, S. (2023) · Artificial intelligence: foundations, theory, and algorithms/Artificial intelligence: Foundations, theory, and algorithms · doi:10.1007/978-3-031-23190-2_6 · Cited by 3
Lessons from shortcomings in machine learning for medical imaging
Varoquaux, G. P., & Cheplygina, V. (2023) · OECD eBooks · doi:10.1787/b885eecd-en · Cited by 2
Yue, L., & Guo, L. (2023) · European Conference on Games Based Learning · doi:10.34190/ecgbl.17.1.1892 · Cited by 1
InstructExcel: A Benchmark for Natural Language Instruction in Excel
Payan, J., Mishra, S. R., Singh, M. P., Negreanu, C., Poelitz, C., Baral, C. R., Roy, S., Chakravarthy, R., Durme, B. V., & Nouri, E. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.265 · Cited by 5
Wang, R., Todd, G., Yuan, X., Xiao, Z., Côté, M.-A., & Jansen, P. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.830 · Cited by 17
Evaluating LLM’s Code Reading Abilities in Big Data Contexts using Metamorphic Testing
Li, Z., Li, Z., Xiao, K., & Li, X. (2023) · International Conference on Big Data and Information Analytics (BigDIA) · doi:10.1109/bigdia60676.2023.10429345 · Cited by 2
Evaluation of Large Language Models on Code Obfuscation (Student Abstract)
Swindle, A., McNealy, D., Krishnan, G. P., & Ramyaa, R. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i21.30517 · Cited by 1
Aligning Crowd-sourced Human Feedback for Code Generation with Bayesian Inference
Wong, M. F., & Tan, C. W. (2024) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai59869.2024.00037 · Cited by 2
CIPAC: A framework of automated software construction based on collective intelligence
Liu, J., Zhang, Y., Li, Y., Ma, T., & Dong, W. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112335 · Cited by 2
DemoCraft: Using In-Context Learning to Improve Code Generation in Large Language Models
Joshua, K. N., & Sreejith, M. (2025) · International Conference on Intelligent and Innovative Technologies in Computing, Electrical and Electronics (IITCEE) · doi:10.1109/iitcee64140.2025.10915349 · Cited by 1
Revisiting the Non-Determinism of Code Generation by the GPT-3.5 Large Language Model
Sawadogo, S., Sabané, A., Kafando, R., Kaboré, A. K., & Bissyandé, T. F. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00012 · Cited by 1
CODEMORPH: Mitigating Data Leakage in Large Language Model Assessment
Rao, H., Zhao, Y., Zhu, W., Xiao, L., Wang, M., & Wang, H. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion66252.2025.00081 · Cited by 1
“For example, Meta AI evaluated Code Llama [29] on HumanEval [16] and MBPP [11], demonstrating its performance superiority over some smaller models such as PaLM-Coder [10] and OpenAI’s code-cushman-001.”
Optimal Neural Program Synthesis from Multimodal SpecificationsPreprint
Ye, X., Chen, Q., Dillig, I., & Durrett, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.01678 · Cited by 10
Controlling Conditional Language Models with Distributional Policy GradientsPreprint
Korbak, T., Elsahar, H., Kruszewski, G., & Dymetman, M. (2021) · arXiv (Cornell University) · Cited by 1
Tagungsband zum 21. Kolloquium Programmiersprachen und Grundlagen der Programmierung
Heinlein, C. (2021) · Kiel computer science series · doi:10.21941/kcss/2021/7 · Cited by 1
Predicting typeScript type annotations and definitions with machine learning
Yee, M. (2024) · doi:10.17760/d20653005 · Cited by 1
Metadata
- Authors
- Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., & Sutton, C.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2108.07732
- OpenAlex
- W3195140297
- Cited by
- 388
- References
- 93
- Citations by year
- 2026: 44 · 2025: 129 · 2024: 92 · 2023: 85 · 2022: 21 · 2021: 15
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- TIB Data Manager
- Funders
- National Natural Science Foundation of China (62472339, 62302375)
- Hankyong National University
- Ministry of Industry and Information Technology of the People's Republic of China
- HORIZON EUROPE Framework Programme (101070305)
- Fundação para a Ciência e a Tecnologia (UIDB/00319, 00319, UIDB/00319/2020, UIDB/05757/2020)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- program synthesis, large language models, Python code generation, MBPP, few-shot learning, fine-tuning, model size scaling, human feedback, error analysis, semantic grounding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
