The DOI matches its DataCite record; 92 of the 93 references it cites were found in a registry or an index, and 1 are not checked yet.
Program Synthesis with Large Language Models
Open accessAbstract
This paper explores the limits of the current generation of large language models for program synthesis in general purpose programming languages. We evaluate a collection of such models (with between 244M and 137B parameters) on two new benchmarks, MBPP and MathQA-Python, in both the few-shot and fine-tuning regimes. Our benchmarks are designed to measure the ability of these models to synthesize short Python programs from natural language descriptions. The Mostly Basic Programming Problems (MBPP) dataset contains 974 programming tasks, designed to be solvable by entry-level programmers. The MathQA-Python dataset, a Python version of the MathQA benchmark, contains 23914 problems that evaluate the ability of the models to synthesize code from more complex text. On both datasets, we find that synthesis performance scales log-linearly with model size. Our largest models, even without finetuning on a code dataset, can synthesize solutions to 59.6 percent of the problems from MBPP using few-shot learning with a well-designed prompt. Fine-tuning on a held-out portion of the dataset improves performance by about 10 percentage points across most model sizes. On the MathQA-Python dataset, the largest fine-tuned model achieves 83.8 percent accuracy. Going further, we study the model's ability to engage in dialog about code, incorporating human feedback to improve its solutions. We find that natural language feedback from a human halves the error rate compared to the model's initial prediction. Additionally, we conduct an error analysis to shed light on where these models fall short and what types of programs are most difficult to generate. Finally, we explore the semantic grounding of these models by fine-tuning them to predict the results of program execution. We find that even our best models are generally unable to predict the output of a program given a specific input.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“In practice, large pools of test cases, Dj , can either originate from human generated documentation (Nassif et al., 2021;Lemieux et al., 2023) or GPTgenerated unit tests (Austin et al., 2021).”
From Does Few-Shot Learning Help LLM Performance in Code Synthesis? (2024) · cites this paper as Austin et al., 2021
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Code data (e.g., from Stack Exchange and GitHub) substantially enhances program synthesis capabilities [37,38].”
From A Survey of Large Language Models (2026) · cites this paper as [38]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Neural code synthesis approaches to code completion generate suggestions by using a language model to predict what the user might type next (the completion) from the context of what they are working on at the moment (the prompt) [2].”
From Productivity Assessment of Neural Code Completion (2022) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Similar studies in the code space are limited to the number of few-shot examples in the prompt, and the effect of length of prompt on performance [9,15].”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [9]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, for the Performance category, existing works have used the Computational Accuracy metric to evaluate generative code models [9,15,33,88], but not other metrics we uncovered regarding the characteristics of the generated artifacts and run-time efficiency.”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [9]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large pre-trained language models perform remarkably well on tasks that can be done 'in one pass', such as generating realistic text (Brown et al., 2020) or synthesizing computer programs (Chen et al., 2021; Austin et al., 2021).”
From Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021) · cites this paper as Austin et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, they struggle with tasks that require unbounded multi-step computation, such as adding integers (Brown et al., 2020) or executing programs (Austin et al., 2021).”
From Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021) · cites this paper as Austin et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The 'MBPP-aug + CodeNet + single line' model achieved an overall synthesis accuracy of 54%, compared to the 62% accuracy of the original few-shot model in Austin et al. (2021).”
From Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021) · cites this paper as Austin et al. (2021)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“MBPP is another large, commonly used benchmark of Python problems.”
From MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current benchmarks for evaluating neural code models focus on only a small subset of programming languages, excluding many popular languages such as Go or Rust.”
From Measuring The Impact Of Programming Language Distribution (2023) · cites this paper as Austin et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“HumanEval [7]-developed by OpenAI for evaluating Codex-and other benchmarks [2,13,20] only consist of programming problems in a single language”
From CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X (2023) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent large language models (LLMs) have demonstrated strong capabilities in understanding and generating code, from competitive programming to repository-level software engineering.”
From Code as Agent Harness (2026) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In fact, existing techniques evaluate on predefined prompt templates with little to no modifications (Austin et al., 2021;Liu et al., 2024b).”
From Does Few-Shot Learning Help LLM Performance in Code Synthesis? (2024) · cites this paper as Austin et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The development of evaluations in tandem with language models (LMs) has substantially contributed to the rapid advancement of these models [29, 11, 7, 25, 81, 27, 72].”
From SciCode: A Research Coding Benchmark Curated by Scientists (2024) · cites this paper as [7]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“HumanEval [5], MBPP [3], and APPS [13] score function-level synthesis; CrossCodeEval [11] targets cross-file completion; CodeClash [33] and PinchBench [18] probe further axes of coding capability.”
From Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks (2026) · cites this paper as [3]
The cited source says what the claim attributes to it.
In the citing paper
“Researchers have already investigated the benefits of natural language feedback to guide program synthesis [2]”
From Productivity Assessment of Neural Code Completion (2022) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“Researchers have already investigated the benefits of natural-language feedback to guide program synthesis, 2 so the conversational framing of coding completions is not a radical proposal.”
From Measuring GitHub Copilot's Impact on Productivity (2024) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“Direct execution prediction has been shown to perform poorly on Python programs in Austin et al. (2021).”
From Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Similarly, large-scale language models struggle to predict the result of executing Python code, even code which is a solution to a programming task the model is able to solve (Austin et al., 2021).”
From Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“However, previous work has found that textual similarity metrics correlate only weakly with code correctness [1,2,27], highlighting the importance of benchmark suites with unit tests.”
From MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“MBPP dataset is proposed by [2], containing 974 problems in Python.”
From CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X (2023) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“Different from prior works on utilizing human feedback for code repair, where the feedback message explains the code errors and how to fix them (Chen et al., 2023a;Austin et al., 2021), SELF-DEBUGGING teaches the model to identify the implementation errors via investigating into execution results and explaining the code by itself.”
From Teaching Large Language Models to Self-Debug (2023) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“MBPP prompts follow a templated structure where the critical task specification appears mid-prompt:”
From Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression (2026) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“The Mostly Basic Python Problems (MBPP) benchmark (Austin et al., 2021) takes a complementary approach, providing 974 crowd-sourced programming problems intended to be solvable by entry-level programmers.”
From Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression (2026) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) evaluate isolated functions.”
From SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents (2026) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“MBPP, on the other hand, comprises 974 programming problems created by crowdsource contributors.”
From Qwen2.5-Coder Technical Report (2024) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“In particular, MBPP -verified is sourced from (Yang et al., 2024;Misu et al., 2024) and contains 78 programs from the original MBPP dataset (Austin et al., 2021).”
From AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinement (2024) · cites this paper as Austin et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Part of this paper’s reference list could not be split into single entries.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“The models we use in this paper are dense left-to-right decoder-only Transformer language models [Vaswani et al., 2017] trained on a combination of web documents, dialog data, and Wikipedia.”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“P L ] 1 6 A ug 2 text [Brown et al., 2020, Raffel et al., 2019] and to solve a rapidly expanding set of modeling and reasoning tasks [Devlin et al., 2019, big-bench collaboration, 2021].”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 1×
“…line of previous work on neural language models of natural language text [Mikolov et al., 2010, Sutskever et al., 2011, Józefowicz et al., 2016, Dai and Le, 2015, Peters et al., 2018, Howard and Ruder, 2018], especially recent large Transformer models [Radford et al., 2018, Brown et al., 2020].”
BackgroundOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · cited 1×
“Our work is inspired by the long line of previous work on neural language models of natural language text [Mikolov et al., 2010, Sutskever et al., 2011, Józefowicz et al., 2016, Dai and Le, 2015, Peters et al., 2018, Howard and Ruder, 2018], especially recent large Transformer models [Radford et…”
MethodKudo, T., & Richardson, J. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-2012 · Cited by 4,454 · cited 1×
“81T BPE tokens with a vocabulary of 32K tokens using the SentencePiece library [Kudo and Richardson, 2018].”
MethodUniversal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 1×
“…line of previous work on neural language models of natural language text [Mikolov et al., 2010, Sutskever et al., 2011, Józefowicz et al., 2016, Dai and Le, 2015, Peters et al., 2018, Howard and Ruder, 2018], especially recent large Transformer models [Radford et al., 2018, Brown et al., 2020].”
BackgroundThe Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · cited 1×
“These results suggest that methods such as prompt-tuning [Li and Liang, 2021, Lester et al., 2021] could yield substantial performance improvements in this domain.”
ResultThe Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092 · cited 1×
“These results suggest that methods such as prompt-tuning [Li and Liang, 2021, Lester et al., 2021] could yield substantial performance improvements in this domain.”
Result
Flagged first, then by citations
Research works printed but not linked: 1
- [10]10. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, Will Guss, Alex Nichol, Igor Babuschkin, Suchir Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, July 2021. URL http://arxiv.org/abs/2107.03374 . · arXiv
Cited by
Sorted by citations
Items 101–200 of 529
Understanding HTML with Large Language Models
Gür, İ., Nachum, O., Miao, Y., Safdari, M., Huang, A., Chowdhery, A., Narang, S., Fiedel, N., & Faust, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.185 · Cited by 44
REFINER: Reasoning Feedback on Intermediate RepresentationsPreprint
Paul, D., Ismayilzada, M., Peyrard, M., Borges, B., Bosselut, A., West, R., & Faltings, B. V. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.01904 · Cited by 35
CRUXEVAL-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution
Xu, R., Cao, J., Lu, Y., Wen, M., Lin, H., Han, X., He, B., Cheung, S.-C., & Sun, L. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.1158 · Cited by 35
Can Large Language Models be Anomaly Detectors for Time Series?
Alnegheimish, S., Nguyen, L., Berti‐Équille, L., & Veeramachaneni, K. K. (2024) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa61799.2024.10722786 · Cited by 44
TheoremQA: A Theorem-driven Question Answering Dataset
Chen, W., Yin, M., Ku, M., Lu, P., Wan, Y., Ma, X., Xu, J., Wang, X., & Xia, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.489 · Cited by 141
Calibration and Correctness of Language Models for Code
Spieß, C. K., Gros, D., Pai, K. S., Pradel, M., Rabin, M. R. I., Alipour, M. A., Jha, S., Dévanbu, P., & Ahmed, T. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00040 · Cited by 102
Large Language Models are Not Models of Natural Language: They are Corpus Models
Veres, C. (2022) · IEEE Access · doi:10.1109/access.2022.3182505 · Cited by 43
Phan, L., Gatti, A., Han, Z., & Li, N. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i1.13973 · Cited by 36
Several Categories of Large Language Models (LLMs): A Short Survey
Pahune, S. A., & Chandrasekharan, M. (2023) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2023.54677 · Cited by 54
SOEN-101: Code Generation by Emulating Software Process Models Using Large Language Model Agents
Lin, F., Kim, D. J., & Chen, T.-H. P. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00140 · Cited by 150
Zhou, X., Liang, P., Zhang, B., Li, Z., Ahmad, A., Shahin, M., & Waseem, M. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112204 · Cited by 30
“…driven by LLMs that have been trained on large amounts of code snippets are increasingly in the spotlight (e.g., AI-augmented development in Gartner Trends 2024 (Gartner, 2023)), making it possible for programmers to automatically generate code with minimized human effort (Austin et al., 2021).”
Three Bricks to Consolidate Watermarks for Large Language Models
Fernandez, P., Chaffin, A., Tit, K., Chappelier, V., & Furon, T. (2023) · IEEE International Workshop on Information Forensics and Security · doi:10.1109/wifs58808.2023.10374576 · Cited by 63
Large circuit models: opportunities and challenges
Chen, L., Chen, Y., Chu, Z., Fang, W., Ho, T.-Y., Huang, R., Huang, Y., Khan, S., Li, M., Li, X., et al. (2024) · Science China Information Sciences · doi:10.1007/s11432-024-4155-7 · Cited by 86
Hussain, R. G., Yow, K., & Gori, M. (2025) · IEEE Access · doi:10.1109/access.2024.3525069 · Cited by 29
Leveraging Prompt Engineering in Large Language Models for Accelerating Chemical Research
Luo, F., Zhang, J., Wang, Q., & Yang, C. (2025) · ACS Central Science · doi:10.1021/acscentsci.4c01935 · Cited by 34
CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation
Peng, J., Cui, L., Huang, K., Yang, J., & Ray, B. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00009 · Cited by 84
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.15877 · Cited by 29
“Complexity Diversity APPS [23] 352.9 137 DS-1000 [32] 138.1 328 ODEX [58] 50.4 230 APIBench [49] 77.5 171 MBPP [9] 181.”
BackgroundMethodInfluentialLarge language models as tool makers
Cai, T., Wang, X., Ma, T., Chen, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2305.17126 · Cited by 28
Learning-based Widget Matching for Migrating GUI Test Cases
Zhang, Y., Zhang, W., Ran, D., Zhu, Q., Dou, C., Hao, D., Xie, T., & Zhang, L. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3623322 · Cited by 42
The Mutators Reloaded: Fuzzing Compilers with Large Language Model Generated Mutation Operators
Ou, X., Li, C., Jiang, Y., & Xu, C. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3622781.3674171 · Cited by 37
SpecRover: Code Intent Extraction via LLMs
Ruan, H., Zhang, Y., & Roychoudhury, A. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00080 · Cited by 45
SpecRover: Code Intent Extraction via LLMs
Ruan, H., Zhang, Y., & Roychoudhury, A. (2024) · arXiv · doi:10.48550/arxiv.2408.02232 · Cited by 28
SLaDe: A Portable Small Language Model Decompiler for Optimized Assembly
Armengol-Estapé, J., Woodruff, J., Cummins, C., & O’boyle, M. F. P. (2024) · Proceedings of the CGO/Proceedings of the ... CGO · doi:10.1109/cgo57630.2024.10444788 · Cited by 52
Benchmarks and Metrics for Evaluations of Code Generation: A Critical Review
Paul, D. G., Zhu, H., & Bayley, I. (2024) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest62860.2024.00019 · Cited by 34
Reasoning Runtime Behavior of a Program with LLM: How Far are We?
Chen, J., Pan, Z., Hu, X., Li, Z., Li, G., & Xia, X. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00012 · Cited by 40
RLCoder: Reinforcement Learning for Repository-Level Code Completion
Wang, Y., Wang, Y., Guo, D., Chen, J., Zhang, R., Ma, Y., & Zheng, Z. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00014 · Cited by 43
Code Execution with Pre-trained Language Models
Liu, C., Lu, S., Chen, W., Jiang, D., Svyatkovskiy, A., Fu, S., Sundaresan, N., & Duan, N. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.308 · Cited by 37
Repository-Level Prompt Generation for Large Language Models of Code
Shrivastava, D., Larochelle, H., & Tarlow, D. (2022) · arXiv · doi:10.48550/arxiv.2206.12839 · Cited by 27
“Codex [8], Google’s model [2], GPT-J-6B [42], GPT-Neo [5], GPT-Neo-X [4], CodeParrot [40], PolyCoder [47] and InCoder [11] are some examples.”
MethodInfluentialHow Important Are Good Method Names in Neural Code Generation? A Model Robustness Perspective
Yang, G., Zhou, Y., Yang, W., Yue, T., Chen, X., & Chen, T. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3630010 · Cited by 34
Gu, Z., Jia, W., Piccardi, M., & Yu, P. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103078 · Cited by 27
Survey on Evaluation of LLM-based Agents
Yehudai, A., Eden, L., Li, A., Uziel, G., Zhao, Y., Bar-Haim, R., Cohan, A., & Shmueli-Scheuer, M. (2025) · arXiv · doi:10.48550/arxiv.2503.16416 · Cited by 26
Metasql: A Generate-Then-Rank Framework for Natural Language to SQL Translation
Fan, Y., He, Z., Ren, T., Huang, C., Jing, Y., Zhang, K., & Wang, X. S. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00143 · Cited by 36
Token-Budget-Aware LLM Reasoning
Han, T., Wang, Z., Fang, C., Zhao, S., Ma, S., & Chen, Z. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.1274 · Cited by 31
Will Code Remain a Relevant User Interface for End-User Programming with Generative AI Models?
Sarkar, A. (2023) · ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) · doi:10.1145/3622758.3622882 · Cited by 32
Learning Performance-Improving Code Edits
Shypula, A., Madaan, A., Zeng, Y., Alon, U., Gardner, J., Hashemi, M., Neubig, G., Ranganathan, P., Bastani, O., & Yazdanbakhsh, A. (2023) · arXiv · doi:10.48550/arxiv.2302.07867 · Cited by 146
“Recent work leverages machine learning to assist program synthesis (Balog et al., 2016; Bunel et al., 2018; Devlin et al., 2017a;b; Kalyan et al., 2018; Parisotto et al., 2016; Odena & Sutton, 2020; Nye et al., 2019; Meng et al., 2011; Jain et al., 2022; Austin et al., 2021; Xiong & Wang, 2022; Cambronero et al., 2023…”
BackgroundContext-Aware Abbreviation Expansion Using Large Language Models
Cai, S., Venugopalan, S., Tomanek, K., Narayanan, A., Morris, M. R., & Brenner, M. P. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.91 · Cited by 26
RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
Gehring, J., Zheng, K., Copet, J., Mella, V., Carbonneaux, Q., Cohen, T., & Synnaeve, G. (2024) · arXiv · doi:10.48550/arxiv.2410.02089 · Cited by 23
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing
Qiu, L., Shaw, P. M., Pasupat, P., Shi, T., Herzig, J., Pitler, E., Sha, F., & Toutanova, K. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.624 · Cited by 41
Towards Understanding the Characteristics of Code Generation Errors Made by Large Language Models
Wang, Z., Zhou, Z., Song, D., Huang, Y., Chen, S., Ma, L., & Zhang, T. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00180 · Cited by 87
LUNA: A Model-Based Universal Analysis Framework for Large Language Models
Song, D., Xie, X., Song, J., Zhu, D., Huang, Y., Juefei-Xu, F., & Ma, L. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3411928 · Cited by 33
Routerdc: Query-based router by dual contrastive learning for assembling large language models
Chen, S., Jiang, W., Lin, B., Kwok, J., & Zhang, Y. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-2120 · Cited by 22
Xiao, C., Cai, J., Zhao, W., Lin, B., Zeng, G., Zhou, J., Zheng, Z., Han, X., Liu, Z., & Sun, M. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01137-0 · Cited by 24
“Following widely used settings, MMLU, BBH, MATH, HumanEval and MBPP were evaluated under the 5-shot, 3-shot, 4-shot, 0-shot and 3-shot settings, respectively.”
InfluentialFakhoury, S., Naik, A., Sakkas, G., Chakraborty, S., Musuvathi, M., & Lahiri, S. K. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3639478.3643525 · Cited by 30
Multi-stage guided code generation for Large Language Models
Han, Y., & Lyu, C. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109491 · Cited by 25
AgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodology
Nguyen, M. H., Chau, T. P., Nguyen, P. X., & Bui, N. D. Q. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00026 · Cited by 68
Natural Language to Code Generation in Interactive Data Science Notebooks
Yin, P., Li, W.-D., Xiao, K., Rao, A., Wen, Y., Shi, K., Howland, J., Bailey, P., Catasta, M., Michalewski, H., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.9 · Cited by 74
Self-citationAutoVerus: Automated Proof Generation for Rust Code
Yang, C., Li, X., Misu, M. R. H., Yao, J., Cui, W., Gong, Y., Hawblitzel, C., Lahiri, S., Lorch, J. R., Lu, S., et al. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3763174 · Cited by 20
The “Code” of Ethics: A Holistic Audit of AI Code Generators
Ma, W., Song, Y., Xue, M., Wen, S., & Xiang, Y. (2024) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2024.3367737 · Cited by 25
Transformers in source code generation: A comprehensive survey
Ghaemi, H., Alizadehsani, Z., Shahraki, A., & Corchado, J. M. (2024) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2024.103193 · Cited by 27
(2022) · doi:10.2139/ssrn.4496380 · Cited by 20
Agentless: Demystifying LLM-based Software Engineering Agents
Xia, C. S., Deng, Y., Dunn, S., & Zhang, L. (2024) · arXiv · doi:10.48550/arxiv.2407.01489 · Cited by 24
“Large language models (LLMs) have become the go-to default choice for code generation [31, 28, 53, 89].”
BackgroundL2CEval : Evaluating Language-to-Code Generation Capabilities of Large Language Models
Ni, A., Yin, P., Zhao, Y., Riddell, M., Feng, T., Shen, R., Yin, S., Ye, L., Yavuz, S., Xiong, C., et al. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00705 · Cited by 49
On the Evaluation of Neural Code Translation: Taxonomy and Benchmark
Jiao, M., Yu, T., Li, X., Qiu, G., Gu, X., & Shen, B. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00114 · Cited by 40
An Open-Source Data Contamination Report for Large Language Models
Li, Y., Guo, Y., Guerin, F., & Lin, C. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.30 · Cited by 19
A Careful Examination of Large Language Model Performance on Grade School Arithmetic
Zhang, H., Da, J., Lee, D., Robinson, V., Wu, C., Song, W., Zhao, T., Raja, P., Zhuang, C., Slack, D., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.00332 · Cited by 19
Licoeval: Evaluating LLMs on License Compliance in Code Generation
Xu, W., Gao, K., He, H., & Zhou, M. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00052 · Cited by 39
The Impact of Prompt Programming on Function-Level Code Generation
Khojah, R., Neto, F. G. D. O., Mohamad, M., & Leitner, P. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3587794 · Cited by 59
Let’s Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs
Aggarwal, P., Madaan, A., Yang, Y., & Mausam, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.761 · Cited by 106
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ac4sp · Cited by 143
Show Your Work: Scratchpads for Intermediate Computation with Language Models
Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.00114 · Cited by 144
“Our evaluation dataset is the Mostly Basic Python Programming Problems (MBPP) dataset, introduced in Austin et al. (2021).”
InfluentialSelf-citationOctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 245
“, 2022b), Jupyter Code Pairs,6 MBPP (Austin et al., 2021), XLCoST (Zhu et al.”
BackgroundDeep learning for code generation: a survey
Zhang, H., Zhang, K., Li, Z., Li, J., Li, J., Li, Y., Zhao, Y., Zhu, Y. S., Liu, F., Li, G., et al. (2024) · Science China Information Sciences · doi:10.1007/s11432-023-3956-3 · Cited by 20
Evaluation of Generative AI Models in Python Code Generation: A Comparative Study
Palla, D., & Slabý, A. (2025) · IEEE Access · doi:10.1109/access.2025.3560244 · Cited by 27
An Empirical Cybersecurity Evaluation of GitHub Copilot's Code ContributionsPreprint
Pearce, H. A., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2021) · arXiv (Cornell University) · Cited by 17
Generating vulnerability security fixes with Code Language Models
Bhandari, G. P., Gavrić, N., & Shalaginov, A. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107786 · Cited by 21
Optimal Neural Program Synthesis from Multimodal Specifications
Ye, X., Chen, Q., Dillig, I., & Durrett, G. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.146 · Cited by 19
LLMEffiChecker : Understanding and Testing Efficiency Degradation of Large Language Models
Feng, X., Han, X., Chen, S. H., & Yang, W. M. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3664812 · Cited by 20
Leveraging large language models for comprehensive locomotion control in humanoid robots design
Sun, S., Li, C., Zhao, Z., Huang, H., & Xu, W. (2024) · Biomimetic Intelligence and Robotics · doi:10.1016/j.birob.2024.100187 · Cited by 18
Are Large Language Models Memorizing Bug Benchmarks?
Ramos, D., Mamede, C., Jain, K., Canelas, P., Gamboa, C., & Goues, C. L. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00005 · Cited by 40
Exploring the Impacts of ChatGPT on Future Scientific WorkPreprint
Irons, J. L., Mason, C., Cooper, P. S., Sidra, S., Reeson, A. F., & Paris, C. L. (2023) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/j2u9x · Cited by 16
Song, D., Liang, J., Xiao, X., & Manocha, D. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3559822 · Cited by 15
Learning to Synthesize Programs as Interpretable and Generalizable PoliciesPreprint
Trivedi, D., Zhang, J. M., Sun, S.-H., & Lim, J. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.13643 · Cited by 22
Measuring AI Ability to Complete Long Software Tasks
Kwa, T., West, B., Becker, J., Deng, A., Garcia, K., Hasin, M., Jawhar, S., Kinniment, M., Rush, N., Von Arx, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.14499 · Cited by 14
“HumanEval [36] and MBPP [37] provide programming challenges of varying complexity, while more complex benchmarks like SWE-bench [28] and APPS [38] test more sophisticated programming abilities.”
Can LLMs Generate Competency Questions?
Rebboud, Y., Tailhardat, L., Lisena, P., & Raphaël Troncy, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-78952-6_7 · Cited by 14
Online dispute resolution: can we leave the initial decision to Large Language Models (LLM)?
Ferrer-Benítez, M. (2022) · Metaverse Basic and Applied Research · doi:10.56294/mr202223 · Cited by 15
Benchmarking Causal Study to Interpret Large Language Models for Source Code
Rodriguez-Cardenas, D., Palacio, D. N., Khati, D., Burke, H., & Poshyvanyk, D. (2023) · 2023 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme58846.2023.00040 · Cited by 14
Lin, Y., Li, C., Ding, M., Tomizuka, M., Zhan, W., & Althoff, M. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3441493 · Cited by 22
Pytester: Deep reinforcement learning for text-to-testcase generation
Takerngsaksiri, W., Charakorn, R., Tantithamthavorn, C. K., & Li, Y.-F. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112381 · Cited by 27
Shaikh, T. A., Rasool, T., & Mir, W. A. (2025) · Computer Standards & Interfaces · doi:10.1016/j.csi.2025.104005 · Cited by 18
INTERTRANS: Leveraging Transitive Intermediate Translations to Enhance LLM-Based Code Translation
Macedo, M., Tian, Y., Nie, P., Cogo, F. R., & Adams, B. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00236 · Cited by 14
Absolute Zero: Reinforced Self-play Reasoning with Zero Data
Zhao, A., Wu, Y., Yue, Y., Wu, T., Xu, Q., Yue, Y., Lin, M., Wang, S., Wu, Q., Zheng, Z., et al. (2025) · arXiv · Cited by 13
ChatGPT and computational-based research: benefits, drawbacks, and machine learning applications
Atkinson, C. F. (2023) · Discover Artificial Intelligence · doi:10.1007/s44163-023-00091-3 · Cited by 18
Validating Semantic Artifacts with Large Language Models
Tüfek, N., Thuluva, A. S., Just, V. P., Ekaputra, F. J., Bandyopadhyay, T., Sabou, M., & Hanbury, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-78952-6_9 · Cited by 15
Piloting Copilot, Codex, and StarCoder2: Hot temperature, cold prompts, or black magic?
Döderlein, J.-B., Kouadio, N. H., Acher, M., Khelladi, D. E., & Combemale, B. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112562 · Cited by 67
Exploring and evaluating personalized models for code generation
Zlotchevski, A., Drain, D., Svyatkovskiy, A., Clement, C. B., Sundaresan, N., & Tufano, M. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3558959 · Cited by 13
BizBench: A Quantitative Reasoning Benchmark for Business and Finance
Krumdick, M., Koncel-Kedziorski, R., Lai, V. D., Reddy, V., Lovering, C., & Tanner, C. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.452 · Cited by 13
EvoR: Evolving Retrieval for Code Generation
Su, H., Jiang, S., Lai, Y., Wu, H., Shi, B., Liu, C., Liu, Q., & Yu, T. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.143 · Cited by 12
“…most of these works set up the evaluation in scenarios LLMs are familiar with, e.g., HumanEval (Chen et al., 2021), HumanEvalPack (Muennighoff et al., 2023) and MBPP (Austin et al., 2021), where they are capable of demonstrating superior zero-shot performance by only utilizing internal knowledge.”
RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation
Wang, Y., Wang, Y., Wang, S., Guo, D., Chen, J., Grundy, J., Liu, X., Ma, Y., Mao, M., Zhang, H., et al. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3645056 · Cited by 12
Review of current and potential uses of large language models in engineering
Nguyen, S. S., & Kittur, J. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1649650 · Cited by 16
Becoming-with-AI: Rethinking Professional Knowledge Through Generative Knowing
Lim, A., & Nicolaides, A. (2024) · Reflective Practice · doi:10.1080/14623943.2024.2321227 · Cited by 14
Democratizing Energy Management with LLM-Assisted Optimization Autoformalism
Jin, M., Sel, B., Hardeep, F., & Yin, W. (2024) · IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm) · doi:10.1109/smartgridcomm60555.2024.10738100 · Cited by 12
Communicating Natural Programs to Humans and Machines
Acquaviva, S., Pu, Y., Kryven, M., Sechopoulos, T., Wong, C., Ecanow, G., Nye, M., Tessler, M. H., & Tenenbaum, J. B. (2022) · neural information processing systems · doi:10.52202/068431-0270 · Cited by 28
Self-citationAn LLM Compiler for Parallel Function Calling
Kim, S., Moon, S., Tabrizi, R., Lee, N., Mahoney, M. W., Keutzer, K., & Gholami, A. (2023) · arXiv · doi:10.48550/arxiv.2312.04511 · Cited by 12
“This limits its applicability in complex scenarios such as coding [7, 33, 14, 3] or math [16, 15] problems, as also stated in the paper [34].”
BackgroundCorrectness assessment of code generated by Large Language Models using internal representations
Bui, T.-D., Vu, T. T., Nguyen, T.-T., Nguyen, S., & Vo, H. D. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112570 · Cited by 39
An Evaluation Method for Large Language Models’ Code Generation Capability
Su, H., Ai, J., Yu, D., & Zhang, H. (2023) · International Conference on Dependable Systems and Their Applications (DSA) · doi:10.1109/dsa59317.2023.00118 · Cited by 13
End-User Development for Human-Robot Interaction: Results and Trends in an Emerging Field
Vaiani, G., & Paternò, F. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3661146 · Cited by 11
An empirical study on capability of Large Language Models in understanding code semantics
Nguyen, T.-T., Vu, T. T., Vo, H. D., & Nguyen, S. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107780 · Cited by 21
Evaluating quantized Large Language Models for code generation on low-resource language benchmarks
Nyamsuren, E. (2025) · Journal of Computer Languages · doi:10.1016/j.cola.2025.101351 · Cited by 18
CrossCodeBench: Benchmarking Cross-Task Generalization of Source Code Models
Niu, C., Li, C., Ng, V., & Luo, B. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00055 · Cited by 11
Aligning Offline Metrics and Human Judgments of Value for Code Generation Models
Dibia, V. C., Fourney, A., Bansal, G., Poursabzi-Sangdeh, F., Liu, H., & Amershi, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.540 · Cited by 24
Metadata
- Authors
- Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., & Sutton, C.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2108.07732
- OpenAlex
- W3195140297
- Cited by
- 388
- References
- 93
- Citations by year
- 2026: 44 · 2025: 129 · 2024: 92 · 2023: 85 · 2022: 21 · 2021: 15
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- TIB Data Manager
- Funders
- National Natural Science Foundation of China (62472339, 62302375)
- Hankyong National University
- Ministry of Industry and Information Technology of the People's Republic of China
- HORIZON EUROPE Framework Programme (101070305)
- Fundação para a Ciência e a Tecnologia (UIDB/00319, 00319, UIDB/00319/2020, UIDB/05757/2020)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- program synthesis, large language models, Python code generation, MBPP, few-shot learning, fine-tuning, model size scaling, human feedback, error analysis, semantic grounding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
