The DOI matches its DataCite record; 92 of the 93 references it cites were found in a registry or an index, and 1 are not checked yet.
Program Synthesis with Large Language Models
Open accessAbstract
This paper explores the limits of the current generation of large language models for program synthesis in general purpose programming languages. We evaluate a collection of such models (with between 244M and 137B parameters) on two new benchmarks, MBPP and MathQA-Python, in both the few-shot and fine-tuning regimes. Our benchmarks are designed to measure the ability of these models to synthesize short Python programs from natural language descriptions. The Mostly Basic Programming Problems (MBPP) dataset contains 974 programming tasks, designed to be solvable by entry-level programmers. The MathQA-Python dataset, a Python version of the MathQA benchmark, contains 23914 problems that evaluate the ability of the models to synthesize code from more complex text. On both datasets, we find that synthesis performance scales log-linearly with model size. Our largest models, even without finetuning on a code dataset, can synthesize solutions to 59.6 percent of the problems from MBPP using few-shot learning with a well-designed prompt. Fine-tuning on a held-out portion of the dataset improves performance by about 10 percentage points across most model sizes. On the MathQA-Python dataset, the largest fine-tuned model achieves 83.8 percent accuracy. Going further, we study the model's ability to engage in dialog about code, incorporating human feedback to improve its solutions. We find that natural language feedback from a human halves the error rate compared to the model's initial prediction. Additionally, we conduct an error analysis to shed light on where these models fall short and what types of programs are most difficult to generate. Finally, we explore the semantic grounding of these models by fine-tuning them to predict the results of program execution. We find that even our best models are generally unable to predict the output of a program given a specific input.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“In practice, large pools of test cases, Dj , can either originate from human generated documentation (Nassif et al., 2021;Lemieux et al., 2023) or GPTgenerated unit tests (Austin et al., 2021).”
From Does Few-Shot Learning Help LLM Performance in Code Synthesis? (2024) · cites this paper as Austin et al., 2021
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Code data (e.g., from Stack Exchange and GitHub) substantially enhances program synthesis capabilities [37,38].”
From A Survey of Large Language Models (2026) · cites this paper as [38]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Neural code synthesis approaches to code completion generate suggestions by using a language model to predict what the user might type next (the completion) from the context of what they are working on at the moment (the prompt) [2].”
From Productivity Assessment of Neural Code Completion (2022) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Similar studies in the code space are limited to the number of few-shot examples in the prompt, and the effect of length of prompt on performance [9,15].”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [9]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, for the Performance category, existing works have used the Computational Accuracy metric to evaluate generative code models [9,15,33,88], but not other metrics we uncovered regarding the characteristics of the generated artifacts and run-time efficiency.”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [9]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large pre-trained language models perform remarkably well on tasks that can be done 'in one pass', such as generating realistic text (Brown et al., 2020) or synthesizing computer programs (Chen et al., 2021; Austin et al., 2021).”
From Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021) · cites this paper as Austin et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, they struggle with tasks that require unbounded multi-step computation, such as adding integers (Brown et al., 2020) or executing programs (Austin et al., 2021).”
From Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021) · cites this paper as Austin et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The 'MBPP-aug + CodeNet + single line' model achieved an overall synthesis accuracy of 54%, compared to the 62% accuracy of the original few-shot model in Austin et al. (2021).”
From Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021) · cites this paper as Austin et al. (2021)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“MBPP is another large, commonly used benchmark of Python problems.”
From MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current benchmarks for evaluating neural code models focus on only a small subset of programming languages, excluding many popular languages such as Go or Rust.”
From Measuring The Impact Of Programming Language Distribution (2023) · cites this paper as Austin et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“HumanEval [7]-developed by OpenAI for evaluating Codex-and other benchmarks [2,13,20] only consist of programming problems in a single language”
From CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X (2023) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent large language models (LLMs) have demonstrated strong capabilities in understanding and generating code, from competitive programming to repository-level software engineering.”
From Code as Agent Harness (2026) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In fact, existing techniques evaluate on predefined prompt templates with little to no modifications (Austin et al., 2021;Liu et al., 2024b).”
From Does Few-Shot Learning Help LLM Performance in Code Synthesis? (2024) · cites this paper as Austin et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The development of evaluations in tandem with language models (LMs) has substantially contributed to the rapid advancement of these models [29, 11, 7, 25, 81, 27, 72].”
From SciCode: A Research Coding Benchmark Curated by Scientists (2024) · cites this paper as [7]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“HumanEval [5], MBPP [3], and APPS [13] score function-level synthesis; CrossCodeEval [11] targets cross-file completion; CodeClash [33] and PinchBench [18] probe further axes of coding capability.”
From Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks (2026) · cites this paper as [3]
The cited source says what the claim attributes to it.
In the citing paper
“Researchers have already investigated the benefits of natural language feedback to guide program synthesis [2]”
From Productivity Assessment of Neural Code Completion (2022) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“Researchers have already investigated the benefits of natural-language feedback to guide program synthesis, 2 so the conversational framing of coding completions is not a radical proposal.”
From Measuring GitHub Copilot's Impact on Productivity (2024) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“Direct execution prediction has been shown to perform poorly on Python programs in Austin et al. (2021).”
From Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Similarly, large-scale language models struggle to predict the result of executing Python code, even code which is a solution to a programming task the model is able to solve (Austin et al., 2021).”
From Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“However, previous work has found that textual similarity metrics correlate only weakly with code correctness [1,2,27], highlighting the importance of benchmark suites with unit tests.”
From MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“MBPP dataset is proposed by [2], containing 974 problems in Python.”
From CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X (2023) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“Different from prior works on utilizing human feedback for code repair, where the feedback message explains the code errors and how to fix them (Chen et al., 2023a;Austin et al., 2021), SELF-DEBUGGING teaches the model to identify the implementation errors via investigating into execution results and explaining the code by itself.”
From Teaching Large Language Models to Self-Debug (2023) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“MBPP prompts follow a templated structure where the critical task specification appears mid-prompt:”
From Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression (2026) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“The Mostly Basic Python Problems (MBPP) benchmark (Austin et al., 2021) takes a complementary approach, providing 974 crowd-sourced programming problems intended to be solvable by entry-level programmers.”
From Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression (2026) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) evaluate isolated functions.”
From SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents (2026) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“MBPP, on the other hand, comprises 974 programming problems created by crowdsource contributors.”
From Qwen2.5-Coder Technical Report (2024) · cites this paper as Austin et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“In particular, MBPP -verified is sourced from (Yang et al., 2024;Misu et al., 2024) and contains 78 programs from the original MBPP dataset (Austin et al., 2021).”
From AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinement (2024) · cites this paper as Austin et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Part of this paper’s reference list could not be split into single entries.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“The models we use in this paper are dense left-to-right decoder-only Transformer language models [Vaswani et al., 2017] trained on a combination of web documents, dialog data, and Wikipedia.”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“P L ] 1 6 A ug 2 text [Brown et al., 2020, Raffel et al., 2019] and to solve a rapidly expanding set of modeling and reasoning tasks [Devlin et al., 2019, big-bench collaboration, 2021].”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 1×
“…line of previous work on neural language models of natural language text [Mikolov et al., 2010, Sutskever et al., 2011, Józefowicz et al., 2016, Dai and Le, 2015, Peters et al., 2018, Howard and Ruder, 2018], especially recent large Transformer models [Radford et al., 2018, Brown et al., 2020].”
BackgroundOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · cited 1×
“Our work is inspired by the long line of previous work on neural language models of natural language text [Mikolov et al., 2010, Sutskever et al., 2011, Józefowicz et al., 2016, Dai and Le, 2015, Peters et al., 2018, Howard and Ruder, 2018], especially recent large Transformer models [Radford et…”
MethodKudo, T., & Richardson, J. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-2012 · Cited by 4,454 · cited 1×
“81T BPE tokens with a vocabulary of 32K tokens using the SentencePiece library [Kudo and Richardson, 2018].”
MethodUniversal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 1×
“…line of previous work on neural language models of natural language text [Mikolov et al., 2010, Sutskever et al., 2011, Józefowicz et al., 2016, Dai and Le, 2015, Peters et al., 2018, Howard and Ruder, 2018], especially recent large Transformer models [Radford et al., 2018, Brown et al., 2020].”
BackgroundThe Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · cited 1×
“These results suggest that methods such as prompt-tuning [Li and Liang, 2021, Lester et al., 2021] could yield substantial performance improvements in this domain.”
ResultThe Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092 · cited 1×
“These results suggest that methods such as prompt-tuning [Li and Liang, 2021, Lester et al., 2021] could yield substantial performance improvements in this domain.”
Result
Flagged first, then by citations
Research works printed but not linked: 1
- [10]10. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, Will Guss, Alex Nichol, Igor Babuschkin, Suchir Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, July 2021. URL http://arxiv.org/abs/2107.03374 . · arXiv
Cited by
Sorted by citations
Items 201–300 of 529
Solving Novel Program Synthesis Problems with Genetic Programming using Parametric Polymorphism
Pantridge, E., & Helmuth, T. (2023) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3583131.3590502 · Cited by 10
Better Context Makes Better Code Language Models: A Case Study on Function Call Argument Completion
Pei, H., Zhao, J., Lausen, L., Zha, S., & Karypis, G. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i4.25653 · Cited by 17
Private-library-oriented code generation with large language models
Zan, D., Chen, B., Gong, Y., Cao, J., Zhang, F., Wu, B., Guan, B., Yin, Y., & Wang, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113934 · Cited by 42
Challenges and Paths Towards AI for Software EngineeringPreprint
Gu, A., Jain, N., Li, W., Shetty, M., Shao, Y., Li, Z., Yang, D., Ellis, K., Sen, K., & Solar-Lezama, A. (2025) · Qeios · doi:10.32388/vv1661 · Cited by 40
Git Merge Conflict Resolution Leveraging Strategy Classification and LLM
Shen, C., Yang, W., Pan, M., & Zhou, Y. (2023) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs60937.2023.00031 · Cited by 10
Automated discovery of algorithms from data
Blazek, P. J., Venkatesh, K., & Lin, M. M. (2024) · Nature Computational Science · doi:10.1038/s43588-024-00593-9 · Cited by 10
Requirements Verification Through the Analysis of Source Code by Large Language Models
Couder, J. O., Gomez, D., & Ochoa, O. (2024) · IEEE SoutheastCon · doi:10.1109/southeastcon52093.2024.10500073 · Cited by 11
Human Language to Analog Layout Using GLayout Layout Automation Framework
Hammoud, A., Goyal, C., Pathen, S., Dai, A., Li, A., Kielian, G., & Saligane, M. (2024) · ACM/IEEE Workshop on Machine Learning for CAD (MLCAD) · doi:10.1109/mlcad62225.2024.10740211 · Cited by 9
Human Evaluation of Procedural Knowledge Graph Extraction from Text with Large Language Models
Carriero, V. A., Azzini, A., Baroni, I., Scrocca, M., & Celino, I. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-77792-9_26 · Cited by 10
A Flexible Template for Edge Generative AI With High-Accuracy Accelerated Softmax and GELU
Belano, A., Tortorella, Y., Garofalo, A., Benini, L., Rossi, D., & Conti, F. (2025) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2025.3562734 · Cited by 13
LLM4TDG: test-driven generation of large language models based on enhanced constraint reasoning
Liu, J., Liang, R., Zhu, X., Zhang, Y., Liu, Y., & Liu, Q. (2025) · Cybersecurity · doi:10.1186/s42400-024-00335-4 · Cited by 10
“With the advent of large models (Brownet et al. 2020), researchers have leveraged the natural language understanding and generation capabilities of large language models (LLMs) to explore their potential in code generation (Austin et al. 2021).”
An Empirical Study on Automatically Detecting AI-Generated Source Code: How Far are We?
Suh, H., Tafreshipour, M., Li, J., Bhattiprolu, A., & Ahmed, I. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00064 · Cited by 17
Learning Interpretable Logic for Agent-Based Models from Domain Independent Primitives
Greig, R., Major, C., Pacholska, M., Bending, S., & Arranz, J. (2023) · Journal of Artificial Societies and Social Simulation · doi:10.18564/jasss.5087 · Cited by 9
Personalized Distillation: Empowering Open-Sourced LLMs with Adaptive Learning for Code Generation
Chen, H., Saha, A., Hoi, S. C. H., & Joty, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.417 · Cited by 18
Code4ML: a large-scale dataset of annotated Machine Learning code
Drozdova, A., Трофимова, Е. В., Guseva, P., Scherbakova, A., & Ustyuzhanin, A. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1230 · Cited by 9
Low-cost language models: Survey and performance evaluation on Python code generation
Espejel, J. L., Alassan, M. S. Y., Bouhandi, M., Dahhane, W., & Ettifouri, E. H. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109490 · Cited by 15
The Fault in our Stars: Quality Assessment of Code Generation Benchmarks
Siddiq, M. L., Dristi, S., Saha, J., & Santos, J. C. S. (2024) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam63643.2024.00028 · Cited by 22
Haque, R., Goh, H.-N., Ting, C., Quek, A., & Hasan, M. R. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100356 · Cited by 13
Code LLMs: A Taxonomy-based Survey
Raihan, N., Newman, C. D., & Zampieri, M. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10826108 · Cited by 11
Can LLMs Generate Higher Quality Code Than Humans? An Empirical Study
Jamil, M. T., Abid, S., & Shamail, S. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00081 · Cited by 15
Your Fix Is My Exploit: Enabling Comprehensive DL Library API Fuzzing with Large Language Models
Zhang, K., Wang, S., Han, J., Zhu, X., Li, X., Wang, S., & Wen, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00041 · Cited by 17
Toward efficient vibe coding: An LLM-based agent for low-code software development
Malamas, N., Tsardoulias, E., Panayiotou, K., & Symeonidis, A. L. (2025) · Journal of Computer Languages · doi:10.1016/j.cola.2025.101367 · Cited by 8
A Static Evaluation of Code Completion by Large Language Models
Ding, H., Kumar, V., Tian, Y., Wang, Z., Kwiatkowski, R., Li, X., Ramanathan, M. K., Ray, B., Bhatia, P., & Sengupta, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.34 · Cited by 14
DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rules
Liu, Y., Held, W., & Yang, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.850 · Cited by 10
Honarvar, S., van der Wilk, M., & Donaldson, A. F. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10989005 · Cited by 8
Alexander, F., Abdiwijaya, E. A., Pherry, F., Gunawan, A. A. S., & Anderies (2022) · International Conference on Software Engineering and Information Technology (ICoSEIT) · doi:10.1109/icoseit55604.2022.10029949 · Cited by 7
Generating Data for Symbolic Language with Large Language Models
Ye, J., Li, C., Kong, L., & Yu, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.523 · Cited by 18
Exploiting Language Models as a Source of Knowledge for Cognitive Agents
Kirk, J. R., Wray, R. E., & Laird, J. E. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v2i1.27690 · Cited by 15
Cascaded Fast and Slow Models for Efficient Semantic Code SearchPreprint
Gotmare, A. D., Li, J., Joty, S., & Hoi, S. C. H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.07811 · Cited by 10
Do large language models know chemistry?Preprint
White, A. D., Hocky, G. M., Gandhi, H. A., Ansari, M., Cox, S., Wellawatte, G. P., Sasmal, S., Yang, Z., Liu, K., Singh, Y., et al. (2022) · ChemRxiv · doi:10.26434/chemrxiv-2022-3md3n · Cited by 8
AFlow: Automating Agentic Workflow Generation
Zhang, J., Xiang, J., Yu, Z., Teng, F., Chen, X., Chen, J., Zhuge, M., Cheng, X., Hong, S., Wang, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.10762 · Cited by 359
“For HumanEval and MBPP, we report the pass@1 metric as presented in (Chen et al., 2021) to assess code accuracy.”
InfluentialEcoAssistant: Using LLM Assistant More Affordably and Accurately
Zhang, J., Krishna, R., Awadallah, A. H., & Wang, C. (2023) · arXiv · doi:10.48550/arxiv.2310.03046 · Cited by 8
“It is also different from generic code generation task (Chen et al., 2021; Hendrycks et al., 2021; Austin et al., 2021; Lu et al., 2021; Yang et al., 2023) by requiring LLMs to exploit domain-specific API based on the user query.”
BackgroundMLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
Chan, J. S., Chowdhury, N., Jaffe, O., Aung, J., Sherburn, D., Mays, E., Starace, G., Liu, K., Maksin, L., Patwardhan, T., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.07095 · Cited by 7
Towards Making Flowchart Images Machine Interpretable
Shukla, S., Gatti, P., Kumar, Y., Yadav, V., & Mishra, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-41734-4_31 · Cited by 8
Evaluating Class Membership Relations in Knowledge Graphs Using Large Language Models
Allen, B. P., & Groth, P. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-78952-6_2 · Cited by 11
Enhancing Code Generation for Low-Resource Languages: No Silver Bullet
Giagnorio, A., Martin-Lopez, A., & Bavota, G. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00058 · Cited by 10
Unseen Horizons: Unveiling the Real Capability of LLM Code Generation Beyond the Familiar
Zhang, Y., Xie, Y., Lit, S., Liu, P., Wang, C., Jia, Z., Huang, X., Song, J., Luo, C., Zheng, Z., et al. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00082 · Cited by 7
Yao, J., Zhang, L., & Huang, J. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1590105 · Cited by 14
“An important approach in integrating Large Language Models (LLMs) with Automated Machine Learning (AutoML) is LLM-as-Translator, where natural language instructions are converted into API calls to control AutoML systems (Trirat et al., 2024; Chen et al., 2024b; Luo D. et al., 2024; Tsai et al.,…”
Promoting open science in test-driven software experiments
Kessel, M., & Atkinson, C. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.111971 · Cited by 7
Boosting LLM-Based Software Generation by Aligning Code with Requirements
Yaacov, T., Elyasaf, A., & Weiss, G. (2024) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew61692.2024.00045 · Cited by 10
Benchmarking Large Language Models for Ethereum Smart Contract Development
Daspe, E., Durand, M., Hatin, J., & Bradai, S. (2024) · Conference on Blockchain Research & Applications for Innovative Networks and Services (BRAINS) · doi:10.1109/brains63024.2024.10732686 · Cited by 7
Tihanyi, N., Bisztray, T., Dubniczky, R. A., Tóth, R., Borsos, B., Cherif, B., Jain, R., Muzsai, L., Ferrag, M. A., Marinelli, R. A., et al. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825051 · Cited by 12
Jiang, X., Dong, Y., Tao, Y., Liu, H., Jin, Z. G., & Li, G. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00133 · Cited by 18
Self-Training Large Language Models for Improved Visual Program Synthesis With Visual Reinforcement
Khan, Z., Vijay Kumar, B. G., Schulter, S., Fu, Y., & Chandraker, M. K. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01360 · Cited by 7
Knowledge Engineering in the Age of Neurosymbolic Systems
Sabou, M., Llugiqi, M., Ekaputra, F. J., Waltersdorfer, L., & Tsaneva, S. (2025) · Neurosymbolic Artificial Intelligence · doi:10.1177/29498732251320078 · Cited by 8
Program Synthesis Guided Reinforcement LearningPreprint
Yang, Y., Inala, J. P., Bastani, O., Pu, Y., Solar-Lezama, A., & Rinard, M. (2021) · arXiv (Cornell University) · Cited by 7
Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows
Lei, F., Chen, J., Ye, Y., Cao, R., Shin, D., Su, H., Suo, Z., Gao, H., Hu, W., Yin, P., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.07763 · Cited by 7
MeshAgent: enabling reliable network management with large language models
Zhou, Y., Hsieh, K., Mani, S. K., Kandula, S., & Liu, Z. (2025) · Proceedings of the ACM on Measurement and Analysis of Computing Systems · doi:10.1145/3771567 · Cited by 7
DafnyBench: A Benchmark for Formal Software Verification
Loughridge, C., Sun, Q., Ahrenbach, S., Cassano, F., Sun, C., Sheng, Y., Mudide, A., Misu, M. R. H., Amin, N., & Tegmark, M. (2024) · arXiv · doi:10.48550/arxiv.2406.08467 · Cited by 6
“Early results indicate that an LLM that is able to converse with a human when producing a program can reduce the error rate against a simple programming benchmark by half [20].”
BackgroundFrom Words to Wires: Generating Functioning Electronic Devices from Natural Language Descriptions
Jansen, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.864 · Cited by 6
Amin, R. M., Kühle, O. H., Buschek, D., & Butz, A. M. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3720243 · Cited by 6
“…line of work on such systems explores user agency via direct manipulation, such as in Spellburst [3], which uses a node-based interface for semantic programming and exploring variations, and DirectGPT [20], which allows users to modify generated content via graphical objects and direct controls.”
Exploring Large Language Models in Information Systems: A Survey
Bianchini, F., & Marinacci, M. (2025) · Springer eBooks · doi:10.1007/978-3-031-92285-5_5 · Cited by 7
Detecting, Creating, Repairing, and Understanding Indivisible Multi-Hunk Bugs
Xin, Q., Wu, H., Tang, J., Liu, X., Reiss, S. P., & Xuan, J. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660828 · Cited by 6
Test Code Generation for Telecom Software Systems Using Two-Stage Generative Model
Nabeel, M., Nimara, D. D., & Zanouda, T. (2024) · IEEE International Conference on Communications workshops · doi:10.1109/iccworkshops59551.2024.10615269 · Cited by 12
Santis, A. D., Marco Balduini, A., Santis, F. D., Proia, A., Leo, A., Brambilla, M., & Valle, E. D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-77847-6_17 · Cited by 9
NARRATE: Versatile Language Architecture for Optimal Control in Robotics
Ismail, S., Arbues, A., Cotterell, R., Zurbrügg, R., & Alonso, C. A. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10801425 · Cited by 10
LoCo: Low-Bit Communication Adaptor for Large-Scale Model Training
Xie, X., Lin, Z., Toh, K.-C., & Zhou, P. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3544764 · Cited by 13
D’Oosterlinck, K., Xu, W., Develder, C., Demeester, T., Singh, A., Potts, C., Kiela, D., & Mehri, S. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00748 · Cited by 31
(2025) · doi:10.1109/llm4code66737.2025.00014 · Cited by 6
3DGen: AI-Assisted Generation of Provably Correct Binary Format Parsers
Fakhoury, S., Kuppe, M. A., Lahiri, S. K., Ramananandro, T., & Swamy, N. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00173 · Cited by 17
On the performance of large language models on introductory programming assignments
Raihan, N., Goswami, D., Puspo, S. S. C., Siddiq, M. L., Newman, C. D., Ranasinghe, T., Santos, J. C. S., & Zampieri, M. (2025) · Journal of Intelligent Information Systems · doi:10.1007/s10844-025-00968-y · Cited by 8
“These models are evaluated on benchmarks like HumanEval and MBPP, consistently out - performing previous models.”
InfluentialA Simple, Yet Effective Approach to Finding Biases in Code Generation
Mouselinos, S., Malinowski, M., & Michalewski, H. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.718 · Cited by 13
Self-citationProgram Synthesis Guided Reinforcement Learning for Partially Observed EnvironmentsPreprint
Yang, Y., Inala, J. P., Bastani, O., Pu, Y., Solar-Lezama, A., & Rinard, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.11137 · Cited by 10
SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?
Miserendino, S., Wang, M., Patwardhan, T., & Heidecke, J. (2025) · arXiv · doi:10.48550/arxiv.2502.12115 · Cited by 5
“…et al., 2024), APPS, LiveCodeBench (Jain et al., 2024), NaturalCodeBench (Zhang et al., 2024), Big-CodeBench (Zhuo et al., 2024), MBPP, MathQA-Python (Austin et al., 2021), CodeXGlue (Lu et al., 2021), Al-phaCode (Li et al., 2022), CODEGEN (Nijkamp et al., 2023), and DS-1000 (Lai et al., 2022).”
Anchor Attention, Small Cache: Code Generation With Large Language Models
Zhang, X., Zhou, Y., Yang, G., Gall, H. C., & Chen, T. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3570680 · Cited by 6
Fuzzy-Assisted Contrastive Decoding Improving Code Generation of Large Language Models
Wang, S., Ding, L., Zhan, Y., Luo, Y., Liu, S., & Ding, W. (2025) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2025.3575060 · Cited by 5
“3) Extensive experiments on HumanEval [7], MBPP [25], and MultiPL-E [26] benchmarks demonstrate that the proposed FCD mechanism outperforms direct contrastive decoding, reducing time required for inference and contrast decoding.”
Large Language Models for C Test Case Generation: A Comparative Analysis
Guzu, A., Nicolae, G., Cucu, H., & Burileanu, C. (2025) · Electronics · doi:10.3390/electronics14112284 · Cited by 7
Jonnala, R., Yang, J. Y., Lee, Y., Liang, G., & Cao, Z. (2025) · IEEE Access · doi:10.1109/access.2025.3585742 · Cited by 5
Lebioda, K., Petrović, N., Pan, F., Zolfaghari, V., Schamschurko, A., & Knoll, A. (2025) · IEEE Access · doi:10.1109/access.2025.3597748 · Cited by 5
HiBenchLLM: Historical Inquiry Benchmarking for Large Language Models
Chartier, M. A., Dakkoune, N., Bourgeois, G., & Jean, S. (2024) · Data & Knowledge Engineering · doi:10.1016/j.datak.2024.102383 · Cited by 5
Evaluating Large Language Models for Code Generation: A Comparative Study on Python, Java, and Swift
Wan, B. (2025) · Applied and Computational Engineering · doi:10.54254/2755-2721/2025.tj22242 · Cited by 5
Comet: Accelerating Private Inference for Large Language Model by Predicting Activation Sparsity
Yan, G., Zhang, Y., Guo, Z., Zhao, L., Chen, X., Wang, C., Wang, W., Meng, D., & Hou, R. (2025) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp61157.2025.00182 · Cited by 6
“(1) Code Generation: HumanEval (0-shot) [21] and MBPP (3-shot) [5].”
Learning Evaluation Models From Large Language Models for Sequence Generation
Wang, C., Zhou, H., Chang, K., Liu, T., Zhang, C., Yang, M., Du, Q., Xiao, T., Zhang, Y., & Zhu, J. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3587460 · Cited by 11
Efficient Text-to-Code Retrieval with Cascaded Fast and Slow Transformer Models
Gotmare, A. D., Li, J., Joty, S., & Hoi, S. C. H. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616369 · Cited by 7
Data Augmentation for Code Translation with Comparable Corpora and Multiple References
Xie, Y., Naik, A., Fried, D., & Rosé, C. P. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.917 · Cited by 6
BatFix: Repairing language model-based transpilation
Ramos, D., Lynce, I., Manquinho, V. M., Martins, R., & Goues, C. L. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3658668 · Cited by 9
NL2Fix: Generating Functionally Correct Code Edits from Bug Descriptions
Fakhoury, S., Chakraborty, S., Musuvathi, M., & Lahiri, S. K. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3639478.3643526 · Cited by 5
Fine-tuning the Gemma model for Kaggle Assistant
Zhalgasbayev, A., Khauazkhan, A., & Sarsenova, Z. N. (2024) · IEEE AITU: Digital Generation · doi:10.1109/ieeeconf61558.2024.10585529 · Cited by 4
Reimagining Enterprise Data Management using Generative Artificial Intelligence
Varma, S., Shivam, S., Ray, B., & Biswas, S. (2024) · IEEE Swiss Conference on Data Science (SDS) · doi:10.1109/sds60720.2024.00023 · Cited by 5
GLUECons: A Generic Benchmark for Learning under Constraints
Faghihi, H. R., Nafar, A., Chen, Z., Mirzaee, R., Zhang, Y., Uszok, A., Wan, A., Premsri, T., Roth, D., & Kordjamshidi, P. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i8.26143 · Cited by 7
Model-Agnostic Syntactical Information for Pre-Trained Programming Language Models
Saberi, I., & Fard, F. H. (2023) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr59073.2023.00036 · Cited by 6
aiXcoder-7B: A Lightweight and Effective Large Language Model for Code CompletionPreprint
Jiang, S., Li, J., Zong, H., Liu, H., Zhu, H., Hu, S., Li, E. M., Ding, J., Han, Y., Ning, W., et al. (2024) · Qeios · doi:10.32388/atahd0 · Cited by 6
Large Language Model Guided Self-Debugging Code GenerationPreprint
Adnan, M., Kuhn, C. C. N., & Xu, Z. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5396508 · Cited by 6
Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving
Zan, D., Huang, Z., Liu, W., Chen, H., Zhang, L., Xin, S., Chen, L., Liu, Q., Zhong, X., Li, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2504.02605 · Cited by 5
“Early efforts in this domain focused on primarily evaluating models in monolingual program-level evaluations [3, 31, 16, 9, 7, 38].”
Agent-as-a-Judge: Evaluate Agents with Agents
Zhuge, M., Zhao, C., Ashley, D., Wang, W., Khizbullin, D., Xiong, Y., Liu, Z., Chang, E., Krishnamoorthi, R., Tian, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.10934 · Cited by 5
Peng, Q., Chai, Y., & Li, X. (2024) · arXiv · doi:10.48550/arxiv.2402.16694 · Cited by 88
“…Prior efforts have predominantly concentrated on English-centric Python generation, evident in benchmarks such as HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), APPS (Hendrycks et al., 2021), DSP (Chandel et al., 2022), MTPB (Nijkamp et al., 2023b), and DS-1000 (Lai et al., 2023).”
BackgroundSWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
Deng, X., Da, J., Pan, E., He, Y. Y., Ide, C., Garg, K., Lauffer, N., Park, A., Pasari, N., Rane, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2509.16941 · Cited by 4
“This was complemented by MBPP [2], which provided approximately 1,000 crowd-sourced Python problems designed for entry-level programmers.”
Gentopia.AI: A Collaborative Platform for Tool-Augmented LLMs
Xu, B., Liu, X., Shen, H., Han, Z., Li, Y., Yue, M., Peng, Z., Liu, Y., Yao, Z., & Xu, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2023.emnlp-demo.20 · Cited by 4
Aggarwal, P., Parno, B., & Welleck, S. (2024) · arXiv · doi:10.48550/arxiv.2412.06176 · Cited by 4
Evaluating the Test Adequacy of Benchmarks for LLMs on Code Generation
Liu, X., Sun, X., Bo, L., Hu, Y., Liu, X., & Ye, Z. (2025) · Journal of Software Evolution and Process · doi:10.1002/smr.70034 · Cited by 4
“The benchmarks studied include HumanEval with 164 tasks [4], MBPP with 974 tasks [18], and their enhanced benchmarks: HumanEval+ with 164 tasks [14] and MBPP+ with 378 tasks [14].”
Bridging HCI and AI Research for the Evaluation of Conversational SE Assistants
Richards, J., & Wessel, M. S. (2025) · IEEE/ACM International Workshop on Bots in Software Engineering (BotSE) · doi:10.1109/botse67031.2025.00009 · Cited by 4
LoRA 2 : Multi-scale low-rank approximations for fine-tuning large language models
Zhang, J.-C., Xiong, Y.-J., Xia, C.-M., Zhu, D.-H., & Zhan, H.-J. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130859 · Cited by 7
Gülmez, B. (2026) · Applied Intelligence · doi:10.1007/s10489-026-07230-0 · Cited by 4
“The survey also addresses domain-specific applications: hardware description languages [1, 26], geospatial computing [12], industrial control systems [18], and emerging domains where automated code generation intersects with autonomous systems.”
Fu, Y., Li, B., Li, L., Zhang, W., & Xie, T. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728947 · Cited by 7
Automatically Testing Functional Properties of Code Translation Models
Enişer, H. F., Wüstholz, V., & Christakis, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i19.30097 · Cited by 4
ScenEval: A Benchmark for Scenario-Based Evaluation of Code Generation
Paul, D. G., Zhu, H., & Bayley, I. (2024) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest62860.2024.00015 · Cited by 4
Measuring code efficiency optimization capabilities with ACEOB
Pan, Y., Shao, X., & Lyu, C. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112250 · Cited by 4
Unit Test Generation Multi-Agent AI System for Enhancing Software Documentation and Code Coverage
Stojanović, D., Pavković, B., Cetic, N. B., Krunić, M., & Vidaković, L. (2024) · Telecommunications Forum (TELFOR) · doi:10.1109/telfor63250.2024.10819096 · Cited by 4
LLM-ProS: Analyzing Large Language Models’ Performance in Competitive Problem Solving
Hossain, M. S., Tabassum, A., Arefin, M. F., & Zaman, T. S. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00015 · Cited by 4
Combining Contexts from Multiple Sources for Documentation-Specific Code Example Generation
Khan, J. Y., & Uddin, G. (2023) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner56733.2023.00071 · Cited by 7
Metadata
- Authors
- Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., & Sutton, C.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2108.07732
- OpenAlex
- W3195140297
- Cited by
- 388
- References
- 93
- Citations by year
- 2026: 44 · 2025: 129 · 2024: 92 · 2023: 85 · 2022: 21 · 2021: 15
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- TIB Data Manager
- Funders
- National Natural Science Foundation of China (62472339, 62302375)
- Hankyong National University
- Ministry of Industry and Information Technology of the People's Republic of China
- HORIZON EUROPE Framework Programme (101070305)
- Fundação para a Ciência e a Tecnologia (UIDB/00319, 00319, UIDB/00319/2020, UIDB/05757/2020)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- program synthesis, large language models, Python code generation, MBPP, few-shot learning, fine-tuning, model size scaling, human feedback, error analysis, semantic grounding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
