Competition-level code generation with AlphaCode
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Abstract
Programming is a powerful and ubiquitous problem-solving tool. Systems that can assist programmers or even generate programs themselves could make programming more productive and accessible. Recent transformer-based neural network models show impressive code generation abilities yet still perform poorly on more complex tasks requiring problem-solving skills, such as competitive programming problems. Here, we introduce AlphaCode, a system for code generation that achieved an average ranking in the top 54.3% in simulated evaluations on recent programming competitions on the Codeforces platform. AlphaCode solves problems by generating millions of diverse programs using specially trained transformer-based networks and then filtering and clustering those programs to a maximum of just 10 submissions. This result marks the first time an artificial intelligence system has performed competitively in programming competitions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AlphaCode, Self-Debugging and CodeRL are effective in fixing less-complex program bugs, but they rely upon ground truth test cases that invalidate pass@1 eligibility, and do not use self-reflection to bridge the gap between error identification and implementation improvement.”
From Reflexion: Language Agents with Verbal Reinforcement Learning (2023) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large language models are increasingly capable of complex programming tasks and are already being used to accelerate AI research, from generating training data to serving as programming tools [1, 2, 3, 4, 5, 6].”
From RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts (2024) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Generative AI, such as OpenAI’s ChatGPT, has rapidly emerged as a disruptive technology capable of achieving human-level performance on a broad range of tasks (1–5).”
From Generative AI without guardrails can harm learning: Evidence from high school mathematics (2025) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Li et al. [27] showed that the accuracy of AlphaCode increases from close to 0% zero-shot to over 15% with 1,000 retries and over 30% with a million retries (accuracy is measured by how often one of the top 10 answers generated by the model is correct).”
From AI Agents That Matter (2024) · cites this paper as Li et al. [27]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Evaluation covered 200 CodeContests tasks [15] across three seeds and up to three feedback rounds, with online tests supplying feedback and hidden tests measuring final success.”
From RSI-Anything: Unlocking Recursive Self-Improvement with World Models (2026) · cites this paper as [15]
Experimental — accuracy not yet measured
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Self-citationDecoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“Following Kaplan et al. (2020), we adjusted the amount of training for other model sizes such that larger models are trained more and smaller models are trained less to optimize the use of compute.”
MethodGrandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455 · cited 2×
“We believe that sampling metadata leads to a more diverse set of model samples, a strategy similar to that used by Vinyals et al. (2019), and allows our model to take advantage of the relative strengths across the metadata distribution.”
MethodSelf-citationResnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., et al. (2009) · Communications of the ACM · doi:10.1145/1592761.1592779 · Cited by 3,333
Kudo, T., & Richardson, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d18-2012 · Cited by 2,869
Research works printed but not linked: 6
- A. Church, Application of recursive arithmetic to the problem of circuit synthesis.. J. Symb. Log. 28, 289–290 (1963).
- Y. Nandwani D. Jindal Mausam P. Singla ?oNeural learning of one-of-many solutions for combinatorial problems in structured output spaces ?? International Conference on Learning Representations (ICLR 2021) 3 to 7 May 2021.
- A. Holtzman J. Buys L. Du M. Forbes Y. Choi ?oThe curious case of neural text degeneration ?? Proceedings of the 8th International Conference on Learning Representations (ICLR 2020) 26 April to 1 May 2020.
- J. Ganitkevitch B. Van Durme C. Callison-Burch ?oPPDB: The paraphrase database ?? Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Atlanta Georgia 9 to 15 June 2013 (Association for Computational Linguistics 2013) pp. 758–764.
- A. Solar-Lezama ?oProgram synthesis by sketching ?? thesis University of California Berkeley (2008).
- J. Cai R. Shin D. Song ?oMaking neural programming architectures generalize via recursion ?? Proceedings of the 5th International Conference on Learning Representations (ICLR) Toulon France 24 to 26 April 2017.
Cited by
Items 101–200 of 1,104
CodePlan: Repository-level Coding using LLMs and Planning
Bairi, R., Sonwane, A., Kanade, A., C, V. D., Iyer, A., Parthasarathy, S., Rajamani, S., Ashok, B., & Shet, S. (2023) · arXiv · doi:10.48550/arxiv.2309.12499 · Cited by 92
Foundation Models and Fair Use
Henderson, P., Li, X., Jurafsky, D., Hashimoto, T., Lemley, M. A., & Liang, P. (2023) · doi:10.2139/ssrn.4404340 · Cited by 57
CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code
Zhou, S., Alon, U., Agarwal, S., & Neubig, G. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.859 · Cited by 100
Large Language Model-Powered Smart Contract Vulnerability Detection: New Perspectives
Hu, S., Huang, T., İlhan, F., Tekin, S. F., & Liu, L. (2023) · 2023 5th IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA) · doi:10.1109/tps-isa58951.2023.00044 · Cited by 85
Understanding the Potential of FPGA-based Spatial Acceleration for Large Language Model Inference
Chen, H., Zhang, J., Du, Y., Xiang, S., Yue, Z., Zhang, N., Cai, Y., & Zhang, Z. (2024) · ACM Transactions on Reconfigurable Technology and Systems · doi:10.1145/3656177 · Cited by 132
CodeKGC: Code Language Model for Generative Knowledge Graph Construction
Bi, Z., Chen, J., Jiang, Y., Xiong, F., Guo, W., Chen, H., & Zhang, N. (2024) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3641850 · Cited by 80
Teaching Large Language Models to Self-Debug
Chen, X., Lin, M., Schärli, N., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2304.05128 · Cited by 812
“, 2023), and performing selection based on the consensus on unit test execution outputs among samples (Chen et al., 2019; 2021b; Roziere et al., 2022; Shi et al., 2022; Li et al., 2022; Chen et al., 2023b).”
BackgroundMethodInfluentialSelf-citationLanguage Model Crossover: Variation through Few-Shot Prompting
Meyerson, E., Nelson, M. J., Bradley, H., Gaier, A., Moradi, A., Hoover, A. K., & Lehman, J. (2024) · ACM Transactions on Evolutionary Learning and Optimization · doi:10.1145/3694791 · Cited by 75
“The enormity and breadth of the dataset on which the LLM was trained, along with its ability to perform in-context learning, enables LMX to generate high-quality offspring across a broad range of domains.”
Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study
Fu, Y., Liang, P., Tahir, A., Li, Z., Shahin, M., Yu, J., & Chen, J. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3716848 · Cited by 74
Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study
Fu, Y., Liang, P., Tahir, A., Li, Z., Shahin, M., Yu, J., & Chen, J. (2023) · arXiv · doi:10.48550/arxiv.2310.02059 · Cited by 75
“Mind [21], and CodeWhisperer by Amazon [3].”
Language model crossover: Variation through few-shot prompting
Meyerson, E., Nelson, M. J., Bradley, H., Gaier, A., Moradi, A., Hoover, A. K., & Lehman, J. (2023) · arXiv · doi:10.48550/arxiv.2302.12170 · Cited by 75
Evolution Through Large Models
Lehman, J., Gordon, J., Jain, S., Ndousse, K., Yeh, C., & Stanley, K. O. (2024) · Genetic and Evolutionary Computation · doi:10.1007/978-981-99-3814-8_11 · Cited by 74
StarCoder 2 and The Stack v2: The Next Generation
Lozhkov, A., Li, R., Allal, L. B., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.19173 · Cited by 74
Evolution through Large Models
Lehman, J., Gordon, J., Jain, S., Ndousse, K., Yeh, C., & Stanley, K. O. (2022) · arXiv · doi:10.48550/arxiv.2206.08896 · Cited by 74
ClarifyGPT: A Framework for Enhancing LLM-Based Code Generation via Requirements Clarification
Mu, F., Shi, L., Wang, S., Yu, Z., Zhang, B., Wang, C., Liu, S., & Wang, Q. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660810 · Cited by 74
“(1) In contrast to prevailing LLM-based code generation methods [8, 23, 27] that leverage post-processing techniques to sample a substantial pool of candidate codes and then select one, ClarifyGPT aims to directly clarify the input requirements by asking clarifying questions.”
Cognitive Architectures for Language Agents
Sumers, T. R., Yao, S., Narasimhan, K., & Griffiths, T. L. (2023) · arXiv · doi:10.48550/arxiv.2309.02427 · Cited by 798
“Unexpectedly, training these models on internet-scale text also made them useful for many tasks beyond generating text, such as writing code (Li et al., 2022; Rozière et al., 2023; Li et al., 2023), modeling proteins (Meier et al.”
Refactoring Programs Using Large Language Models with Few-Shot Examples
Shirafuji, A., Oda, Y., Suzuki, J., Morishita, M., & Watanobe, Y. (2023) · 2023 30th Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec60848.2023.00025 · Cited by 67
Program Code Generation with Generative AIs
Idrisov, B., & Schlippe, T. (2024) · Algorithms · doi:10.3390/a17020062 · Cited by 66
Synthetic Data Generation Using Large Language Models: Advances in Text and Code
Nadas, M. D., Dioşan, L. S., & Tomescu, A. (2025) · IEEE Access · doi:10.1109/access.2025.3589503 · Cited by 227
A Survey on LLM-Based Code Generation for Low-Resource and Domain-Specific Programming Languages
Joel, S., Wu, J. J., & Fard, F. H. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3770084 · Cited by 209
Piccolo, S. R., Denny, P., Luxton-Reilly, A., Payne, S. H., & Ridge, P. G. (2023) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011511 · Cited by 69
Many bioinformatics programming tasks can be automated with ChatGPT
Piccolo, S. R., Denny, P., Luxton-Reilly, A., Payne, S., & Ridge, P. G. (2023) · arXiv · doi:10.48550/arxiv.2303.13528 · Cited by 69
A Practical Survey on Zero-shot Prompt Design for In-context Learning
Columbia University, N. Y. C., & Li, Y. (2023) · Proceedings of the Conference Recent Advances in Natural Language Processing - Large Language Models for Natural Language Processings · doi:10.26615/978-954-452-092-2_069 · Cited by 68
Self-citationKnowledge, Perceptions and Attitude of Researchers Towards Using ChatGPT in Research
Abdelhafiz, A. S., Ali, A., Maaly, A. M., Ziady, H. H., Sultan, E. A., & Mahgoub, M. A. (2024) · Journal of Medical Systems · doi:10.1007/s10916-024-02044-4 · Cited by 68
Yuan, A., Colato, E. G., Pescosolido, B. A., Song, H., & Samtani, S. (2025) · ACM Transactions on Management Information Systems · doi:10.1145/3701041 · Cited by 109
Kuhail, M. A., Mathew, S. S., Khalil, A., Berengueres, J., & Shah, S. J. H. (2024) · Science of Computer Programming · doi:10.1016/j.scico.2024.103111 · Cited by 68
CERT: Continual Pre-training on Sketches for Library-oriented Code Generation
Zan, D., Chen, B., Yang, D., Lin, Z., Kim, M., Guan, B., Wang, Y., Chen, W., & Lou, J.-G. (2022) · Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/329 · Cited by 65
CodeScore: Evaluating Code Generation by Learning Code Execution
Dong, Y., Ding, J., Jiang, X., Li, G., Li, Z., & Jin, Z. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695991 · Cited by 64
Sarbay, İ., Berikol, G. B., & Özturan, İ. U. (2023) · Turkish Journal of Emergency Medicine · doi:10.4103/tjem.tjem_79_23 · Cited by 61
Generative AIPreprint
Feuerriegel, S., Hartmann, J., Janiesch, C., & Zschech, P. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4443189 · Cited by 65
An Empirical Study of Code Smells in Transformer-based Code Generation Techniques
Siddiq, M. L., Majumder, S. H., Mim, M. R., Jajodia, S., & Santos, J. C. S. (2022) · 2022 IEEE 22nd International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam55253.2022.00014 · Cited by 70
“AlphaCode [51] is another type of code generation model that generates code for competitive programming problems.”
BackgroundPhotonic Neural Networks Based on Integrated Silicon Microresonators
Biasi, S., Donati, G. P., Lugnan, A., Mancinelli, M., Staffoli, E., & Pavesi, L. (2024) · Intelligent Computing · doi:10.34133/icomputing.0067 · Cited by 80
Economic impacts of AI-augmented R&D
Besiroglu, T., Emery-Xu, N., & Thompson, N. (2024) · Research Policy · doi:10.1016/j.respol.2024.105037 · Cited by 118
Gemini-the most powerful LLM: Myth or Truth
Islam, R., & Ahmed, I. (2024) · Information Communication Technologies Conference (ICTC) · doi:10.1109/ictc61510.2024.10602253 · Cited by 118
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
Laurençon, H., Saulnier, L., Wang, T., Akiki, C., del Moral, A. V., Scao, T. L., Von Werra, L., Mou, C., Ponferrada, E. G., Nguyen, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03915 · Cited by 93
“We collected a code dataset from BigQuery7 using the same language selection as AlphaCode (Li et al., 2022).”
MethodSignificant Productivity Gains through Programming with Large Language Models
Weber, T., Brandmaier, M., Schmidt, A., & Mayer, S. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3661145 · Cited by 64
“While it is possible with more general LLMs like the family of GPT models [6], there are dedicated models specifically tailored for this use case [6, 9–11, 15, 29].”
BackgroundMethodFoundation models and intelligent decision-making: Progress, challenges, and perspectives
Huang, J., Xu, Y., Wang, Q., Wang, Q., Liang, X., Wang, F., Zhang, Z., Wei, W., Zhang, B., Huang, L., et al. (2025) · The Innovation · doi:10.1016/j.xinn.2025.100948 · Cited by 192
Self-citationLearning Deep Semantics for Test Completion
Nie, P., Banerjee, R., Li, J. J., Mooney, R. J., & Gligoric, M. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00178 · Cited by 57
ChatGPT and other AI Users: Innovative and Creative Utilitarian Value and Mindset Shift
Duhaylungsod, A. V., & Chavez, J. V. (2023) · Journal of Namibian Studies : History Politics Culture · doi:10.59670/jns.v33i.2791 · Cited by 59
Can Students without Prior Knowledge Use ChatGPT to Answer Test Questions? An Empirical Study
Shoufan, A. (2023) · ACM Transactions on Computing Education · doi:10.1145/3628162 · Cited by 59
“Viable AI-powered code generation has become available for the public since 2021 with several popular models such as OpenAI Codex, DeepMind AlphaCode, and Amazon CodeWhisperer, promising to make programming more productive and accessible [24].”
MethodCRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Duan, N., & Chen, W. (2023) · arXiv · doi:10.48550/arxiv.2305.11738 · Cited by 561
“[13] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al.”
BackgroundHPC-GPT: Integrating Large Language Model for High-Performance Computing
Ding, X., Chen, L., Emani, M. K., Liao, C., Lin, P., Vanderbruggen, T., Xie, Z., Cerpa, A., & Du, W. (2023) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3624062.3624172 · Cited by 82
Assessing and understanding creativity in large language models
Zhao, Y., Zhang, R., Li, W., & Li, L. (2025) · Machine Intelligence Research · doi:10.1007/s11633-025-1546-4 · Cited by 57
“From transforming rudimentary text into compelling narratives[7, 8], unlocking a new realm of storytelling, to solving complex algorithmic problems[9], these models have shown a semblance of what could be interpreted as creativity.”
BackgroundCCTEST: Testing and Repairing Code Completion Systems
Li, Z., Wang, C., Liu, Z., Wang, H., Chen, D., Wang, S., & Gao, C. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00110 · Cited by 52
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2310.06770 · Cited by 72
The role of large language models in agriculture: harvesting the future with LLM intelligence
Shaikh, T. A., Rasool, T., Veningston, K., & Yaseen, S. M. (2024) · Progress in Artificial Intelligence · doi:10.1007/s13748-024-00359-4 · Cited by 86
Prompt Engineering or Fine-Tuning: An Empirical Assessment of LLMs for Code
Shin, J., Tang, C. J., Mohati, T., Nayebi, M., Wang, S., & Hemmati, H. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00082 · Cited by 182
Foundation Models in Agriculture: A Comprehensive Review
Yin, S., Xi, Y., Zhang, X., Sun, C., & Mao, Q. (2025) · Agriculture · doi:10.3390/agriculture15080847 · Cited by 76
Evaluating Contextually Personalized Programming Exercises Created with Generative AI
Logacheva, E., Hellas, A., Prather, J. E., Sarsa, S., & Leinonen, J. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671103 · Cited by 105
Chain-of-Thought in Neural Code Generation: From and for Lightweight Language Models
Yang, G., Zhou, Y., Chen, X., Zhang, X., Zhuo, T. Y., & Chen, T. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3440503 · Cited by 112
Self-citationAutomatic Programming: Large Language Models and Beyond
Lyu, M. R.-T., Ray, B., Roychoudhury, A., Tan, S. H., & Thongtanunam, P. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708519 · Cited by 132
Wang, W. Y., Zhang, S., Li, G., Lu, J., Ren, Y. S., Wang, X., Gao, X., Su, Y., Song, H., & Li, J. (2024) · Materials Genome Engineering Advances · doi:10.1002/mgea.56 · Cited by 82
ChatGPT for PLC/DCS Control Logic Generation
Koziolek, H., Grüner, S., & Ashiwal, V. (2023) · IEEE International Conference on Emerging Technologies and Factory Automation (ETFA) · doi:10.1109/etfa54631.2023.10275411 · Cited by 67
HPC-Coder: Modeling Parallel Programs using Large Language Models
Nichols, D., Marathe, A., Menon, H., Gamblin, T., & Bhatele, A. S. (2024) · ISC High Performance (ISC) · doi:10.23919/isc.2024.10528929 · Cited by 94
Comparing Large Language Models and Human Programmers for Generating Programming Code
Hou, W., & Ji, Z. (2025) · Advanced Science · doi:10.1002/advs.202412279 · Cited by 48
Natural Language to Code Translation with Execution
Shi, F., Fried, D., Ghazvininejad, M., Zettlemoyer, L., & Wang, S. I. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.231 · Cited by 46
Natural Language to Code Translation with Execution
Shi, F., Fried, D., Ghazvininejad, M., Zettlemoyer, L., & Wang, S. I. (2022) · arXiv · doi:10.48550/arxiv.2204.11454 · Cited by 49
Don’t Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environments
Gu, Y., Deng, X. B., & Su, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.270 · Cited by 167
Pan, Z., Xie, Z., Liu, T., & Xia, T. (2024) · Systems · doi:10.3390/systems12050176 · Cited by 46
A Survey on Symbolic Knowledge Distillation of Large Language Models
Acharya, K., Velasquez, A., & Song, H. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3428519 · Cited by 80
An Empirical Study of the Code Generation of Safety-Critical Software Using LLMs
Liu, M., Wang, J., Lin, T., Ma, Q., Fang, Z., & Wu, Y. (2024) · Applied Sciences · doi:10.3390/app14031046 · Cited by 79
Ataei, M., Cheong, H., Grandi, D., Wang, Y., Morris, N., & Tessier, A. (2024) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4067388 · Cited by 75
DreamCodeVR: Towards Democratizing Behavior Design in Virtual Reality with Speech-Driven Programming
Giunchi, D., Numan, N., Gatti, E., & Steed, A. (2024) · IEEE Conference on Virtual Reality and 3D User Interfaces · doi:10.1109/vr58804.2024.00078 · Cited by 63
Empowering the Metaverse with Generative AI: Survey and Future Directions
Qin, H. X., & Hui, P. (2023) · IEEE International Conference on Distributed Computing Systems Workshops (ICDCSW) · doi:10.1109/icdcsw60045.2023.00022 · Cited by 58
TrojanPuzzle: Covertly Poisoning Code-Suggestion Models
Aghakhani, H., Dai, W., Manoel, A., Fernandes, X., Kharkar, A., Kruegel, C., Vigna, G., Evans, D., Zorn, B., & Sim, R. (2024) · 2024 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp54263.2024.00140 · Cited by 41
Large scale foundation models for intelligent manufacturing applications: a survey
Zhang, H., Semujju, S. D., Wang, Z., Lv, X., Xu, K., Wu, L., Jia, Y., Wu, J., Liang, W., Zhuang, R., et al. (2025) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-024-02536-7 · Cited by 84
Extending the Frontier of ChatGPT: Code Generation and Debugging
Sakib, F. A., Khan, S. H., & Karim, A. H. M. R. (2023) · 2024 International Conference on Electrical, Computer and Energy Technologies (ICECET · doi:10.1109/icecet61485.2024.10698405 · Cited by 62
Resource-efficient Algorithms and Systems of Foundation Models: A Survey
Xu, M., Cai, D., Yin, W., Wang, S., Jin, X., & Liu, X. (2025) · ACM Computing Surveys · doi:10.1145/3706418 · Cited by 278
GAI identity threat: When and why do individuals feel threatened?
Zhou, J., Lu, Y., & Chen, Q. (2024) · Information & Management · doi:10.1016/j.im.2024.104093 · Cited by 63
Fully Autonomous Programming with Large Language Models
Liventsev, V., Grishina, A., Härmä, A., & Moonen, L. (2023) · Proceedings of the Genetic and Evolutionary Computation Conference · doi:10.1145/3583131.3590481 · Cited by 45
Shi, J., Yang, Z., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708525 · Cited by 43
Large Language Models to generate meaningful feature model instances
Galindo, J. Á., Dominguez, A. J., White, J., & Benavides, D. (2023) · International Software Product Line Conference (SPLC) · doi:10.1145/3579027.3608973 · Cited by 49
A survey on machine learning techniques applied to source code
Sharma, T., Kechagia, M., Georgiou, S., Tiwari, R., Vats, I., Moazen, H., & Sarro, F. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111934 · Cited by 173
PyDex: Repairing Bugs in Introductory Python Assignments using LLMs
Zhang, J., Cambronero, J., Gulwani, S., Le, V., Piskač, R., Soares, G., & Verbruggen, G. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3649850 · Cited by 165
Clover: Closed-Loop Verifiable Code Generation
Sun, C., Sheng, Y., Padon, O., & Barrett, C. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-65112-0_7 · Cited by 43
Clover: Closed-Loop Verifiable Code Generation
Sun, C., Sheng, Y., Padon, O., & Barrett, C. (2023) · arXiv · doi:10.48550/arxiv.2310.17807 · Cited by 43
“Code Generation Besides well-known work [14, 16, 38] on code generation using LLMs, [26] is a survey on program synthesis before the era of LLMs.”
BackgroundMethodReCode: Robustness Evaluation of Code Generation Models
Wang, S., Li, Z., Qian, H., Yang, C., Wang, Z., Shang, M., Kumar, V., Tan, S., Ray, B., Bhatia, P., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.773 · Cited by 42
Regulative development as a model for origin of life and artificial life studies
Fields, C., & Levin, M. (2023) · Biosystems · doi:10.1016/j.biosystems.2023.104927 · Cited by 40
Aljedaani, W., Habib, A., Aljohani, A., Eler, M., & Feng, Y. (2024) · Proceedings of the 21st International Web for All Conference · doi:10.1145/3677846.3677854 · Cited by 42
Hajipour, H., Hassler, K., Holz, T., Schönherr, L., & Fritz, M. (2024) · 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00040 · Cited by 40
Could ChatGPT get an engineering degree? Evaluating higher education vulnerability to AI assistants
Borges, B., Foroutan, N., Bayazit, D., Sotnikova, A., Montariol, S., Nazaretsky, T., Banaei, M., Sakhaeirad, A., Servant, P., Neshaei, S. P., et al. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2414955121 · Cited by 76
How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study
Gu, K., Grunde-McLaughlin, M., McNutt, A., Heer, J., & Althoff, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641891 · Cited by 40
“LLMs trained on large-scale code corpora [29, 31, 43, 73, 89] have shown proficiency in learning programming concepts, solving programming challenges [29], and covering various domains like web development [130] and data science problems [27, 71].”
BackgroundTeaching Code LLMs to Use Autocompletion Tools in Repository-Level Code Generation
Wang, C., Zhang, J., Feng, Y., Li, T., Sun, W., Liu, Y., & Peng, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3714462 · Cited by 40
“Recent advancements have introduced a variety of code large language models (LLMs) [7, 12, 13, 15, 17, 21, 28, 29, 33, 37, 46, 53, 54] constructed upon the Transformer model architecture [45], achieving promising performance in code-related applications [19, 20, 47—50, 52, 57, 59, 60].”
Challenges and opportunities in bioimage analysis
Li, X., Zhang, Y., Wu, J., & Dai, Q. (2023) · Nature Methods · doi:10.1038/s41592-023-01900-4 · Cited by 44
When Language Model Meets Private Library
Zan, D., Chen, B., Lin, Z., Guan, B., Yongji, W., & Lou, J.-G. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.21 · Cited by 36
When Language Model Meets Private Library
Zan, D., Chen, B., Lin, Z., Guan, B., Wang, Y., & Lou, J.-G. (2022) · arXiv · doi:10.48550/arxiv.2210.17236 · Cited by 39
CYCLE: Learning to Self-Refine the Code Generation
Ding, Y., Min, M. J., Kaiser, G. E., & Ray, B. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3649825 · Cited by 94
Bias Testing and Mitigation in LLM-based Code Generation
Huang, D., Zhang, J. M., Bu, Q., Xie, X., Chen, J., & Cui, H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3724117 · Cited by 174
Automated scholarly paper review: Concepts, technologies, and challenges
Lin, J., Song, J., Zhou, Z., Chen, Y., & Shi, X. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101830 · Cited by 37
Odeh, A. H., Odeh, N., & Mohammed, A. S. (2024) · TEM Journal · doi:10.18421/tem131-76 · Cited by 74
Hot or Cold? Adaptive Temperature Sampling for Code Generation with Large Language Models
Zhu, Y., Li, J., Li, G., Zhao, Y., Li, J., Jin, Z. G., & Mei, H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i1.27798 · Cited by 141
Can generative AI motivate management students? The role of perceived value and information literacy
Jose, E. M. K., Prasanna, A., Kushwaha, B. P., & Das, M. (2024) · The International Journal of Management Education · doi:10.1016/j.ijme.2024.101082 · Cited by 69
An Empirical Study of Retrieval-Augmented Code Generation: Challenges and Opportunities
Yang, Z., Chen, S., Gao, C., Li, Z., Hu, X., Liu, K., & Xia, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3717061 · Cited by 37
Open-ended Knowledge Tracing for Computer Science Education
Liu, N., Wang, Z., Baraniuk, R., & Lan, A. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.254 · Cited by 34
Wickramasekara, A., Breitinger, F., & Scanlon, M. (2025) · Forensic Science International Digital Investigation · doi:10.1016/j.fsidi.2024.301859 · Cited by 88
Data Formulator: AI-powered Concept-driven Visualization Authoring
Wang, C., Thompson, J., & Lee, B. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3326585 · Cited by 104
Navigating Ontology Development with Large Language Models
Saeedizade, M. J., & Blomqvist, E. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-60626-7_8 · Cited by 81
Agents in software engineering: survey, landscape, and vision
Wang, Y., Zhong, W., Huang, Y., Shi, E., Yang, M., Chen, J., Li, H., Ma, Y., Wang, Q., & Zheng, Z. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00544-2 · Cited by 131
Autonomous LLM-Driven Research — from Data to Human-Verifiable Research Papers
Ifargan, T., Hafner, L., Kern, M., Alcalay, O., & Kishony, R. (2025) · NEJM AI · doi:10.1056/aioa2400555 · Cited by 34
Framework for evaluating code generation ability of large language models
Yeo, S., Ma, Y., Kim, S. C., Jun, H., & Kim, T. (2024) · ETRI Journal · doi:10.4218/etrij.2023-0357 · Cited by 54
Metadata
- Authors
- Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., Hubert, T., Choy, P., de Masson d’Autume, C., Babuschkin, I., Chen, X., Huang, P.-S., Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D. J., Sutherland Robson, E., Kohli, P., de Freitas, N., Kavukcuoglu, K., & Vinyals, O.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- DeepMind, London, UK.
- Corresponding author
- Li, Yujia
- Choi, David
- Chung, Junyoung
- Kushman, Nate
- Schrittwieser, Julian
- Leblond, Rémi
- Eccles, Tom
- Keeling, James
- Gimeno, Felix
- Choy, Peter
- Vinyals, Oriol
- ORCID
- Venue
- Science
- Published
- 2022-12-09
- Volume
- 378
- Issue
- 6624
- Pages
- 1092-1097
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W4226287673
- Cited by
- 1,043
- References
- 34
- Citations by year
- 2026: 269 · 2025: 319 · 2024: 237 · 2023: 186 · 2022: 30
- FWCI
- 191.10
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- automatic code generation, competitive programming, programming competition, Codeforces, transformer language models, AI-assisted programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.