Competition-level code generation with AlphaCode
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Abstract
Programming is a powerful and ubiquitous problem-solving tool. Systems that can assist programmers or even generate programs themselves could make programming more productive and accessible. Recent transformer-based neural network models show impressive code generation abilities yet still perform poorly on more complex tasks requiring problem-solving skills, such as competitive programming problems. Here, we introduce AlphaCode, a system for code generation that achieved an average ranking in the top 54.3% in simulated evaluations on recent programming competitions on the Codeforces platform. AlphaCode solves problems by generating millions of diverse programs using specially trained transformer-based networks and then filtering and clustering those programs to a maximum of just 10 submissions. This result marks the first time an artificial intelligence system has performed competitively in programming competitions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AlphaCode, Self-Debugging and CodeRL are effective in fixing less-complex program bugs, but they rely upon ground truth test cases that invalidate pass@1 eligibility, and do not use self-reflection to bridge the gap between error identification and implementation improvement.”
From Reflexion: Language Agents with Verbal Reinforcement Learning (2023) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large language models are increasingly capable of complex programming tasks and are already being used to accelerate AI research, from generating training data to serving as programming tools [1, 2, 3, 4, 5, 6].”
From RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts (2024) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Generative AI, such as OpenAI’s ChatGPT, has rapidly emerged as a disruptive technology capable of achieving human-level performance on a broad range of tasks (1–5).”
From Generative AI without guardrails can harm learning: Evidence from high school mathematics (2025) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Li et al. [27] showed that the accuracy of AlphaCode increases from close to 0% zero-shot to over 15% with 1,000 retries and over 30% with a million retries (accuracy is measured by how often one of the top 10 answers generated by the model is correct).”
From AI Agents That Matter (2024) · cites this paper as Li et al. [27]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Evaluation covered 200 CodeContests tasks [15] across three seeds and up to three feedback rounds, with online tests supplying feedback and hidden tests measuring final success.”
From RSI-Anything: Unlocking Recursive Self-Improvement with World Models (2026) · cites this paper as [15]
Experimental — accuracy not yet measured
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Self-citationDecoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“Following Kaplan et al. (2020), we adjusted the amount of training for other model sizes such that larger models are trained more and smaller models are trained less to optimize the use of compute.”
MethodGrandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455 · cited 2×
“We believe that sampling metadata leads to a more diverse set of model samples, a strategy similar to that used by Vinyals et al. (2019), and allows our model to take advantage of the relative strengths across the metadata distribution.”
MethodSelf-citationResnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., et al. (2009) · Communications of the ACM · doi:10.1145/1592761.1592779 · Cited by 3,333
Kudo, T., & Richardson, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d18-2012 · Cited by 2,869
Research works printed but not linked: 6
- A. Church, Application of recursive arithmetic to the problem of circuit synthesis.. J. Symb. Log. 28, 289–290 (1963).
- Y. Nandwani D. Jindal Mausam P. Singla ?oNeural learning of one-of-many solutions for combinatorial problems in structured output spaces ?? International Conference on Learning Representations (ICLR 2021) 3 to 7 May 2021.
- A. Holtzman J. Buys L. Du M. Forbes Y. Choi ?oThe curious case of neural text degeneration ?? Proceedings of the 8th International Conference on Learning Representations (ICLR 2020) 26 April to 1 May 2020.
- J. Ganitkevitch B. Van Durme C. Callison-Burch ?oPPDB: The paraphrase database ?? Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Atlanta Georgia 9 to 15 June 2013 (Association for Computational Linguistics 2013) pp. 758–764.
- A. Solar-Lezama ?oProgram synthesis by sketching ?? thesis University of California Berkeley (2008).
- J. Cai R. Shin D. Song ?oMaking neural programming architectures generalize via recursion ?? Proceedings of the 5th International Conference on Learning Representations (ICLR) Toulon France 24 to 26 April 2017.
Cited by
Items 201–300 of 1,104
Deep Learning for Code Intelligence: Survey, Benchmark and Toolkit
Wan, Y., Bi, Z., He, Y., Zhang, J., Zhang, H., Sui, Y., Xu, G., Jin, H., & Yu, P. S. (2024) · ACM Computing Surveys · doi:10.1145/3664597 · Cited by 76
Statler: State-Maintaining Language Models for Embodied Reasoning
Yoneda, T., Fang, J., Li, P., Zhang, H., Jiang, T., Lin, S., Picker, B., Yunis, D., Mei, H., & Walter, M. R. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610634 · Cited by 92
Pirzado, F. A., Ahmed, A., Mendoza-Urdiales, R. A., & Terashima‐Marín, H. (2024) · IEEE Access · doi:10.1109/access.2024.3443621 · Cited by 65
Evaluating LLMs for Code Generation in HRI: A Comparative Study of ChatGPT, Gemini, and Claude
Sobo, A., Mubarak, A., Baimagambetov, A., & Polatidis, N. (2024) · Applied Artificial Intelligence · doi:10.1080/08839514.2024.2439610 · Cited by 71
A comprehensive overview of Generative AI (GAI): Technologies, applications, and challenges
Zhang, Z., Zhang, J., Zhang, X., & Mai, W. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129645 · Cited by 59
CodeRAG-Bench: Can Retrieval Augment Code Generation?
Wang, Z. Z., Asai, A., Yu, X. V., Xu, F. F., Xie, Y., Neubig, G., & Fried, D. (2024) · arXiv · doi:10.48550/arxiv.2406.14497 · Cited by 34
Code Difference Guided Adversarial Example Generation for Deep Code Models
Tian, Z., Chen, J., & Jin, Z. G. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00149 · Cited by 70
Guo, A., Pataranutaporn, P., & Maes, P. (2024) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613905.3650892 · Cited by 32
“The integration of large language models (LLMs) into everyday tasks, such as writing, programming, and customer service, is becoming increasingly common [12, 22, 33].”
From Misuse to Mastery: Enhancing Code Generation with Knowledge-Driven AI Chaining
Ren, X., Ye, X., Zhao, D., Xing, Z., & Yang, X. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00143 · Cited by 66
Robust GNN-Based Representation Learning for HLS
Sohrabizadeh, A., Bai, Y., Sun, Y., & Cong, J. (2023) · doi:10.1109/iccad57390.2023.10323853 · Cited by 69
AceCoder : An Effective Prompting Technique Specialized in Code Generation
Li, J., Zhao, Y., Li, Y., Li, G., & Jin, Z. G. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3675395 · Cited by 149
Self-citationSALLM: Security Assessment of Generated Code
Siddiq, M. L., Santos, J. C. S., Devareddy, S., & Muller, A. (2024) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1145/3691621.3694934 · Cited by 124
The sociolinguistic foundations of language modeling
Grieve, J., Bartl, S., Fuoli, M., Grafmiller, J., Huang, W., Jawerbaum, A., Murakami, A., Perlman, M., Rœmling, D., & Winter, B. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1472411 · Cited by 87
A LLM-informed multi-agent AI system for drone-based visual inspection for infrastructure
Liu, J., Li, H., Chai, C., Chen, K., & Wang, D. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103643 · Cited by 32
Exploring Generative Artificial Intelligence Research: A Bibliometric Analysis Approach
Dwivedi, R., & Elluri, L. (2024) · IEEE Access · doi:10.1109/access.2024.3450629 · Cited by 50
LLM Compiler: Foundation Language Models for Compiler Optimization
Cummins, C., Seeker, V., Grubisic, D., Rozière, B., Gehring, J., Synnaeve, G., & Leather, H. (2025) · ACM SIGPLAN International Conference on Compiler Construction (CC) · doi:10.1145/3708493.3712691 · Cited by 164
PADL: Language-Directed Physics-Based Character Control
Juravsky, J., Guo, Y., Fidler, S., & Peng, X. B. (2022) · SIGGRAPH Asia Conference Papers · doi:10.1145/3550469.3555391 · Cited by 69
BioCoder: a benchmark for bioinformatics code generation with large language models
Tang, X., Qian, B., Gao, R., Chen, J., Chen, X., & Gerstein, M. B. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae230 · Cited by 30
“In response, execution-based evaluation approaches have become more prevalent (Chen et al. 2021, Khlaaf et al. 2022, Lai et al. 2022, Li et al. 2022, Wang et al. 2022b, Athiwaratkun et al. 2023).”
Self-citationIdentifying algorithm in program code based on structural features using CNN classification model
Watanobe, Y., Rahman, M. M., Amin, M. F. I., & Kabir, R. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04078-y · Cited by 43
Evaluating Diverse Large Language Models for Automatic and General Bug Reproduction
Kang, S., Yoon, J., Askarbekkyzy, N., & Yoo, S. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3450837 · Cited by 54
Questions Are All You Need to Train a Dense Passage Retriever
Sachan, D. S., Lewis, M., Yogatama, D., Zettlemoyer, L., Pineau, J., & Zaheer, M. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00564 · Cited by 76
Bai, X., Huang, S., Wei, C., & Wang, R. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126357 · Cited by 48
Tedeschi, L. O. (2023) · animal · doi:10.1016/j.animal.2023.100813 · Cited by 37
Liao, D., Pan, S., Sun, X., Ren, X., Huang, Q., Xing, Z., Jin, H., & Li, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3486195 · Cited by 23
Wu, Y., Li, Z., Zhang, J. M., & Liu, Y. (2024) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3663529.3663815 · Cited by 51
Design pattern recognition: a study of large language models
Pandey, S. K., Chand, S., Horkoff, J., Staron, M., Ochodek, M., & Durisic, D. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10625-1 · Cited by 46
Passport: Improving Automated Formal Verification Using Identifiers
Sanchez-Stern, A., First, E., Zhou, T., Kaufman, Z., Brun, Y., & Ringer, T. (2023) · ACM Transactions on Programming Languages and Systems · doi:10.1145/3593374 · Cited by 51
Entailer: Answering Questions with Faithful and Truthful Chains of Reasoning
Tafjord, O., Dalvi Mishra, B., & Clark, P. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.134 · Cited by 27
Codepori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technology
Rasheed, Z., Malik, S., Kemell, K.-K., Muhammad, W., Saari, M., Systä, K., & Abrahamsson, P. (2024) · doi:10.2139/ssrn.4979510 · Cited by 27
Repository-Level Prompt Generation for Large Language Models of Code
Shrivastava, D., Larochelle, H., & Tarlow, D. (2022) · arXiv · doi:10.48550/arxiv.2206.12839 · Cited by 27
“We also have encoder-decoder models that generally use a bidirectional encoding of a context to decode a series of masked tokens (Wang et al., 2021b; Li et al., 2022). et al., 2020).”
MethodMagicoder: Empowering Code Generation with OSS-Instruct
Wei, Y., Wang, Z., Liu, J., Ding, Y., & Zhang, L. (2023) · arXiv · doi:10.48550/arxiv.2312.02120 · Cited by 27
InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback
Yang, J., Prabhakar, A., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2306.14898 · Cited by 26
How Important Are Good Method Names in Neural Code Generation? A Model Robustness Perspective
Yang, G., Zhou, Y., Yang, W., Yue, T., Chen, X., & Chen, T. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3630010 · Cited by 45
Self-citationAn Empirical Evaluation of Competitive Programming AI: A Case Study of AlphaCode
Lertbanjongngam, S., Chinthanet, B., Ishio, T., Kula, R. G., Leelaprute, P., Manaskasemsak, B., Rungsawang, A., & Matsumoto, K. (2022) · IEEE International Workshop on Software Clones (IWSC) · doi:10.1109/iwsc55060.2022.00010 · Cited by 38
(2023) · doi:10.52202/075280-1035 · Cited by 26
Public attitudes toward chatgpt on twitter: sentiments, topics, and occupations
Koonchanok, R., Pan, Y., & Jang, H. (2024) · Social Network Analysis and Mining · doi:10.1007/s13278-024-01260-7 · Cited by 25
“It has captured people’s imagination about a revolutionized future of programming (Chen et al. 2021; Li et al. 2022), writing (Mirowski et al. 2022; Lee et al. 2022), and many more (Kocoń et al. 2023; Fatouros et al. 2023).”
Metasql: A Generate-Then-Rank Framework for Natural Language to SQL Translation
Fan, Y., He, Z., Ren, T., Huang, C., Jing, Y., Zhang, K., & Wang, X. S. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00143 · Cited by 51
Search-Based LLMs for Code Optimization
Gao, S., Gao, C., Gu, W., & Lyu, M. R.-T. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00021 · Cited by 79
Harnessing the Power of General-Purpose LLMs in Hardware Trojan Design
Kokolakis, G., Moschos, A., & Keromytis, A. D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-61486-6_11 · Cited by 39
Automated Test Case Repair Using Language Models
Yaraghi, A. S., Holden, D., Kahani, N., & Briand, L. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3541166 · Cited by 57
Opportunities and Challenges of Foundation Models in Industrial Manufacturing
Ayyat, M., Osman, M., & Nadeem, T. (2025) · IEEE Access · doi:10.1109/access.2025.3589777 · Cited by 31
An agent-based approach for the automatic generation of valid SysMLv2 Models in industrial contexts
Cibrián, E., Olivert-Iserte, J., Lloréns, J., & Álvarez-Rodríguez, J. M. (2025) · Computers in Industry · doi:10.1016/j.compind.2025.104350 · Cited by 27
Learning Performance-Improving Code Edits
Shypula, A., Madaan, A., Zeng, Y., Alon, U., Gardner, J., Hashemi, M., Neubig, G., Ranganathan, P., Bastani, O., & Yazdanbakhsh, A. (2023) · arXiv · doi:10.48550/arxiv.2302.07867 · Cited by 146
“To improve coverage, we include additional test cases from AlphaCode (Li et al., 2022) generated with a fine-tuned LLM.”
BackgroundMethodInfluentialGrounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23
“…12 billion model parameters and is trained on 54 million software repositories on GitHub, is able to correctly solve 30–70% of novel Python problems, while DeepMind’s AlphaCode [Li et al. 2022] ranked in the top 54.3% among 5000 human programmers on the competitive programming platform Codeforces.”
BackgroundInfluentialWhen to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2306.04930 · Cited by 22
“…performance of code recommendation models, but these typically assess how well the model can complete code in an offline setting without developer input rather than evaluating how well it assists programmers in situ (Ziegler et al. 2022; Li et al. 2022; Evtikhiev et al. 2022; Dakhel et al. 2022).”
BackgroundMFTCoder: Boosting Code LLMs with Multitask Fine-Tuning
Liu, B., Chen, C., Gong, Z., Liao, C., Wang, H., Lei, Z., Liang, M., Chen, D., Shen, M., Zhou, H., et al. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671609 · Cited by 23
Feng, K., Luo, L., Xia, Y., Luo, B., He, X., Li, K., Zha, Z., Xu, B., & Peng, K. (2024) · Symmetry · doi:10.3390/sym16111470 · Cited by 33
Programming with AI: Evaluating ChatGPT, Gemini, AlphaCode, and GitHub Copilot for Programmers
Siam, K., Gu, H., & Cheng, J. Q. (2024) · International Conference on Computing Advancements (ICCA) · doi:10.1145/3723178.3723224 · Cited by 39
Espejel, J. L., Alassan, M. S. Y., Chouham, E. M., Dahhane, W., & Ettifouri, E. H. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100013 · Cited by 28
On Extracting Specialized Code Abilities from Large Language Models: A Feasibility Study
Li, Z., Wang, C., Ma, P., Liu, C., Wang, S., Wu, D., Gao, C., & Liu, Y. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639091 · Cited by 67
Comparative Analysis of Generative AI Models
Rani, G., Singh, J., & Khanna, A. (2023) · International Conference on Advances in Computation, Communication and Information Technology (ICAICCIT) · doi:10.1109/icaiccit60255.2023.10465941 · Cited by 33
Data Authenticity, Consent, and Provenance for AI Are All Broken: What Will It Take to Fix Them?
Longpre, S., Mahari, R., Obeng-Marnu, N., Brannon, W., South, T., Kabbara, J., & Pentland, A. (2024) · doi:10.21428/e4baedd9.a650f77d · Cited by 41
ChatGPT Code Detection: Techniques for Uncovering the Source of Code
Oedingen, M., Engelhardt, R. C., Denz, R., Hammer, M., & Konen, W. (2024) · AI · doi:10.3390/ai5030053 · Cited by 37
Kumar, J., & Chimalakonda, S. (2024) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3661167.3661210 · Cited by 37
Technology adoption performance evaluation applied to testing industrial REST APIs
Poth, A., Rrjolli, O., & Arcuri, A. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00477-2 · Cited by 38
CodeFuse-13B: A Pretrained Multi-lingual Code Large Language Model
Di, P., Li, J., Yu, H., Jiang, W., Cai, W., Cao, Y., Chen, C., Chen, D., Chen, H., Chen, L., et al. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3639477.3639719 · Cited by 59
LUNA: A Model-Based Universal Analysis Framework for Large Language Models
Song, D., Xie, X., Song, J., Zhu, D., Huang, Y., Juefei-Xu, F., & Ma, L. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3411928 · Cited by 41
When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i9.28878 · Cited by 23
Zhang, H., Cheng, W., Wu, Y., & Hu, W. P. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695506 · Cited by 73
Multi-stage guided code generation for Large Language Models
Han, Y., & Lyu, C. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109491 · Cited by 36
AgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodology
Nguyen, M. H., Chau, T. P., Nguyen, P. X., & Bui, N. D. Q. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00026 · Cited by 96
Steps towards an Ecology for the Internet: Computing through crisis into the next decade of growth
Madhavapeddy, A., Reynolds, S., Christie, A. P., Coomes, D. A., Dales, M., Ferris, P., Gibb, R. T., Haddadi, H., Jaffer, S., Millar, J., et al. (2025) · Decennial Aarhus Conference · doi:10.1145/3744169.3744180 · Cited by 22
OmniTester: Multimodal Large Language Model Driven Scenario Testing for Autonomous Vehicles
Lu, Q., Wang, X., Jiang, Y., Zhao, G., Ma, M., & Feng, S. (2025) · Automotive Innovation · doi:10.1007/s42154-025-00364-w · Cited by 69
Agentic AI and the rise of in silico team science in biomedical research
Li, B., Saini, A. K., Hernandez, J. G., & Moore, J. H. (2026) · Nature Biotechnology · doi:10.1038/s41587-026-03035-1 · Cited by 45
Generative AI: Overview, Economic Impact, and Applications in Asset ManagementPreprint
Luk, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4574814 · Cited by 26
Rasheed, Z., Waseem, M., Ahmad, A., Kemell, K., Wang, X., NguyenDuc, A., & Abrahamsson, P. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5149683 · Cited by 27
AgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodology
Nguyen, M. H., Chau, T. P., Nguyen, P. X., & Bui, N. D. Q. (2024) · arXiv · doi:10.48550/arxiv.2406.11912 · Cited by 21
Assessing Learning of Computer Programing Skills in the Age of Generative Artificial Intelligence
Wilson, S. E., & Nishimoto, M. (2024) · Journal of Biomechanical Engineering · doi:10.1115/1.4064364 · Cited by 20
The “Code” of Ethics: A Holistic Audit of AI Code Generators
Ma, W., Song, Y., Xue, M., Wen, S., & Xiang, Y. (2024) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2024.3367737 · Cited by 31
RTL++: Graph-enhanced LLM for RTL Code Generation
Akyash, M., Azar, K. Z., & Kamali, H. (2025) · IEEE International Conference on LLM-Aided Design (ICLAD) · doi:10.1109/iclad65226.2025.00020 · Cited by 45
Liu, Z., Zeng, R., Wang, D., Peng, G., Liu, X., Liu, Q., Liu, P., Wang, W., & Wang, J. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3667895 · Cited by 20
“Emerging LLMs for code generation (a.k.a. code LLMs), such as Openai Codex [10], AlphaCode [37], CodeLlama [46], are shown (see our empirical results later) not ideal for PLC code generation due to the following reasons.”
Storhaug, A., Li, J., & Hu, T. (2023) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre59848.2023.00035 · Cited by 47
Execution-Based Evaluation for Open-Domain Code Generation
Wang, Z., Zhou, S., Fried, D., & Neubig, G. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.89 · Cited by 134
AI-Assisted Programming Tasks Using Code Embeddings and Transformers
Kotsiantis, S. B., Verykios, V. S., & Tzagarakis, M. M. (2024) · Electronics · doi:10.3390/electronics13040767 · Cited by 39
Penney, J., Pimentel, J. F., Steinmacher, I., & Gerosa, M. A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-54975-5_12 · Cited by 29
Transformers in source code generation: A comprehensive survey
Ghaemi, H., Alizadehsani, Z., Shahraki, A., & Corchado, J. M. (2024) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2024.103193 · Cited by 31
Network model with internal complexity bridges artificial intelligence and neuroscience
He, L., Xu, Y., He, W., Lin, Y., Tian, Y., Wu, Y., Wang, W., Zhang, Z., Han, J., Tian, Y., et al. (2024) · Nature Computational Science · doi:10.1038/s43588-024-00674-9 · Cited by 24
Execution-Based Evaluation for Open-Domain Code Generation
Wang, Z., Zhou, S., Fried, D., & Neubig, G. (2022) · arXiv · doi:10.48550/arxiv.2212.10481 · Cited by 20
“These findings suggest that common practices to improve LLMs may not address the complexities inherent in open-domain code generation problems.”
BackgroundMethodInfluentialAgentless: Demystifying LLM-based Software Engineering Agents
Xia, C. S., Deng, Y., Dunn, S., & Zhang, L. (2024) · arXiv · doi:10.48550/arxiv.2407.01489 · Cited by 24
“This is done by providing the previously described repository structure and asking the LLM to produce a list of irrelevant folders that do not need to be further inspected or modified to resolve the issue.”
MethodReading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2022) · arXiv · doi:10.48550/arxiv.2210.14306 · Cited by 114
“Popular systems, including Copilot [Github, 2022], CodeWhisperer [Amazon, 2022], and AlphaCode[Li et al., 2022], herald the start of a tectonic shift in how software is developed.”
MethodOlympiad-level formal mathematical reasoning with reinforcement learning
Hubert, T., Mehta, R., Sartran, L., Horváth, M. Z., Žužić, G., Wieser, E., Huang, A., Schrittwieser, J., Schroecker, Y., Masoom, H., et al. (2026) · Nature · doi:10.1038/s41586-025-09833-y · Cited by 19
“At its core is the proof network, a 3-billion-parameter encoder–decoder transformer model 15,16 , that learns to interpret the observed Lean tactic state (Fig.”
Self-citationL2CEval : Evaluating Language-to-Code Generation Capabilities of Large Language Models
Ni, A., Yin, P., Zhao, Y., Riddell, M., Feng, T., Shen, R., Yin, S., Ye, L., Yavuz, S., Xiong, C., et al. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00705 · Cited by 76
Kemell, K., Saarikallio, M., Nguyen‐Duc, A., & Abrahamsson, P. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107805 · Cited by 42
Greengard, S. (2023) · Communications of the ACM · doi:10.1145/3583083 · Cited by 25
On the Evaluation of Neural Code Translation: Taxonomy and Benchmark
Jiao, M., Yu, T., Li, X., Qiu, G., Gu, X., & Shen, B. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00114 · Cited by 90
Analyzing Prompt Influence on Automated Method Generation: An Empirical Study with Copilot
Fagadau, I. D., Mariani, L., Micucci, D., & Riganelli, O. (2024) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3643916.3644409 · Cited by 38
Knowledge-Aware Code Generation with Large Language Models
Huang, T., Sun, Z., Jin, Z. G., Li, G., & Lyu, C. (2024) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3643916.3644418 · Cited by 43
SciSciGPT: advancing human–AI collaboration in the science of science
Shao, E., Wang, Y., Qian, Y., Pan, Z., Liu, H., & Wang, D. (2025) · Nature Computational Science · doi:10.1038/s43588-025-00906-6 · Cited by 77
PTM-APIRec: Leveraging Pre-trained Models of Source Code in API Recommendation
Li, Z., Li, C., Tang, Z. J., Huang, W., Ge, J., Luo, B., Ng, V., Wang, T., Hu, Y., & Zhang, X. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3632745 · Cited by 24
Let’s Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs
Aggarwal, P., Madaan, A., Yang, Y., & Mausam, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.761 · Cited by 234
Rahman, M. M., & Watanobe, Y. (2023) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2023.3336920 · Cited by 25
Integration of cognitive tasks into artificial general intelligence test for large models
Qu, Y., Wei, C., Du, P., Che, W., Zhang, C., Ouyang, W., Bian, Y., Xu, F., Hu, B., Du, K., et al. (2024) · iScience · doi:10.1016/j.isci.2024.109550 · Cited by 25
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ac4sp · Cited by 157
Self-citationOctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 245
HALO: an ontology for representing and categorizing hallucinations in large language models
Nananukul, N., & Kejriwal, M. (2023) · Defense + Commercial Sensing · doi:10.1117/12.3014048 · Cited by 7
Wu, H., Shen, Z., Hou, S., Liang, J., Jiao, H., Qing, Y., Zhang, X., Li, X., Gui, Z., Guan, X., et al. (2025) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi14070256 · Cited by 26
Large Language Model Powered Symbolic Execution
Li, Y., Meng, R., & Duck, G. J. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3763163 · Cited by 48
Self-citationA User-centered Security Evaluation of Copilot
Asare, O., Nagappan, M., & Asokan, N. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639154 · Cited by 32
COLDECO: An End User Spreadsheet Inspection Tool for AI-Generated Code
Ferdowsi, K., Williams, J., Drosos, I., Gordon, A. D., Negreanu, C., Polikarpova, N., Sarkar, A., & Zorn, B. (2023) · 2023 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc57772.2023.00017 · Cited by 17
REEF: A Framework for Collecting Real-World Vulnerabilities and Fixes
Wang, C., Li, Z., Pena, Y., Gao, S., Chen, S., Wang, S., Gao, C., & Lyu, M. R.-T. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00199 · Cited by 33
Metadata
- Authors
- Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., Hubert, T., Choy, P., de Masson d’Autume, C., Babuschkin, I., Chen, X., Huang, P.-S., Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D. J., Sutherland Robson, E., Kohli, P., de Freitas, N., Kavukcuoglu, K., & Vinyals, O.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- DeepMind, London, UK.
- Corresponding author
- Li, Yujia
- Choi, David
- Chung, Junyoung
- Kushman, Nate
- Schrittwieser, Julian
- Leblond, Rémi
- Eccles, Tom
- Keeling, James
- Gimeno, Felix
- Choy, Peter
- Vinyals, Oriol
- ORCID
- Venue
- Science
- Published
- 2022-12-09
- Volume
- 378
- Issue
- 6624
- Pages
- 1092-1097
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W4226287673
- Cited by
- 1,043
- References
- 34
- Citations by year
- 2026: 269 · 2025: 319 · 2024: 237 · 2023: 186 · 2022: 30
- FWCI
- 191.10
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- automatic code generation, competitive programming, programming competition, Codeforces, transformer language models, AI-assisted programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.