Competition-level code generation with AlphaCode
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Abstract
Programming is a powerful and ubiquitous problem-solving tool. Systems that can assist programmers or even generate programs themselves could make programming more productive and accessible. Recent transformer-based neural network models show impressive code generation abilities yet still perform poorly on more complex tasks requiring problem-solving skills, such as competitive programming problems. Here, we introduce AlphaCode, a system for code generation that achieved an average ranking in the top 54.3% in simulated evaluations on recent programming competitions on the Codeforces platform. AlphaCode solves problems by generating millions of diverse programs using specially trained transformer-based networks and then filtering and clustering those programs to a maximum of just 10 submissions. This result marks the first time an artificial intelligence system has performed competitively in programming competitions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AlphaCode, Self-Debugging and CodeRL are effective in fixing less-complex program bugs, but they rely upon ground truth test cases that invalidate pass@1 eligibility, and do not use self-reflection to bridge the gap between error identification and implementation improvement.”
From Reflexion: Language Agents with Verbal Reinforcement Learning (2023) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large language models are increasingly capable of complex programming tasks and are already being used to accelerate AI research, from generating training data to serving as programming tools [1, 2, 3, 4, 5, 6].”
From RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts (2024) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Generative AI, such as OpenAI’s ChatGPT, has rapidly emerged as a disruptive technology capable of achieving human-level performance on a broad range of tasks (1–5).”
From Generative AI without guardrails can harm learning: Evidence from high school mathematics (2025) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Li et al. [27] showed that the accuracy of AlphaCode increases from close to 0% zero-shot to over 15% with 1,000 retries and over 30% with a million retries (accuracy is measured by how often one of the top 10 answers generated by the model is correct).”
From AI Agents That Matter (2024) · cites this paper as Li et al. [27]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Evaluation covered 200 CodeContests tasks [15] across three seeds and up to three feedback rounds, with online tests supplying feedback and hidden tests measuring final success.”
From RSI-Anything: Unlocking Recursive Self-Improvement with World Models (2026) · cites this paper as [15]
Experimental — accuracy not yet measured
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Self-citationDecoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“Following Kaplan et al. (2020), we adjusted the amount of training for other model sizes such that larger models are trained more and smaller models are trained less to optimize the use of compute.”
MethodGrandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455 · cited 2×
“We believe that sampling metadata leads to a more diverse set of model samples, a strategy similar to that used by Vinyals et al. (2019), and allows our model to take advantage of the relative strengths across the metadata distribution.”
MethodSelf-citationResnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., et al. (2009) · Communications of the ACM · doi:10.1145/1592761.1592779 · Cited by 3,333
Kudo, T., & Richardson, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d18-2012 · Cited by 2,869
Research works printed but not linked: 6
- A. Church, Application of recursive arithmetic to the problem of circuit synthesis.. J. Symb. Log. 28, 289–290 (1963).
- Y. Nandwani D. Jindal Mausam P. Singla ?oNeural learning of one-of-many solutions for combinatorial problems in structured output spaces ?? International Conference on Learning Representations (ICLR 2021) 3 to 7 May 2021.
- A. Holtzman J. Buys L. Du M. Forbes Y. Choi ?oThe curious case of neural text degeneration ?? Proceedings of the 8th International Conference on Learning Representations (ICLR 2020) 26 April to 1 May 2020.
- J. Ganitkevitch B. Van Durme C. Callison-Burch ?oPPDB: The paraphrase database ?? Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Atlanta Georgia 9 to 15 June 2013 (Association for Computational Linguistics 2013) pp. 758–764.
- A. Solar-Lezama ?oProgram synthesis by sketching ?? thesis University of California Berkeley (2008).
- J. Cai R. Shin D. Song ?oMaking neural programming architectures generalize via recursion ?? Proceedings of the 5th International Conference on Learning Representations (ICLR) Toulon France 24 to 26 April 2017.
Cited by
Items 301–400 of 1,104
Exploring Large Language Models for Multi-Modal Out-of-Distribution Detection
Dai, Y. Q., Lang, H., Zeng, K., Huang, F., & Li, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.351 · Cited by 38
Self-citationFostering computer programming and digital skills development: An experimental approach
Omeh, C. B., Olelewe, C. J., & Nwangwu, E. C. (2024) · Computer Applications in Engineering Education · doi:10.1002/cae.22711 · Cited by 30
Code Gradients: Towards Automated Traceability of LLM-Generated Code
North, M., Atapour–Abarghouei, A., & Bencomo, N. (2024) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re59067.2024.00038 · Cited by 34
Deep learning for code generation: a survey
Zhang, H., Zhang, K., Li, Z., Li, J., Li, J., Li, Y., Zhao, Y., Zhu, Y. S., Liu, F., Li, G., et al. (2024) · Science China Information Sciences · doi:10.1007/s11432-023-3956-3 · Cited by 27
Self-citationBetween Lines of Code: Unraveling the Distinct Patterns of Machine and Human Programmers
Shi, Y., Zhang, H., Wan, C., & Gu, X. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00005 · Cited by 67
Islam, M. A., Ali, M. E., & Parvez, M. R. (2025) · arXiv · doi:10.48550/arxiv.2502.05664 · Cited by 17
Generating vulnerability security fixes with Code Language Models
Bhandari, G. P., Gavrić, N., & Shalaginov, A. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107786 · Cited by 28
Refactoring to Pythonic Idioms: A Hybrid Knowledge-Driven Approach Leveraging Large Language Models
Zhang, Z., Xing, Z., Ren, X., Lu, Q., & Xu, X. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3643776 · Cited by 25
Reasoning and Planning with Large Language Models in Code Development
Ding, H., Fan, Z., Guehring, I., Gupta, G. P., Ha, W., Huan, J., Liu, L. Q., Omidvar-Tehrani, B., Wang, S., & Zhou, H. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671452 · Cited by 40
Leveraging large language models for comprehensive locomotion control in humanoid robots design
Sun, S., Li, C., Zhao, Z., Huang, H., & Xu, W. (2024) · Biomimetic Intelligence and Robotics · doi:10.1016/j.birob.2024.100187 · Cited by 28
An assessment of large language models for OpenMP-based code parallelization: a user perspective
Mišić, M. J., & Dodović, M. (2024) · Journal Of Big Data · doi:10.1186/s40537-024-01019-z · Cited by 21
Skill-Based Few-Shot Selection for In-Context Learning
An, S., Zhou, B., Lin, Z., Fu, Q., Chen, B., Zheng, N., Chen, W., & Lou, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.831 · Cited by 76
Exploring the Impacts of ChatGPT on Future Scientific WorkPreprint
Irons, J. L., Mason, C., Cooper, P. S., Sidra, S., Reeson, A. F., & Paris, C. L. (2023) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/j2u9x · Cited by 18
Public Attitudes Toward ChatGPT on Twitter: Sentiments, Topics, and OccupationsPreprint
Koonchanok, R., Pan, Y., & Jang, H. (2024) · Research Square · doi:10.21203/rs.3.rs-3945065/v1 · Cited by 20
International AI Safety Report
Bengio, Y., Mindermann, S., Privitera, D., Besiroglu, T., Bommasani, R., Casper, S., Choi, Y., Fox, P., Garfinkel, B., Goldfarb, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.17805 · Cited by 16
AlphaEvolve: A coding agent for scientific and algorithmic discovery
Novikov, A., Vũ, N., Eisenberger, M., Dupont, E., Huang, P.-S., Wagner, A. Z., Shirobokov, S., Kozlovskii, B., Ruiz, F. J. R., Mehrabian, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.13131 · Cited by 15
“This literature builds on the success of LLMs in coding tasks, perhaps best illustrated by their success in (simulated) programming competitions as in the case of AlphaCode [60].”
Self-citationCase Scenario Generators for Trauma Surgery Simulation Utilizing Autoregressive Language Models
Chung, P. J., Boodoo, M., & Doboli, S. (2022) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102635 · Cited by 17
Coherence boosting: When your pretrained language model is not paying enough attention
Malkin, N., Wang, Z., & Jojić, N. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.565 · Cited by 56
Salikutluk, V., Koert, D., & Jäkel, F. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230081 · Cited by 22
Generative AI for Software Development: A Family of Studies on Code Generation
Dakhel, A. M., Nikanjam, A., Khomh, F., Desmarais, M. C., & Washizaki, H. (2024) · Springer eBooks · doi:10.1007/978-3-031-55642-5_7 · Cited by 20
Sänger, M., Mecquenem, N. D., Lewińska, K. E., Bountris, V., Lehmann, F., Leser, U., & Kosch, T. (2024) · GigaScience · doi:10.1093/gigascience/giae030 · Cited by 23
APIGen: Generative API Method Recommendation
Chen, Y., Gao, C., Zhu, M., Liao, Q., Wang, Y., & Xu, G. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00025 · Cited by 30
CoSec: On-the-Fly Security Hardening of Code LLMs via Supervised Co-decoding
Li, D., Yan, M. L., Zhang, Y. X., Liu, Z., Liu, C. C., Zhang, X., Chen, T., & Lo, D. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680371 · Cited by 39
Qu, Y., Huang, S., Li, Y., Bai, T., Chen, X., Wang, X., Li, L., & Yao, Y. (2025) · Automated Software Engineering · doi:10.1007/s10515-024-00485-2 · Cited by 24
Self-citationCOFFE: A Code Efficiency Benchmark for Code Generation
Peng, Y., Wan, J., Li, Y., & Ren, X. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715727 · Cited by 61
Self-citationPerfCodeGen: Improving Performance of LLM Generated Code with Execution Feedback
Peng, Y. F., Gotmare, A. D., Lyu, M. R.-T., Xiong, C., Savarese, S., & Sahoo, D. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00008 · Cited by 77
Chamola, V., Dave, D., Goyal, I., & Sharma, S. (2026) · Expert Systems · doi:10.1111/exsy.70201 · Cited by 28
Standing on the Shoulders of AI Giants
Lin, H.-Y. (2023) · Computer · doi:10.1109/mc.2022.3218176 · Cited by 18
Zero-shot Prompting for Code Complexity Prediction Using GitHub Copilot
Siddiq, M. L., Samee, A., Azgor, S. R., Haider, M. A., Sawraz, S. I., & Santos, J. C. S. (2023) · IEEE/ACM International Workshop on Natural Language-Based Software Engineering (NLBSE) · doi:10.1109/nlbse59153.2023.00018 · Cited by 18
Gar: A Generate-and-Rank Approach for Natural Language to SQL Translation
Fan, Y., He, Z., Ren, T., Guo, D., Chen, L., Zhu, R., Chen, G., Jing, Y., Zhang, K., & Wang, X. (2023) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde55515.2023.00016 · Cited by 24
Sun, Z., Du, X., Song, F., Wang, S., & Li, L. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639120 · Cited by 29
Tao, N., Ventresque, A., Nallur, V., & Saber, T. (2024) · Algorithms · doi:10.3390/a17070287 · Cited by 32
Rethinking AI code generation: a one-shot correction approach based on user feedback
Le, K. T., & Andrzejak, A. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00451-y · Cited by 23
He, Y., Huang, J., Rong, Y., Guo, Y., Wang, E., & Chen, H. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680342 · Cited by 32
GIA: LLM-Enabled Generative Intent Abstraction to Enhance Adaptability for Intent-Driven Networks
Kou, S., Yang, C., & Gurusamy, M. (2025) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2025.3545589 · Cited by 21
LogoMotion: Visually-Grounded Code Synthesis for Creating and Editing Animation
Liu, V., Kazi, R. H., Wei, L., Fisher, M., Langlois, T., Walker, S., & Chilton, L. B. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714155 · Cited by 46
Can large language models generate geospatial code?
Hou, S., Shen, Z., Liang, J., Jiao, H., Zhao, A., Qing, Y., Peng, D., Gui, Z., Guan, X., Xiang, L., et al. (2025) · Geo-spatial Information Science · doi:10.1080/10095020.2025.2535523 · Cited by 26
AIGC for Industrial Time Series: From Deep-Generative Models to Large-Generative Models
Ren, L., Wang, H., Li, J., Tang, Y. S., & Yang, C. (2025) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2025.3598252 · Cited by 36
Executable World Models for ARC-AGI-3 in the Era of Coding Agents
Rodionov, S. (2026) · arXiv · doi:10.48550/arxiv.2605.05138 · Cited by 14
Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., & Mirhoseini, A. (2024) · arXiv · doi:10.48550/arxiv.2407.21787 · Cited by 14
AI-Generated Code Not Considered Harmful
Kendon, T., Wu, L., & Aycock, J. (2023) · Proceedings of the 25th Western Canadian Conference on Computing Education · doi:10.1145/3593342.3593349 · Cited by 13
(2022) · doi:10.2139/ssrn.4496380 · Cited by 20
Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code
Liu, F., Liu, Y., Shi, L., Yang, Z., Zhang, L., Lian, X., Li, Z., & Ma, Y. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3657432 · Cited by 412
Pair programming with ChatGPT for sampling and estimation of copulas
Górecki, J. (2023) · Computational Statistics · doi:10.1007/s00180-023-01437-2 · Cited by 17
Advances and challenges in artificial intelligence text generation
Li, B., Yang, P., Sun, Y., Hu, Z., & Yi, M. (2024) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2300410 · Cited by 25
Metamorphic Malware Evolution: The Potential and Peril of Large Language Models
Madani, P. (2023) · IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA) · doi:10.1109/tps-isa58951.2023.00019 · Cited by 24
Piloting Copilot, Codex, and StarCoder2: Hot temperature, cold prompts, or black magic?
Döderlein, J.-B., Kouadio, N. H., Acher, M., Khelladi, D. E., & Combemale, B. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112562 · Cited by 70
EvoR: Evolving Retrieval for Code Generation
Su, H., Jiang, S., Lai, Y., Wu, H., Shi, B., Liu, C., Liu, Q., & Yu, T. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.143 · Cited by 12
“Additionally, we are connected to the line of code execution since we also leverage it as an important retrieval source. generation (Li et al., 2022; Nijkamp et al., 2022; Li et al., 2023b; Roziere et al., 2023; Wei et al., 2023).”
TextGrad: Automatic "Differentiation" via Text
Yuksekgonul, M., Bianchi, F., Boen, J., Liu, S., Huang, Z., Guestrin, C., & Zou, J. (2024) · arXiv · doi:10.48550/arxiv.2406.07496 · Cited by 12
Machine Learning for Translating Pseudocode to Python: A Comprehensive Review
Tiwari, S. P., Prasad, S., & Thushara, M. G. (2023) · International Conference on Intelligent Computing and Control Systems (ICICCS) · doi:10.1109/iciccs56967.2023.10142254 · Cited by 13
Rule-Based Error Classification for Analyzing Differences in Frequent Errors
Shirafuji, A., Matsumoto, T., Amin, M. F. I., & Watanobe, Y. (2023) · IEEE International Conference on Teaching, Assessment, and Learning for Engineering · doi:10.1109/tale56641.2023.10398341 · Cited by 15
MindLLM: Lightweight large language model pre-training, evaluation and domain application
Yang, Y., Sun, H., Li, J., Liu, R., Li, Y., Liu, Y., Gao, Y., & Huang, H. (2024) · AI Open · doi:10.1016/j.aiopen.2024.08.001 · Cited by 18
Self-citationFrom Triumph to Uncertainty: The Journey of Software Engineering in the AI Era
Mastropaolo, A., Escobar‐Velásquez, C., & Linares‐Vásquez, M. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709360 · Cited by 30
VRank: Enhancing Verilog Code Generation from Large Language Models via Self-Consistency
Zhao, Z., Qiu, R., Lin, I.-C., Zhang, G. L., Li, B., & Schlichtmann, U. (2025) · International Symposium on Quality Electronic Design (ISQED) · doi:10.1109/isqed65160.2025.11014398 · Cited by 17
Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decoding
Süzgün, M., Melas-Kyriazi, L., & Jurafsky, D. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.262 · Cited by 59
Miceli Barone, A. V., Barez, F., Cohen, S. B., & Konstas, I. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.19 · Cited by 12
Verified Code Transpilation with LLMs
Bhatia, S., Qiu, J., Hasabnis, N., Seshia, S., & Cheung, A. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1310 · Cited by 12
A Survey on Data Selection for Language Models
Albalak, A., Elazar, Y., Xie, S. M., Longpre, S., Lambert, N., Wang, X., Muennighoff, N., Hou, B., Pan, L., Jeong, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.16827 · Cited by 12
$π_0$: A Vision-Language-Action Flow Model for General Robot Control
Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.24164 · Cited by 13
“While such models have been shown to exhibit broad instruction-following and problem-solving abilities [53, 27], they are not truly situated in a physical world the way that people are, and their understanding of physical interaction is based entirely on abstract descriptions.”
SGLang: Efficient Execution of Structured Language Model Programs
Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.07104 · Cited by 145
“In addition, it is common in LLM programs to generate multiple outputs from a single prompt or to fork a new prompt from the current state [25].”
Revolutionizing Healthcare with Generative Artificial Intelligence Technologies
Rashidieranjbar, F., Farhadi, A., & Zamanifar, A. (2025) · Information systems engineering and management · doi:10.1007/978-3-031-76710-4_10 · Cited by 21
Shvetsova, O. A., Katalshov, D., & Lee, S.-K. (2025) · Applied Sciences · doi:10.3390/app15137298 · Cited by 26
Luo, Q., Lin, Q., Xu, L., Wu, S., Mao, R., Wang, C., Feng, H., Huang, B., & Du, Z. (2026) · Big Earth Data · doi:10.1080/20964471.2026.2615511 · Cited by 18
Diverse title generation for Stack Overflow posts with multiple-sampling-enhanced transformer
Zhang, F., Liu, J., Wan, Y., Yu, X., Liu, X., & Keung, J. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111672 · Cited by 14
Towards Integrating Emerging AI Applications in SE Education
Vierhauser, M., Groher, I., Antensteiner, T., & Sauerwein, C. (2024) · IEEE International Conference on Software Engineering Education and Training (CSEE&T) · doi:10.1109/cseet62301.2024.10663045 · Cited by 30
Language-Driven Policy Distillation for Cooperative Driving in Multi-Agent Reinforcement Learning
Liu, J., Xu, C., Hang, P., Sun, J., Ding, M., Zhan, W., & Tomizuka, M. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3551098 · Cited by 22
An empirical study on capability of Large Language Models in understanding code semantics
Nguyen, T.-T., Vu, T. T., Vo, H. D., & Nguyen, S. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107780 · Cited by 33
Quality In, Quality Out: Investigating Training Data's Role in AI Code Generation
Improta, C., Tufano, R., Liguori, P., Cotroneo, D., & Bavota, G. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00056 · Cited by 21
Lessons from the trenches on evaluating machine learning systems in materials science
Alampara, N., Schilling-Wilhelmi, M., & Jablonka, K. M. (2025) · Computational Materials Science · doi:10.1016/j.commatsci.2025.114041 · Cited by 22
MAO: A Framework for Process Model Generation With Multi-Agent Orchestration
Lin, L., Jin, Y., Zhou, Y., Chen, W., & Qian, C. (2025) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2025.3611816 · Cited by 34
Aligning Offline Metrics and Human Judgments of Value for Code Generation Models
Dibia, V. C., Fourney, A., Bansal, G., Poursabzi-Sangdeh, F., Liu, H., & Amershi, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.540 · Cited by 37
Zheng, D., Wang, Y., Shi, E., Zhang, R., Ma, Y., Zhang, H., & Zheng, Z. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) · doi:10.1109/icse55347.2025.00228 · Cited by 11
Sifting through the Chaff: On Utilizing Execution Feedback for Ranking the Generated Code Candidates
Sun, Z., Wan, Y., Li, J., Zhang, H., Jin, Z., Li, G., & Lyu, C. (2024) · Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3691620.3695000 · Cited by 10
A Preference-Driven Methodology for Efficient Code Generation
Li, Y., Zhou, Z., Peng, Z., Dong, J., You, H., Yan, R., Wen, S., Tian, Y., & Huang, T. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3639527 · Cited by 36
Self-citationPandoGen: Generating complete instances of future SARS-CoV-2 sequences using Deep Learning
Ramachandran, A., Lumetta, S. S., & Chen, D. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011790 · Cited by 15
A Comparison of Large Language Models and Genetic Programming for Program Synthesis
Sobania, D., Petke, J., Briesch, M., & Rothlauf, F. (2024) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2024.3410873 · Cited by 13
CoachLM: Automatic Instruction Revisions Improve the Data Quality in LLM Instruction Tuning
Liu, Y., Tao, S., Zhao, X., Zhu, M. L., Ma, W., Zhu, J., Su, C., Hou, Y., Zhang, M., Zhang, M., et al. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00390 · Cited by 46
Multimodal Deep Representation Learning for Quantum Cross-Platform Verification
Qian, Y., Du, Y., He, Z., Hsieh, M.-H., & Tao, D. (2024) · Physical Review Letters · doi:10.1103/physrevlett.133.130601 · Cited by 24
Wang, Q., & Li, H. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9030050 · Cited by 17
Reindex-Then-Adapt: Improving Large Language Models for Conversational Recommendation
He, Z., Xie, Z., Steck, H., Liang, D., Jha, R., Kallus, N., & McAuley, J. (2025) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3701551.3703573 · Cited by 36
Kohn, T. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70043 · Cited by 15
LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generation
Ma, Z., Gong, Y., Guo, H., Chen, J., Ma, Y., Cao, Z., & Zhang, J. (2026) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2026.3656374 · Cited by 78
Better Context Makes Better Code Language Models: A Case Study on Function Call Argument Completion
Pei, H., Zhao, J., Lausen, L., Zha, S., & Karypis, G. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i4.25653 · Cited by 28
Chatgpt With Risks And Opportunities
Kabak, T., & Kırbaş, İ. (2023) · Mehmet Akif Ersoy Üniversitesi Fen Bilimleri Enstitüsü Dergisi · doi:10.29048/makufebed.1271477 · Cited by 10
Enhancing Code Language Models for Program Repair by Curricular Fine-tuning Framework
Hao, S., Shi, X., Liu, H., & Shu, Y. (2023) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58846.2023.00024 · Cited by 19
Private-library-oriented code generation with large language models
Zan, D., Chen, B., Gong, Y., Cao, J., Zhang, F., Wu, B., Guan, B., Yin, Y., & Wang, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113934 · Cited by 48
Tao, N., Ventresque, A., & Saber, T. (2022) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec55065.2022.9870312 · Cited by 10
Reasoning in Large Language Models Through Symbolic Math Word Problems
Gaur, V., & Saunshi, N. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.364 · Cited by 50
Challenges and Paths Towards AI for Software EngineeringPreprint
Gu, A., Jain, N., Li, W., Shetty, M., Shao, Y., Li, Z., Yang, D., Ellis, K., Sen, K., & Solar-Lezama, A. (2025) · Qeios · doi:10.32388/vv1661 · Cited by 50
Scaling Instructable Agents Across Many Simulated Worlds
SIMA Team, Raad, M. A., Ahuja, A., Barros, C., Besse, F., Bolt, A., Bolton, A., Brownfield, B., Buttimore, G., Cant, M., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.10179 · Cited by 9
“Modern AI can write computer programs (Li et al., 2022) or play chess at super-human level (Silver et al., 2018), but the ability of AI to perceive and act in the world remains far below human level.”
Evaluation of a Visual Question Answering Architecture for Pedestrian Attribute Recognition
Castrillón-Santana, M., Sánchez‐Nielsen, E., Freire-Obregón, D., Santana, O. J., Hernández-Sosa, D., & Lorenzo-Navarro, J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44237-7_2 · Cited by 9
Hybrid API Migration: A Marriage of Small API Mapping Models and Large Language Models
Zhou, B., Wang, X., Xu, S., Yao, Y., Pan, M., Xu, F., & Ma, X. (2023) · Asia-Pacific Symposium on Internetware · doi:10.1145/3609437.3609466 · Cited by 18
Automated discovery of algorithms from data
Blazek, P. J., Venkatesh, K., & Lin, M. M. (2024) · Nature Computational Science · doi:10.1038/s43588-024-00593-9 · Cited by 14
A search-and-fill strategy to code generation for complex software requirements
Dong, Y., Kong, L., Zhang, L., Wang, S., Liu, X., Liu, S., & Chen, M.-C. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107584 · Cited by 11
User Centric Evaluation of Code Generation Tools (Invited Paper)
Miah, T., & Zhu, H. (2024) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest62860.2024.00022 · Cited by 25
MAGECODE: Machine-Generated Code Detection Method Using Large Language Models
Pham, H. N., Ha, H., Tong, V., Hoang, D., Tran, D., & Le, N. T. (2024) · IEEE Access · doi:10.1109/access.2024.3509987 · Cited by 18
LLMSecCode: Evaluating Large Language Models for Secure Coding
Rydén, A., Näslund, E. B., Schiller, E. M., & Almgren, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-76934-4_7 · Cited by 21
LLM4TDG: test-driven generation of large language models based on enhanced constraint reasoning
Liu, J., Liang, R., Zhu, X., Zhang, Y., Liu, Y., & Liu, Q. (2025) · Cybersecurity · doi:10.1186/s42400-024-00335-4 · Cited by 20
Scaling Llama 3 Training with Efficient Parallelism Strategies
Chu, W., Xie, X., Yu, J., Wang, J., Phanishayee, A., Tang, C., Hao, Y., Huang, J., Özdal, M. M., Wang, J., et al. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731410 · Cited by 29
Metadata
- Authors
- Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., Hubert, T., Choy, P., de Masson d’Autume, C., Babuschkin, I., Chen, X., Huang, P.-S., Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D. J., Sutherland Robson, E., Kohli, P., de Freitas, N., Kavukcuoglu, K., & Vinyals, O.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- DeepMind, London, UK.
- Corresponding author
- Li, Yujia
- Choi, David
- Chung, Junyoung
- Kushman, Nate
- Schrittwieser, Julian
- Leblond, Rémi
- Eccles, Tom
- Keeling, James
- Gimeno, Felix
- Choy, Peter
- Vinyals, Oriol
- ORCID
- Venue
- Science
- Published
- 2022-12-09
- Volume
- 378
- Issue
- 6624
- Pages
- 1092-1097
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W4226287673
- Cited by
- 1,043
- References
- 34
- Citations by year
- 2026: 269 · 2025: 319 · 2024: 237 · 2023: 186 · 2022: 30
- FWCI
- 191.10
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- automatic code generation, competitive programming, programming competition, Codeforces, transformer language models, AI-assisted programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.