Competition-level code generation with AlphaCode
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Abstract
Programming is a powerful and ubiquitous problem-solving tool. Systems that can assist programmers or even generate programs themselves could make programming more productive and accessible. Recent transformer-based neural network models show impressive code generation abilities yet still perform poorly on more complex tasks requiring problem-solving skills, such as competitive programming problems. Here, we introduce AlphaCode, a system for code generation that achieved an average ranking in the top 54.3% in simulated evaluations on recent programming competitions on the Codeforces platform. AlphaCode solves problems by generating millions of diverse programs using specially trained transformer-based networks and then filtering and clustering those programs to a maximum of just 10 submissions. This result marks the first time an artificial intelligence system has performed competitively in programming competitions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AlphaCode, Self-Debugging and CodeRL are effective in fixing less-complex program bugs, but they rely upon ground truth test cases that invalidate pass@1 eligibility, and do not use self-reflection to bridge the gap between error identification and implementation improvement.”
From Reflexion: Language Agents with Verbal Reinforcement Learning (2023) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large language models are increasingly capable of complex programming tasks and are already being used to accelerate AI research, from generating training data to serving as programming tools [1, 2, 3, 4, 5, 6].”
From RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts (2024) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Generative AI, such as OpenAI’s ChatGPT, has rapidly emerged as a disruptive technology capable of achieving human-level performance on a broad range of tasks (1–5).”
From Generative AI without guardrails can harm learning: Evidence from high school mathematics (2025) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Li et al. [27] showed that the accuracy of AlphaCode increases from close to 0% zero-shot to over 15% with 1,000 retries and over 30% with a million retries (accuracy is measured by how often one of the top 10 answers generated by the model is correct).”
From AI Agents That Matter (2024) · cites this paper as Li et al. [27]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Evaluation covered 200 CodeContests tasks [15] across three seeds and up to three feedback rounds, with online tests supplying feedback and hidden tests measuring final success.”
From RSI-Anything: Unlocking Recursive Self-Improvement with World Models (2026) · cites this paper as [15]
Experimental — accuracy not yet measured
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Self-citationDecoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“Following Kaplan et al. (2020), we adjusted the amount of training for other model sizes such that larger models are trained more and smaller models are trained less to optimize the use of compute.”
MethodGrandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455 · cited 2×
“We believe that sampling metadata leads to a more diverse set of model samples, a strategy similar to that used by Vinyals et al. (2019), and allows our model to take advantage of the relative strengths across the metadata distribution.”
MethodSelf-citationResnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., et al. (2009) · Communications of the ACM · doi:10.1145/1592761.1592779 · Cited by 3,333
Kudo, T., & Richardson, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d18-2012 · Cited by 2,869
Research works printed but not linked: 6
- A. Church, Application of recursive arithmetic to the problem of circuit synthesis.. J. Symb. Log. 28, 289–290 (1963).
- Y. Nandwani D. Jindal Mausam P. Singla ?oNeural learning of one-of-many solutions for combinatorial problems in structured output spaces ?? International Conference on Learning Representations (ICLR 2021) 3 to 7 May 2021.
- A. Holtzman J. Buys L. Du M. Forbes Y. Choi ?oThe curious case of neural text degeneration ?? Proceedings of the 8th International Conference on Learning Representations (ICLR 2020) 26 April to 1 May 2020.
- J. Ganitkevitch B. Van Durme C. Callison-Burch ?oPPDB: The paraphrase database ?? Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Atlanta Georgia 9 to 15 June 2013 (Association for Computational Linguistics 2013) pp. 758–764.
- A. Solar-Lezama ?oProgram synthesis by sketching ?? thesis University of California Berkeley (2008).
- J. Cai R. Shin D. Song ?oMaking neural programming architectures generalize via recursion ?? Proceedings of the 5th International Conference on Learning Representations (ICLR) Toulon France 24 to 26 April 2017.
Cited by
Items 401–500 of 1,104
LLM Assistance for Memory Safety
Mohammed, N., Lal, A., Rastogi, A., Sharma, R., & Roy, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00023 · Cited by 22
An Empirical Study on Automatically Detecting AI-Generated Source Code: How Far are We?
Suh, H., Tafreshipour, M., Li, J., Bhattiprolu, A., & Ahmed, I. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00064 · Cited by 26
Legal Requirements Translation from Law
Singhal, A., & Breaux, T. D. (2025) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re63999.2025.00028 · Cited by 13
Making large language models reliable data science programming copilots for biomedical research
Wang, Z., Danek, B. P., Yang, Z., Chen, Z., & Sun, J. (2026) · Nature Biomedical Engineering · doi:10.1038/s41551-025-01587-2 · Cited by 25
On the effectiveness of pretrained models for API learning
Hadi, M. A., Yusuf, I. N. B., Thung, F., Luong, K., Jiang, L., Fard, F. H., & Lo, D. (2022) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3524610.3527886 · Cited by 18
Learning Interpretable Logic for Agent-Based Models from Domain Independent Primitives
Greig, R., Major, C., Pacholska, M., Bending, S., & Arranz, J. (2023) · Journal of Artificial Societies and Social Simulation · doi:10.18564/jasss.5087 · Cited by 11
Conversational Text-to-SQL: An Odyssey into State-of-the-Art and Challenges Ahead
Parthasarathi, S. H. K., Zeng, L., & Hakkani-Tur, D. Z. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096170 · Cited by 11
Continuous Error Timing in Automation: The Peak-End Effect on Human-Automation Trust
Wang, K., Lü, J., Ruan, S., & Qi, Y. (2023) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2023.2223954 · Cited by 13
INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformers
Karmakar, A., & Robbes, R. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3341624 · Cited by 15
Análise das Respostas do ChatGPT em Relação ao Conteúdo de Programação para Iniciantes
Filho, L. C. P., de Souza, T. D. P. C., & de Paula, L. B. (2023) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/sbie.2023.234870 · Cited by 10
Code Detection for Hardware Acceleration Using Large Language Models
Martínez, P. A., Bernabé, G., & García, J. M. (2024) · IEEE Access · doi:10.1109/access.2024.3372853 · Cited by 12
PwnPilot: Reflections on Trusting Trust in the Age of Large Language Models and AI Code Assistants
Horne, D. (2023) · Congress in Computer Science, Computer Engineering, and Applied Computing (CSCE) · doi:10.1109/csce60160.2023.00396 · Cited by 15
Chen, L., Liu, H., Jia, N., Ren, N., Cui, R., & Wei, W. (2024) · International Journal of Production Research · doi:10.1080/00207543.2024.2361441 · Cited by 10
Generating Java code pairing with ChatGPT
Zhao, Z., Zhang, N., Yu, B., & Duan, Z. (2024) · Theoretical Computer Science · doi:10.1016/j.tcs.2024.114879 · Cited by 11
do Nascimento, N. M., Alencar, P., & Cowan, D. D. (2023) · Research Square · doi:10.21203/rs.3.rs-3126005/v1 · Cited by 18
The Robots are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.00658 · Cited by 9
Gemini-the most powerful LLM: Myth or TruthPreprint
Ahmed, I., & Islam, R. (2024) · doi:10.36227/techrxiv.171177477.70151414/v1 · Cited by 9
Case study: using AI-assisted code generation in mobile teams
Vasiliniuc, M.-S., & Groza, A. P. (2023) · IEEE International Conference on Intelligent Computer Communication and Processing (ICCP) · doi:10.1109/iccp60212.2023.10398656 · Cited by 24
Guo, J., Zhao, S., Huang, B., Wang, H., He, Y., Zhang, C., Zhang, C., & Shao, T. (2024) · IEEE Transactions on Plasma Science · doi:10.1109/tps.2024.3382320 · Cited by 8
Su, J., Deng, L., Wen, C., Qin, S., & Tian, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-64626-3_22 · Cited by 13
FDI: Attack Neural Code Generation Systems through User Feedback Channel
Sun, Z., Du, X., Luo, X., Song, F., Lo, D., & Li, L. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680300 · Cited by 10
Practical and Ethical Considerations for Generative AI in Medical Imaging
Jha, D., Rauniyar, A., Hagos, D. H., Sharma, V., Tomar, N. K., Zhang, Z., Isler, I., Durak, G., Wallace, M. B., Yazıcı, C., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72787-0_17 · Cited by 8
Low-cost language models: Survey and performance evaluation on Python code generation
Espejel, J. L., Alassan, M. S. Y., Bouhandi, M., Dahhane, W., & Ettifouri, E. H. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109490 · Cited by 19
Code LLMs: A Taxonomy-based Survey
Raihan, N., Newman, C. D., & Zampieri, M. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10826108 · Cited by 15
A Conjecture on a Fundamental Trade-Off Between Certainty and Scope in Symbolic and Generative AI
Floridi, L. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00927-z · Cited by 28
Wu, H., Shen, Z., Hou, S., Jiao, H., Liu, Z., Xie, L., Liang, J., Qing, Y., Zhang, X., Gui, Z., et al. (2025) · Big Earth Data · doi:10.1080/20964471.2025.2581425 · Cited by 10
DecoRTL: A Run-Time Decoding Framework for RTL Code Generation with LLMs
Akyash, M., Azar, K. Z., & Kamali, H. M. (2025) · doi:10.1109/iccad66269.2025.11240830 · Cited by 10
Code Generation From Flowcharts with Texts: A Benchmark Dataset and An Approach
Liu, Z., Hu, X., Zhou, D., Li, L., Zhang, X., & Xiang, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.449 · Cited by 21
Program Repair with Minimal Edits Using CodeT5
Shirafuji, A., Rahman, M. M., Amin, M. F. I., & Watanobe, Y. (2023) · International Conference on Awareness Science and Technology (iCAST) · doi:10.1109/icast57874.2023.10359288 · Cited by 15
Answering Uncertain, Under-Specified API Queries Assisted by Knowledge-Aware Human-AI Dialogue
Huang, Q. Y., Li, Z., Xing, Z., Zuo, Z., Peng, X., Xu, X., & Lu, Q. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3346954 · Cited by 9
Top Pass: improve code generation by pass@k-maximized code ranking
Lyu, Z., Li, X., Xie, Z., & Li, M. (2025) · Frontiers of Computer Science · doi:10.1007/s11704-024-40415-9 · Cited by 30
Honarvar, S., van der Wilk, M., & Donaldson, A. F. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10989005 · Cited by 25
Rosas, M. R., Sanchez, M. A. T., & Eigenmann, R. (2025) · doi:10.1145/3718350.3718357 · Cited by 26
Alexander, F., Abdiwijaya, E. A., Pherry, F., Gunawan, A. A. S., & Anderies (2022) · International Conference on Software Engineering and Information Technology (ICoSEIT) · doi:10.1109/icoseit55604.2022.10029949 · Cited by 9
The Covid-19 Pandemic Spurred Growth in Automation: What Does this Mean for Minority Workers?
Broady, K. E., Booth-Bell, D., Barr, A., & Perry, R. (2023) · doi:10.21033/wp-2023-06 · Cited by 10
MeshAgent: enabling reliable network management with large language models
Zhou, Y., Hsieh, K., Mani, S. K., Kandula, S., & Liu, Z. (2025) · Proceedings of the ACM on Measurement and Analysis of Computing Systems · doi:10.1145/3771567 · Cited by 7
Enhancing LLM Reasoning Capabilities Through Brokered Multi-Expert Reflection
Sheokand, T., Jain, G., Bahga, A., & Madisetti, V. K. (2025) · IEEE Access · doi:10.1109/access.2025.3561693 · Cited by 7
Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., & Narayanan, A. (2024) · arXiv · doi:10.48550/arxiv.2407.01502 · Cited by 13
“Li et al. [27] showed that the accuracy of AlphaCode increases from close to 0% zero-shot to over 15% with 1,000 retries and over 30% with a million retries (accuracy is measured by how often one of the top 10 answers generated by the model is correct).”
MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
Chan, J. S., Chowdhury, N., Jaffe, O., Aung, J., Sherburn, D., Mays, E., Starace, G., Liu, K., Maksin, L., Patwardhan, T., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.07095 · Cited by 7
CrossCodeBench: Benchmarking Cross-Task Generalization of Source Code Models
Niu, C., Li, C., Ng, V., & Luo, B. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00055 · Cited by 11
Softmax Bottleneck Makes Language Models Unable to Represent Multi-mode Word Distributions
Chang, H.-S., & McCallum, A. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.554 · Cited by 11
Priority Sampling of Large Language Models for Compilers
Grubisic, D., Seeker, V., Synnaeve, G., Leather, H., Mellor‐Crummey, J., & Cummins, C. (2024) · Workshop on Machine Learning and Systems (EuroMLSys) · doi:10.1145/3642970.3655831 · Cited by 19
TrickyBugs: A Dataset of Corner-case Bugs in Plausible Programs
Liu, K., Han, Y., Liu, Y., Chen, Z., Zhang, J. M., Sarro, F., Huang, G., & Ma, Y. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3644870 · Cited by 18
Attacks and Defenses for Large Language Models on Coding Tasks
Zhang, C., Wang, Z., Zhao, R., Mangal, R., Fredrikson, M., Jia, L., & Păsăreanu, C. S. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695297 · Cited by 31
Unearthing Semantic Checks for Cloud Infrastructure-as-Code Programs
Qiu, Y., Kon, P. T. J., Beckett, R., & Chen, A. (2024) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3694715.3695974 · Cited by 14
S2AF : An action framework to self-check the Understanding Self-Consistency of Large Language Models
Shao, H., Wang, F., & Xie, Z. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107365 · Cited by 10
Emerging Data Practices: Data Work in the Era of Large Language Models
Garcia, A. A., Candello, H. C. D. S. P., Badillo-Urquiola, K. A., & Wong-Villacrés, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714069 · Cited by 23
A Medical Multimodal Large Language Model for Pediatric Pneumonia
Tian, W., Huang, X., Cheng, T., He, W., Fang, J., Feng, R., Geng, D., & Zhang, X. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3569361 · Cited by 14
Unseen Horizons: Unveiling the Real Capability of LLM Code Generation Beyond the Familiar
Zhang, Y., Xie, Y., Lit, S., Liu, P., Wang, C., Jia, Z., Huang, X., Song, J., Luo, C., Zheng, Z., et al. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00082 · Cited by 29
LLM-Aided Automatic Modeling for Security Protocol Verification
Mao, Z., Wang, J., Sun, J., Qin, S., & Xiong, J. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00197 · Cited by 22
Accurate and Consistent Graph Model Generation from Text with Large Language Models
Chen, B., Ou, W., Zheng, B., & Mussbacher, G. (2025) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1109/models67397.2025.00018 · Cited by 19
Ding, G., Yang, S., Lin, H., Chen, Z., & Yang, J. C. (2026) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2026.3667549 · Cited by 25
Wang, X., Liu, X., Zhou, P., Liu, Q., Liu, J., Wu, H., & Cui, X. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3559549 · Cited by 13
Fine-Tuning Large Language Models for Answering Programming Questions with Code Snippets
Lomshakov, V., Kovalchuk, S. V., Omelchenko, M. O., Nikolenko, S., & Aliev, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-36021-3_15 · Cited by 9
Building Persona Consistent Dialogue Agents with Offline Reinforcement Learning
Shea, R., & Yu, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.110 · Cited by 36
AskIt: Unified Programming Interface for Programming with Large Language Models
Okuda, K., & Amarasinghe, S. P. (2024) · Proceedings of the CGO/Proceedings of the ... CGO · doi:10.1109/cgo57630.2024.10444830 · Cited by 21
Promoting open science in test-driven software experiments
Kessel, M., & Atkinson, C. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.111971 · Cited by 14
Unveiling the Potential of Large Language Models in Generating Semantic and Cross-Language Clones
Roy, P. R., Alam, A. I., Al-omari, F., Roy, B., Roy, C. K., & Schneider, K. A. (2023) · IEEE International Workshop on Software Clones (IWSC) · doi:10.1109/iwsc60764.2023.00011 · Cited by 17
Large Language Models for Software Engineering: A Systematic Mapping Study
Görmez, M. K., Yılmaz, M., & Clarke, P. M. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-71139-8_5 · Cited by 18
Fully Autonomous Programming Using Iterative Multi-Agent Debugging with Large Language Models
Grishina, A., Liventsev, V., Härmä, A., & Moonen, L. (2025) · ACM Transactions on Evolutionary Learning and Optimization · doi:10.1145/3719351 · Cited by 17
FoC: Figure Out the Cryptographic Functions in Stripped Binaries with LLMs
Shang, X., Chen, G., Cheng, S., Guo, S., Zhang, Y., Zhang, W., & Yu, N. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3731449 · Cited by 16
Jiang, X., Dong, Y., Tao, Y., Liu, H., Jin, Z. G., & Li, G. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00133 · Cited by 42
AI-Driven DevOps Automation for Cloud-Native Application Modernization
Mittal, A. (2025) · Springer eBooks · doi:10.1007/978-3-032-02853-2_9 · Cited by 11
Reinforcement learning for scaffold-free construction of spanning structures
Vallat, G., Wang, J., Maddux, A. M., Kamgarpour, M., & Parascho, S. (2023) · ACM Symposium on Computational Fabrication (SCF) · doi:10.1145/3623263.3623359 · Cited by 11
It’s MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Risk
Bertsch, A., Xie, A., Neubig, G., & Gormley, M. R. (2023) · doi:10.18653/v1/2023.bigpicture-1.9 · Cited by 62
Cognitive Apprenticeship and Artificial Intelligence Coding Assistants
Poitras, E., Crane, B. G. C., Dempsey, D., Bragg, T. A., Siegel, A. A., & Lin, M. P.-C. (2024) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-1066-3.ch013 · Cited by 8
RunBugRun: An executable dataset for automated program repair
Prenner, J. A., & Robbes, R. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10790-3 · Cited by 36
Šavelka, J., Agarwal, A., Bogart, C., Song, Y., & Sakr, M. F. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.09325 · Cited by 7
Imb-FinDiff: Conditional Diffusion Models for Class Imbalance Synthesis of Financial Tabular Data
Schreyer, M., Sattarov, T., Sim, A., & Wu, K. J. (2024) · ACM International Conference on AI in Finance (ICAIF) · doi:10.1145/3677052.3698659 · Cited by 13
Rakhilin, N., Morris, H. D., Pham, D. L., Hood, M. N., & Ho, V. B. (2025) · Bioengineering · doi:10.3390/bioengineering12050519 · Cited by 10
Code Red! On the Harmfulness of Applying Off-the-Shelf Large Language Models to Programming Tasks
Al-Kaswan, A., Deatc, S., Koç, B., van Deursen, A., & Izadi, M. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729380 · Cited by 11
Legal knowledge infusion for large language models: A survey
Liu, Q., Yu, H., Wang, Q., Xu, Q., Li, J., Zou, Z., Mao, R., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103426 · Cited by 15
CoCoEvo: Co-Evolution of Programs and Test Cases to Enhance Code Generation
Li, K., Yuan, Y., Yu, H., Guo, T., & Cao, S. (2025) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2025.3593272 · Cited by 27
Brender, J., El‐Hamamsy, L., Uittenhove, K. L., Mondada, F., & Bumbacher, E. W. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-03870-8_7 · Cited by 10
Li, C., Treude, C., & Turel, O. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112634 · Cited by 9
The Impact of AI on Academic Research and Publishing
Lund, B. D., Lamba, M., & Oh, S. H. (2025) · Springer Reference Sozialwissenschaften · doi:10.1007/978-3-658-08460-8_85-1 · Cited by 33
Kusper, G., & Szabó, C. (2025) · International Conference on Emerging eLearning Technologies and Applications (ICETA) · doi:10.1109/iceta67772.2025.11280285 · Cited by 18
Realistic Corner Case Generation for Autonomous Vehicles with Multimodal Large Language Model
Lu, Q., Ma, M., Wang, Z., Lu, W., Ma, M., Zhao, G., & Feng, S. (2025) · Tsinghua Science & Technology · doi:10.26599/tst.2025.9010178 · Cited by 33
CSLLM: Code-Specific Large Language Models—A Survey
Umre, J., Parihar, A. S., & Gupta, A. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.130991 · Cited by 10
Methods and Techniques of Agentic Software Engineering: A Systematic Literature Review
Otoum, N., & Elkhalili, N. (2026) · IEEE Access · doi:10.1109/access.2026.3652325 · Cited by 10
ContraCLM: Contrastive Learning For Causal Language Model
Jain, N., Zhang, D., Ahmad, W. U., Wang, Z., Nan, F., Li, X., Tan, M., Nallapati, R. M., Ray, B., Bhatia, P., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.355 · Cited by 25
On the Challenges of Using Large Language Models for NCL Code Generation
Moraes, D. D. S., da Costa, P. B., Busson, A. J. G., Boaro, J. M. C., Neto, C. D. S. S., & Colcher, S. (2023) · Brazilian Symposium on Multimedia and the Web · doi:10.5753/webmedia_estendido.2023.236175 · Cited by 10
Multimodal Retrieval Augmented Generation Evaluation Benchmark
Sun, T., Somalwar, A., & Chan, H. (2024) · IEEE Vehicular Technology Conference · doi:10.1109/vtc2024-spring62846.2024.10683437 · Cited by 12
AI-assisted Programming and AI Literacy in Computer Science Education
Tan, C. W., Khan, M. A. M., & Yu, P.-D. (2024) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-83608-852-320241020 · Cited by 7
NARRATE: Versatile Language Architecture for Optimal Control in Robotics
Ismail, S., Arbues, A., Cotterell, R., Zurbrügg, R., & Alonso, C. A. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10801425 · Cited by 16
Use of Transfer Learning for Affordable In-Context Fake Review Generation
Ibañez-Lissen, L., González‐Manzano, L., Fuentes, J. M. D., & Goyanes, M. (2025) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2025.3536927 · Cited by 8
VEGA: Automatically Generating Compiler Backends using a Pre-trained Transformer Model
Zhong, M., Lv, F., Wang, L., Qiu, L., Wang, Y., Liu, Y., Cui, H., Feng, X., & Xue, J. (2025) · Proceedings of the CGO/Proceedings of the ... CGO · doi:10.1145/3696443.3708931 · Cited by 15
3DGen: AI-Assisted Generation of Provably Correct Binary Format Parsers
Fakhoury, S., Kuppe, M. A., Lahiri, S. K., Ramananandro, T., & Swamy, N. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00173 · Cited by 19
Specializing Neural Networks for Cryptographic Code Completion Applications
Xiao, Y., Song, W., Qi, J., Viswanath, B., McDaniel, P. D., & Yao, D. D. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3265362 · Cited by 6
A Zero-Shot Language Agent for Computer Control with Structured Reflection
Li, T., Li, G., Deng, Z., Wang, B. K., & Li, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.753 · Cited by 44
Self-citationMorescient GAI for Software Engineering
Kessel, M., & Atkinson, C. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709354 · Cited by 11
Joint Generator-Ranker Learning for Natural Language Generation
Shen, W., Gong, Y., Shen, Y., Wang, S., Quan, X., Duan, N., & Chen, W. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.486 · Cited by 8
Solving Bongard Problems With a Visual Language and Pragmatic Constraints
Depeweg, S., Rothkopf, C. A., & Jäkel, F. (2024) · Cognitive Science · doi:10.1111/cogs.13432 · Cited by 43
Complex QA and language models hybrid architectures, SurveyPreprint
Daull, X., Bellot, P., Bruno, E., Martin, V., & Murisasco, É. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.09051 · Cited by 43
Public Attitudes Toward ChatGPT on Twitter: Sentiments, Topics, and Occupations
Koonchanok, R., Pan, Y., & Jang, H. (2023) · arXiv · doi:10.48550/arxiv.2306.12951 · Cited by 5
SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?
Miserendino, S., Wang, M., Patwardhan, T., & Heidecke, J. (2025) · arXiv · doi:10.48550/arxiv.2502.12115 · Cited by 154
“…et al., 2024), APPS, LiveCodeBench (Jain et al., 2024), NaturalCodeBench (Zhang et al., 2024), Big-CodeBench (Zhuo et al., 2024), MBPP, MathQA-Python (Austin et al., 2021), CodeXGlue (Lu et al., 2021), Al-phaCode (Li et al., 2022), CODEGEN (Nijkamp et al., 2023), and DS-1000 (Lai et al., 2022).”
Zhang, H., Maillo, A., Khan, S. A., Martinez‐de‐Morentin, X., Lehmann, R., Gómez-Cabrero, D., & Tegner, J. N. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.10.034 · Cited by 7
Deng, L., Zhong, Q., Song, J., Lei, H., & Li, W. (2025) · Sensors · doi:10.3390/s25092683 · Cited by 12
Anchor Attention, Small Cache: Code Generation With Large Language Models
Zhang, X., Zhou, Y., Yang, G., Gall, H. C., & Chen, T. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3570680 · Cited by 10
Zhang, X., Gong, Y., Zhong, Y.-T., Huang, T., & Zhang, J. (2025) · Information Sciences · doi:10.1016/j.ins.2025.122762 · Cited by 13
Metadata
- Authors
- Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., Hubert, T., Choy, P., de Masson d’Autume, C., Babuschkin, I., Chen, X., Huang, P.-S., Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D. J., Sutherland Robson, E., Kohli, P., de Freitas, N., Kavukcuoglu, K., & Vinyals, O.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- DeepMind, London, UK.
- Corresponding author
- Li, Yujia
- Choi, David
- Chung, Junyoung
- Kushman, Nate
- Schrittwieser, Julian
- Leblond, Rémi
- Eccles, Tom
- Keeling, James
- Gimeno, Felix
- Choy, Peter
- Vinyals, Oriol
- ORCID
- Venue
- Science
- Published
- 2022-12-09
- Volume
- 378
- Issue
- 6624
- Pages
- 1092-1097
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W4226287673
- Cited by
- 1,043
- References
- 34
- Citations by year
- 2026: 269 · 2025: 319 · 2024: 237 · 2023: 186 · 2022: 30
- FWCI
- 191.10
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- automatic code generation, competitive programming, programming competition, Codeforces, transformer language models, AI-assisted programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.