Competition-level code generation with AlphaCode
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Abstract
Programming is a powerful and ubiquitous problem-solving tool. Systems that can assist programmers or even generate programs themselves could make programming more productive and accessible. Recent transformer-based neural network models show impressive code generation abilities yet still perform poorly on more complex tasks requiring problem-solving skills, such as competitive programming problems. Here, we introduce AlphaCode, a system for code generation that achieved an average ranking in the top 54.3% in simulated evaluations on recent programming competitions on the Codeforces platform. AlphaCode solves problems by generating millions of diverse programs using specially trained transformer-based networks and then filtering and clustering those programs to a maximum of just 10 submissions. This result marks the first time an artificial intelligence system has performed competitively in programming competitions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AlphaCode, Self-Debugging and CodeRL are effective in fixing less-complex program bugs, but they rely upon ground truth test cases that invalidate pass@1 eligibility, and do not use self-reflection to bridge the gap between error identification and implementation improvement.”
From Reflexion: Language Agents with Verbal Reinforcement Learning (2023) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large language models are increasingly capable of complex programming tasks and are already being used to accelerate AI research, from generating training data to serving as programming tools [1, 2, 3, 4, 5, 6].”
From RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts (2024) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Generative AI, such as OpenAI’s ChatGPT, has rapidly emerged as a disruptive technology capable of achieving human-level performance on a broad range of tasks (1–5).”
From Generative AI without guardrails can harm learning: Evidence from high school mathematics (2025) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Li et al. [27] showed that the accuracy of AlphaCode increases from close to 0% zero-shot to over 15% with 1,000 retries and over 30% with a million retries (accuracy is measured by how often one of the top 10 answers generated by the model is correct).”
From AI Agents That Matter (2024) · cites this paper as Li et al. [27]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Evaluation covered 200 CodeContests tasks [15] across three seeds and up to three feedback rounds, with online tests supplying feedback and hidden tests measuring final success.”
From RSI-Anything: Unlocking Recursive Self-Improvement with World Models (2026) · cites this paper as [15]
Experimental — accuracy not yet measured
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Self-citationDecoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“Following Kaplan et al. (2020), we adjusted the amount of training for other model sizes such that larger models are trained more and smaller models are trained less to optimize the use of compute.”
MethodGrandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455 · cited 2×
“We believe that sampling metadata leads to a more diverse set of model samples, a strategy similar to that used by Vinyals et al. (2019), and allows our model to take advantage of the relative strengths across the metadata distribution.”
MethodSelf-citationResnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., et al. (2009) · Communications of the ACM · doi:10.1145/1592761.1592779 · Cited by 3,333
Kudo, T., & Richardson, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d18-2012 · Cited by 2,869
Research works printed but not linked: 6
- A. Church, Application of recursive arithmetic to the problem of circuit synthesis.. J. Symb. Log. 28, 289–290 (1963).
- Y. Nandwani D. Jindal Mausam P. Singla ?oNeural learning of one-of-many solutions for combinatorial problems in structured output spaces ?? International Conference on Learning Representations (ICLR 2021) 3 to 7 May 2021.
- A. Holtzman J. Buys L. Du M. Forbes Y. Choi ?oThe curious case of neural text degeneration ?? Proceedings of the 8th International Conference on Learning Representations (ICLR 2020) 26 April to 1 May 2020.
- J. Ganitkevitch B. Van Durme C. Callison-Burch ?oPPDB: The paraphrase database ?? Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Atlanta Georgia 9 to 15 June 2013 (Association for Computational Linguistics 2013) pp. 758–764.
- A. Solar-Lezama ?oProgram synthesis by sketching ?? thesis University of California Berkeley (2008).
- J. Cai R. Shin D. Song ?oMaking neural programming architectures generalize via recursion ?? Proceedings of the 5th International Conference on Learning Representations (ICLR) Toulon France 24 to 26 April 2017.
Cited by
Items 501–600 of 1,104
Koohestani, R., de Bekker, P., Koç, B., & Izadi, M. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3644183 · Cited by 16
AdaCoder: An Adaptive Planning and Multi-Agent Framework for Function-Level Code Generation
Zhu, Y., Liu, C., He, X., Ren, X., Liu, Z., Pan, R., & Zhang, H. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3642621 · Cited by 19
An Overview of Recent Advances in Natural Language Processing for Information Systems
O’Shaughnessy, D. D. (2026) · Applied Sciences · doi:10.3390/app16021122 · Cited by 14
PALM: Synergizing Program Analysis and LLMs to Enhance Rust Unit Test Coverage
Chu, B., Feng, Y., Liu, K., Shi, H., Nan, Z., Guo, Z., & Xu, B. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00223 · Cited by 12
Assessing small language models for code generation: An empirical study with benchmarks
Hasan, M. M., Waseem, M., Kemell, K., Rasku, J., Ala-Rantala, J., & Abrahamsson, P. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112815 · Cited by 31
Discovering quantum circuit components with program synthesis
Sarra, L., Ellis, K., & Marquardt, F. (2024) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ad4252 · Cited by 14
Dumitran, A. M., Badea, A. C., & Muscalu, S.-G. (2024) · International Conference on Innovations in Intelligent Systems and Applications (INISTA) · doi:10.1109/inista62901.2024.10683837 · Cited by 7
AACEGEN: Attention Guided Adversarial Code Example Generation for Deep Code Models
Li, Z., Zhang, C., Pan, M., Zhang, T., & Li, X. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695500 · Cited by 16
Qu, Y., Huang, S., & Nie, P. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107707 · Cited by 18
SCodeSearcher: soft contrastive learning for code search
Li, J., Zheng, F., Xianjie, S., Jin, Z. G., Liu, F., Li, J., Zhao, Y., & Li, G. (2025) · Empirical Software Engineering · doi:10.1007/s10664-024-10603-z · Cited by 6
Evaluating Large Language Models for Code Generation: A Comparative Study on Python, Java, and Swift
Wan, B. (2025) · Applied and Computational Engineering · doi:10.54254/2755-2721/2025.tj22242 · Cited by 5
DesignRepair: Dual-Stream Design Guideline-Aware Frontend Repair with Large Language Models
Yuan, M., Chen, J., Xing, Z., Quigley, A. J., Luo, Y., Luo, T., Mohammadi, G., Lu, Q., & Zhu, L. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00109 · Cited by 25
A Deep Learning Framework for Verilog Autocompletion Towards Design and Verification Automation
Dehaerne, E., Dey, B., Halder, S., & Gendt, S. D. (2025) · IEEE International System-on-Chip Conference (SOCC) · doi:10.1109/socc66126.2025.11235413 · Cited by 21
Hierarchical Chain-of-Thought and Mixture-of-Experts for Efficient Code Generation
Huang, T. (2025) · International Conference on Artificial Intelligence and Intelligent Information Processing (AIIIP) · doi:10.1145/3778534.3778592 · Cited by 12
AI and atoms: How artificial intelligence is revolutionizing nuclear material production
He, J., & Degtyarev, N. (2023) · Bulletin of the Atomic Scientists · doi:10.1080/00963402.2023.2245251 · Cited by 6
Measuring Efficient Code Generation with GEC
Pan, Y., & Lyu, C. (2023) · Asia-Pacific Symposium on Internetware · doi:10.1145/3609437.3609459 · Cited by 10
Efficient Text-to-Code Retrieval with Cascaded Fast and Slow Transformer Models
Gotmare, A. D., Li, J., Joty, S., & Hoi, S. C. H. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616369 · Cited by 8
Training AI Model that Suggests Python Code from Student Requests in Natural Language
Kuramitsu, K., Obara, M., Sato, M., & Akinobu, Y. (2024) · Journal of Information Processing · doi:10.2197/ipsjjip.32.69 · Cited by 8
Oracle-Guided Program Selection from Large Language Models
Fan, Z., Ruan, H., Mechtaev, S., & Roychoudhury, A. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680308 · Cited by 24
Symbolic Execution with Test Cases Generated by Large Language Models
Xu, J., Xu, J., Chen, T., & Ma, X. (2024) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs62785.2024.00031 · Cited by 10
Yue, S. (2024) · Innovations in Systems and Software Engineering · doi:10.1007/s11334-024-00591-0 · Cited by 12
Robustness evaluation of code generation systems via concretizing instructions
Yan, M. X., Chen, J., Zhang, J. M., Cao, X., Yang, C., & Harman, M. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107645 · Cited by 22
Auto-Logging: AI-centred Logging Instrumentation
Bogatinovski, J., & Kao, O. (2023) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1109/icse-nier58687.2023.00023 · Cited by 11
How much can ChatGPT really help computational biologists in programming?
Rahman, C. R., & Wong, L. (2024) · Journal of Bioinformatics and Computational Biology · doi:10.1142/s021972002471001x · Cited by 5
Follow-Up Attention: An Empirical Study of Developer and Neural Model Code Exploration
Paltenghi, M., Pandita, R., Henley, A. Z., & Ziegler, A. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3445338 · Cited by 15
Siegel, Z. S., Kapoor, S., Nadgir, N., Stroebl, B., & Narayanan, A. (2024) · arXiv · doi:10.48550/arxiv.2409.11363 · Cited by 4
“Past work has shown retrying or increasing temperature between retries can be enough to drastically improve performance (Kapoor et al., 2024; Hassid et al., 2024; Brown et al., 2024; Li et al., 2022).”
LIMO: Less is More for Reasoning
Ye, Y., Huang, Z., Xiao, Y., Chern, E., Xia, S., & Liu, P. (2025) · arXiv · doi:10.48550/arxiv.2502.03387 · Cited by 627
“This can be achieved by augmenting LLMs with methods such as parallel sampling (Brown et al., 2024; Wang et al., 2022; Li et al., 2022) or symbolic tree search (Hao et al., 2023; Chen et al., 2024; Yao et al., 2023) to enhance reasoning ability.”
Aggarwal, P., Parno, B., & Welleck, S. (2024) · arXiv · doi:10.48550/arxiv.2412.06176 · Cited by 4
“…(LLMs) Chen et al. (2021); Austin et al. (2021); Li et al. (2023); Rozi`ere et al. (2024); Team (2024), leading to LLM-powered agents that can perform tasks ranging from fixing bugs in software repositories to solving Olympiad-level algorithmic problems Jimenez et al. (2023); Li et al. (2022a).”
AI-driven income inequality and preferences for redistribution
Veselý, Š., & Amaris, G. (2025) · Economic Analysis and Policy · doi:10.1016/j.eap.2025.06.019 · Cited by 7
Evaluating the Test Adequacy of Benchmarks for LLMs on Code Generation
Liu, X., Sun, X., Bo, L., Hu, Y., Liu, X., & Ye, Z. (2025) · Journal of Software Evolution and Process · doi:10.1002/smr.70034 · Cited by 9
Advancing LLM-Generated Code Reliability: A Hybrid Approach for Hallucination Detection
Yang, B., Dang, J., Liu, H., & Jin, Z. Y. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3640641 · Cited by 9
Measuring and mitigating debugging effectiveness decay in code language models
Adnan, M., & Kuhn, C. C. N. (2025) · Scientific Reports · doi:10.1038/s41598-025-27846-5 · Cited by 15
Li, S., Cheng, Y., Chen, J., Xuan, J., He, S., & Shang, W. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10776-1 · Cited by 12
QuanBench: Benchmarking Quantum Code Generation with Large Language Models
Guo, X., Wang, M., & Zhao, J. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00218 · Cited by 36
Gambo, O. O., Onifade, O. J., Olanrewaju-Smart, W., Abuya, O. J., Salawu, R. E., Massenon, R., & Gambo, I. (2026) · Discover Education · doi:10.1007/s44217-026-01142-4 · Cited by 14
Föhr, T. L., Schreyer, M., Moffitt, K. C., & Marten, K.-U. (2026) · International Journal of Accounting Information Systems · doi:10.1016/j.accinf.2025.100758 · Cited by 16
Luu, N. T., Luu, D. T., Pham, N. N., & Truong, T. C. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.3554 · Cited by 12
Composing Smart Data Services in Shop Floors Through Large Language Models
Mathew, J. G., Monti, F., Firmani, D., Leotta, F., Mandreoli, F., & Mecella, M. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0808-9_21 · Cited by 7
RefleXGen:The unexamined code is not worth using
Wang, B., Li, H., Liu, A., Yang, B., Yang, A., Zhong, Y., Huang, W., Huang, R., Zeng, W., & Zhang, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890824 · Cited by 6
Cao, D., Hong, Y., Pan, Q., & Wu, J. (2025) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2025.3581213 · Cited by 6
Fu, Y., Li, B., Li, L., Zhang, W., & Xie, T. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728947 · Cited by 14
Thematic Analysis of Expert Opinions on the Use of Large Language Models in Software Development
Yadav, S., Kaushik, A., & Qureshi, A. M. (2025) · Applied AI Letters · doi:10.1002/ail2.127 · Cited by 7
Evaluating LLMs for Multimodal GUI Test Generation in Android Applications
Fagundes, N., & Teixeira, L. (2025) · Simpósio Brasileiro de Testes de Software Sistemático e Automatizado (SAST) · doi:10.5753/sast.2025.13852 · Cited by 5
On Simulation-Guided LLM-based Code Generation for Safe Autonomous Driving Software
Nouri, A., Andersson, J., Hornig, K. D. J., Fei, Z., Knabe, E., Sivencrona, H., Cabrero‐Daniel, B., & Berger, C. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3756681.3756987 · Cited by 15
E2ETune: End-to-End Knob Tuning via Fine-Tuned Generative Language Model
Huang, X., Li, H., Zhang, J., Zhao, X., Yao, Z., Li, Y., Zhang, T., Chen, J., Chen, H., & Li, C. (2025) · Proceedings of the VLDB Endowment · doi:10.14778/3773731.3773732 · Cited by 39
Self-citationAn Extensive Study of the Structure Features in Transformer-based Code Semantic Summarization
Yang, K., Mao, X., Wang, S., Qin, Y., Zhang, T., Lu, Y., & Al‐Sabahi, K. (2023) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc58990.2023.00024 · Cited by 4
Generating and Reviewing Programming Codes with Large Language Models: A Systematic Mapping Study
de Albuquerque, B. V. L., da Cunha, A. F. S., Souza, L., Siqueira, S. W. M., & dos Santos, R. P. (2024) · Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI) · doi:10.1145/3658271.3658342 · Cited by 7
Large Language Models for SoC Security
Tehranipoor, M. M., Azar, K. Z., Asadizanjani, N., Rahman, F., Kamali, H. M., & Farahmandi, F. (2024) · Springer eBooks · doi:10.1007/978-3-031-58687-3_6 · Cited by 6
Towards Functional Verification of eBPF Programs
Lu, D., Tang, B., Paper, M., & Kogias, M. (2024) · doi:10.1145/3672197.3673435 · Cited by 5
Clustering-Driven State Embedding for Reinforcement Learning Under Visual Distractions
Wang, R., Cheng, Y., & Wang, X. (2024) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2024.3449294 · Cited by 6
Empirical Evaluation of Large Language Models for Novice Program Fault Localization
Liu, Y., Liu, H., Yang, Z., Li, Z., & Liu, Y. (2024) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs62785.2024.00027 · Cited by 13
Measuring code efficiency optimization capabilities with ACEOB
Pan, Y., Shao, X., & Lyu, C. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112250 · Cited by 7
Understanding Defects in Generated Codes by Language Models
Esfahani, A. M., Kahani, N., & Ajila, S. A. (2024) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1109/cascon62161.2024.10837857 · Cited by 11
Attributing ChatGPT-Generated Source Codes
Choi, S., & Mohaisen, A. (2025) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2025.3535218 · Cited by 12
QueryArtisan: Generating Data Manipulation Codes for Ad-hoc Analysis in Data Lakes
Xiu, T., Liu, W., Wu, S., Yao, C., Yuan, G., Ying, S., & Chen, G. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3705829.3705832 · Cited by 5
Borghoff, U. M., Minas, M., & Schopp, J. (2025) · European Conference on Software Engineering Education (ECSEE) · doi:10.1145/3723010.3723012 · Cited by 10
Gupta, R. D., Hosain, M. T., Mridha, M. F., & Ahmed, S. U. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-93864-1_5 · Cited by 10
Fagbohun, O., Yashwanth, S., Akintola, A. S., Wurola, I., Shittu, L., Inyang, A., Odubola, O., Offia, U. I., Olanrewaju, S., Toluwaleke, O., et al. (2025) · International Conference on Machine Learning Technologies (ICMLT) · doi:10.1109/icmlt65785.2025.11193211 · Cited by 5
Cheng, J., Liu, F., Wu, C., & Zhang, L. (2025) · Asia-Pacific Symposium on Internetware · doi:10.1145/3755881.3755925 · Cited by 9
XTest: A Parallel Multilingual Corpus with Test Cases for Code Translation and Its Evaluation*
Rithy, I. J., Shakil, H. H., Mondal, N., Sultana, F., & Shah, F. M. (2022) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit57492.2022.10055851 · Cited by 8
Code classification with graph neural networks: Have you ever struggled to make it work?
Yu, Q., Liu, X., Zhou, Q., Zhuge, J., & Wu, C. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120978 · Cited by 7
Python Code Generation by Asking Clarification Questions
Li, H.-S., Mesgar, M., Martins, A. F. T., & Gurevych, I. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.799 · Cited by 35
Self supervised learning and the poverty of the stimulus
Veres, C., & Sampson, J. (2023) · Data & Knowledge Engineering · doi:10.1016/j.datak.2023.102208 · Cited by 5
Flows: Building Blocks of Reasoning and Collaborating AIPreprint
Josifoski, M., Klein, L., Peyrard, M., Baldwin, N., Li, Y., Geng, S., Schnitzler, J. P., Yao, Y., Wei, J., Paul, D., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.01285 · Cited by 41
Self-citationGPT-4 vs. GPT-3.5 AS CODING ASSISTANTSPreprint
Moussiades, L., Zografos, G., & Papakostas, G. A. (2024) · Research Square · doi:10.21203/rs.3.rs-3920214/v1 · Cited by 5
AIDE: AI-Driven Exploration in the Space of Code
Jiang, Z., Schmidt, D., Srikanth, D., Xu, D., Kaplan, I., Jacenko, D., & Wu, Y. (2025) · arXiv · doi:10.48550/arxiv.2502.13138 · Cited by 4
“In contrast, specialized agents like Voyager (Wang et al., 2023) and AlphaCode (Li and et al., 2022) are tailored to specific domains such as embodied reasoning and competitive code generation.”
A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems
Ke, Z., Jiao, F., Ming, Y., Nguyen, X.-P., Xu, A., Long, D. X., Li, M., Qin, C., Wang, P., Savarese, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2504.09037 · Cited by 4
SecRepoBench: Benchmarking Code Agents for Secure Code Completion in Real-World Repositories
Shen, C., Dilgren, C., Chiniya, P., Griffith, L., Ding, Y., & Chen, Y. (2026) · Proceedings of the 3rd International Workshop on Large Language Models For Code · doi:10.1145/3786181.3788703 · Cited by 4
SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement
Antoniades, A., Örwall, A., Zhang, K., Xie, Y., Goyal, A., & Wang, W. (2024) · arXiv · doi:10.48550/arxiv.2410.20285 · Cited by 3
“While machine learning has made progress in automating parts of this workflow (Li et al., 2022; OpenAI et al., 2024; Ouyang et al., 2022; Yang et al., 2024b), replicating the adaptive and strategic behavior of human engineers remains a significant challenge.”
Meta-designing quantum experiments with language models
Arlt, S., Duan, H., Li, F., Xie, S. M., Wu, Y., & Krenn, M. (2026) · Nature Machine Intelligence · doi:10.1038/s42256-025-01153-0 · Cited by 30
Metamorphic Testing of Deep Code Models: A Systematic Literature Review
Asgari, A., de Koning, M., Derakhshanfar, P., & Panichella, A. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3766552 · Cited by 10
Is Quantization a Deal-Breaker? Empirical Insights From Large Code Models
Afrin, S., Xu, B., & Mastropaolo, A. (2025) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme64153.2025.00049 · Cited by 18
Esmaeili, A., Saberi, I., & Fard, F. H. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10740-z · Cited by 6
Steer Your Model: Secure Code Generation With Contrastive Decoding
Huang, L., Yan, M. L., Yin, T., Sun, W., Liu, Z., Zhang, H., & Lo, D. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3650127 · Cited by 10
Banitaan, S., Daoud, M., Alquran, H., & Akour, M. A. (2026) · Information · doi:10.3390/info17010073 · Cited by 8
Automated Repair of Ambiguous Problem Descriptions for LLM-Based Code Generation
Jia, H., Morris, R., Ye, H., Sarro, F., & Mechtaev, S. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00038 · Cited by 33
Yan, F., Liu, J., Guo, L. H., Liu, L., & Geng, X. (2026) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2026.01.016 · Cited by 5
Few-shot molecular property optimization via a domain-specialized large language model
Guo, Y., Luo, M., Zhang, W., Liu, P., Liu, J., Huang, S., Lv, J., Ke, B., & Liu, X. (2026) · Chemical Science · doi:10.1039/d5sc08859c · Cited by 7
Are LLMs reliable code reviewers? systematic overcorrection in requirement conformance judgement
Jin, H., & Chen, H. (2026) · Automated Software Engineering · doi:10.1007/s10515-026-00638-5 · Cited by 16
A Balance of Power: Exploring the Opportunities and Challenges of AI for a Nation
Yu, S., & Carroll, F. (2023) · Advanced sciences and technologies for security applications · doi:10.1007/978-3-031-40118-3_2 · Cited by 6
Eberhardinger, M., Maucher, J., & Maghsudi, S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment · doi:10.1609/aiide.v19i1.27516 · Cited by 10
Exploring the Potential of Pre-Trained Language Models of Code for Automated Program Repair
Hao, S., Shi, X., & Liu, H. (2024) · Electronics · doi:10.3390/electronics13071200 · Cited by 9
Fine-Tuning LLMs for Code Mutation: A New Era of Cyber Threats
Setak, M., & Madani, P. (2024) · IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA) · doi:10.1109/tps-isa62245.2024.00043 · Cited by 12
Borghoff, U. M., Minas, M., & Mönch, K. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-81596-6_4 · Cited by 4
Vito, G. D., D’Amici, D., Izzo, F., Ferrucci, F., & Nucci, D. D. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00024 · Cited by 8
AI-Assisted Autoformalization of Combinatorics Problems in Proof Assistants
Doan, L., & Nguyen, T. H. (2025) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1109/icse-nier66352.2025.00006 · Cited by 4
Lee, H., An, G., & Yoo, S. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00020 · Cited by 12
Nnadi, L. C., Ding, D., Muepu, D. M., Amin, M. F. I., & Kabir, R. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-8892-0_7 · Cited by 5
AlphaCode and “data-driven” programming
Kolter, J. Z. (2022) · Science · doi:10.1126/science.add8258 · Cited by 3
Pseudocode to Code Based on Adaptive Global and Local Information
Yu, Q., Huang, Z., & Gu, N. (2023) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner56733.2023.00016 · Cited by 5
Survey of intelligent program synthesis techniques
Zhou, B., & Ding, G. (2023) · International Conference on Algorithms, High Performance Computing and Artificial Intelligence (AHPCAI) · doi:10.1117/12.3011627 · Cited by 3
Chen, N., Sun, Q., Wang, J., Gao, M. Y., Li, X., & Li, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.991 · Cited by 17
A Pilot Study in Surveying Data Challenges of Automatic Software Engineering Tasks
Dong, L., Lu, Q., & Zhu, L. (2024) · doi:10.1145/3663530.3665020 · Cited by 4
Code Generation Using Self-Interactive Assistant
Zhao, Z., Sun, J., Cai, C., & Wei, Z. (2024) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00377 · Cited by 6
Are Large Language Models a Threat to Programming Platforms? An Exploratory Study
Billah, M. M., Roy, P. R., Codabux, Z., & Roy, B. (2024) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3674805.3686689 · Cited by 13
B4: Towards Optimal Assessment of Plausible Code Solutions with Plausible Tests
Chen, M., Liu, Z., Tao, H., Hong, Y., Lo, D., Xia, X., & Sun, J. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695536 · Cited by 21
E-code: Mastering efficient code generation through pretrained models and expert encoder group
Pan, Y., Lyu, C., Yang, Z., Li, L., Liu, Q., & Shao, X. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107602 · Cited by 8
Utomo, M. S., Utami, E., Kusrini, K., & Setyanto, A. (2024) · International Conference on Information Technology and Computing (ICITCOM) · doi:10.1109/icitcom62788.2024.10762291 · Cited by 5
AdvFusion: Adapter-based Knowledge Transfer for Code Summarization on Code Language Models
Saberi, I., Esmaeili, A., Fard, F. H., & Chen, F. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00059 · Cited by 16
Using Large Language Models for Aerospace Code Generation: Methods, Benchmarks, and Potential Values
He, R., Zhang, L., Lyu, M., Lyu, L., & Xue, C. (2025) · Aerospace · doi:10.3390/aerospace12060498 · Cited by 9
Metadata
- Authors
- Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., Hubert, T., Choy, P., de Masson d’Autume, C., Babuschkin, I., Chen, X., Huang, P.-S., Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D. J., Sutherland Robson, E., Kohli, P., de Freitas, N., Kavukcuoglu, K., & Vinyals, O.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- DeepMind, London, UK.
- Corresponding author
- Li, Yujia
- Choi, David
- Chung, Junyoung
- Kushman, Nate
- Schrittwieser, Julian
- Leblond, Rémi
- Eccles, Tom
- Keeling, James
- Gimeno, Felix
- Choy, Peter
- Vinyals, Oriol
- ORCID
- Venue
- Science
- Published
- 2022-12-09
- Volume
- 378
- Issue
- 6624
- Pages
- 1092-1097
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W4226287673
- Cited by
- 1,043
- References
- 34
- Citations by year
- 2026: 269 · 2025: 319 · 2024: 237 · 2023: 186 · 2022: 30
- FWCI
- 191.10
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- automatic code generation, competitive programming, programming competition, Codeforces, transformer language models, AI-assisted programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.