Competition-level code generation with AlphaCode
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Abstract
Programming is a powerful and ubiquitous problem-solving tool. Systems that can assist programmers or even generate programs themselves could make programming more productive and accessible. Recent transformer-based neural network models show impressive code generation abilities yet still perform poorly on more complex tasks requiring problem-solving skills, such as competitive programming problems. Here, we introduce AlphaCode, a system for code generation that achieved an average ranking in the top 54.3% in simulated evaluations on recent programming competitions on the Codeforces platform. AlphaCode solves problems by generating millions of diverse programs using specially trained transformer-based networks and then filtering and clustering those programs to a maximum of just 10 submissions. This result marks the first time an artificial intelligence system has performed competitively in programming competitions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AlphaCode, Self-Debugging and CodeRL are effective in fixing less-complex program bugs, but they rely upon ground truth test cases that invalidate pass@1 eligibility, and do not use self-reflection to bridge the gap between error identification and implementation improvement.”
From Reflexion: Language Agents with Verbal Reinforcement Learning (2023) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large language models are increasingly capable of complex programming tasks and are already being used to accelerate AI research, from generating training data to serving as programming tools [1, 2, 3, 4, 5, 6].”
From RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts (2024) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Generative AI, such as OpenAI’s ChatGPT, has rapidly emerged as a disruptive technology capable of achieving human-level performance on a broad range of tasks (1–5).”
From Generative AI without guardrails can harm learning: Evidence from high school mathematics (2025) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Li et al. [27] showed that the accuracy of AlphaCode increases from close to 0% zero-shot to over 15% with 1,000 retries and over 30% with a million retries (accuracy is measured by how often one of the top 10 answers generated by the model is correct).”
From AI Agents That Matter (2024) · cites this paper as Li et al. [27]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Evaluation covered 200 CodeContests tasks [15] across three seeds and up to three feedback rounds, with online tests supplying feedback and hidden tests measuring final success.”
From RSI-Anything: Unlocking Recursive Self-Improvement with World Models (2026) · cites this paper as [15]
Experimental — accuracy not yet measured
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Self-citationDecoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“Following Kaplan et al. (2020), we adjusted the amount of training for other model sizes such that larger models are trained more and smaller models are trained less to optimize the use of compute.”
MethodGrandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455 · cited 2×
“We believe that sampling metadata leads to a more diverse set of model samples, a strategy similar to that used by Vinyals et al. (2019), and allows our model to take advantage of the relative strengths across the metadata distribution.”
MethodSelf-citationResnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., et al. (2009) · Communications of the ACM · doi:10.1145/1592761.1592779 · Cited by 3,333
Kudo, T., & Richardson, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d18-2012 · Cited by 2,869
Research works printed but not linked: 6
- A. Church, Application of recursive arithmetic to the problem of circuit synthesis.. J. Symb. Log. 28, 289–290 (1963).
- Y. Nandwani D. Jindal Mausam P. Singla ?oNeural learning of one-of-many solutions for combinatorial problems in structured output spaces ?? International Conference on Learning Representations (ICLR 2021) 3 to 7 May 2021.
- A. Holtzman J. Buys L. Du M. Forbes Y. Choi ?oThe curious case of neural text degeneration ?? Proceedings of the 8th International Conference on Learning Representations (ICLR 2020) 26 April to 1 May 2020.
- J. Ganitkevitch B. Van Durme C. Callison-Burch ?oPPDB: The paraphrase database ?? Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Atlanta Georgia 9 to 15 June 2013 (Association for Computational Linguistics 2013) pp. 758–764.
- A. Solar-Lezama ?oProgram synthesis by sketching ?? thesis University of California Berkeley (2008).
- J. Cai R. Shin D. Song ?oMaking neural programming architectures generalize via recursion ?? Proceedings of the 5th International Conference on Learning Representations (ICLR) Toulon France 24 to 26 April 2017.
Cited by
Items 601–700 of 1,104
Enhancing Code Generation via Bidirectional Comment-Level Mutual Grounding
Di, Y., & Zhang, T. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00165 · Cited by 11
Föhr, T. L., Schreyer, M., Moffitt, K. C., & Marten, K.-U. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4488271 · Cited by 3
An Empirical Study on Automatically Detecting AI-Generated Source Code: How Far Are We?Preprint
Suh, H., Tafreshipour, M., Li, J., Bhattiprolu, A., & Ahmed, I. (2025) · Qeios · doi:10.32388/sbl97o · Cited by 5
Assessing Small Language Models for Code Generation: An Empirical Study with BenchmarksPreprint
Hasan, M. M., Waseem, M., Kemell, K., Rasku, J., Ala-Rantala, J., & Abrahamsson, P. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5472927 · Cited by 3
Floridi, L. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00968-4 · Cited by 9
Enhancing Code Generation via Bidirectional Comment-Level Mutual Grounding
Di, Y., & Zhang, T. (2025) · arXiv · doi:10.48550/arxiv.2505.07768 · Cited by 3
Seed-Coder: Let the Code Model Curate Data for Itself
Seed, B., Zhang, Y., Su, J., Sun, Y., Xi, C., Xiao, X., Zheng, S., Zhang, A., Liu, K., Zan, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.03524 · Cited by 3
“We collected challenging real-world coding problems, including CodeContests [Li et al., 2022] and ICPC problems 2 , and gathered model-generated solutions from DeepSeek-R1 [DeepSeek-AI et al., 2025] on these tasks.”
From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Models
Welleck, S., Bertsch, A., Finlayson, M., Schoelkopf, H., Xie, A., Neubig, G., Kulikov, I., & Harchaoui, Z. (2024) · arXiv · doi:10.48550/arxiv.2406.16838 · Cited by 3
Wang, Z., Chen, J., Wei, Z., Lin, L., & Tong, G. (2026) · Sensors · doi:10.3390/s26113502 · Cited by 2
“%) metric is adopted as the gold standard for code functional correctness, following the definition introduced together with HumanEval [15], which has since been used as the de facto functional metric in subsequent code LLM studies [16,17].”
Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation
Zelikman, E., Lorch, E., Mackey, L., & Kalai, A. T. (2023) · arXiv · doi:10.48550/arxiv.2310.02304 · Cited by 4
RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts
Wijk, H., Lin, T., Becker, J., Jawhar, S., Parikh, N., Broadley, T., Chan, L., Chen, M., Clymer, J., Dhyani, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.15114 · Cited by 187
Code sharing and artificial intelligence can help decolonise public health modelling
Hooda, Y., & Saha, S. (2024) · BMJ · doi:10.1136/bmj.q27 · Cited by 2
Chang, H., & Lin, C. (2024) · Journal of Computer Assisted Learning · doi:10.1111/jcal.13003 · Cited by 4
A Method for Processing Static Analysis Alarms Based on Deep Learning
Tan, Y., & Tian, J. (2024) · Applied Sciences · doi:10.3390/app14135542 · Cited by 4
PPM: Automated Generation of Diverse Programming Problems for Benchmarking Code Generation Models
Chen, S. H., Feng, X., Han, X., Liu, C., & Yang, W. M. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3643780 · Cited by 19
NoviCode: Generating Programs from Natural Language Utterances by Novices
Mordechai, A. A., Goldberg, Y., & Tsarfaty, R. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00694 · Cited by 12
Protecting the Whisper: A Security Assessment of Amazon CodeWhisperer’s Generated Code
Araujo, G. F., & Hayajneh, T. S. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-85856-7_36 · Cited by 3
Themes of Building LLM-Based Applications for Production: A Practitioner's View
Mailach, A., Simon, S., Dorn, J., & Siegmund, N. (2025) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain66642.2025.00011 · Cited by 15
Mitigating data bias and ensuring reliable evaluation of AI models with shortcut hull learning
Zhou, W., Liu, F., Zheng, H., & Zhao, R. J. (2025) · Nature Communications · doi:10.1038/s41467-025-60801-6 · Cited by 13
Filho, L. C. P., de Souza, T. D. P. C., & de Paula, L. B. (2025) · Revista Brasileira de Informática na Educação · doi:10.5753/rbie.2025.4477 · Cited by 3
Chang, H.-S., Peng, N., Bansal, M., Ramakrishna, A., & Chung, T. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00757 · Cited by 29
Kokkinakis, I. W., & Drikakis, D. (2025) · IEEE Access · doi:10.1109/access.2025.3607265 · Cited by 4
Regression in Earth Observation: Are vision–language models up to the challenge?
Xue, X., & Zhu, X. X. (2025) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2025.3596243 · Cited by 10
Data Service Composition in Cyber-Physical Systems Adopting LLMs
Izzi, A., Mathew, J. G., Monti, F., Firmani, D., Leotta, F., Mandreoli, F., & Mecella, M. (2025) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws67624.2025.00085 · Cited by 4
AGON: Automated Design Framework for Customizing Processors From ISA Documents
Li, C., Huang, D. S., Jin, P., Ma, T., Han, H., Cheng, S., Hao, Y., Zhao, Y., Xu, G., Du, Z., et al. (2025) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2025.3617851 · Cited by 2
Let’s Take Esoteric Programming Languages Seriously
Singer, J., & DRAPER, S. W. (2025) · ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) · doi:10.1145/3759429.3762632 · Cited by 3
CASP: An Evaluation Dataset for Formal Verification of C Code
Hertzberg, N., Sevenhuijsen, M., Kåreborn, L., & Lokrantz, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-07132-3_5 · Cited by 2
A Federated Adaptive Large Language Model Fine-Tuning Framework for Software Development
Chen, J., Cai, Z., Chen, W., Wang, W., Zheng, Z., & Yu, P. S. (2025) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2025.3623626 · Cited by 2
Athena: A Plug-and-Play Advisor for Retrieval-Augmented Generation using VectorDB
Liang, N., Wenz, F., Giceva, J., & Wills, L. W. (2025) · IEEE International Symposium on Workload Characterization · doi:10.1109/iiswc66894.2025.00013 · Cited by 4
Evaluating and improving LLM-based competitive program generation
Wei, M., Li, Z., Chen, X., Zheng, M., Qu, Z., Yu, C., Chen, S. C. J., & Ju, X. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107977 · Cited by 8
Self-citationLee, K., Yoon, S., & Lim, H. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.130633 · Cited by 6
AgentTune: An Agent-Based Large Language Model Framework for Database Knob Tuning
Li, Y., Li, H., Zhang, J., Borovica‐Gajic, R., Wang, S., Zhang, T., Chen, J., Shi, R., Li, C., & Chen, H. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3769758 · Cited by 14
Self-citationCertificates in AI: Learn but Verify
Barrett, C., Henzinger, T. A., & Seshia, S. A. (2025) · Communications of the ACM · doi:10.1145/3737447 · Cited by 9
Navas-Loro, M., Takeda, H., & Satoh, K. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.131031 · Cited by 2
Instruction-Following Evaluation of Large Vision-Language Models
Shiono, D., Miyawaki, S., Tanaka, R., & Suzuki, J. (2026) · New Generation Computing · doi:10.1007/s00354-025-00308-2 · Cited by 4
Aldosari, S. S., & Aldawsari, L. S. (2026) · Science of Computer Programming · doi:10.1016/j.scico.2026.103446 · Cited by 6
DLBench: A Comprehensive Benchmark for SQL Translation with Large Language Models
Lin, L., Chen, H., Zhu, Q., Chen, L., Tang, L., & Wu, R. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00076 · Cited by 4
PrefGen: A Preference-Driven Methodology for Secure Yet Gas-Efficient Smart Contract Generation
Peng, Z., Yin, X., Zhou, Z., Ying, C., Ni, C., & Luo, Y. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00034 · Cited by 10
ODEL: An Experience-Augmented Self-Evolving Framework for Efficient Python-to-C++ Code Translation
Feng, K., Peng, F., & Wu, J. (2026) · Applied Sciences · doi:10.3390/app16031506 · Cited by 2
Semantic Mastery: Enhancing LLMs with Advanced Natural Language Understanding
Hariharan, M. (2026) · IEEE International Conference on AI in Cybersecurity (ICAIC) · doi:10.1109/icaic67076.2026.11395740 · Cited by 4
Hierarchical Syntax-Aware Modeling for Code Generation and Program Repair
Qiu, C. (2025) · International Conference on Artificial Intelligence, Robotics, and Communication (ICAIRC) · doi:10.1109/icairc68035.2025.11385172 · Cited by 4
Automatically Engineering Trusted Software: A Research Roadmap
Brun, Y., Chakraborty, S., Goues, C. L., Păsăreanu, C. S., & Singla, A. K. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3779132 · Cited by 5
AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding
Li, Z., Chen, Z., Delacourt, R., Oliaro, G., Wang, Z., Chen, Q., LIN, S.-J., Yang, A., Zhang, Z., Chen, Z., et al. (2026) · European Conference on Computer Systems · doi:10.1145/3767295.3769315 · Cited by 50
A Transformer-Based Intrusion Detection System for Zero-Day Attack Detection in IoT Networks
Hssayeni, M. D., & Mahgoub, I. (2026) · Future Internet · doi:10.3390/fi18060282 · Cited by 7
Unveiling the power of code pre-trained models in automated program repair: A systematic review
Zhan, S., Wang, X. J., Wei, D., Xiang, X., Lin, Y., Wu, H., & Lu, J. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.113015 · Cited by 3
Li, J., Deng, H., Zhang, Y., Zhang, K., Shao, T., Zhao, T., Wang, W., Jin, Z. G., Li, G., Liu, Y., et al. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3797106 · Cited by 6
Xiang, W., Zhang, J., Zhang, P. C., Wang, M., Li, J., Li, C., Feng, L., & Pan, Z. (2026) · Advanced Engineering Informatics · doi:10.1016/j.aei.2026.104946 · Cited by 2
An efficient probabilistic hardware architecture for diffusion-like models
Jelinčič, A., Lockwood, O., Garlapati, A., Schillinger, P., Chuang, I. L., Verdon, G., & McCourt, T. (2026) · npj Unconventional Computing · doi:10.1038/s44335-026-00075-3 · Cited by 21
AI for science: Progress, challenges, and perspectives
Xu, Y., Shao, Z., Liu, X., Li, Y., An, Z., Fu, C., Huang, C., Zhang, L., Gao, F., Li, J., et al. (2026) · The Innovation · doi:10.1016/j.xinn.2026.101530 · Cited by 14
Self-citationCode Generation from Natural Language Using Two-Way Pre-Training
Wang, J., Peng, Y., Le, X., Chen, C., & Guan, X. (2023) · International Conference on Advanced Computational Intelligence (ICACI) · doi:10.1109/icaci58115.2023.10146131 · Cited by 3
Review of ways to apply machine learning methods in software engineering
Asaad, J., & Avksentieva, E. Y. (2023) · E3S Web of Conferences · doi:10.1051/e3sconf/202344907018 · Cited by 3
Improving Transformer-based Program Repair Model through False Behavior Diagnosis
Kim, Y., Kim, M., & Lee, E. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.865 · Cited by 2
Information Visualization with ChatGPT
Stöckl, A. (2024) · Studies in computational intelligence · doi:10.1007/978-3-031-46549-9_17 · Cited by 2
SwapVid: Integrating Video Viewing and Document Exploration with Direct Manipulation
Murakami, T., Fujita, K., Hara, K., Takashima, K., & Kitamura, Y. (2024) · International Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642515 · Cited by 5
DiffCoder: Enhancing Large Language Model on API Invocation via Analogical Code Exercises
Zan, D., Yu, A., Shen, B., Chen, B. C. J., Li, W., Gong, Y., Chen, X., Yao, Y., Luo, W., Guan, B., et al. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3643745 · Cited by 6
Self-citationRetrospective Analysis of Google Gemini
Yadav, N. S., Goar, V. K., Yadav, P. S., Rajpurohit, J., & Kotecha, K. V. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-97-6106-7_35 · Cited by 6
The Powerful AI: An Exploration of Generative Artificial Intelligence Taxonomy and Applications
Vadhil, F. A., Nanne, M. F., & Salihi, M. L. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-71429-0_18 · Cited by 3
Pattern memory cannot be completely and truly realized in deep neural networks
Li, T., Lyu, R., & Xie, Z. (2024) · Scientific Reports · doi:10.1038/s41598-024-80647-0 · Cited by 2
Generating highly customizable python code for data processing with large language models
Trummer, I. (2025) · The VLDB Journal · doi:10.1007/s00778-025-00900-4 · Cited by 3
Tree Variational Autoencoder for Code
Liventsev, V., de Bruin, S., Härmä, A., & Petković, M. (2025) · IEEE Access · doi:10.1109/access.2025.3532522 · Cited by 2
GPT-Based Automated Induction: Vulnerability Detection in Medical Software
Deng, L., Lei, H., Khan, F., Srivastava, G., Chen, J., & Haque, M. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3544560 · Cited by 3
Mass Generation of Programming Learning Problems from Public Code Repositories
Александрович, С. О., & Shashkov, D. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9030057 · Cited by 8
LLM-based Interactive Code Generation: Empirical Evaluation
Shaikhelislamov, D., Drobyshevskiy, M., & Belevantsev, A. A. (2024) · Ivannikov ISPRAS Open Conference (ISPRAS) · doi:10.1109/ispras64596.2024.10899123 · Cited by 7
SpeedGen: Enhancing Code Efficiency through Large Language Model-Based Performance Optimization
Purschke, N., Kirchner, S., & Knoll, A. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00045 · Cited by 4
Breaking Barriers in Mobile Accessibility: A Study of LLM-Generated Native Android Interfaces
Rabelo, D. M. F., Martins, R. D. S., Santos, I., Silva, P. H. G. D., da Gama, K. S., & Viana, W. (2025) · IEEE/ACM International Conference on Mobile Software Engineering and Systems (MOBILESoft) · doi:10.1109/mobilesoft66462.2025.00010 · Cited by 10
Bessas, N., Tzanaki, E., Vavougios, D., & Plagianakos, V. P. (2025) · Social Sciences & Humanities Open · doi:10.1016/j.ssaho.2025.101610 · Cited by 7
AI support for data scientists: An empirical study on workflow and alternative code recommendations
Ramasamy, D., Sarasua, C., & Bernstein, A. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10622-4 · Cited by 3
Exploring Human and Language Model Alignment in Perceived Design Similarity Using Ordinal Embeddings
Keeler, M., Fuge, M., Peng, A., & Miller, S. R. (2025) · Journal of Mechanical Design · doi:10.1115/1.4069129 · Cited by 3
New Deep Learning-Based Approach for Source Code Generation: Application to Computer Vision Systems
Alshehri, W. S., Jarraya, S. K., & Allinjawi, A. (2025) · AI · doi:10.3390/ai6070162 · Cited by 2
Building an Intelligent Data Exploring Assistant for Geoscientists
Widlansky, M. J., & Komar, N. (2025) · Journal of Geophysical Research Machine Learning and Computation · doi:10.1029/2025jh000649 · Cited by 5
Generalized Adversarial Code-Suggestions: Exploiting Contexts of LLM-based Code-Completion
Rubel, K., Noppel, M., & Wressnegger, C. (2025) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3708821.3733870 · Cited by 3
Evaluation of the Code Generated By Large Language Models: The State of the Art
Ying, Z., Towey, D., & Zhang, Y. (2025) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac65507.2025.00064 · Cited by 3
Language Model Evolutionary Algorithms for Recommender Systems: Benchmarks and Algorithm Comparisons
Liu, J., Sun, Z., Feng, S., Chen, C., & Ong, Y.-S. (2025) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2025.3609058 · Cited by 8
An Empirical Study of Exploring the Capabilities of Large Language Models in Code Learning
Liu, S., Guo, D., Zhang, J., Ma, W., Li, Y., & Liu, Y. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3609876 · Cited by 6
Self-citationKnoll: Creating a Knowledge Ecosystem for Large Language Models
Zhao, D., Yang, D., & Bernstein, M. S. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747711 · Cited by 11
Transforming Code Development with Large Language Models
Kaplunovich, A. (2025) · International Conference on Intelligent Computing, Communication, Networking and Services (ICCNS) · doi:10.1109/iccns66249.2025.11428578 · Cited by 2
Program Synthesis Through Learning the Input-Output Behavior of Commands
Lee, S., Nam, S. Y., & Kim, J. (2022) · IEEE Access · doi:10.1109/access.2022.3183091 · Cited by 2
Kejriwal, M. (2022) · Future of business and finance · doi:10.1007/978-3-031-19039-1_3 · Cited by 4
Does ChatGPT Add Value to University Students’ Academic Performance?
Kaur, P., Jawaid, A., Qi, R. O. J., Rokman, R. D. B. C. M., Awai, S. S. A., Liew, S. S., & Yunus, S. N. B. M. (2023) · International Journal of Membrane Science and Technology · doi:10.15379/ijmst.v10i3.2583 · Cited by 3
Big Transformers for Code Generation
Arutyunov, G. A., & Avdoshin, S. M. (2022) · Proceedings of the Institute for System Programming of RAS · doi:10.15514/ispras-2022-34(4)-6 · Cited by 4
Iterative improvements from feedback for language modelsPreprint
Li, Y. (2023) · doi:10.14293/pr2199.000220.v1 · Cited by 5
Self-citationAI Outputs Are Not Protected SpeechPreprint
Salib, P. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4687558 · Cited by 4
The Human-AI Contracting ParadoxPreprint
Bastani, H., & Cachon, G. P. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5962739 · Cited by 5
KernelBench: Can LLMs Write Efficient GPU Kernels?
Ouyang, A., Guo, S., Arora, S., Zhang, A. L., Hu, W., Ré, C., & Mirhoseini, A. (2025) · arXiv · doi:10.48550/arxiv.2502.10517 · Cited by 2
“In the past year, there have been several efforts to build LMs that can automate algorithmic coding [5, 19, 31], resolving GitHub issues [43, 44], and domain-specific coding [17, 46].”
McEval: Massively Multilingual Code Evaluation
Chai, L., Liu, S., Yang, J., Yin, Y., Jin, K., Liu, J., Sun, T., Zhang, G., Ren, C., Guo, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.07436 · Cited by 73
“For the field of soft engineering, code LLMs (Feng et al., 2020; Chen et al., 2021; Scao et al., 2022; Li et al., 2022; Allal et al., 2023; Fried et al., 2022; Wang et al., 2021; Zheng et al., 2024; Guo et al., 2024) pre-trained on billions of code snippets, such as StarCoder (Li et al., 2023;…”
MethodA benchmark for vericoding: formally verified program synthesis
Bursuc, S., Ehrenborg, T., Lin, S., Astefanoaei, L., Chiosa, I. E., Kukovec, J., Singh, A., Butterley, O., Bizid, A., Dougherty, Q., et al. (2025) · arXiv · doi:10.48550/arxiv.2509.22908 · Cited by 1
R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents
Jain, N., Singh, J., Shetty, M., Zheng, L., Sen, K., & Stoica, I. (2025) · arXiv · doi:10.48550/arxiv.2504.07164 · Cited by 1
A Multi-Agent Framework for Accessible Web Code Generation
Aljedaani, W., Shaik, M. A., & Aljohani, A. (2026) · Proceedings of the 23rd International Web for All Conference · doi:10.1145/3800424.3800453 · Cited by 1
AbDPP: Target‐oriented antibody design with pretraining and prior biological structure knowledge
Yu, C., Lin, X., Cheng, Y., Xu, J., Wang, H., Yan, Y., Huang, Y., Liu, L., Zhao, W., Zhao, Q., et al. (2024) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.26676 · Cited by 1
Harnessing GNNs for Robust Representation Learning in High-Level Synthesis
Sohrabizadeh, A., Bai, Y., Sun, Y., & Cong, J. (2024) · IEEE transactions on circuits and systems for artificial intelligence. · doi:10.1109/tcasai.2024.3485522 · Cited by 1
De-fine: De composing and Re fin ing Visual Programs with Auto-Feedback
Gao, M., Li, J., Fei, H., Pang, L., Ji, W., Wang, G., Lv, Z., Zhang, W., Tang, S., & Zhuang, Y. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681082 · Cited by 14
Lee, M.-J., & Kim, H. (2024) · The Journal of Korean Association of Computer Education · doi:10.32431/kace.2024.27.9.005 · Cited by 1
Kátai, Z., & Iclănzan, D. (2025) · Acta Universitatis Sapientiae Informatica · doi:10.47745/ausi-2024-0014 · Cited by 3
Iterative Self-training for Code Generation via Reinforced Re-ranking
Sorokin, N., Sedykh, I., & Malykh, V. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88714-7_21 · Cited by 3
CoverNexus: Multi-agent LLM System for Automated Code Coverage Enhancement
Ba, T. N., Thanh, B. N., & Tran, V.-T. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-4291-5_38 · Cited by 4
Revisiting SWE-Bench: On the Importance of Data Quality for LLM-Based Code Models
Aleithan, R. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion66252.2025.00075 · Cited by 10
Code Ranking with Structure Awareness Contrastive Learning
Huang, H., Cao, L., Wang, J., Yu, T., & Cai, Y. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00053 · Cited by 1
ReproCopilot: LLM-Driven Failure Reproduction with Dynamic Refinement
Leesatapornwongsa, T., Faisal, F. E., & Nath, S. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729399 · Cited by 4
L’IA générative : une révolution technologique entre promesses et défis
Moulines, É. (2025) · Annales des Mines - Réalités industrielles · doi:10.3917/rindu1.252.0007 · Cited by 1
Chen, L., Ahmed, N. K., Capotă, M., Willke, T., Hasabnis, N., & Jannesari, A. (2025) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps64566.2025.00055 · Cited by 9
Metadata
- Authors
- Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., Hubert, T., Choy, P., de Masson d’Autume, C., Babuschkin, I., Chen, X., Huang, P.-S., Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D. J., Sutherland Robson, E., Kohli, P., de Freitas, N., Kavukcuoglu, K., & Vinyals, O.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- DeepMind, London, UK.
- Corresponding author
- Li, Yujia
- Choi, David
- Chung, Junyoung
- Kushman, Nate
- Schrittwieser, Julian
- Leblond, Rémi
- Eccles, Tom
- Keeling, James
- Gimeno, Felix
- Choy, Peter
- Vinyals, Oriol
- ORCID
- Venue
- Science
- Published
- 2022-12-09
- Volume
- 378
- Issue
- 6624
- Pages
- 1092-1097
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W4226287673
- Cited by
- 1,043
- References
- 34
- Citations by year
- 2026: 269 · 2025: 319 · 2024: 237 · 2023: 186 · 2022: 30
- FWCI
- 191.10
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- automatic code generation, competitive programming, programming competition, Codeforces, transformer language models, AI-assisted programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.