Competition-level code generation with AlphaCode
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Abstract
Programming is a powerful and ubiquitous problem-solving tool. Systems that can assist programmers or even generate programs themselves could make programming more productive and accessible. Recent transformer-based neural network models show impressive code generation abilities yet still perform poorly on more complex tasks requiring problem-solving skills, such as competitive programming problems. Here, we introduce AlphaCode, a system for code generation that achieved an average ranking in the top 54.3% in simulated evaluations on recent programming competitions on the Codeforces platform. AlphaCode solves problems by generating millions of diverse programs using specially trained transformer-based networks and then filtering and clustering those programs to a maximum of just 10 submissions. This result marks the first time an artificial intelligence system has performed competitively in programming competitions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AlphaCode, Self-Debugging and CodeRL are effective in fixing less-complex program bugs, but they rely upon ground truth test cases that invalidate pass@1 eligibility, and do not use self-reflection to bridge the gap between error identification and implementation improvement.”
From Reflexion: Language Agents with Verbal Reinforcement Learning (2023) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large language models are increasingly capable of complex programming tasks and are already being used to accelerate AI research, from generating training data to serving as programming tools [1, 2, 3, 4, 5, 6].”
From RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts (2024) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Generative AI, such as OpenAI’s ChatGPT, has rapidly emerged as a disruptive technology capable of achieving human-level performance on a broad range of tasks (1–5).”
From Generative AI without guardrails can harm learning: Evidence from high school mathematics (2025) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Li et al. [27] showed that the accuracy of AlphaCode increases from close to 0% zero-shot to over 15% with 1,000 retries and over 30% with a million retries (accuracy is measured by how often one of the top 10 answers generated by the model is correct).”
From AI Agents That Matter (2024) · cites this paper as Li et al. [27]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Evaluation covered 200 CodeContests tasks [15] across three seeds and up to three feedback rounds, with online tests supplying feedback and hidden tests measuring final success.”
From RSI-Anything: Unlocking Recursive Self-Improvement with World Models (2026) · cites this paper as [15]
Experimental — accuracy not yet measured
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Self-citationDecoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“Following Kaplan et al. (2020), we adjusted the amount of training for other model sizes such that larger models are trained more and smaller models are trained less to optimize the use of compute.”
MethodGrandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455 · cited 2×
“We believe that sampling metadata leads to a more diverse set of model samples, a strategy similar to that used by Vinyals et al. (2019), and allows our model to take advantage of the relative strengths across the metadata distribution.”
MethodSelf-citationResnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., et al. (2009) · Communications of the ACM · doi:10.1145/1592761.1592779 · Cited by 3,333
Kudo, T., & Richardson, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d18-2012 · Cited by 2,869
Research works printed but not linked: 6
- A. Church, Application of recursive arithmetic to the problem of circuit synthesis.. J. Symb. Log. 28, 289–290 (1963).
- Y. Nandwani D. Jindal Mausam P. Singla ?oNeural learning of one-of-many solutions for combinatorial problems in structured output spaces ?? International Conference on Learning Representations (ICLR 2021) 3 to 7 May 2021.
- A. Holtzman J. Buys L. Du M. Forbes Y. Choi ?oThe curious case of neural text degeneration ?? Proceedings of the 8th International Conference on Learning Representations (ICLR 2020) 26 April to 1 May 2020.
- J. Ganitkevitch B. Van Durme C. Callison-Burch ?oPPDB: The paraphrase database ?? Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Atlanta Georgia 9 to 15 June 2013 (Association for Computational Linguistics 2013) pp. 758–764.
- A. Solar-Lezama ?oProgram synthesis by sketching ?? thesis University of California Berkeley (2008).
- J. Cai R. Shin D. Song ?oMaking neural programming architectures generalize via recursion ?? Proceedings of the 5th International Conference on Learning Representations (ICLR) Toulon France 24 to 26 April 2017.
Cited by
Items 701–800 of 1,104
LLaVul: A Multimodal LLM for Interpretable Vulnerability Reasoning about Source Code
Jararweh, A., Adams, M., Sahu, A. D., Mueen, A., & Anwar, A. (2025) · Intelligent Cybersecurity Conference (ICSC) · doi:10.1109/icsc65596.2025.11140501 · Cited by 2
The “Question Neighbourhood” Approach for Systematic Evaluation of Code-Generating LLMs
Honarvar, S., Rei, M., & Donaldson, A. F. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3612251 · Cited by 6
Zero-Shot Detection of LLM-Generated Code via Approximated Task Conditioning
Ashkenazi, M., Brenner, O., Shohet, T. F., & Treister, E. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-06078-5_11 · Cited by 8
Nie, P., Huang, S., Zheng, C., & Gu, M. (2025) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs65678.2025.00072 · Cited by 1
Wu, S., Xiao, J., & Sang, J. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3620146 · Cited by 3
Mao, K., Hu, B., Lin, R., Li, Z., Lu, G., & Zhang, Z. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1660912 · Cited by 10
Generating Robotic Control Strategies With LLMs : Via Human–Robot Voice Interaction
Wang, X., Zhou, G., Hua, H., Tang, Z., & Cao, Y. (2025) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.70372 · Cited by 2
TaskEval: Assessing Difficulty of Code Generation Tasks for Large Language Models
Tambon, F., Nikanjam, A., Zid, C., Khomh, F., & Antoniol, G. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3773285 · Cited by 15
Riaz, A., Marzolla, M., Rossi, D., & Sinigardi, S. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228660 · Cited by 2
VFocus: Better Verilog Generation from Large Language Model via Focused Reasoning
Zhao, Z., Li, B., Zhang, G. L., & Schlichtmann, U. (2025) · IEEE International System-on-Chip Conference (SOCC) · doi:10.1109/socc66126.2025.11235465 · Cited by 1
Kim, J., Kim, S., Jeong, G., Lee, E., & Lee, J. (2025) · IEEE Access · doi:10.1109/access.2025.3638671 · Cited by 2
Fully automated processor chip design: motivation, challenges and future directions
Zhang, R., Guo, J., Cheng, S., & Chen, Y. (2025) · National Science Review · doi:10.1093/nsr/nwaf532 · Cited by 1
Outperforming Classical PI Tuning Methods via AlphaZero Algorithm
Queiroz, K. I. P. D. M., Dias, S. M., Oliveira, T. R., & Araújo, A. (2025) · Journal of Control Automation and Electrical Systems · doi:10.1007/s40313-025-01225-x · Cited by 1
Ma, H., Ji, Z., Al-Hashimy, T., Allen, A., Cen, N., Chen, O., Chen, Y., Chen, Y., Cheng, T., Gu, Y., et al. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2521062122 · Cited by 2
The Power of Small LLMs: A Multi-Agent for Code Generation via Dynamic Precaution Tuning
Zhang, J., Liao, J., Tang, J., & Zhao, X. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3632508 · Cited by 3
Reinforcement Learning vs Supervised Learning: A tug of war to generate refactored code accurately
Palit, I., & Sharma, T. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3756681.3756971 · Cited by 11
From Legal Text to Tech Specs: Generative AI’s Interpretation of Consent in Privacy Law
Kesari, A., Breaux, T. D., Santos, S., Norton, T. B., & Singhal, A. (2025) · International Conference on Artificial Intelligence and Law · doi:10.1145/3769126.3769241 · Cited by 4
Gnanakumar, D., Sainath Reddy, C., Naga Purna Sekhar, D., Hemanth Kumar, B., & Tarun Kumar Reddy, D. (2025) · International Conference on Automation, Computing and Renewable Systems (ICACRS) · doi:10.1109/icacrs67045.2025.11324363 · Cited by 1
The Accuracy of AI-Generated Code Towards Productivity
Akbar, M. R., & Anderies (2025) · International Conference on Converging Technology in Electrical and Information Engineering (ICCTEIE) · doi:10.1109/iccteie66144.2025.11341839 · Cited by 1
Performance of LLMS on Stochastic Modeling Operations Research Problems: From Theory to Practice
Kumar, A., Peng, T., Wu, Y., & Zeevi, A. J. (2025) · Winter Simulation Conference · doi:10.1109/wsc68292.2025.11339139 · Cited by 5
SemGuard: Real-Time Semantic Evaluator for Correcting LLM-Generated Code
Wang, Q., Sun, Z., Wang, R., Huang, T., Jin, Z., Li, G., & Lyu, C. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00160 · Cited by 4
RealisticCodeBench: Towards More Realistic Evaluation of Large Language Models for Code Generation
Yu, X., Chen, H., Liu, L., Hu, X., Keung, J., & Xia, X. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00248 · Cited by 3
Securing AI Code Generation - A Prompt Rectification Approach for Mitigating Cyber Risks
Dong, J., Ni, Z., Sun, N., Jha, S., Zhang, Y., Bertino, E., Nepal, S., & Ma, S. (2025) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom66490.2025.00106 · Cited by 2
The Impact of AI on Academic Research and Publishing
Lund, B. D., Lamba, M., & Oh, S. H. (2026) · Springer eBooks · doi:10.1007/978-3-658-44521-8_85 · Cited by 1
Rizqullah, M., & Albassam, E. (2026) · IEEE Access · doi:10.1109/access.2026.3662817 · Cited by 5
A Large-Scale Empirical Evaluation of LLMs for Automated Self-Admitted Technical Debt Repayment
Sheikhaei, M. S., Tian, Y., Wang, S., & Xu, B. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3796704 · Cited by 5
Fan, G., Wei, D., & Yin, B. (2026) · Electronics · doi:10.3390/electronics15061148 · Cited by 2
Research on FPGA Acceleration of Transformer Model for Modulation Recognition
Lan, J., Lin, Y., Zhuang, J., & Peng, S. (2025) · International Conference on Information Communication and Software Engineering (ICICSE) · doi:10.1109/icicse66971.2025.11430095 · Cited by 1
Many-objective hyper-heuristics: A state-of-the-art survey
Anwar, A. A., & Zhang, X. (2026) · Computer Science Review · doi:10.1016/j.cosrev.2026.100971 · Cited by 1
H2A-BPMN: A Hierarchical and Hybrid Agent Framework for Industrial BPMN Automated Modeling
Xu, Q., Zhang, Y., Tang, H., Ran, Q., Gao, H., & Xu, Y. (2026) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-032-21171-2_10 · Cited by 2
Lu, Z., Zhang, P., Nie, Y., Yang, Y., Tang, Y., Chong, C. Y., & Zhou, Y. (2026) · Proceedings of the ACM on Programming Languages · doi:10.1145/3798251 · Cited by 3
Joshi, O. S., Mehta, K., Kashyap, S., Purohit, A., & Das, S. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3798880 · Cited by 1
ArchAgent: Scalable Legacy Software Architecture Recovery with LLMs
Pan, R., Mao, B., Ma, T., & Ling, Z. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462552 · Cited by 7
Patel, T., Bayyavarapu, S. R. K. V., Soni, V. R., Purushothaman, R., Thokala, G. B., & Ranganathan, V. (2026) · International Conference on Advances in Artificial Intelligence and Machine Learning (AAIML) · doi:10.1109/aaiml67890.2026.11498111 · Cited by 1
KCoEvo: A Knowledge Graph Augmented Framework for Evolutionary Code Generation
Kang, J., Lu, Y., Jiang, C., Liu, J., Zhang, T., Jiang, B., Sun, N., Wu, T., & Qi, G. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-0378-9_30 · Cited by 3
Exploring the role of Large Language Models in High-Performance Computing programming: A survey
Ljaljevic, S., Jorba, J., & Iserte, S. (2026) · Future Generation Computer Systems · doi:10.1016/j.future.2026.108618 · Cited by 5
InteriorAgent: LLM Agent for Interior Design-Aware 3D Layout Generation
Gupta, K., Mehta, I., Wang, K., Chua, N., Krishna, A., Deng, Y., Ramamoorthi, R., & Chandraker, M. K. (2026) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv69130.2026.00138 · Cited by 3
LLM-Assisted Software Development in Aerospace Engineering Education: A Structured Workflow
Kafafy, R., Okasha, M., & Abdelkhalik, O. O. (2026) · International Conference on Artificial Intelligence, Systems, and Emerging Technologies (ICAISET) · doi:10.1109/icaiset66439.2026.11541825 · Cited by 1
Chen, S. T., & LIU, Y. F. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104957 · Cited by 2
Focus Session: Large Language Models in Physical Design: From Data Generation to Intelligent Agents
Wu, B.-Y., Dey, A., Rovinski, A., & Chhabria, V. A. (2026) · Design, Automation, and Test in Europe · doi:10.23919/date69613.2026.11539185 · Cited by 2
Saleem, A., Adnan, Z., Ryan, S. A., Jamıl, A., Hameed, A. A., & Fargetta, G. (2026) · IEEE International Conference on Secure IoT, Assured and Trusted Computing (SATC) · doi:10.1109/satc69565.2026.11542535 · Cited by 1
Correctness isn’t Efficiency: Runtime Memory Divergence in LLM-Generated Code
Rajput, P. K., Song, Y., Bonkoungou, A. A., Olatunji, I. E., Kaboré, A. K., Klein, J., & Bissyandé, T. F. (2026) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3786583.3786908 · Cited by 7
A Comprehensive Survey of Generative AI: Applications and Future Directions Across Domains
Javadpour, A., Ja'fari, F., Taleb, T., & Benzaïd, C. (2026) · IEEE Access · doi:10.1109/access.2026.3715238 · Cited by 2
Code for Machines, Not Just Humans: Quantifying AI-Friendliness with Code Health Metrics
Borg, M., Hagatulah, N., Tornhill, A., & Söderberg, E. (2026) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1145/3793655.3793722 · Cited by 10
A multi-agent framework for standards-aligned semantic validation of SysML v2 models
Olivert-Iserte, J., Cibrián, E., Mendieta, R., & Álvarez-Rodríguez, J. M. (2026) · Computer Standards & Interfaces · doi:10.1016/j.csi.2026.104191 · Cited by 2
Younes, Y., & Elsheikh, Y. (2026) · IEEE Jordan Conference on Applied Electrical Engineering and Computing Technologies · doi:10.1109/aeect69724.2026.11657848 · Cited by 1
Chen, L.-G., Wang, X., Chen, J.-Y., Liang, R.-Z., Zeng, Z., Li, Y.-N., Li, Y.-H., Wang, Y.-D., Yi-jiang, X., Gao, Q., et al. (2026) · Journal of Computer Science and Technology · doi:10.1007/s11390-025-5518-5 · Cited by 1
Self-citationXiang, J., Ye, T., Liu, P., Zhang, Y. Z., & Wang, W. (2026) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1145/3822455.3830313 · Cited by 1
From Code to Low-Code: An LLM-Driven Pipeline for Lifting Code Clones into Reusable Abstractions
Zefferer, R., Schenkenfelder, B., & Wagner, Ş. (2026) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1145/3837062.3839368 · Cited by 1
Deepfake-Agent: Aggregating Semantic Forgery Clues for Generalizable Detection
Guo, X., Zhang, Y., Bansal, M., & Liu, X. (2026) · 2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition- FINDINGS Track (CVPRF) · doi:10.1109/cvprf71909.2026.00471 · Cited by 4
Enhancing Code Similarity with Augmented Data Filtering and Ensemble Strategies
Kim, G., Kim, M., & Jo, J. (2022) · JOIV International Journal on Informatics Visualization · doi:10.30630/joiv.6.3.1259 · Cited by 2
Chattopadhyay, I. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.3086 · Cited by 1
Analysis on ChatGPT Technology Ethics Governance
Li, J. (2023) · International Journal of Frontiers in Sociology · doi:10.25236/ijfs.2023.050511 · Cited by 1
Research and Application of AI-Enabled Education
Wang, Z., Tian, Y., Chen, R., & Kong, L. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-5971-6_30 · Cited by 1
Assessing ChatGPT’s Proficiency in CS1-Level Problem Solving
Sánchez, M., & Herrera, A. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-47372-2_7 · Cited by 1
InstructExcel: A Benchmark for Natural Language Instruction in Excel
Payan, J., Mishra, S. R., Singh, M. P., Negreanu, C., Poelitz, C., Baral, C. R., Roy, S., Chakravarthy, R., Durme, B. V., & Nouri, E. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.265 · Cited by 13
Wang, R., Todd, G., Yuan, X., Xiao, Z., Côté, M.-A., & Jansen, P. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.830 · Cited by 26
Evaluating LLM’s Code Reading Abilities in Big Data Contexts using Metamorphic Testing
Li, Z., Li, Z., Xiao, K., & Li, X. (2023) · International Conference on Big Data and Information Analytics (BigDIA) · doi:10.1109/bigdia60676.2023.10429345 · Cited by 2
Evaluation of Large Language Models on Code Obfuscation (Student Abstract)
Swindle, A., McNealy, D., Krishnan, G. P., & Ramyaa, R. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i21.30517 · Cited by 1
Sylva: Sparse Embedded Adapters via Hierarchical Approximate Second-Order Information
Mu, B., Giannoula, C., Wang, S., & Pekhimenko, G. G. (2024) · International Conference on Supercomputing · doi:10.1145/3650200.3656619 · Cited by 1
Usage of Regular Features of Partially Defined Functional Neural Networks to Find a Solution
Betin, V. N., Ivashchenko, V. F., & Suprun, A. P. (2024) · Automatic Documentation and Mathematical Linguistics · doi:10.3103/s0005105524700067 · Cited by 1
Modeling processes and sensitivity analysis of machine learning methods for environmental data
Wang, Y., Song, Y., Yin, W.-X., Li, H., Lv, J.-Q., Wang, A., & Wang, H. (2024) · Elsevier eBooks · doi:10.1016/b978-0-443-14170-6.00016-0 · Cited by 1
Communicating Design Intent Using Drawing and Text
McCarthy, W. P., Matejka, J., Willis, K. D. D., Fan, J. E., & Pu, Y. (2024) · Creativity and Cognition · doi:10.1145/3635636.3664261 · Cited by 3
Revolutionizing Software Development: Autonomous Software Evolution
Mlinarić, D., Dončević, J., Brčić, M., & Botički, I. (2024) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro60963.2024.10569871 · Cited by 1
Aligning Crowd-sourced Human Feedback for Code Generation with Bayesian Inference
Wong, M. F., & Tan, C. W. (2024) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai59869.2024.00037 · Cited by 8
AutoTest: Evolutionary Code Solution Selection with Test Cases
Duan, Z., & Wang, J. (2024) · IEEE International Conference on Cyber Security and Cloud Computing (CSCloud) · doi:10.1109/cscloud62866.2024.00030 · Cited by 2
Comparing Programming Language Models for Design Pattern Recognition
Pandey, S. K., Staron, M., Horkoff, J., Ochodek, M., & Durisic, D. (2024) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c63560.2024.00041 · Cited by 1
Leveraging Large Language Models for Efficient Failure Analysis in Game Development
Marini, L., Gisslén, L., & Sestini, A. (2024) · IEEE Conference on Games (CoG) · doi:10.1109/cog60054.2024.10645540 · Cited by 2
Evaluation Metrics in LLM Code Generation
Hartung, K., Mallick, S., Gröttrup, S., & Georges, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70563-2_17 · Cited by 4
Trofimova, E., Sataev, E., & Ustyuzhanin, A. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.2328
Artificial Intelligence and Quantum Computing
Wirtz, B. W. (2024) · Springer texts in business and economics · doi:10.1007/978-3-658-44675-8_13 · Cited by 1
Research on robust decision making for intelligent connected vehicle at highway on-ramp
Fang, H., Liu, L., Gu, Q., Xiao, X., & Meng, Y. (2024) · Proceedings of the Institution of Mechanical Engineers Part D Journal of Automobile Engineering · doi:10.1177/09544070241281396 · Cited by 1
Code Optimization Chain-of-Thought: Structured Understanding and Self-Checking
Xu, Q., Yang, D., & Zhang, L. (2024) · International Conference on Artificial Intelligence, Big Data and Algorithms (CAIBDA) · doi:10.1145/3690407.3690479 · Cited by 5
Automating TODO-missed Methods Detection and Patching
Gao, Z., Su, Y., Hu, X., & Xia, X. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3700793 · Cited by 4
Big Coding Data: Analysis, Insights, and Applications
Rahman, M. M., Shirafuji, A., & Watanobe, Y. (2024) · IEEE Access · doi:10.1109/access.2024.3521383 · Cited by 5
CodeContrast: A Contrastive Learning Approach for Generating Coherent Programming Exercises
Torres, N. (2025) · Education Sciences · doi:10.3390/educsci15010080 · Cited by 3
DemoCraft: Using In-Context Learning to Improve Code Generation in Large Language Models
Joshua, K. N., & Sreejith, M. (2025) · International Conference on Intelligent and Innovative Technologies in Computing, Electrical and Electronics (IITCEE) · doi:10.1109/iitcee64140.2025.10915349 · Cited by 7
ParaCoder: Parallel Code Generation with Large Language Model
Huang, X., Zhang, X., Tao, L., Mao, R., Zhou, N., Zhu, W., Deng, M., Meng, J., Wei, Y., Zhou, A. C., et al. (2025) · doi:10.1145/3711708.3723442 · Cited by 3
How to Select Pre-Trained Code Models for Reuse? A Learning Perspective
Bi, Z., Wan, Y., Chu, Z., Hu, Y., Zhang, J., Zhang, H., Xu, G., & Jin, H. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00065 · Cited by 2
Revisiting the Non-Determinism of Code Generation by the GPT-3.5 Large Language Model
Sawadogo, S., Sabané, A., Kafando, R., Kaboré, A. K., & Bissyandé, T. F. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00012 · Cited by 5
Stoica, A. Ş., Băbiceanu, D., Rebedea, T., & Mihăescu, M. C. (2025) · Mathematics · doi:10.3390/math13223589 · Cited by 1
Less is more: Towards green code large language models via unified structural pruning
Yang, G., Zhou, Y., Zhang, X., Cheng, W., Liu, K., Chen, X., Zhuo, T. Y., & Chen, T. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2025.104580 · Cited by 8
Self-citationModel Checking for AlphaCode-Generated Programs
Zhu, W. (2022) · International Conference on Intelligent Computing and Signal Processing (ICSP) · doi:10.1109/icsp54964.2022.9778472 · Cited by 1
Early Experience with Transformer-Based Similarity Analysis for DataRaceBench
Chen, W., Vanderbruggen, T., Lin, P., Liao, C., & Emani, M. K. (2022) · IEEE/ACM International Workshop on Software Correctness for HPC Applications (Correctness) · doi:10.1109/correctness56720.2022.00011 · Cited by 3
Zum Einsatz von KI als Methode und Werkzeug in der Medienwissenschaft
Sudmann, A. (2023) · Springer eBooks · doi:10.1007/978-3-658-36625-4_16 · Cited by 4
Deduplicating and Ranking Solution Programs for Suggesting Reference Solutions
Shirafuji, A., & Watanobe, Y. (2023) · Asia Service Sciences and Software Engineering Conference (ASSE) · doi:10.1145/3634814.3634828 · Cited by 1
Artificial Intelligence and Quantum Computing
Wirtz, B. W. (2024) · Springer texts in business and economics · doi:10.1007/978-3-031-50289-7_8 · Cited by 1
Diffeomorphic transformations for time series analysis: An efficient approach to nonlinear warping.
Lopez, I. (. M. (2003) · Depósito Académico Digital Universidad de Navarra (University of Navarra) · doi:10.15581/10171/67243 · Cited by 1
Potential of Chatgpt in Academic Research: Exploring Innovative Thinking SkillsPreprint
Songkram, N., Chootongchai, S., Keereerat, C., & Songkram, A. P. N. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4598067 · Cited by 1
Zhou, B. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/qswzc · Cited by 1
AlphaPortfolio: Discovery of Portfolio Optimization and Allocation Methods Using LLMsPreprint
Yüksel, K. A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5118317 · Cited by 2
Development and Evaluation of an AI-Assisted, Privacy-Preserving Surgical Risk CalculatorPreprint
Wolfrath, N., SenthilKumar, G., Ramamurthi, A., & Kothari, A. N. (2025) · medRxiv · doi:10.1101/2025.04.18.25325614 · Cited by 1
Proceedings of the 1st FastCode Programming Challenge
(2025) · doi:10.1145/3711708 · Cited by 1
Self-Programming AI: Code-Learning Agents for Autonomous Refactoring and Architectural EvolutionPreprint
Khemani, K. (2025) · Research Square · doi:10.21203/rs.3.rs-6688473/v1 · Cited by 2
Ao, R., Simchi‐Levi, D., & Wang, X. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6145307 · Cited by 2
Automating the semantic validation of SysML v2models through a multi-agent frameworkPreprint
Olivert-Iserte, J., Cibrián, E., Mendieta, R., & Álvarez-Rodríguez, J. M. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6553831 · Cited by 1
Executable World Models for ARC-AGI-3 in the Era of Coding Agents
Rodionov, S. (2026) · Lecture Notes in Computer Science · doi:10.1007/978-3-032-33195-3_15 · Cited by 1
“This pattern appears in recent successes such as AlphaCode, where generated programs are filtered by execution [8], FunSearch, where proposed programs are selected by automated evaluators [10], and AlphaGeometry, where a neural model proposes auxiliary geometric constructions and an exact symbolic…”
Research, C., :, Chan, A., Shalaby, A., Wettig, A., Sanger, A., Zhai, A., Ajay, A., Nair, A., Snell, C., et al. (2026) · arXiv · doi:10.48550/arxiv.2603.24477 · Cited by 1
Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability
Borg, M., Hewett, D., Hagatulah, N., Couderc, N., Söderberg, E., Graham, D., Kini, U., & Farley, D. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10889-1 · Cited by 4
SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios
Le, T., Thai, M. V. T., Manh, D. N., Nhat, H. P., & Bui, N. D. Q. (2025) · arXiv · doi:10.48550/arxiv.2512.18470 · Cited by 79
“…have achieved remarkable progress in automating software engineering (SE) tasks, including code generation (Bui et al., 2023; Chen et al., 2021a; Li et al., 2022; Manh et al., 2023; To et al., 2023; Wang et al., 2023; Wei et al., 2023; Zhuo et al., 2024), bug fixing (Jimenez et al., 2023; Xia…”
Metadata
- Authors
- Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., Hubert, T., Choy, P., de Masson d’Autume, C., Babuschkin, I., Chen, X., Huang, P.-S., Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D. J., Sutherland Robson, E., Kohli, P., de Freitas, N., Kavukcuoglu, K., & Vinyals, O.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- DeepMind, London, UK.
- Corresponding author
- Li, Yujia
- Choi, David
- Chung, Junyoung
- Kushman, Nate
- Schrittwieser, Julian
- Leblond, Rémi
- Eccles, Tom
- Keeling, James
- Gimeno, Felix
- Choy, Peter
- Vinyals, Oriol
- ORCID
- Venue
- Science
- Published
- 2022-12-09
- Volume
- 378
- Issue
- 6624
- Pages
- 1092-1097
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W4226287673
- Cited by
- 1,043
- References
- 34
- Citations by year
- 2026: 269 · 2025: 319 · 2024: 237 · 2023: 186 · 2022: 30
- FWCI
- 191.10
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- automatic code generation, competitive programming, programming competition, Codeforces, transformer language models, AI-assisted programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.