Competition-level code generation with AlphaCode
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Abstract
Programming is a powerful and ubiquitous problem-solving tool. Systems that can assist programmers or even generate programs themselves could make programming more productive and accessible. Recent transformer-based neural network models show impressive code generation abilities yet still perform poorly on more complex tasks requiring problem-solving skills, such as competitive programming problems. Here, we introduce AlphaCode, a system for code generation that achieved an average ranking in the top 54.3% in simulated evaluations on recent programming competitions on the Codeforces platform. AlphaCode solves problems by generating millions of diverse programs using specially trained transformer-based networks and then filtering and clustering those programs to a maximum of just 10 submissions. This result marks the first time an artificial intelligence system has performed competitively in programming competitions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AlphaCode, Self-Debugging and CodeRL are effective in fixing less-complex program bugs, but they rely upon ground truth test cases that invalidate pass@1 eligibility, and do not use self-reflection to bridge the gap between error identification and implementation improvement.”
From Reflexion: Language Agents with Verbal Reinforcement Learning (2023) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large language models are increasingly capable of complex programming tasks and are already being used to accelerate AI research, from generating training data to serving as programming tools [1, 2, 3, 4, 5, 6].”
From RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts (2024) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Generative AI, such as OpenAI’s ChatGPT, has rapidly emerged as a disruptive technology capable of achieving human-level performance on a broad range of tasks (1–5).”
From Generative AI without guardrails can harm learning: Evidence from high school mathematics (2025) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Li et al. [27] showed that the accuracy of AlphaCode increases from close to 0% zero-shot to over 15% with 1,000 retries and over 30% with a million retries (accuracy is measured by how often one of the top 10 answers generated by the model is correct).”
From AI Agents That Matter (2024) · cites this paper as Li et al. [27]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Evaluation covered 200 CodeContests tasks [15] across three seeds and up to three feedback rounds, with online tests supplying feedback and hidden tests measuring final success.”
From RSI-Anything: Unlocking Recursive Self-Improvement with World Models (2026) · cites this paper as [15]
Experimental — accuracy not yet measured
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Self-citationDecoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“Following Kaplan et al. (2020), we adjusted the amount of training for other model sizes such that larger models are trained more and smaller models are trained less to optimize the use of compute.”
MethodGrandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455 · cited 2×
“We believe that sampling metadata leads to a more diverse set of model samples, a strategy similar to that used by Vinyals et al. (2019), and allows our model to take advantage of the relative strengths across the metadata distribution.”
MethodSelf-citationResnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., et al. (2009) · Communications of the ACM · doi:10.1145/1592761.1592779 · Cited by 3,333
Kudo, T., & Richardson, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d18-2012 · Cited by 2,869
Research works printed but not linked: 6
- A. Church, Application of recursive arithmetic to the problem of circuit synthesis.. J. Symb. Log. 28, 289–290 (1963).
- Y. Nandwani D. Jindal Mausam P. Singla ?oNeural learning of one-of-many solutions for combinatorial problems in structured output spaces ?? International Conference on Learning Representations (ICLR 2021) 3 to 7 May 2021.
- A. Holtzman J. Buys L. Du M. Forbes Y. Choi ?oThe curious case of neural text degeneration ?? Proceedings of the 8th International Conference on Learning Representations (ICLR 2020) 26 April to 1 May 2020.
- J. Ganitkevitch B. Van Durme C. Callison-Burch ?oPPDB: The paraphrase database ?? Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Atlanta Georgia 9 to 15 June 2013 (Association for Computational Linguistics 2013) pp. 758–764.
- A. Solar-Lezama ?oProgram synthesis by sketching ?? thesis University of California Berkeley (2008).
- J. Cai R. Shin D. Song ?oMaking neural programming architectures generalize via recursion ?? Proceedings of the 5th International Conference on Learning Representations (ICLR) Toulon France 24 to 26 April 2017.
Cited by
Items 901–1,000 of 1,104
Towards Trustworthy Legal AI: A Multi-Agent Approach to Integrating Legislative Knowledge
Matak, M., & Chudziak, J. A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-09318-9_1
Evaluating SAP Joule for Code Generation
Heisler, J., Reisinger, J., & Fischer, A. (2025) · International Generative AI and Computational Language Modelling Conference (GACLM) · doi:10.1109/gaclm67198.2025.11232153 · Cited by 1
SmellDetector: Multi-Label Code Smell Detection and Refactoring with Large Language Models
Liang, W., Wang, J., Zheng, H., Li, Y., Lin, H., Kim, H., An, B., Zhao, W., & Xu, Y. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11227837 · Cited by 1
Self-citationRepository-Level Code Smell Detection Based on Multi-Scale Code Information and LLM Assistance
Liang, W., Zeng, Y., Zheng, H., Wang, J., Lin, H., Kim, H., An, B., Wei, Z., & Xu, Y. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228932
MSCoT: Structured Chain-of-Thought Generation for Multiple Programming Languages
Jin, N., Li, Z., Zhang, T., & Zeng, Q. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228673 · Cited by 4
Ju, X., Zhang, Y., Zhou, X., & Zhang, M. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228272
Atalay, A. S., Öztin, Ö., Kanak, A., Arif, İ., Badii, A., Dasari, N., Ergün, S., Ergün, S., Herkiloğlu, O., Mermoud, G., et al. (2025) · Open Research Europe · doi:10.12688/openreseurope.21192.1
109PwnPilot: could an adversary be pair programming with our most trusted software engineers?
Horne, D. (2025) · De Gruyter eBooks · doi:10.1515/9783111436548-006
Exploring the Scaling Properties of RLHF on Data, Model, and MethodPreprint
Hou, Z., Du, P., Niu, Y., Du, Z., Zeng, A., Liu, X., Xiong, N., Liu, W., Huang, M., Wang, H., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5819183
G-STAR: A Threat Modeling Framework for General-Purpose AI Systems
Xiong, P., Lohrasbi, S., Kotian, P., & Buffett, S. (2025) · Conference on Privacy, Security and Trust · doi:10.1109/pst65910.2025.11268824
AI-Powered Code Completion with LSTM Networks
Bani-Salameh, H., Fattoum, Y., & Hjeela, F. A. (2025) · National Conference on Electrical and Electronics Engineering (ELECO) · doi:10.1109/iceee67194.2025.11261923
MAC: Multi-Agent LLM Coder is All You NeedPreprint
Kodati, A., Khomh, F., & Sami, A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5887028 · Cited by 1
Gondowijoyo, F. J., & Girsang, A. S. (2025) · International Conference on Software Engineering and Information Technology (ICoSEIT) · doi:10.1109/icoseit67010.2025.11290944
Tan, W., Jiang, C., Duan, Y., Lei, M., Li, J., Hong, Y., Wang, X., & An, B. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37132-4_14 · Cited by 6
SEAlign: Alignment Training for Software Engineering Agent
Zhang, K., Zhang, H., Li, G., You, J., Li, J., Zhao, Y., & Jin, Z. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3764563 · Cited by 14
MACEDON : Supporting Programmers with Real-Time Multi-Dimensional Code Evaluation and Optimization
Liu, X., You, Y., Qiu, X., Ma, T., & Zhao, J. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747655 · Cited by 1
The Uniqueness of Digital Transformation and Data-Based Systems (DS)
Kirchschlaeger, P. G. (2025) · Springer eBooks · doi:10.1007/978-3-031-98704-5_3
Beliakin, E., & Sagirova, A. (2025) · Studies in computational intelligence · doi:10.1007/978-3-032-07690-8_8
RKE-Coder: A LLMs-Based Code Generation Framework with Algorithmic and Code Knowledge Integration
Deng, Y., Zhang, Z., Huang, Q., & Wang, M. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3343-5_41
Abela, J., Cachia, E., & Layfield, C. (2025) · Mathematics · doi:10.3390/math14010041 · Cited by 1
Generating Test Cases with Large Language Model: Frameworks and Challenges
Hosseini, M. R., Janjua, N. K., Mohsin, A., & Rahman, S. U. (2025) · IEEE International Conference on e-Business Engineering (ICEBE) · doi:10.1109/icebe68123.2025.00041 · Cited by 1
CallNavi, A challenge and empirical study on LLM function calling and routing
Song, Y., Tang, X., Lothritz, C., Ezzini, S., Klein, J., Bissyandé, T. F., Boytsov, A., Ble, U., & Goujon, A. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3756681.3756975 · Cited by 7
AI in Orthopaedics: The Wisdom of “Should We?” Versus “Can We?”
Murphy, M. P., Weinstein, J. N., Guyton, G. P., & Porter, S. E. (2025) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.25.00829 · Cited by 2
Mitigating Syntax and Logic Errors in LLM Based Code Generation via XML-Structured Prompts
Sambaraju, S., Boman, J., Wu, H., & Zong, Z. (2025) · IEEE International Performance, Computing, and Communications Conference · doi:10.1109/ipccc66453.2025.11304661 · Cited by 2
RAGdeterm: Deterministic Retrieval-Augmented Generation for Code GeneratorPreprint
Bochenek, A., Protasiewicz, J., & Pedrycz, W. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5986423
MCPybarra: A Multi-agent Framework for Low-Cost, High-Quality MCP Service Generation
Peng, B., Liu, M., Liu, Y., Tian, C., Yu, S., & Wang, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-5012-8_4
Atomizer: An LLM-based Collaborative Multi-Agent Framework for Intent-Driven Commit Untangling
Zhu, K., Tian, Z., Wang, S., Leng, M., & Mao, X. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3787773 · Cited by 7
Toward Safeguarding GenAI with Graph Transformation: Defining an Exchange Format in JSON
Hofmann, L. S., Lauer, A., Requeno, J. I., & Taentzer, G. (2026) · Electronic Proceedings in Theoretical Computer Science · doi:10.4204/eptcs.440.0.3
Valencia, J. M. C., Martínez-Rolán, X., & Seoane, J. P. (2026) · EDU REVIEW International Education and Learning Review / Revista Internacional de Educación y Aprendizaje · doi:10.62701/revedu.v13.5514 · Cited by 6
Consensus planning boosts LLM code generation
Wen, C., Liu, J., & Du, L. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108030 · Cited by 4
Hanif, H., Maffeis, S., & Anuar, N. B. (2025) · IEEE International Conference on Computing (ICOCO) · doi:10.1109/icoco67189.2025.11334069
Test Case Enhanced Self-Iterative Code Generation Framework
Chang, T., Zhang, Y., & Fang, Y. (2025) · IEEE International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/icpads67057.2025.11322891
GRACG: Graph Retrieval Augmented Code Generation
Fedorov, K., Zarubin, B., & Ivanov, V. V. (2025) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew67777.2025.00060 · Cited by 3
Environment-Aware Code Generation: How far are We?
Wu, T., Chen, R., Du, W., Ma, S., Qi, G., Xing, Z., Khadivi, S., Periyathambi, R., & Haffari, G. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3787784 · Cited by 6
A Novel Approach to Improve Latency and Accuracy of NLP Models Involving Quantum-Inspired Algorithms
Kumar, C. S. S., Chakravarthy, M., & Mohan, B. V. (2025) · International Conference on Artificial Intelligence for Sustainable Innovation (AI-SI) · doi:10.1109/ai-si66213.2025.11341047
Hierarchical Subspace Adaptation for Efficient Fine-Tuning of Large Language Models
Dong, J., Wang, Y., Wang, S., & Chen, X. (2025) · International Conference on Machine Intelligence and Nature-Inspired Computing (MIND) · doi:10.1109/mind67540.2025.11351573
Self-citationIvanov, A., Pakhorukova, A., Solodkaya, M. A., Konstantinov, A. M., Avdyushina, A. E., & Markina, T. A. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-14674-8_8
Cyber Insurance, Software Diversity, and AI Coders
Barreto, C., & Franke, U. (2026) · Communications of the ACM · doi:10.1145/3746131
M2QCode: A Model-Driven Framework for Generating Multi-Platform Quantum Programs
Guo, X., Saito, S., & Zhao, J. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00301
KiC: Keyword-inspired Cascade for Cost-Efficient Text Generation with LLMs
Kim, W.-C., Park, J.-H., & Lee, S.-W. (2025) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc58881.2025.11343731
FGit: Fault-Guided Fine-Tuning for Code Generation
Fan, L., Liu, Z., Wang, H., Bao, L., Xia, X., & Li, S. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00114 · Cited by 5
Espinoza, N., Soto, A., & Navarrete, E. (2025) · International Conference on Soft Computing & Machine Intelligence (ISCMI) · doi:10.1109/iscmi67495.2025.11358555
LLMs as Continuous Learners: Improving Software Issue Reproduction by Distilling ExperiencesPreprint
Lin, Y., CHEN, S., Ma, Y., Cao, R., Li, B., Huang, F., li, Y., & Gu, X. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6170084
Self-citationYapay Zeka Destekli Kod Üretimi İle İnsan Yazımı Kodların Karşılaştırılması: Python Örneği
EYGÜ, S., & Keskinkılıç, M. (2026) · Bilgisayar bilimleri ve mühendisliği dergisi. · doi:10.54525/bbmd.1781375
Laaksonen, A. (2026) · Undergraduate topics in computer science · doi:10.1007/978-3-032-07453-9_5
shboul, B. A., Almakadmeh, K., Bani-Salameh, H., & Bishara, M. (2026) · AIP conference proceedings · doi:10.1063/5.0319013
A Logic-Driven Workflow Based on LLM Agents for High-Fidelity SMT Code Generation
Liu, Z., & Wang, K. (2025) · International Conference on Embodied Intelligence and Large Models · doi:10.1145/3788731.3788734
Design and Implementation of a Software Tool to Support Agile-SOFL
Chen, R., & Liu, S. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-6032-5_9
Hybrid Real-Time Intelligent Code Translator
Malar., T., Nitheeshwari, S., & Abirami, R. (2025) · IEEE Conference on Information and Communication Technology (CICT) · doi:10.1109/cict67193.2025.11399005
Gurukulya: An AI-Powered Smart Campus Education Platform
Prof. Deepak, B., Setturu, B., & Naidu, P. A. S. (2026) · Vidhyayana · doi:10.58213/4zpqc821
Plan-Execute-Generate-Judge: A Self Verifying Multi Agent LLM Framework for Complex NoSQL Querying
Kaplunovich, A. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11400819
Sultan, M. F., Akter, S., & Cuzzocrea, A. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402539
Performance-Aligned LLMs for Generating Fast HPC Code
Nichols, D., Polasam, P., Menon, H., Marathe, A., Gamblin, T., & Bhatele, A. S. (2026) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2026.3675550 · Cited by 31
TheBotCompany: Self-Organizing Multi-agent Systems for Continuous Software Development
Lyu, W., Xiao, Y., Zhang, Y., & Sun, Y. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834358 · Cited by 6
Foundation Vision-Language models
Laurençon, H. (2025) · theses.fr (ABES) · doi:10.70675/ec0311a0zb6a4z4249zb9fbz01bf51a02c7c · Cited by 1
Hyper Web AI: A Natural Language Processing Based AI-Powered Web Code Generator
Kolli Neharika, V. H., & Ganapathi, G. S. A. (2026) · International Journal of Advanced Research in Science Communication and Technology · doi:10.48175/ijarsct-32590
Yu, F. X., Adam, G. C., Bastian, N. D., & Lan, T. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6568443
Direct Synthesis of Ultra-Constrained Bare-Metal Firmware from Structured IntentPreprint
Wang, Y., & Shi, Z. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6576157
Hao, H., Zheng, X., & Yu, X. (2026) · Scientific Reports · doi:10.1038/s41598-026-47693-2 · Cited by 1
Wang, P., Qi, K., Zhen, F., Yan, B., & Guo, T. (2026) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.70705
Sur, D. A., & Jain, R. K. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6603315
CSF: A curriculum-based scaffolding framework for heuristic evolution in combinatorial optimization
Zhang, Z., Li, S., Chen, X., Chen, X., Tianle, Z., Zhao, J. W., & Liu, P. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.114659
Self-citationVGHTCoder: Multi-agent Code Generation with Hypothesis Testing and Verification Guidance
Yin, B., & Wang, X. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7078-2_50
Dare: Dual-Aspect Reflective Evolution for Prompt Optimization
Qiu, X., Zhang, J., Liu, B., Chen, Z., & Wang, M. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462574
LLM-based multi-agent systems for code generation: A multi-vocal literature review
Rasheeda, Z., Waseema, M., Kemella, K.-K., Saari, M., & Abrahamsson, P. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.113142 · Cited by 3
Coding with Eyes: Visual Feedback Unlocks Reliable GUI Code Generating and Debugging
Liu, Z., Huang, Y., Xie, T., (752030), R. Z., Li, W., & Duan, L. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37314-4_2 · Cited by 1
Integrating Retrieval Augmentation and Decoding Intervention for Automated Program Repair
Wang, S., Lu, L., Qiu, S., Liang, Z., & Suo, S. (2026) · Expert Systems · doi:10.1111/exsy.70251 · Cited by 1
Vibe Coding on Trial: Operating Characteristics of Unanimous LLM Juries
Ullah, M. A., & Serwadda, A. (2026) · IEEE International Conference on Semantic Computing (ICSC) · doi:10.1109/icsc67292.2026.00039
MongoTalk: An Agentic AI Framework for Conversational and Autonomous No-Code MongoDB Management
Patle, A., Ajalkar, D. A., Siddheshwar, S., Vijay, V., & Shirsath, D. (2025) · Joint International Conference on Data Science & Management of Data (CODS-COMAD) · doi:10.1145/3799830.3799881
Acharya, V., Pisharodi, A., Pasi, R., Mondal, R., & Batra, N. (2025) · Joint International Conference on Data Science & Management of Data (CODS-COMAD) · doi:10.1145/3799830.3799884
Asavathongkul, T., Sa-nga-ngam, P., & Kiattisin, S. (2025) · Technology Innovation Management and Engineering Science International Conference (TIMES-iCON) · doi:10.1109/times-icon67125.2025.11488122
A Documentation-Driven Framework for AI-Assisted Full-Circle Agile Software Development
Meng, H., Yu, X., Xu, M., & Zhang, W. (2025) · International Conference on Intelligent Education and Intelligent Research (IEIR) · doi:10.1109/ieir67391.2025.11491045 · Cited by 1
Mew, L. (2026) · Journal of Information Systems Applied Research and Analytics · doi:10.62273/hmad7698
Benchmarking AI Models for Automated Code Generation and Testing Using HumanEval
Manzoor, M., Rehman, M. U., Kazim, M., Deka, L., Khattak, A. A., & Usmani, I. A. (2026) · Global Conference on Wireless and Optical Technologies (GCWOT) · doi:10.1109/gcwot69191.2026.11499481
Ghojogh, B., & Ghodsi, A. (2026) · Springer eBooks · doi:10.1007/978-3-032-10738-1_11
Fedorov, A. M., Datyev, I. O., Илясов, М. О., & Вишняков, И. Г. (2026) · Transaction Kola Science Centre · doi:10.37614/2949-1215.2026.17.1.012
An overview of evaluation and enhancement methods for code generation by large language models
Truong, J., Nguyen, V., & Nguyen, T. T. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108185 · Cited by 4
AdaptGen: A Problem-Adaptive Solution Template Generation Technique for Online Programming Platforms
Zhang, G. T., Zou, W., Zhang, X., Zhang, J., & Xuan, J. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3692192
Chen, S., & Liu, F. (2026) · International Conference on Educational and Information Technology · doi:10.1109/iceit68991.2026.11521279 · Cited by 1
Ghorbel, A., Ayed, N. E. B., Tinkicht, N., & Bouchahma, M. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.116044
Caldeira, T., Antunes, T., Khonji, M., Dias, J. M. M., & Lima, P. U. (2026) · IEEE International Conference on Autonomous Robot Systems and Competitions · doi:10.1109/icarsc70216.2026.11523235
Huang, C., Han, H., & Tian, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-20080-8_14
Effective Fine-tuning for Low-resource Languages: A Case Study of Cangjie
Lin, Z., Liu, Z., Zhou, M., Huang, Z., Chen, C., Ma, W., & Li, L. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10878-4 · Cited by 2
TraceFix: Repairing Agent Coordination Protocols with TLA+ Counterexamples
Xia, S., Li, Q., Ehsan, T., & Ortiz, J. (2026) · ACM Conference on AI and Agentic Systems (CAIS) · doi:10.1145/3786335.3813159 · Cited by 2
Lee, J. D., Jung, S., & Kung, J. (2026) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2026.3693713
MagicGeo: Training-free text-guided geometric diagram generation
Zhang, T., Wang, J., Yu, H., Xie, Q., Wang, J., & Huang, H. (2026) · Graphical Models · doi:10.1016/j.gmod.2026.101331 · Cited by 14
Su, Z., Wang, J., Quan, F., & Wang, S. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-7299-1_36
Shi, B., & Kristensson, P. O. (2026) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3817601
Yang, K., Sun, J., Wang, Z., & Chang, X. (2026) · Mathematics · doi:10.3390/math14111891
Demo: Mobile Vibe-Coding with Fully Local Models and Memory
Radu, V., Foley, M., Pour, M. V., Pegoraro, M., Faricy, F., & Girolami, M. (2026) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3812836.3814797 · Cited by 1
LLM-Augmented Mutation Testing: Intelligent Generation and Triage of Mutants
Bajaj, S., Dantuluri, V. S. R., Rajput, D., Kolhe, A., & Sharma, A. (2026) · International Conference on Artificial Intelligence, Systems, and Emerging Technologies (ICAISET) · doi:10.1109/icaiset66439.2026.11541746
Beyond Browser Agents: Distilling Experience for Accessible Web Automation
Yitalo, A., Jovanovic-Hacon, S., Strathman, M., Gill, B., & Khurana, R. (2026) · International Web for All Conference (W4A) · doi:10.1145/3800424.3800431
Diamond of Thoughts: Synergistic Merging for Efficient Large Language Model Reasoning
Zhang, Y. (2026) · International Conference on Intelligent Computing and Signal Processing (ICSP) · doi:10.1109/icsp69961.2026.11540838
GPU Kernel Optimization Beyond Full Builds: An LLM Framework with Minimal Executable Programs
Chu, R., Wang, A., Bai, X., Liu, S., & Dong, X. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-1468-6_19 · Cited by 3
Vu, S. M., Pham, T. T., & Tran, V. H. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6906675
Kátai, Z., & Iclănzan, D. (2026) · Acta Universitatis Sapientiae Informatica · doi:10.1007/s44427-026-00033-8
Evaluating LLMs for Explaining Insecure Code and Identifying Vulnerabilities in Java and Python
Mahmud, A., Rawajfih, Y., & Wu, F. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-22211-4_12
Design Methodology and Performance Trade-offs Management for Distributed and Compound AI Systems
Gravara, M., Stanisic, A., & Nastic, S. (2026) · IEEE ... International Conference on Cloud Computing · doi:10.1109/cloud72782.2026.00049 · Cited by 3
Reasoning Meets Adaptation: Bridging Global LLM Planning and Distributed UAV Control
Cha, C., Jin, S., & Park, H. (2026) · IEEE Wireless Communications and Networking Conference · doi:10.1109/wcnc65185.2026.11555260 · Cited by 1
Exploring AI-Generated Code for Health Monitoring Modeling in Complex Industrial Systems
Zhao, C., Sun, Y., Jiang, W.-M., Yang, Y., Liu, L.-L., & Gao, Z.-G. (2025) · International Conference on Mechanical Engineering and Aerospace Engineering (MEAE) · doi:10.1109/meae68077.2025.11557847
Metadata
- Authors
- Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., Hubert, T., Choy, P., de Masson d’Autume, C., Babuschkin, I., Chen, X., Huang, P.-S., Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D. J., Sutherland Robson, E., Kohli, P., de Freitas, N., Kavukcuoglu, K., & Vinyals, O.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- DeepMind, London, UK.
- Corresponding author
- Li, Yujia
- Choi, David
- Chung, Junyoung
- Kushman, Nate
- Schrittwieser, Julian
- Leblond, Rémi
- Eccles, Tom
- Keeling, James
- Gimeno, Felix
- Choy, Peter
- Vinyals, Oriol
- ORCID
- Venue
- Science
- Published
- 2022-12-09
- Volume
- 378
- Issue
- 6624
- Pages
- 1092-1097
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W4226287673
- Cited by
- 1,043
- References
- 34
- Citations by year
- 2026: 269 · 2025: 319 · 2024: 237 · 2023: 186 · 2022: 30
- FWCI
- 191.10
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- automatic code generation, competitive programming, programming competition, Codeforces, transformer language models, AI-assisted programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.