Competition-level code generation with AlphaCode
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Abstract
Programming is a powerful and ubiquitous problem-solving tool. Systems that can assist programmers or even generate programs themselves could make programming more productive and accessible. Recent transformer-based neural network models show impressive code generation abilities yet still perform poorly on more complex tasks requiring problem-solving skills, such as competitive programming problems. Here, we introduce AlphaCode, a system for code generation that achieved an average ranking in the top 54.3% in simulated evaluations on recent programming competitions on the Codeforces platform. AlphaCode solves problems by generating millions of diverse programs using specially trained transformer-based networks and then filtering and clustering those programs to a maximum of just 10 submissions. This result marks the first time an artificial intelligence system has performed competitively in programming competitions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AlphaCode, Self-Debugging and CodeRL are effective in fixing less-complex program bugs, but they rely upon ground truth test cases that invalidate pass@1 eligibility, and do not use self-reflection to bridge the gap between error identification and implementation improvement.”
From Reflexion: Language Agents with Verbal Reinforcement Learning (2023) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large language models are increasingly capable of complex programming tasks and are already being used to accelerate AI research, from generating training data to serving as programming tools [1, 2, 3, 4, 5, 6].”
From RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts (2024) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Generative AI, such as OpenAI’s ChatGPT, has rapidly emerged as a disruptive technology capable of achieving human-level performance on a broad range of tasks (1–5).”
From Generative AI without guardrails can harm learning: Evidence from high school mathematics (2025) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Li et al. [27] showed that the accuracy of AlphaCode increases from close to 0% zero-shot to over 15% with 1,000 retries and over 30% with a million retries (accuracy is measured by how often one of the top 10 answers generated by the model is correct).”
From AI Agents That Matter (2024) · cites this paper as Li et al. [27]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Evaluation covered 200 CodeContests tasks [15] across three seeds and up to three feedback rounds, with online tests supplying feedback and hidden tests measuring final success.”
From RSI-Anything: Unlocking Recursive Self-Improvement with World Models (2026) · cites this paper as [15]
Experimental — accuracy not yet measured
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Self-citationDecoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“Following Kaplan et al. (2020), we adjusted the amount of training for other model sizes such that larger models are trained more and smaller models are trained less to optimize the use of compute.”
MethodGrandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455 · cited 2×
“We believe that sampling metadata leads to a more diverse set of model samples, a strategy similar to that used by Vinyals et al. (2019), and allows our model to take advantage of the relative strengths across the metadata distribution.”
MethodSelf-citationResnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., et al. (2009) · Communications of the ACM · doi:10.1145/1592761.1592779 · Cited by 3,333
Kudo, T., & Richardson, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d18-2012 · Cited by 2,869
Research works printed but not linked: 6
- A. Church, Application of recursive arithmetic to the problem of circuit synthesis.. J. Symb. Log. 28, 289–290 (1963).
- Y. Nandwani D. Jindal Mausam P. Singla ?oNeural learning of one-of-many solutions for combinatorial problems in structured output spaces ?? International Conference on Learning Representations (ICLR 2021) 3 to 7 May 2021.
- A. Holtzman J. Buys L. Du M. Forbes Y. Choi ?oThe curious case of neural text degeneration ?? Proceedings of the 8th International Conference on Learning Representations (ICLR 2020) 26 April to 1 May 2020.
- J. Ganitkevitch B. Van Durme C. Callison-Burch ?oPPDB: The paraphrase database ?? Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Atlanta Georgia 9 to 15 June 2013 (Association for Computational Linguistics 2013) pp. 758–764.
- A. Solar-Lezama ?oProgram synthesis by sketching ?? thesis University of California Berkeley (2008).
- J. Cai R. Shin D. Song ?oMaking neural programming architectures generalize via recursion ?? Proceedings of the 5th International Conference on Learning Representations (ICLR) Toulon France 24 to 26 April 2017.
Cited by
Items 1,001–1,100 of 1,104
Evaluating LLM Coding Agents on SZ-Family Lossy Compression Across Architectures
Li, C., Gao, S., Zhao, K., Di, S., & Dong, W. (2026) · IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum · doi:10.1109/ipdpsw71298.2026.00111
Dursunoglu, H., & Sulkalar, K. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6958668
On the complexity of least general generalization with commutative functions
Akutsu, T., & Takasu, A. (2026) · Annals of Mathematics and Artificial Intelligence · doi:10.1007/s10472-026-10014-4
Development of a domain-tailored on-premise dialogue system for accelerated materials analysis
Goto, S., Kamiyama, M., Sudo, E., & Kimura, H. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12254-1
GeoMind: An Autonomous Multi-Agent Framework for Verified Geospatial AnalysisPreprint
Habib, J., Srabon, M. I., Nahar, M. L., Lutfy, M., Khatun, M. A., Shammo, M. B. A. Z., & Hasan, M. A. (2026) · Research Square · doi:10.21203/rs.3.rs-10034367/v1
Lin, H. W., & 謝, Z.-F. 宗. S. (2026) · Publications of the Astronomical Society of the Pacific · doi:10.1088/1538-3873/ae768e
DCR-Code: A Difficulty-Aware Multi-Agent Framework for Code Generation Via Decompose-Compose-Refine
Liu, H. (2026) · International Conference on Artificial Intelligence and Industrial Technology Applications (AIITA) · doi:10.1109/aiita69518.2026.11567118
GEN-CSAF: A Generative Autonomy Framework for End-to-End Task Scheduling in Container-Based Systems
Zhu, L., Huang, K., Wang, Y., & Su, Y. W. (2026) · IEEE Transactions on Cloud Computing · doi:10.1109/tcc.2026.3707138
An Empirical Study of Waterfall-style Multi-Agent Workflows for Class-Level Code Generation
Shafin, W. I., Rafi, M. N., Li, Z., & Chen, T.-H. P. (2026) · doi:10.1145/3803846.3807461
VidToCode: LLM-Based Source Code Reconstruction From Programming Video Tutorial Transcripts
Tayeb, A. J., Alahmadi, M. D., Alsulami, A. A., Alturki, B., Alakhtar, R. A., & Alsobhi, H. (2026) · IEEE Access · doi:10.1109/access.2026.3741380
Understanding and Predicting Accepted Code Suggestions in AI-Assisted Programming
Jiang, J., Li, L., Hou, J., Tan, X. L., & Zhang, L. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808125
PlayCoder: Making LLM-Generated GUI Code Playable
Peng, Z., Tao, W., Yin, X., Ying, C., Luo, Y., & Guo, Y. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808097 · Cited by 16
DualCodeDetect: Zero-Shot LLM-Generated Code Detection via Dual-Channel Perturbation
Li, Z., Shang, X., Fu, Z., Guo, S., Zhang, W., Yu, N., & Chen, K. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808166 · Cited by 1
CertiCoder: Towards MISRA-Compliant C Code Generation with LLMs
Gou, M., Yao, Z., Ma, H., Zhang, E., Zhou, J., & He, F. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3797120
WasmWeaver: A Framework for Runtime-Aware WebAssembly Program Generation with Reinforcement Learning
Müller, K., Mane, S., Wägemann, P., & Franchi, N. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner67736.2026.00045 · Cited by 1
FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving
Gao, S., Yin, J., Wang, F., & Dong, W. (2026) · International Conference on Supercomputing · doi:10.1145/3797905.3800525 · Cited by 6
Programming Language Confusion: When Code LLMs Can't Keep Their Languages Straight
Moumoula, M. B., Nikiema, S. L., Kaboré, A. K., Klein, J., & Bissyandé, T. F. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner67736.2026.00087 · Cited by 7
Leveraging Enhanced Test-Driven Development for Accurate Code Generation in LLMs
Zhang, R., Shan, W., Fu, C., & Long, T. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner67736.2026.00089
On the Time Efficiency of Code Generated and Optimized by Large Language Models
Peng, Y., Wan, J., Ren, X., Menzies, T., & Lyu, M. R.-T. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3709872
When AI Coding Assistants Leak Training Data: A Study of LLM Memorization in Code Generation
Cheng, X., Yao, K., Nie, P., & Shang, W. (2026) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1145/3805760.3814902 · Cited by 1
Learning to Navigate (L2N): A Human-Centered Framework for Local Code Search
Leon, S., Zhang, Z., Tadepalli, P. V., & Kuttal, S. K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31048-4_21
LLM-Driven SQL Remediation: Towards Safe and Explainable Code for Automated Schema Refactoring
Seabra, A., Cavalcante, C., Ruberg, N., & Lifschitz, S. (2026) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1145/3786181.3788716
VulJSFormer: Learning to Detect JavaScript Vulnerabilities with Vulnerability-Relevant Graphs
Ren, K., He, H., Wang, H., Qiang, W., Wu, Y., & Zou, D. (2026) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn69566.2026.00064
Autofy: Automated Synthesis of Formally Verified Code
Purschke, N., Schamschurko, A., Kirchner, S., Wu, C., & Knoll, A. (2026) · IEEE/ACM International Conference on Formal Methods in Software Engineering (FormaliSE) · doi:10.1145/3793656.3793688
Ke, Q., Zhao, M., Dong, Y., Woźniak, M., Zhang, T., & Tian, S. (2026) · Applied Soft Computing · doi:10.1016/j.asoc.2026.115934
Zhu, H., Wu, J., Ling, X., Luo, T., & Zhao, C. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2856-0_32
Omnia: Efficient RAG Serving through Speculative Scheduling
Fu, R., Li, S., Xu, Y., Wu, T., Ma, Z., & Shi, J. (2026) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1145/3806645.3807814 · Cited by 1
DOTA: Intelligent Debugging with D elta o f T houghts A gents
Yang, R., Gupta, R., Zhang, Q., & Kundu, A. (2026) · ACM SIGOPS Operating Systems Review · doi:10.1145/3830422.3830424 · Cited by 1
Song, Y.-T., Sun, Y., Jiang, J., Liu, L.-L., Gao, Z., & Qin, W. (2026) · Advanced Engineering Informatics · doi:10.1016/j.aei.2026.105060
Fu, C., Zhang, W., Zhang, J., & Si, N. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3417-2_13
XPJudge: Explanation-Aware Pairwise Judging for Cost-Effective Code LLM Selection
Braiek, Y. B., Braiek, H. B., & Khomh, F. (2026) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst69053.2026.00094
PracAPR: A Test-Free, Execution Trace-Driven Automated Program Repair Framework
Harsita, S. S., Devi, S. J., Sathyapriya, S., Jeniba, S., & Sitara, N. S. (2026) · International Conference on Secure Cyber Computing and Communications (ICSCCC) · doi:10.1109/icsccc69031.2026.11600180
Mapping text to multiplex graph: Prompt compression as Lévy walk-guided graph pruning
Gao, Y., Lu, Y., Deng, J., Nie, J., Tang, Z., Zhang, J., Zhu, Z., Yu, S., Xuan, Q., & Zhou, J. T. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116661
DRRAR: Domain-Aware Requirement Refinement via Adaptive RAG
Wei, L. (2026) · International Conference on Artificial Intelligence and Digital Ethics (ICAIDE) · doi:10.1109/icaide69370.2026.11604517
Rusting Volatility: Accelerating memory forensics through LLM-assisted cross-language migration
Gharaibeh, T., & Baggili, I. (2026) · Forensic Science International Digital Investigation · doi:10.1016/j.fsidi.2026.302129
Multi-Agent LLM Pipeline for Code Writing: An Experimental Study of Writer-Reviewer Architecture
Ekiz, T. K., & Konyar, M. Z. (2026) · ADBA computer science. · doi:10.69882/adba.cs.2026071
Java academic benchmark: Exam-based evaluation of LLMs on object-oriented programming
Cocchieri, A., Ragazzi, L., Aguzzi, G., Frisoni, G., Molfetta, L., Moro, G., & Viroli, M. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.113033
A Code Ranking Framework with Human-Inspired Agent-Based Tests Generation
Cao, L., Liang, J., Wang, J., & Cai, Y. (2026) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3794763.3794820
Self-Improving Code Generation via Semantic Entropy and Behavioral Consensus
Zhang, H., Cheng, W., & Hu, W. P. (2026) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3794763.3794798 · Cited by 2
Fu, Y., Zhou, Z., Huang, C., Wang, J., Zeng, B., Dongke, H., Wang, X., Hong, W., & Fan, J. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770854.3783937
TANGRAM-AI: An Intelligent Assistant for Performance Modeling and Learning-Based Control
Chambers, Q., Gounot, T. C., Bruno, G. G. E., Silva, E. D. S. E., & Leão, R. M. M. (2026) · Mediterranean Artificial Intelligence and Networking Conference (MAIN) · doi:10.1109/main71116.2026.11622397
Uso de LLMs no Agrupamento e Classificação de Estratégias de Programação em Juízes On-line
Aguiar, D., Melo, R., Pires, F., Pessoa, M., & Fernandes, D. (2026) · Anais do Workshop sobre Educação em Computação (WEI) · doi:10.5753/wei.2026.22274
A Comprehensive Survey of Generative AI: Modalities, Systems, and Applications
Hans, S., & Ranjan, P. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-28304-7_15
Yang, Y., Huang, E., Xia, C., Yang, B., Pan, T., Zhang, Z., Dong, D., & Qin, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31335-5_22
KernelEvolve: Scaling Agentic Kernel Coding for Heterogeneous AI Accelerators at Meta
Liao, G., Qin, H., Wang, Y., Golden, A., Kuchnik, M., Yetim, Y., Xiao, R., Ang, J. J., Fu, C., He, Y., et al. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00063 · Cited by 65
SaaSGenAI: Micro-SaaS Generation and Deployment Framework
Uke, S. N., Walunjkar, S., Ukirade, S. S., Patil, V., & Wagh, G. (2026) · International Conference on Inventive Computation and Information Technologies (ICICIT) · doi:10.1109/icicit69063.2026.11634153
Feng, S., Cheng, J., He, G., Zheng, J., & Zhang, Y. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7231715
CAMP: Cognition-Aligned Multi-stage Automated Program Repair
Wu, J., Shen, B., & Dong, Y. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108282 · Cited by 1
BiVCoder: A Multi-Agent Framework for Code Generation via Bidirectional Code-Test Diagnosis
Li, X., Dong, J., Shi, W., Lü, W., & Du, X. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818107
Competitive Learning: A Paradigm for Excellence, Anchored in Competitive Programming
Lim, Y. (2026) · The Journal of Competitive Learning · doi:10.66044/5qzbhd69
Hasan, I. H., & Potrus, M. Y. (2026) · Automated Software Engineering · doi:10.1007/s10515-026-00658-1
Wan, Z., Tang, D., Zhang, Z., Zhao, Z., & Wang, L. (2026) · International Conference on Artificial Intelligence and Electromechanical Automation (AIEA) · doi:10.1109/aiea70743.2026.11632325
How well do LLMs understand code?
Xu, J., Sun, R., Ding, Z., Li, W., Han, K., Zhang, H., Wang, C., Wang, H., Wang, W. W., Sun, Y., et al. (2026) · Communications AI & Computing · doi:10.1038/s44488-026-00014-y
Towards Optimizing LLM-based Sensor Data Processing: An Empirical Study of Evolutionary Algorithms
Quan, P., Wang, B., Ouyang, X., Yang, K., & Srivastava, M. B. (2026) · IEEE International Workshop on Foundation Models for Cyber-Physical Systems & Internet of Things (FMSys) · doi:10.1109/fmsys72110.2026.00019
RL–LLMfuzzer: reinforcement learning–guided dual–model differential fuzzing for compiler testing
Li, D., Li, Y., Zhao, B., Xu, J., Yao, J., Wang, N., & Liu, L. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-01115-5
Self-citationVibeCodeHPC: A Multi LLM Agent System for HPC Code Auto-Tuning
Hayashi, S., Morita, K., Mukunoki, D., Hoshino, T., & Katagiri, T. (2026) · IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum · doi:10.1109/ipdpsw71298.2026.00036
Bhardwaj, K., & Menon, H. (2026) · IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum · doi:10.1109/ipdpsw71298.2026.00116
Cao, Y., Keung, J., Wang, Y., Sun, Y., & Mao, Z. (2026) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac69091.2026.00062 · Cited by 1
Torello, D., & Fennell, J. (2026) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--59625
Conditional VAE framework for lens starting-point generation
Xu, H., Yang, T., Cheng, D., & Wang, Y. (2026) · Optics Express · doi:10.1364/oe.611723
Cross-Model Collaboration for Enhancing LLM-Based Code Generation
Huang, J., Ye, W., Sun, W., Feng, Y., & Liu, Y. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3840382
TensorAgent: Multi-agent Framework for Automated Tensor Core Code Generation
Liu, B., Zheng, Z., & Zhang, X. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-4805-6_16
Zibaeirad, A., Nogueira, R. P., & Vieira, M. (2026) · IEEE Transactions on Reliability · doi:10.1109/tr.2026.3716597
Evaluation and Distillation of Source Code Generation Tasks by Large Language Models
Brahman, D. (2026) · Digital Commons - DU (University of Denver) · doi:10.56902/etdcrp.2026.69
Cluster-Aware Model Selection and Sequential Refinement for Code GenerationPreprint
Taya, M., Cohen, S., & Rokach, L. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7355699
Zhang, S., Shu, Y., Yuan, X., & Xiao, Y. (2026) · Journal on Computing and Cultural Heritage · doi:10.1145/3842757
Machine Common Sense: A Grand Challenge in Artificial Intelligence
Kejriwal, M. (2026) · ACM eBooks · doi:10.1145/3789270.3789272
Kejriwal, M. (2026) · ACM eBooks · doi:10.1145/3789270.3789276
Commonsense Knowledge versus Reasoning
Kejriwal, M. (2026) · ACM eBooks · doi:10.1145/3789270.3789275
Kejriwal, M. (2026) · ACM eBooks · doi:10.1145/3789270.3789277
Kejriwal, M. (2026) · ACM eBooks · doi:10.1145/3789270.3789274
(2026) · ACM eBooks · doi:10.1145/3789270.3789271
Multi-disciplinary Perspectives on Defining Common Sense
Kejriwal, M. (2026) · ACM eBooks · doi:10.1145/3789270.3789273
(2026) · ACM eBooks · doi:10.1145/3789270.3844419
EG-MAP: Evidence-Grounded Multi-Agent Smart Contract Auditing
Song, Z., Yue, L., 陶泽家, Han, Q., Zeng, W., Xu, Z., Pal, S., & Dong, C. (2026) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse72781.2026.00018
Common Sense in Large Language Models
Kejriwal, M. (2026) · ACM eBooks · doi:10.1145/3789270
Kejriwal, M. (2026) · ACM eBooks · doi:10.1145/3789270.3789279
(2026) · ACM eBooks · doi:10.1145/3789270.3844418
Kejriwal, M. (2026) · ACM eBooks · doi:10.1145/3789270.3789278
Bench of Euler: A Benchmark for Evaluating the Problem-Solving Abilities of Large Language Models
Dutta, A., Priya, S., Ramamoorthy, A., & Kumar, P. K. (2026) · AppliedMath · doi:10.3390/appliedmath6090143
Ghorbel, A., Ayed, N. E. B., Letsholo, K. J., Jemmali, M., Hosam, O., & Osman, N. (2026) · IEEE Access · doi:10.1109/access.2026.3730807 · Cited by 1
Vu, M.-S., Pham, T.-T., & Hong-Viet, T. (2026) · Machine Learning with Applications · doi:10.1016/j.mlwa.2026.100987 · Cited by 1
So, C. C., Lee, C. H., Yang, D., Loh, A. W.-K., & Wu, C. (2026) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2026.3731978
Skill-Based Architecture for Reusable, AI-Native Enterprise Modules: An Empirical Study
Jathar, N. (2026) · IEEE International Conference on Information Reuse and Integration (IRI) · doi:10.1109/iri69576.2026.00012
Dong, Y., Lv, Z., Liu, S., Chen, M., Meng, X., Wang, S., & Liu, X. (2026) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s021819402650083x
Zheng, D., Wang, Y., Shi, E., Liu, X., Ma, Y., Zhang, H., & Zheng, Z. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773271 · Cited by 41
TACO: Trust Assessment of Large Language Models in Coding Assistance Tasks
Weng, S., Feng, Y., Li, J., Yin, Y., Zhang, Z., Liu, L., & Liu, J. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3787835
Enhancing Issue Localization Agent with Tool-Interactive Training
Ma, Z., Peng, C., Zeng, Q., Gao, P., Zou, Y., & Xie, B. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773137 · Cited by 2
Robsky, A., Lavitas, L., & Wang, Y. (2026) · Springer eBooks · doi:10.1007/978-3-032-26749-8_4
SpecGuru: Hierarchical LLM-Driven API Points-to Specification Generation with Self-Validation
Kan, S., Li, Y., Cheng, X. Y., & Sui, Y. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773209 · Cited by 1
Self-citationWAQ-LLM: Optimizing Multi-Instance LLM Deployment via Workload-Aware Queueing Model
Lai, J., Luo, Y., & Wang, Q. (2026) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3832810.3832865
Code generation for legal metadata extraction: a decomposition-based in-context learning approach
Singhal, A., & Breaux, T. D. (2026) · Requirements Engineering · doi:10.1007/s00766-026-00468-7 · Cited by 4
SEBGen: A Self-Evolving Benchmark Generation Framework for Software Engineering Agents
Huang, J., Feng, Z., Xiong, D., Ren, C., & Geng, Y. (2026) · Electronics · doi:10.3390/electronics15204663
A Multi-Modal Intelligent Vehicle System with Unified Visual Perception and Motion Control
Wang, H., Cao, L., & Zhang, H. (2026) · Actuators · doi:10.3390/act15090496
RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL System
Wang, Y., Xie, T., Shen, K., Wang, M., & Yang, L. (2026) · arXiv · doi:10.48550/arxiv.2602.02488
Twin: Playing an Unknown Game with a Test-Time Digital Twin
Skoutnev, A., Acharya, K., Longhitano, G., Udell, M., Ellis, K., & Drori, I. (2026) · arXiv · doi:10.48550/arxiv.2608.14490
NeuroProlog: Multi-Task Fine-Tuning for Neurosymbolic Mathematical Reasoning via the Cocktail Effect
Zunjare, P., & Hsiao, M. (2026) · arXiv · doi:10.48550/arxiv.2603.02504
Pham, M. V. T., Phan, H. N., Phan, H. N., Chi, C. L., Nguyen, T. N., & Bui, N. D. Q. (2025) · arXiv · doi:10.48550/arxiv.2504.14757
“…(LLMs) have made significant strides in automating software engineering (SE) tasks, from code generation (Bui et al., 2023; Chen et al., 2021; Li et al., 2022b; Manh et al., 2023; To et al., 2023; Wang et al., 2023b; Wei et al., 2023; Zhuo et al., 2024) to bug fixing (Jimenez et al., 2023;…”
LLM-as-a-Verifier: A General-Purpose Verification Framework
Kwok, J., Li, S., Atreya, P., Liu, Y., Jiang, Y., Finn, C., Pavone, M., Stoica, I., & Mirhoseini, A. (2026) · arXiv · doi:10.48550/arxiv.2607.05391
“Repeated sampling and best-of-𝑁 selection further scale candidate pools for code generation [40], general reasoning [3, 41], parallel self-verification [5], and inference-aware training [42].”
Metadata
- Authors
- Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., Hubert, T., Choy, P., de Masson d’Autume, C., Babuschkin, I., Chen, X., Huang, P.-S., Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D. J., Sutherland Robson, E., Kohli, P., de Freitas, N., Kavukcuoglu, K., & Vinyals, O.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- DeepMind, London, UK.
- Corresponding author
- Li, Yujia
- Choi, David
- Chung, Junyoung
- Kushman, Nate
- Schrittwieser, Julian
- Leblond, Rémi
- Eccles, Tom
- Keeling, James
- Gimeno, Felix
- Choy, Peter
- Vinyals, Oriol
- ORCID
- Venue
- Science
- Published
- 2022-12-09
- Volume
- 378
- Issue
- 6624
- Pages
- 1092-1097
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W4226287673
- Cited by
- 1,043
- References
- 34
- Citations by year
- 2026: 269 · 2025: 319 · 2024: 237 · 2023: 186 · 2022: 30
- FWCI
- 191.10
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- automatic code generation, competitive programming, programming competition, Codeforces, transformer language models, AI-assisted programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.