Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.

Competition-level code generation with AlphaCode

Open access
Li, Yujia, Choi, David, Chung, Junyoung, Kushman, Nate, Schrittwieser, Julian, Leblond, Rémi, Eccles, Tom, Keeling, James, Gimeno, Felix, Dal Lago, Agustin, Hubert, Thomas, Choy, Peter, de Masson d’Autume, Cyprien, Babuschkin, Igor, Chen, Xinyun, Huang, Po-Sen, Welbl, Johannes, Gowal, Sven, Cherepanov, Alexey, Molloy, James, Mankowitz, Daniel J., Sutherland Robson, Esme, Kohli, Pushmeet, de Freitas, Nando, Kavukcuoglu, Koray, Vinyals, Oriol · 2022 · Science

Abstract

Programming is a powerful and ubiquitous problem-solving tool. Systems that can assist programmers or even generate programs themselves could make programming more productive and accessible. Recent transformer-based neural network models show impressive code generation abilities yet still perform poorly on more complex tasks requiring problem-solving skills, such as competitive programming problems. Here, we introduce AlphaCode, a system for code generation that achieved an average ranking in the top 54.3% in simulated evaluations on recent programming competitions on the Codeforces platform. AlphaCode solves problems by generating millions of diverse programs using specially trained transformer-based networks and then filtering and clustering those programs to a maximum of just 10 submissions. This result marks the first time an artificial intelligence system has performed competitively in programming competitions.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1126/science.abq1158

Cites

60 of 60 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

5 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 6

  1. A. Church, Application of recursive arithmetic to the problem of circuit synthesis.. J. Symb. Log. 28, 289–290 (1963).
  2. Y. Nandwani D. Jindal Mausam P. Singla ?oNeural learning of one-of-many solutions for combinatorial problems in structured output spaces ?? International Conference on Learning Representations (ICLR 2021) 3 to 7 May 2021.
  3. A. Holtzman J. Buys L. Du M. Forbes Y. Choi ?oThe curious case of neural text degeneration ?? Proceedings of the 8th International Conference on Learning Representations (ICLR 2020) 26 April to 1 May 2020.
  4. J. Ganitkevitch B. Van Durme C. Callison-Burch ?oPPDB: The paraphrase database ?? Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Atlanta Georgia 9 to 15 June 2013 (Association for Computational Linguistics 2013) pp. 758–764.
  5. A. Solar-Lezama ?oProgram synthesis by sketching ?? thesis University of California Berkeley (2008).
  6. J. Cai R. Shin D. Song ?oMaking neural programming architectures generalize via recursion ?? Proceedings of the 5th International Conference on Learning Representations (ICLR) Toulon France 24 to 26 April 2017.

Cited by

See all 1,083

Metadata

Authors
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., Hubert, T., Choy, P., de Masson d’Autume, C., Babuschkin, I., Chen, X., Huang, P.-S., Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D. J., Sutherland Robson, E., Kohli, P., de Freitas, N., Kavukcuoglu, K., & Vinyals, O.
Affiliations
  • Google DeepMind (United Kingdom) (GB)
  • DeepMind, London, UK.
Corresponding author
  • Li, Yujia
  • Choi, David
  • Chung, Junyoung
  • Kushman, Nate
  • Schrittwieser, Julian
  • Leblond, Rémi
  • Eccles, Tom
  • Keeling, James
  • Gimeno, Felix
  • Choy, Peter
  • Vinyals, Oriol
Venue
Science
Published
2022-12-09
Volume
378
Issue
6624
Pages
1092-1097
ISSN
  • 0036-8075 (ISSN-L)
  • 1095-9203
OpenAlex
W4226287673
Cited by
1,043
References
34
Citations by year
2026: 269 · 2025: 319 · 2024: 237 · 2023: 186 · 2022: 30
FWCI
191.10
Type
journal article
Language
en
Open access
Locations
  • Science (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
automatic code generation, competitive programming, programming competition, Codeforces, transformer language models, AI-assisted programming

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.