The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Competition-level code generation with AlphaCode
Open accessAbstract
Programming is a powerful and ubiquitous problem-solving tool. Systems that can assist programmers or even generate programs themselves could make programming more productive and accessible. Recent transformer-based neural network models show impressive code generation abilities yet still perform poorly on more complex tasks requiring problem-solving skills, such as competitive programming problems. Here, we introduce AlphaCode, a system for code generation that achieved an average ranking in the top 54.3% in simulated evaluations on recent programming competitions on the Codeforces platform. AlphaCode solves problems by generating millions of diverse programs using specially trained transformer-based networks and then filtering and clustering those programs to a maximum of just 10 submissions. This result marks the first time an artificial intelligence system has performed competitively in programming competitions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AlphaCode, Self-Debugging and CodeRL are effective in fixing less-complex program bugs, but they rely upon ground truth test cases that invalidate pass@1 eligibility, and do not use self-reflection to bridge the gap between error identification and implementation improvement.”
From Reflexion: Language Agents with Verbal Reinforcement Learning (2023) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large language models are increasingly capable of complex programming tasks and are already being used to accelerate AI research, from generating training data to serving as programming tools [1, 2, 3, 4, 5, 6].”
From RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts (2024) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Generative AI, such as OpenAI’s ChatGPT, has rapidly emerged as a disruptive technology capable of achieving human-level performance on a broad range of tasks (1–5).”
From Generative AI without guardrails can harm learning: Evidence from high school mathematics (2025) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Li et al. [27] showed that the accuracy of AlphaCode increases from close to 0% zero-shot to over 15% with 1,000 retries and over 30% with a million retries (accuracy is measured by how often one of the top 10 answers generated by the model is correct).”
From AI Agents That Matter (2024) · cites this paper as Li et al. [27]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Evaluation covered 200 CodeContests tasks [15] across three seeds and up to three feedback rounds, with online tests supplying feedback and hidden tests measuring final success.”
From RSI-Anything: Unlocking Recursive Self-Improvement with World Models (2026) · cites this paper as [15]
Experimental — accuracy not yet measured
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Self-citationDecoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“Following Kaplan et al. (2020), we adjusted the amount of training for other model sizes such that larger models are trained more and smaller models are trained less to optimize the use of compute.”
MethodGrandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455
Self-citationResnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., et al. (2009) · Communications of the ACM · doi:10.1145/1592761.1592779 · Cited by 3,333
Kudo, T., & Richardson, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d18-2012 · Cited by 2,869
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Self-citationHierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242
Understanding Back-Translation at Scale
Edunov, S., Ott, M., Auli, M., & Grangier, D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1045 · Cited by 993
Extracting Training Data from Large Language Models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07805 · Cited by 727
A Syntactic Neural Model for General-Purpose Code Generation
Yin, P., & Neubig, G. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1041 · Cited by 623
A Syntactic Neural Model for General-Purpose Code Generation
Yin, P., & Neubig, G. (2017) · arXiv · doi:10.48550/arxiv.1704.01696 · Cited by 585
Automating string processing in spreadsheets using input-output examples
Gulwani, S. (2011) · ACM SIGPLAN Notices · doi:10.1145/1925844.1926423 · Cited by 581
Training Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575
Ethical and social risks of harm from Language Models
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04359 · Cited by 477
Self-citationAsleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions
Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833571 · Cited by 476
IntelliCode compose: code generation using transformer
Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417058 · Cited by 474 · cited 2×
“Lanza, 2008) and large amounts of existing code data (Aye et al., 2021; Svyatkovskiy et al., 2020).”
BackgroundOn the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Devanbu, P. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227135 · Cited by 425
Program Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388
Improving language models by retrieving from trillions of tokens
Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Driessche, G. V. D., Lespiau, J.-B., Damoc, B., Clark, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04426 · Cited by 382
Self-citationLearning from examples to improve code completion systems
Bruch, M., Monperrus, M., & Mezini, M. (2009) · Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering · doi:10.1145/1595696.1595728 · Cited by 378
Ten Lessons From Three Generations Shaped Google’s TPUv4i : Industrial Product
Jouppi, N. P., Hyun Yoon, D., Ashcraft, M., Gottscho, M., Jablin, T. B., Kurian, G., Laudon, J., Li, S., Ma, P., Ma, X., et al. (2021) · 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca52012.2021.00010 · Cited by 360
Gulwani, S., Polozov, O., & Singh, R. (2017) · Foundations and Trends in Programming Languages · doi:10.1561/2500000010 · Cited by 319
Latent Predictor Networks for Code Generation
Ling, W., Blunsom, P., Grefenstette, E., Hermann, K. M., Kočiský, T., Wang, F., & Senior, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1057 · Cited by 307
RobustFill: Neural Program Learning under Noisy I/OPreprint
Devlin, J., Uesato, J., Bhupatiraju, S., Singh, R., Mohamed, A., & Kohli, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.07469 · Cited by 305
Self-citationApplication of Theorem Proving to Problem Solving
Green, C. (1969) · doi:10.21236/ada459656 · Cited by 289
Toward automatic program synthesis
Manna, Z., & Waldinger, R. J. (1971) · Communications of the ACM · doi:10.1145/362566.362568 · Cited by 285
The adverse effects of code duplication in machine learning models of code
Allamanis, M. (2019) · Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software · doi:10.1145/3359591.3359735 · Cited by 284
Matsakis, N. D., & Klock, F. S. (2014) · ACM SIGAda Ada Letters · doi:10.1145/2692956.2663188 · Cited by 268
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.11446 · Cited by 235
Self-citationCodeBLEU: a Method for Automatic Evaluation of Code Synthesis
Ren, S., Guo, D., Lu, S., Zhou, L., Liu, S., Tang, D., Sundaresan, N., Zhou, M., Blanco, A., & Ma, S. (2020) · arXiv · doi:10.48550/arxiv.2009.10297 · Cited by 214
DeepCoder: Learning to Write Programs
Balog, M., Gaunt, A. L., Brockschmidt, M., Nowozin, S., & Tarlow, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01989 · Cited by 212
Code completion with statistical language models
Raychev, V., Vechev, M., & Yahav, E. (2014) · ACM SIGPLAN Notices · doi:10.1145/2666356.2594321 · Cited by 189
Estimating Training Data Influence by Tracing Gradient DescentPreprint
Pruthi, G., Liu, F., Sundararajan, M., & Kale, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.08484 · Cited by 175
A Systematic Review on Code Clone Detection
Ain, Q. U., Butt, W. H., Anwar, M. W., Azam, F., & Maqbool, B. (2019) · IEEE Access · doi:10.1109/access.2019.2918202 · Cited by 165
How Program History Can Improve Code Completion
Robbes, R., & Lanza, M. (2008) · 2008 23rd IEEE/ACM International Conference on Automated Software Engineering · doi:10.1109/ase.2008.42 · Cited by 155 · cited 1×
“More recent intelligent code completion systems can learn from history (Robbes and Lanza, 2008) and large amounts of existing code data (Aye et al., 2021; Svyatkovskiy et al., 2020).”
BackgroundPyMT5: multi-mode translation of natural language and Python code with transformers
Clement, C. B., Drain, D., Timcheck, J., Svyatkovskiy, A., & Sundaresan, N. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.728 · Cited by 148
Measuring Coding Challenge Competence With APPS
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.09938 · Cited by 142
Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagation
Huang, P.-S., Stanforth, R., Welbl, J., Dyer, C., Yogatama, D., Gowal, S., Dvijotham, K., & Kohli, P. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1419 · Cited by 129
Self-citationNeural Sketch Learning for Conditional Program GenerationPreprint
Murali, V., Qi, L., Chaudhuri, S., & Jermaine, C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.05698 · Cited by 111
SPoC: Search-based Pseudocode to Code
Kulal, S., Pasupat, P., Chandra, K., Lee, M., Padon, O., Aiken, A., & Liang, P. (2019) · arXiv · doi:10.48550/arxiv.1906.04908 · Cited by 103
Fast Transformer Decoding: One Write-Head is All You Need
Shazeer, N. (2019) · arXiv · doi:10.48550/arxiv.1911.02150 · Cited by 101
CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks
Puri, R., Kung, D. S., Janssen, G., Zhang, W., Domeniconi, G., Zolotov, V., Dolby, J., Chen, J., Choudhury, M., Decker, L., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.12655 · Cited by 92
Learning to Synthesize Programs as Interpretable and Generalizable PoliciesPreprint
Trivedi, D., Zhang, J. M., Sun, S.-H., & Lim, J. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.13643 · Cited by 64
Text Generation by Learning from DemonstrationsPreprint
Pang, R. Y., & He, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.07839 · Cited by 53
Learning to Complete Code with SketchesPreprint
Guo, D., Svyatkovskiy, A., Yin, J., Duan, N., Brockschmidt, M., & Allamanis, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.10158 · Cited by 35
Learning Autocompletion from Real-World Datasets
Aye, G. A., Kim, S., & Li, H. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip52600.2021.00022 · Cited by 29
NAPS: Natural Program Synthesis DatasetPreprint
Zavershynskyi, M., Skidanov, A., & Polosukhin, I. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.03168 · Cited by 29
Solving Linear Algebra by Program SynthesisPreprint
Drori, I., & Verma, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.08171 · Cited by 18
Solving Probability and Statistics Problems by Program SynthesisPreprint
Tang, L., Ke, E., Singh, N., Verma, N., & Drori, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.08267 · Cited by 13
Softmax Tempering for Training Neural Machine Translation ModelsPreprint
Dabre, R., & Fujita, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.09372 · Cited by 6
Caballero, E., OpenAI, & Sutskever, I. (2016) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.5664030 · Cited by 5
Caballero, E., OpenAI, & Sutskever, I. (2016) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.5665051 · Cited by 4
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno Gil, F. A., Dal Lago, A., et al. (2022) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.6975436 · Cited by 1
Self-citation
Research works printed but not linked: 6
- A. Church, Application of recursive arithmetic to the problem of circuit synthesis.. J. Symb. Log. 28, 289–290 (1963).
- Y. Nandwani D. Jindal Mausam P. Singla ?oNeural learning of one-of-many solutions for combinatorial problems in structured output spaces ?? International Conference on Learning Representations (ICLR 2021) 3 to 7 May 2021.
- A. Holtzman J. Buys L. Du M. Forbes Y. Choi ?oThe curious case of neural text degeneration ?? Proceedings of the 8th International Conference on Learning Representations (ICLR 2020) 26 April to 1 May 2020.
- J. Ganitkevitch B. Van Durme C. Callison-Burch ?oPPDB: The paraphrase database ?? Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Atlanta Georgia 9 to 15 June 2013 (Association for Computational Linguistics 2013) pp. 758–764.
- A. Solar-Lezama ?oProgram synthesis by sketching ?? thesis University of California Berkeley (2008).
- J. Cai R. Shin D. Song ?oMaking neural programming architectures generalize via recursion ?? Proceedings of the 5th International Conference on Learning Representations (ICLR) Toulon France 24 to 26 April 2017.
Cited by
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
“Recent work has shown that large language models can be useful for coding tasks including competitive programming (Li et al., 2022), code completion (Chen et al., 2021), and program synthesis from natural language specifications (Chen et al., 2021; Austin et al., 2021).”
BackgroundFeuerriegel, S., Hartmann, J., Janiesch, C., & Zschech, P. (2024) · Business & Information Systems Engineering · doi:10.1007/s12599-023-00834-7 · Cited by 2,267
“…(Rombach et al. 2022) for text-to-image generation, MusicLM (Agostinelli et al. 2023) for text-to-music generation, Codex (Chen et al. 2021) and AlphaCode (Li et al. 2022) for text-to-code generation, and as mentioned above GPT-4 for image-to-text as well as text-to-text generation (OpenAI 2023a).”
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3744746 · Cited by 1,257
ChatGPT for Education and Research: Opportunities, Threats, and Strategies
Rahman, M. M., & Watanobe, Y. (2023) · Applied Sciences · doi:10.3390/app13095783 · Cited by 1,080
“Alphacode [30] is a state-of-the-art LLM developed and trained specifically to support competitive-level programming.”
BackgroundLarge Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695988 · Cited by 1,029
Metadata
- Authors
- Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., Hubert, T., Choy, P., de Masson d’Autume, C., Babuschkin, I., Chen, X., Huang, P.-S., Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D. J., Sutherland Robson, E., Kohli, P., de Freitas, N., Kavukcuoglu, K., & Vinyals, O.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- DeepMind, London, UK.
- Corresponding author
- Li, Yujia
- Choi, David
- Chung, Junyoung
- Kushman, Nate
- Schrittwieser, Julian
- Leblond, Rémi
- Eccles, Tom
- Keeling, James
- Gimeno, Felix
- Choy, Peter
- Vinyals, Oriol
- ORCID
- Venue
- Science
- Published
- 2022-12-09
- Volume
- 378
- Issue
- 6624
- Pages
- 1092-1097
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W4226287673
- Cited by
- 1,043
- References
- 34
- Citations by year
- 2026: 269 · 2025: 319 · 2024: 237 · 2023: 186 · 2022: 30
- FWCI
- 191.10
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- automatic code generation, competitive programming, programming competition, Codeforces, transformer language models, AI-assisted programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.