Measuring Coding Challenge Competence With APPS
Open accessAbstract
While programming is one of the most broadly applicable skills in modern society, modern machine learning models still cannot code solutions to basic problems. Despite its importance, there has been surprisingly little work on evaluating code generation, and it can be difficult to accurately assess code generation performance rigorously. To meet this challenge, we introduce APPS, a benchmark for code generation. Unlike prior work in more restricted settings, our benchmark measures the ability of models to take an arbitrary natural language specification and generate satisfactory Python code. Similar to how companies assess candidate software developers, we then evaluate models by checking their generated code on test cases. Our benchmark includes 10,000 problems, which range from having simple one-line solutions to being substantial algorithmic challenges. We fine-tune large language models on both GitHub and our training set, and we find that the prevalence of syntax errors is decreasing exponentially as models improve. Recent models such as GPT-Neo can pass approximately 20% of the test cases of introductory problems, so we find that machine learning models are now beginning to learn how to code. As the social significance of automatic code generation increases over the coming years, our benchmark can provide an important measure for tracking advancements.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
HumanEval, MBPP, and APPS score function-level synthesis.
“HumanEval [5], MBPP [3], and APPS [13] score function-level synthesis; CrossCodeEval [11] targets cross-file completion; CodeClash [33] and PinchBench [18] probe further axes of coding capability.”
From Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks (2026) · cites this paper as [13]
The APPS Dataset (p. 4) says some APPS problems use a call-based format with a function header, supporting the function-level description for those problems. Model Output (p. 5) also describes standard-input problems that require output to STDOUT, so APPS is not exclusively a function-level synthesis benchmark. This source does not establish what HumanEval or MBPP score; those benchmarks have separate citations in the sentence. (LLM, conf=0.88)
Experimental — accuracy not yet measuredExisting works have used the Computational Accuracy metric to evaluate generative code models, but not other metrics regarding characteristics of generated artifacts and run-time efficiency.
“For example, for the Performance category, existing works have used the Computational Accuracy metric to evaluate generative code models [9,15,33,88], but not other metrics we uncovered regarding the characteristics of the generated artifacts and run-time efficiency.”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [33]
Section 4.2 (p. 7) supports the computational-accuracy part: APPS evaluates generated code using test-case average and strict accuracy. But Section 4.3 (p. 8) also measures the frequency of syntax errors, a characteristic of generated code, so the claim that this work used no such other metric is too broad. The cited passages do not establish a run-time-efficiency metric. (LLM, conf=0.88)
Experimental — accuracy not yet measuredAPPS, CodeEditorBench-switch-java, CodeEditorBench-debug-py, CodeEditorBench-switchc, and CodeEditorBench-switch-py are mainly derived from LeetCode.
“Benchmarks such as APPS, CodeEditorBench -switch-java , CodeEditorBench -debug-py , CodeEditorBench -switchc , and CodeEditorBench -switch-py are mainly derived from LeetCode.”
From LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks (2025) · cites this paper as [31]
The Auxiliary Dataset Information (p. 13) lists LeetCode among seven websites from which APPS problems were scraped, but does not say LeetCode supplied most of them. Dataset Construction (p. 4) also names several other sources. The paper does not address the named CodeEditorBench variants, so it cannot support the claim about their origins. (LLM, conf=0.86)
Experimental — accuracy not yet measuredThe development of evaluations in tandem with language models has substantially contributed to the rapid advancement of these models.
“The development of evaluations in tandem with language models (LMs) has substantially contributed to the rapid advancement of these models [29, 11, 7, 25, 81, 27, 72].”
From SciCode: A Research Coding Benchmark Curated by Scientists (2024) · cites this paper as [27]
Related Work (p. 3) describes rapidly improving language-model performance across benchmarks and introduces APPS as a new evaluation benchmark. It does not say that developing evaluations substantially contributed to those advances; the claimed causal contribution is unsupported. (LLM, conf=0.78)
Experimental — accuracy not yet measuredExecution-based metrics that require providing test cases are expensive to conduct in practice.
“Executionbased metrics (Kulal et al., 2019;Hao et al., 2022;Hendrycks et al., 2021;Rozière et al., 2020;Dong et al., 2023) that require providing test cases are expensive to conduct in practice”
From CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation (2023) · cites this paper as Hendrycks et al., 2021
Section 4.2 (p. 7) confirms that APPS uses execution-based metrics requiring test cases. Section 3 (p. 4) describes six months of dataset preparation and says executing arbitrary Python code is challenging, but the paper does not establish that these metrics are generally expensive to conduct in practice; Section 4.2 instead emphasizes that test cases permit automatic evaluation without manual analysis. (LLM, conf=0.81)
Experimental — accuracy not yet measured
Researchers have used software testing to evaluate the behaviour of completions, using competitive programming sites and hand-written programming problems as data sources.
“Due to the diversity of potential solutions to a multi-line completion task, researchers have used software testing to evaluate the behaviour of completions.Competitive programming sites have been used as a source of such data [8,9] as well as hand-written programming problems [5].”
From Productivity Assessment of Neural Code Completion (2022) · cites this paper as [8]
HumanEval and other benchmarks only consist of programming problems in a single language.
“HumanEval [7]-developed by OpenAI for evaluating Codex-and other benchmarks [2,13,20] only consist of programming problems in a single language”
From CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X (2023) · cites this paper as [13]
The APPS dataset includes comprehensive test cases designed to evaluate the functional correctness of generated code.
“We specifically selected the APPS dataset because it includes comprehensive test cases designed to evaluate the functional correctness of generated code.”
From LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks (2025) · cites this paper as [31]
On APPS Introductory and LiveCodeBench v6, JEPA-Anything-guided candidate selection achieved final full-test success rates of 46.37% and 33.05%, respectively, exceeding random selection and the static verifier.
“On APPS Introductory [17] and LiveCodeBench v6 [18], JEPA-Anything-guided candidate selection achieved final full-test success rates of 46 . 37% and 33 . 05%, respectively, exceeding random selection and the static verifier (Figure 3e,f).”
From RSI-Anything: Unlocking Recursive Self-Improvement with World Models (2026) · cites this paper as [17]
The APPS test split instances have more tests per solution and are generally harder than other APPS instances.
“For APPS, we focused on the 5,000 instances in the test split (Hendrycks et al., 2021), as these have more tests per solution and generally harder tasks.”
From A benchmark for vericoding: formally verified program synthesis (2025) · cites this paper as Hendrycks et al., 2021
The significant overhead of executing numerous test cases and the heightened security risks during the execution process remain unresolved.
“the significant overhead of executing numerous test cases and the heightened security risks during the execution process remain unresolved.”
From CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation (2023) · cites this paper as Hendrycks et al., 2021
Section 3 (p. 4) says executing and evaluating arbitrary Python code is challenging, while the Introduction (p. 2) and Metrics (p. 7) describe using a large bank of test cases. None of these passages says that test-case execution has significant overhead or that security risks during execution remain unresolved; searches for execution costs and security risks found no such passage. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 32 · Cited by 142
Research works printed but not linked: 14
- [2]2. Hui Liu, Mingzhu Shen, Jiaqi Zhu, Nan Niu, Ge Li, and Lu Zhang. Deep learning based program generation from requirements text: Are we there yet? IEEE Transactions on Software Engineering , 2020a. - J. Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. LogiQA: A challenge dataset for machine reading comprehension with logical reasoning. In IJCAI , 2020b. - I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR , 2019.
- [6]6. Yusuke Oda, Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. Learning to generate pseudo-code from source code using statistical machine translation. In International Conference on Automated Software Engineering (ASE) , 2015.
- [8]8. Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem proving. ArXiv , abs/2009.03393, 2020. · arXiv
- [9]9. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.
- [11]11. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , 2020.
- [12]12. Veselin Raychev, Pavol Bielik, and Martin T. Vechev. Probabilistic model for code with decision trees. Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications , 2016.
- [13]13. Mohammad Raza, Sumit Gulwani, and Natasa Milic-Frayling. Compositional program synthesis from natural language and examples. In IJCAI , 2015.
- [14]14. Shuo Ren, Daya Guo, Shuai Lu, L. Zhou, Shujie Liu, Duyu Tang, M. Zhou, A. Blanco, and S. Ma. Codebleu: a method for automatic evaluation of code synthesis. ArXiv , abs/2009.10297, 2020. - L. Tang and R. Mooney. Using multiple clause constructors in inductive logic programming for semantic parsing. In ECML , 2001. · arXiv
- [16]16. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, L. Kaiser, and Illia Polosukhin. Attention is all you need. ArXiv , abs/1706.03762, 2017. · arXiv
- [17]17. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In NeurIPS , 2019a.
- [18]18. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In NeurIPS , 2019b.
- [19]19. Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. Ratsql: Relation-aware schema encoding and linking for text-to-sql parsers. arXiv preprint arXiv:1911.04942 , 2019c. · arXiv
- [20]20. Michihiro Yasunaga and Percy Liang. Graph-based, self-supervised program repair from diagnostic feedback. ArXiv , abs/2005.10636, 2020. · arXiv
- [21]21. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint arXiv:1809.08887 , 2018. - Maksym Zavershynskyi, A. Skidanov, and Illia Polosukhin. Naps: Natural program synthesis dataset. 2nd Workshop on Neural Abstract Machines and Program Induction , 2018. - J. Zelle and R. Mooney. Learning to parse database queries using inductive logic programming. In AAAI/IAAI, Vol. 2 , 1996. - Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?, 2019. | | Hearthstone | Django | NAPS | APPS | |--------------------------|---------------|----------|------------|----------------------| | Programming Language | Python | Python | UAST | Python | | Test Cases | 5 | 5 | | | | Number of Programs | 665 | 18,805 | 17,477 | 232,421 | | Lines per Program (Avg.) | 7.7 | 1 | 21.7 | 18.0 | | Number of Exercises | 665 | 18,805 | 2,231 | 10,000 | | Text Input | Card Text | Comment | Pseudocode | Problem Descriptions | Table 4: Further comparisons of APPS with previous datasets. | Model | Introductory | Top-5 Test Case Average Interview | Competitive | Average | Introductory | Top-5 Strict Interview | Accuracy Competition | Average | |--------------|----------------|-------------------------------------|---------------|-----------|----------------|--------------------------|------------------------|-----------| | GPT-2 0.1B | 13.81 | 10.97 | 7.03 | 10.75 | 2.70 | 0.73 | 0.00 | 1.02 | | GPT-2 1.5B | 16.86 | 13.84 | 9.01 | 13.48 | 3.60 | 1.03 | 0.00 | 1.34 | | GPT-Neo 2.7B | 19.89 | 13.19 | 9.90 | 13.87 | 5.50 | 0.80 | 0.00 | 1.58 | Table 5: Top-5 performance of GPT-2 models and GPT-Neo. Taking the best of five candidate solutions markedly improves performance. · arXiv
- Authors
- Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., & Steinhardt, J.
- Affiliations
- University of California, Berkeley (US)
- University of Chicago (US)
- University of Illinois Urbana-Champaign (US)
- Cornell University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2105.09938
- OpenAlex
- W3161457214
- Cited by
- 142
- References
- 32
- Citations by year
- 2026: 9 · 2025: 24 · 2024: 27 · 2023: 69 · 2022: 8 · 2021: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- GPT-Neo, autoregressive language models, GPT-2, GPT-3, model parallelism, data parallelism, TPU, distributed training, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
