The DOI matches its DataCite record; 91 of the 96 references it cites were found in a registry or an index.
Evaluating Large Language Models Trained on Code
Open accessAbstract
We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“OpenAI's Codex model [Chen et al. 2021], which contains 12 billion model parameters and is trained on 54 million software repositories on GitHub, is able to correctly solve 30-70% of novel Python problems”
From Grounded Copilot: How Programmers Interact with Code-Generating Models (2023) · cites this paper as [Chen et al. 2021]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Codex was initially tested on the HumanEval dataset containing 164 programming problems, where it is asked to write the function body from a docstring [8] and achieves 37.7% accuracy with a single generation.”
From Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming (2024) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Offline evaluation where the system is shown a partial snippet of code and then asked to complete it is difficult not least because for longer completions there are many acceptable alternatives and no straightforward mechanism for labeling them automatically [5].”
From Productivity Assessment of Neural Code Completion (2022) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, for the Performance category, existing works have used the Computational Accuracy metric to evaluate generative code models [9,15,33,88], but not other metrics we uncovered regarding the characteristics of the generated artifacts and run-time efficiency.”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Similar studies in the code space are limited to the number of few-shot examples in the prompt, and the effect of length of prompt on performance [9,15].”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [15]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Offline evaluation, where the system is shown a partial snippet of code and then asked to complete it, is difficult not least because for longer completions there are many acceptable alternatives and no straightforward mechanism for labeling them automatically.5”
From Measuring GitHub Copilot's Impact on Productivity (2024) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, LLMs struggle to produce the correct action code consistently in one shot [41].”
From Voyager: An Open-Ended Embodied Agent with Large Language Models (2023) · cites this paper as [41]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With this formulation, we can easily estimate pass@ k with low variance across all k ≤ n .”
From Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (2025) · cites this paper as Chen et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, existing coding benchmarks, such as HumanEval (Chen et al., 2021), mostly involve self-contained problems that can be solved in a few lines of code.”
From SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023) · cites this paper as Chen et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It is a diverse collection of 164 problems, where all problems have tests to check correctness, and most have examples or doctests as part of the prompt.”
From MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, previous work has found that textual similarity metrics correlate only weakly with code correctness [1,2,27], highlighting the importance of benchmark suites with unit tests.”
From MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current benchmarks for evaluating neural code models focus on only a small subset of programming languages, excluding many popular languages such as Go or Rust.”
From Measuring The Impact Of Programming Language Distribution (2023) · cites this paper as Chen et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Alpha-Code (Li et al., 2022), Codex (Chen et al., 2021), PaLM (Chowdhery et al., 2022), and other works (Nijkamp et al., 2022;Fried et al., 2022;Allal et al., 2023;Christopoulou et al., 2022) have shown that decoder-only code language models achieve exceptional performance on a wide range of tasks.”
From Measuring The Impact Of Programming Language Distribution (2023) · cites this paper as Chen et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Previous works [7,17] have already discovered that there's a trade-off between exploration and exploitation: When the budget is small, it is better to use a low temperature to ensure accuracy on easy problems.When the budget is large, instead, adjusting a higher temperature makes the model more likely to find at least one solution for difficult problems.”
From CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X (2023) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Specifically, each task can have different requirements, operations, and dependencies, which makes it difficult to represent them in a unified yet flexible manner (Chen et al., 2021;Cobbe et al., 2021;Yang et al., 2018;Luo et al., 2018).”
From AFlow: Automating Agentic Workflow Generation (2024) · cites this paper as Chen et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The majority of evaluation protocols largely simplify experimental assumptions and design static, pre-defined rules, with relatively short and isolated task settings (Cobbe et al., 2021; Chen et al., 2021; Budzianowski et al., 2018).”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Chen et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs such as the GPT-type transformers which underpin Codex [3] function by building probabilistic sequences of tokens based on the frequency of observed tokens in the training data [2]. In other words, they act as an 'autocomplete' tool.”
From Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent large language models (LLMs) have demonstrated strong capabilities in understanding and generating code, from competitive programming to repository-level software engineering.”
From Code as Agent Harness (2026) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Our experiments mostly target the Code-Davinci model, particularly Code-Davinci-002, which excels at translating natural language to code [14] and supports code completion as well as code insertion 4 .”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large language models are increasingly capable of complex programming tasks and are already being used to accelerate AI research, from generating training data to serving as programming tools [1, 2, 3, 4, 5, 6].”
From RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts (2024) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The development of evaluations in tandem with language models (LMs) has substantially contributed to the rapid advancement of these models [29, 11, 7, 25, 81, 27, 72].”
From SciCode: A Research Coding Benchmark Curated by Scientists (2024) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“HumanEval prompts are code-dense from the first token, with function signatures and type hints appearing early:”
From Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression (2026) · cites this paper as Chen et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Other studies further find that the bugs introduced by LLM-powered code completion are often more subtle [4, 34], and difficult to revert [42].”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as [4]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While it is highly unlikely that model-generated code will perform overtly malicious actions in our current settings with the Foundation Models (FMs) we employ, such code could still act destructively due to limitations in model capability or alignment (Rokon et al., 2020; Chen et al., 2021).”
From Automated Design of Agentic Systems (2024) · cites this paper as Chen et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Simultaneously, language models have made significant strides in coding tasks, solving most tasks in benchmarks such as HumanEval (Chen et al., 2021).”
From CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark (2024) · cites this paper as Chen et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“CLEVER includes hand-crafted Lean specifications of 161 programming tasks from the HUMANEVAL benchmark [4].”
From CLEVER: A Curated Benchmark for Formally Verified Code Generation (2025) · cites this paper as [4]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Modern LLMs substantially lower this barrier by enabling researchers to describe analytical tasks in plain language and receive executable code in return 20,21 .”
From Reimagining biomedical science workflows in the age of large language models (2026) · cites this paper as [20]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“GitHub Copilot provides code completions using OpenAI Codex [5] , which is a version of GPT-3 that has been tuned on publicly available source code.”
From Productivity Assessment of Neural Code Completion (2022) · cites this paper as [5]
The cited source says what the claim attributes to it.
In the citing paper
“Copilot is powered by the OpenAI Codex family of models [Chen et al. 2021], which are derived by fine-tuning GPT-3 [Brown et al. 2020] on publicly available Github repositories.”
From Grounded Copilot: How Programmers Interact with Code-Generating Models (2023) · cites this paper as [Chen et al. 2021]
The cited source says what the claim attributes to it.
In the citing paper
“HumanEval [5], MBPP [3], and APPS [13] score function-level synthesis; CrossCodeEval [11] targets cross-file completion; CodeClash [33] and PinchBench [18] probe further axes of coding capability.”
From Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks (2026) · cites this paper as [5]
The cited source says what the claim attributes to it.
In the citing paper
“Code data (e.g., from Stack Exchange and GitHub) substantially enhances program synthesis capabilities [37,38].”
From A Survey of Large Language Models (2026) · cites this paper as [37]
The cited source says what the claim attributes to it.
In the citing paper
“Copilot is powered by the OpenAI Codex family of models [Chen et al. 2021], which are derived by fine-tuning GPT-3 [Brown et al. 2020] on publicly available Github repositories.”
From Grounded Copilot: How Programmers Interact with Code-Generating Models (2023) · cites this paper as Chen et al. 2021
The cited source says what the claim attributes to it.
In the citing paper
“OpenAI's Codex model [Chen et al. 2021], which contains 12 billion model parameters and is trained on 54 million software repositories on GitHub, is able to correctly solve 30-70% of novel Python problems”
From Grounded Copilot: How Programmers Interact with Code-Generating Models (2023) · cites this paper as Chen et al. 2021
The cited source says what the claim attributes to it.
In the citing paper
“Due to the diversity of potential solutions to a multi-line completion task, researchers have used software testing to evaluate the behaviour of completions.Competitive programming sites have been used as a source of such data [8,9] as well as hand-written programming problems [5].”
From Productivity Assessment of Neural Code Completion (2022) · cites this paper as [5]
The cited source says what the claim attributes to it.
In the citing paper
“Large pre-trained language models perform remarkably well on tasks that can be done 'in one pass', such as generating realistic text (Brown et al., 2020) or synthesizing computer programs (Chen et al., 2021; Austin et al., 2021).”
From Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021) · cites this paper as Chen et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Prominent examples include Stable Diffusion (Rombach et al. 2022) for text-to-image generation, MusicLM (Agostinelli et al. 2023) for text-to-music generation, Codex (Chen et al. 2021) and AlphaCode (Li et al. 2022) for text-to-code generation, and as mentioned above GPT-4 for image-to-text as well as text-to-text generation (OpenAI 2023a).”
From Generative AI (2024) · cites this paper as Chen et al. 2021
The cited source says what the claim attributes to it.
In the citing paper
“[1] describe a 12B parameter version of Codex finetuned on 159 GB of deduplicated, filtered Python code from Github.”
From MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“Codex is a GPT-3 language model fine-tuned on code.[1] describe a 12B parameter version of Codex finetuned on 159 GB of deduplicated, filtered Python code from Github.”
From MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“For statistical reliability, we take 200 completions at each temperature and calculate average pass rate using the unbiased sampling estimator presented in [1].”
From MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“We evaluate a more recent Codex model (codedavinci-002) than the original paper and observe a large improvement on Python: a pass@1 rate of 45.9%, compared to 28.8% reported earlier [1].”
From MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“Prior work has shown that sampling the output of a code generation model significantly improves the correctness of generated code [1].”
From MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“Empirical results show these settings are optimal for Python [1], but it is possible that different settings would be better for other programming languages.”
From MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“HumanEval [7]-developed by OpenAI for evaluating Codex-and other benchmarks [2,13,20] only consist of programming problems in a single language”
From CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X (2023) · cites this paper as [7]
The cited source says what the claim attributes to it.
In the citing paper
“Subsequently, the same model is tasked to regenerate the code given only its own explanation.The second step allows us to score this task via code execution and measure pass@k (Chen et al., 2021) instead of evaluating the explanation itself using heuristic-based metrics like BLEU (Papineni et al., 2002) or ROUGE (Lin, 2004) which have major limitations”
From OctoPack: Instruction Tuning Code Large Language Models (2023) · cites this paper as Chen et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“We also checked for any contamination with HumanEval (Chen et al., 2021) but did not find any solution or docstring present in COMMITPACKFT.”
From OctoPack: Instruction Tuning Code Large Language Models (2023) · cites this paper as Chen et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Frontier models have achieved impressive performance in mathematics, coding, general knowledge, and commonsense reasoning (Hendrycks et al., 2021a;b; Chen et al., 2021; Sakaguchi et al., 2021; Yue et al., 2024).”
From Does Spatial Cognition Emerge in Frontier Models? (2024) · cites this paper as Chen et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“In code-generation tasks, SELF-REFINE improves the initial generation by up to absolute 13% when applied to strong code models such as Codex (code-davinci-002; Chen et al., 2021).”
From Self-Refine: Iterative Refinement with Self-Feedback (2023) · cites this paper as Chen et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“In tasks with automatic verifiers available, such as math [Hendrycks et al., 2021] and code [Chen et al., 2021], simply sampling LLMs multiple times can significantly improve the pass@k metric and hence boost the success rate of solving the tasks”
From Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial? (2025) · cites this paper as Chen et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“The modern pass@k metric and its unbiased estimator are due to Chen et al. [2021], with the earlier idea of counting success when at least one of k test-validated candidates passes under a fixed budget appearing in SPoC [Kulal et al., 2019].”
From DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks (2026) · cites this paper as Chen et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“We report two task-success metrics, both following the pass@ k lineage of Chen et al. [2021], whose 'solved if at least one of k sampled candidates passes' framing in turn builds on the budget-constrained candidate-selection idea of Kulal et al. [2019].”
From DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks (2026) · cites this paper as Chen et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“With four rollouts per task this is exactly the fixed-budget pass@4 of Chen et al. [2021]: when the number of samples equals the budget ( n = k = 4), the unbiased pass@ k estimator reduces to the indicator that at least one sample passed, so no separate estimator is needed.”
From DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks (2026) · cites this paper as Chen et al. [2021]
The cited source says what the claim attributes to it.
In the citing paper
“As shown in Table 2, we employ four metrics to evaluate different tasks in our benchmark, i.e., accuracy for question answering and language modeling, concordance index (Shaham et al., 2023) for sorting, pass@1 (Chen et al., 2021) for code completion, and precision, recall, and F1 for hallucination detection.”
From BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models (2023) · cites this paper as Chen et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Representative products, including ChatGPT [278], DALL-E-2 [279], Codex [280] and Midjourney [281], have attaining significant attention from society.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [280]
The cited source says what the claim attributes to it.
In the citing paper
“Our approach prioritizes objective evaluation through execution accuracy , providing more rigorous validation than non-executable assessment methodologies (Wang et al., 2022; Chen et al., 2021).”
From SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers (2025) · cites this paper as Chen et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Similar behavior holds for benchmarks such as HumanEval (Chen et al., 2021), which contains only 164 examples.”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Chen et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Early studies have shown that large language models (LLMs) exhibit basic reasoning capabilities in mathematics (Cobbe et al., 2021), coding (Chen et al., 2021), and general reasoning tasks.”
From Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning (2025) · cites this paper as Chen et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Previous studies (Chen et al., 2021;Nijkamp et al., 2022;Chen et al., 2022) have demonstrated the effectiveness of utilizing large pre-trained language models for code generation in a zero-shot completion manner, conditioned on the provided context.”
From RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation (2023) · cites this paper as Chen et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Following existing works evaluating LLM code generation [12, 23] and common evaluation practices over LiveCodeBench and DevEval, we use Pass@1 as the evaluation metric for all RQs rather than code similarity scores, as it is more appropriate for problem-solving and code generation tasks.”
From What Makes In-Context Examples Effective for Code Generation? (2026) · cites this paper as [12]
The cited source says what the claim attributes to it.
In the citing paper
“The pattern indicates a correlation between the LLMs' coding ability (demonstrated by the performance on code generation benchmarks [12, 90]) and context code leveraging with information elimination.”
From What Makes In-Context Examples Effective for Code Generation? (2026) · cites this paper as [12]
The cited source says what the claim attributes to it.
In the citing paper
“This dataset augments the original HumanEval (Chen et al., 2021) dataset with GPT-generated test cases and further test cases from mutation genetic algorithms.”
From Does Few-Shot Learning Help LLM Performance in Code Synthesis? (2024) · cites this paper as Chen et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Among the fully open-source options, StarCoder [45] and StarCoder2 [51] ranked highest on two widely used code generation benchmarks, HumanEval [17] and MBPP [6], thereby fulfilling our second criterion.”
From LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks (2025) · cites this paper as [17]
The cited source says what the claim attributes to it.
In the citing paper
“Chen et al. (2021) introduced HumanEval as a hand-crafted benchmark comprising 164 programming problems designed to assess functional correctness of synthesized code.”
From Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression (2026) · cites this paper as Chen et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“We also briefly investigated whether our RLHF-finetuned code models have any comparative advantage when exposed to prompts including buggy code [Chen et al., 2021], but we did not find any benefits there.”
From Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback (2022) · cites this paper as Chen et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) evaluate isolated functions.”
From SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents (2026) · cites this paper as Chen et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“HumanEval consists of 164 manually written programming tasks, each providing a Python function signature and a docstring as input to the model.”
From Qwen2.5-Coder Technical Report (2024) · cites this paper as Chen et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“but match-based metrics cannot cover the vast space of equivalent programs (Chen et al., 2021).”
From Planetarium: A Rigorous Benchmark for Translating Text to Structured Planning Languages (2025) · cites this paper as Chen et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“In this section, we test the HumanEval dataset (Chen et al., 2021), which is known to be sensitive to prompt design.”
From Language Agents as Optimizable Graphs (2024) · cites this paper as Chen et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“In particular, AlphaVerus is the first method to achieve non-zero formally verified code generation performance on a verified version of HumanEval (Chen et al., 2021), thus establishing a starting point for code generation models that generate increasingly complex-yet trustworthy-code.”
From AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinement (2024) · cites this paper as Chen et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“CLEVER builds on HUMANEVAL [4] by adapting 161 2 of its 164 programming problems for formal verification in Lean 4.”
From CLEVER: A Curated Benchmark for Formally Verified Code Generation (2025) · cites this paper as [4]
The cited source says what the claim attributes to it.
In the citing paper
“In our work, we employ HUMANEVAL [4] to create CLEVER, our formal verification and synthesis benchmark.”
From CLEVER: A Curated Benchmark for Formally Verified Code Generation (2025) · cites this paper as [4]
The cited source says what the claim attributes to it.
In the citing paper
“In parallel, LLMs have achieved strong performance in generating source code from natural language descriptions [16, 22, 48, 76, 77].”
From A Multi-Agent Framework for Accessible Web Code Generation (2026) · cites this paper as [22]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Natural science education: LLM-based agents can communicate with humans fluently, often being utilized to develop agent-based educational tools [115,117-119].”
From A survey on large language model based autonomous agents (2024) · cites this paper as [119]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Today, safety measures are typically task-specific heuristics”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Chen et al., 2021
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“We set the temperature to 0.6 based on the results reported in the original Codex work [3] as well as in [47], which show that the best pass@10 rate is at temperature=0.6.”
From Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022) · cites this paper as [3]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“This dataset, encompassing over 6TB of code in 358 languages, is extensively used for training large language models due to its permissive licensing and wide coverage.”
From Full Line Code Completion: Bringing AI to Desktop (2024) · cites this paper as [8]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Top agents now cluster near ceiling-level scores and are often statistically indistinguishable from one another [2, 30, 11, 10, 26].”
From Life After Benchmark Saturation: A Case Study of CORE-Bench (2026) · cites this paper as [10]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“We introduce CLEVER 1 , a high-quality, curated benchmark of 161 problems for end-to-end verified code generation in Lean.”
From CLEVER: A Curated Benchmark for Formally Verified Code Generation (2025) · cites this paper as [4]
Experimental — accuracy not yet measuredIn the citing paper
“However, there is an unignorable programming language bias in existing code benchmarks -over 95% code generation benchmarks are dominated by Python, leaving the LLMs' capabilities in other programming languages such as Java and C/C++ unknown.”
From CRUXEVAL-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution (2025) · cites this paper as Chen et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“Scalable sequence prediction models (Graves, 2014; Vaswani et al., 2017; Child et al., 2019) have become a general-purpose method for generation and representation learning in many domains, including natural language processing (Mikolov et al., 2013; Sutskever et al., 2014; Dai & Le, 2015; Peters…”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 5×
“Following the success of large natural language models (Devlin et al., 2018; Radford et al., 2019; Liu et al., 2019; Raffel et al., 2020; Brown et al., 2020) large scale Transformers have also been applied towards program synthesis.”
BackgroundMethodInfluentialDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“…al., 2019) have become a general-purpose method for generation and representation learning in many domains, including natural language processing (Mikolov et al., 2013; Sutskever et al., 2014; Dai & Le, 2015; Peters et al., 2018; Radford et al., 2018; Devlin et al., 2018), computer vision (Van…”
MethodSelf-citationSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 2×
“, 2019) have become a general-purpose method for generation and representation learning in many domains, including natural language processing (Mikolov et al., 2013; Sutskever et al., 2014; Dai & Le, 2015; Peters et al., 2018; Radford et al., 2018; Devlin et al., 2018), computer vision (Van Oord et al.”
MethodSelf-citationRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 1×
“Following the success of large natural language models (De-vlin et al., 2018; Radford et al., 2019; Liu et al., 2019; Raffel et al., 2020; Brown et al., 2020) large scale Transformers have also been applied towards program synthesis.”
MethodLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 3×
“Generative models have been shown to encode bias in modalities such as natural language (Brown et al., 2020; Blodgett et al., 2020) and images (Radford et al., 2021), and we find that the same is true of models like Codex that generate code.”
BackgroundMethodSelf-citationDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 1×
“…and representation learning in many domains, including natural language processing (Mikolov et al., 2013; Sutskever et al., 2014; Dai & Le, 2015; Peters et al., 2018; Radford et al., 2018; Devlin et al., 2018), computer vision (Van Oord et al., 2016; Menick & Kalchbrenner, 2018; Chen et al.,…”
BackgroundHandbook of applied cryptography
(1997) · Choice Reviews Online · doi:10.5860/choice.34-4512 · Cited by 10,357
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 2×
“model test loss follows a power law in model size (Kaplan et al., 2020), test loss after code fine-tuning follows a similar power law with functional form ( N 5.”
BackgroundMethodSelf-citationOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 2×
“Mirroring what has been found in the case of other language models trained on Internet data (Bender et al., 2021; Blod-gett et al., 2020; Abid et al., 2021; Brown et al., 2020”
BackgroundRepresentation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242 · cited 1×
“…al., 2018), computer vision (Van Oord et al., 2016; Menick & Kalchbrenner, 2018; Chen et al., 2020; Bao et al., 2021), audio and speech processing (Oord et al., 2016; 2018; Dhari-wal et al., 2020; Baevski et al., 2020), biology (Alley et al., 2019; Rives et al., 2021), and even across multiple…”
BackgroundWaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911 · cited 2×
“…(Mikolov et al., 2013; Sutskever et al., 2014; Dai & Le, 2015; Peters et al., 2018; Radford et al., 2018; Devlin et al., 2018), computer vision (Van Oord et al., 2016; Menick & Kalchbrenner, 2018; Chen et al., 2020; Bao et al., 2021), audio and speech processing (Oord et al., 2016; 2018; Dhari-wal…”
BackgroundRives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280 · cited 1×
“…et al., 2020; Bao et al., 2021), audio and speech processing (Oord et al., 2016; 2018; Dhari-wal et al., 2020; Baevski et al., 2020), biology (Alley et al., 2019; Rives et al., 2021), and even across multiple modalities (Das et al., 2017; Lu et al., 2019; Ramesh et al., 2021; Zellers et al., 2021).”
BackgroundGenerating Sequences With Recurrent Neural Networks
Graves, A. (2013) · arXiv · doi:10.48550/arxiv.1308.0850 · Cited by 2,894 · cited 1×
“Scalable sequence prediction models (Graves, 2014; Vaswani et al., 2017; Child et al., 2019) have become a general-purpose method for generation and representation learning in many domains, including natural language processing (Mikolov et al., 2013; Sutskever et al., 2014; Dai & Le, 2015; Peters…”
MethodCodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824
Zero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417 · cited 2×
“…et al., 2020; Bao et al., 2021), audio and speech processing (Oord et al., 2016; 2018; Dhari-wal et al., 2020; Baevski et al., 2020), biology (Alley et al., 2019; Rives et al., 2021), and even across multiple modalities (Das et al., 2017; Lu et al., 2019; Ramesh et al., 2021; Zellers et al., 2021).”
BackgroundSelf-citationwav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020) · arXiv · doi:10.48550/arxiv.2006.11477 · Cited by 2,369 · cited 1×
“…Menick & Kalchbrenner, 2018; Chen et al., 2020; Bao et al., 2021), audio and speech processing (Oord et al., 2016; 2018; Dhari-wal et al., 2020; Baevski et al., 2020), biology (Alley et al., 2019; Rives et al., 2021), and even across multiple modalities (Das et al., 2017; Lu et al., 2019;…”
BackgroundViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109
DeBERTa: Decoding-enhanced BERT with Disentangled AttentionPreprint
He, P., Liu, X., Gao, J., & Chen, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03654 · Cited by 1,916 · cited 1×
“A common approach to adjusting the behavior of Trans-formers is to fine-tune large pre-trained models with curated or human-generated datasets of the desired behavior (e.g., Raffel et al. (2020); He et al. (2020)).”
BackgroundPixel Recurrent Neural NetworksPreprint
van den Oord, A., Kalchbrenner, N., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1601.06759 · Cited by 1,793 · cited 2×
“…(Mikolov et al., 2013; Sutskever et al., 2014; Dai & Le, 2015; Peters et al., 2018; Radford et al., 2018; Devlin et al., 2018), computer vision (Van Oord et al., 2016; Menick & Kalchbrenner, 2018; Chen et al., 2020; Bao et al., 2021), audio and speech processing (Oord et al., 2016; 2018; Dhari-wal…”
BackgroundRecalibrating global data center energy-use estimates
Masanet, E., Shehabi, A., Lei, N., Smith, S., & Koomey, J. (2020) · Science · doi:10.1126/science.aba3758 · Cited by 1,545 · cited 1×
“8 While data center energy usage has become much more effi-cient in recent years (Masanet et al., 2020), the production, use, and disposal of semiconductors still imposes environmental and human costs.”
BackgroundHybrid computing using a neural network with dynamic external memory
Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwińska, A., Colmenarejo, S. G., Grefenstette, E., Ramalho, T., Agapiou, J., et al. (2016) · Nature · doi:10.1038/nature20101 · Cited by 1,412 · cited 2×
“…incorporated inductive biases based on modern computing devices, such as the Neural Turing Machine (Graves et al., 2014), memory networks (Weston et al., 2015; Sukhbaatar et al., 2015), the Neural GPU (Kaiser & Sutskever, 2015), and the differentiable neural computer (Graves et al., 2016).”
BackgroundMethodBEiT: BERT Pre-Training of Image Transformers
Bao, H., Dong, L., Piao, S., & Wei, F. (2021) · arXiv · doi:10.48550/arxiv.2106.08254 · Cited by 1,392 · cited 2×
“, 2018), computer vision (Van Oord et al., 2016; Menick & Kalchbrenner, 2018; Chen et al., 2020; Bao et al., 2021), audio and speech processing (Oord et al.”
BackgroundUnified rational protein engineering with sequence-based deep representation learning
Alley, E. C., Khimulya, G., Biswas, S., AlQuraishi, M., & Church, G. M. (2019) · Nature Methods · doi:10.1038/s41592-019-0598-1 · Cited by 1,131 · cited 2×
“, 2020), biology (Alley et al., 2019; Rives et al., 2021), and even across multiple modalities (Das et al.”
BackgroundJukebox: A Generative Model for MusicPreprint
Dhariwal, P., Jun, H., Payne, C., Kim, J. W., Radford, A., & Sutskever, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00341 · Cited by 1,101 · cited 1×
“, 2021), audio and speech processing (Oord et al., 2016; 2018; Dhariwal et al., 2020; Baevski et al., 2020), biology (Alley et al.”
BackgroundSelf-citationGenerating Long Sequences with Sparse Transformers
Child, R., Gray, S., Radford, A., & Sutskever, I. (2019) · arXiv · doi:10.48550/arxiv.1904.10509 · Cited by 1,063 · cited 2×
“Scalable sequence prediction models (Graves, 2014; Vaswani et al., 2017; Child et al., 2019) have become a general-purpose method for generation and representation learning in many domains, including natural language processing (Mikolov et al., 2013; Sutskever et al., 2014; Dai & Le, 2015; Peters…”
MethodSelf-citationLanguage (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050 · cited 3×
“Generative models have been shown to encode bias in modalities such as natural language (Brown et al., 2020; Blodgett et al., 2020) and images (Radford et al., 2021), and we find that the same is true of models like Codex that generate code.”
BackgroundMethod(2020) · International Conference on Machine Learning · Cited by 1,012
Self-citationSemi-supervised Sequence Learning
Dai, A. M., & Le, Q. V. (2015) · arXiv · doi:10.48550/arxiv.1511.01432 · Cited by 994 · cited 1×
“…for generation and representation learning in many domains, including natural language processing (Mikolov et al., 2013; Sutskever et al., 2014; Dai & Le, 2015; Peters et al., 2018; Radford et al., 2018; Devlin et al., 2018), computer vision (Van Oord et al., 2016; Menick & Kalchbrenner, 2018;…”
BackgroundRobots and Jobs: Evidence from US Labor Markets
Acemoğlu, D., & Restrepo, P. (2017) · National Bureau of Economic Research · doi:10.3386/w23285 · Cited by 924
Genetic Programming III: Darwinian Invention and Problem Solving
Koza, J. R., André, D., Bennett, F. H., & Keane, M. A. (1999) · Medical Entomology and Zoology · Cited by 763
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751 · cited 1×
“Another possibility is to label the pre-training data based on code quality, then condition the model on the ’high quality’ label at deployment time (Keskar et al., 2019).”
BackgroundThe Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence
CRAWFORD, K. (2021) · doi:10.2307/j.ctv1ghv45t · Cited by 742
DeepCoder: Learning to Write ProgramsPreprint
Balog, M., Gaunt, A. L., Brockschmidt, M., Nowozin, S., & Tarlow, D. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1611.01989 · Cited by 641 · cited 1×
“DeepCoder (Balog et al., 2017) trained a model to predict the functions appearing in source code, which could be used to guide program search.”
Methodcode2seq: Generating Sequences from Structured Representations of CodePreprint
Alon, U., Brody, S., Levy, O., & Yahav, E. (2018) · arXiv (Cornell University) · doi:10.4230/oasics.slate.2021.14 · Cited by 622
Automating string processing in spreadsheets using input-output examples
Gulwani, S. (2011) · ACM SIGPLAN Notices · doi:10.1145/1925844.1926423 · Cited by 581
On the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Dévanbu, P. (2012) · doi:10.5555/2337223.2337322 · Cited by 580
Learning to summarize from human feedback
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P. (2020) · arXiv · doi:10.48550/arxiv.2009.01325 · Cited by 550
Self-citationCodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.04664 · Cited by 536 · cited 1×
“Recently, CodeXGLUE (Lu et al., 2021) aggregated several programming benchmarks, making use of the recently proposed CodeBLEU metric (Ren et al., 2020).”
BackgroundLearning to ExecutePreprint
Zaremba, W., & Sutskever, I. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.4615 · Cited by 525 · cited 1×
“Learning to Execute (Zaremba & Sutskever, 2014) demonstrated that models could execute simple tasks like addition and memo-rization.”
BackgroundSelf-citationCalibrate Before Use: Improving Few-Shot Performance of Language Models
Zhao, T. Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021) · arXiv · doi:10.48550/arxiv.2102.09690 · Cited by 513 · cited 1×
“And research shows that so-called prompt engineering can enable stronger results from AI systems (Zhao et al., 2021).”
BackgroundThe Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492 · cited 2×
“Two recent works similar in spirit to Codex are GPT-Neo (Black et al., 2021) and GPT-J (Wang & Komatsuzaki, 2021), which are trained on The Pile (Gao et al., 2020), a dataset containing text from a variety of sources as well as 8% GitHub code.”
BackgroundMethodCarbon Emissions and Large Neural Network Training
Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., So, D., Texier, M., & Dean, J. (2021) · arXiv · doi:10.48550/arxiv.2104.10350 · Cited by 486 · cited 1×
“Codex, like other large generative models, has an energy footprint from both training and inference (Schwartz et al., 2019; Bender et al., 2021; Patterson et al., 2021).”
BackgroundPersistent Anti-Muslim Bias in Large Language Models
Abid, A., Farooqi, M., & Zou, J. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462624 · Cited by 476 · cited 1×
“Mirroring what has been found in the case of other language models trained on Internet data (Bender et al., 2021; Blod-gett et al., 2020; Abid et al., 2021; Brown et al., 2020”
Background(2020) · arXiv (Cornell University) · Cited by 461
Das, A., Kottur, S., Gupta, K., Singh, A., Jain, D., Moura, J. M. F., Parikh, D., & Batra, D. (2017) · Cited by 432
An analysis of patch plausibility and correctness for generate-and-validate patch generation systems
Qi, Z., Long, F., Achour, S., & Rinard, M. (2015) · Proceedings of the 2015 International Symposium on Software Testing and Analysis · doi:10.1145/2771783.2771791 · Cited by 420 · cited 1×
“However, these works used an exact match against a reference instead of functional correctness, citing Qi et al. (2015)’s finding that most of the proposed solutions by genetic search in (Goues et al., 2012) passed through weak test suites by deleting functionality that failed.”
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, Y., Mohamed, A., & Auli, M. (2020) · neural information processing systems · Cited by 386
An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation
Tufano, M., Watson, C., Bavota, G., Penta, M. D., White, M., & Poshyvanyk, D. (2019) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3340544 · Cited by 370 · cited 2×
“More recent works (Tufano et al., 2019; Drain et al., 2021) considered bug-fixing as neural machine translation from buggy to correct programs.”
BackgroundDataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses
Goldblum, M., Tsipras, D., Xie, C., Chen, X., Schwarzschild, A., Song, D., Madry, A., Li, B., & Goldstein, T. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3162397 · Cited by 362 · cited 1×
“Similarly, the public data should generally be treated as untrusted, as previous work (Goldblum et al., 2021; Schuster et al., 2020) has found that attackers may be able to corrupt training data to trigger specific model behaviors at runtime.”
BackgroundNeural GPUs Learn AlgorithmsPreprint
Kaiser, Ł., & Sutskever, I. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.08228 · Cited by 333 · cited 2×
“…incorporated inductive biases based on modern computing devices, such as the Neural Turing Machine (Graves et al., 2014), memory networks (Weston et al., 2015; Sukhbaatar et al., 2015), the Neural GPU (Kaiser & Sutskever, 2015), and the differentiable neural computer (Graves et al., 2016).”
MethodSelf-citationGPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow
Black, S., Leo, G., Wang, P., Leahy, C., & Biderman, S. (2021) · Zenodo · doi:10.5281/zenodo.5297715 · Cited by 324 · cited 7×
“Two recent works similar in spirit to Codex are GPT-Neo (Black et al., 2021) and GPT-J (Wang & Komatsuzaki, 2021), which are trained on The Pile (Gao et al., 2020), a dataset containing text from a variety of sources as well as 8% GitHub code.”
MethodInfluentialLatent Predictor Networks for Code Generation
Ling, W., Blunsom, P., Grefenstette, E., Hermann, K. M., Kočiský, T., Wang, F., & Senior, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1057 · Cited by 307 · cited 2×
“Latent Predictor Networks (Ling et al., 2016) showed that character-level language models could generate working code for implementing Magic the Gathering cards in an online arena, when aided with a latent mode that allows card attributes to be copied into code.”
BackgroundSpreadsheet data manipulation using examples
Gulwani, S., Harris, W. R., & Singh, R. K. (2012) · Communications of the ACM · doi:10.1145/2240236.2240260 · Cited by 296 · cited 1×
“Two early domain-specific datasets used to benchmark neural programming systems were FlashFill (Gulwani, 2011; Gulwani et al., 2012) and Hearthstone (Ling et al., 2016), though the community has trended towards broader and more difficult datasets.”
BackgroundFault localization using execution slices and dataflow tests
Agrawal, H., Horgan, J. R., London, S., & Wong, W. E. (2002) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.1995.497652 · Cited by 288 · cited 2×
“Early works used static or dynamic code analysis (Agrawal et al., 1995; Korel & Rilling, 1997), learned association rules (Jeffrey et al., 2009), and genetic programming (Goues et al., 2012) to debug faulty code.”
MethodToward automatic program synthesis
Manna, Z., & Waldinger, R. J. (1971) · Communications of the ACM · doi:10.1145/362566.362568 · Cited by 285 · cited 1×
“More recently, language models have also fueled progress towards the longstanding challenge of program synthesis (Simon, 1963; Manna & Waldinger, 1971), spurred by the presence of code in large datasets (Husain et al., 2019; Gao et al., 2020) and the resulting programming capabilities of language…”
BackgroundTemporal Logics for Hyperproperties
Clarkson, M. R., Finkbeiner, B., Koleini, M., Micinski, K., Rabe, M. N., & Sánchez, C. (2014) · Lecture notes in computer science · doi:10.1007/978-3-642-54792-8_15 · Cited by 283
Unit Test Case Generation with Transformers and Focal ContextPreprint
Tufano, M., Drain, D., Svyatkovskiy, A., Deng, S. K., & Sundaresan, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.05617 · Cited by 263 · cited 1×
“Tufano et al. (2020) use Transformers to generate unit tests for code which outperformed commercial offerings.”
MethodBackstabber’s Knife Collection: A Review of Open Source Software Supply Chain Attacks
Ohm, M., Plate, H., Sykosch, A., & Meier, M. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-52683-2_2 · Cited by 259 · cited 1×
“20 Also worrying is the possibility of Codex suggesting malicious or typosquatted packages (Ohm et al., 2020).”
BackgroundMERLOT: Multimodal Neural Script Knowledge ModelsPreprint
Zellers, R., Lu, X., Hessel, J., Yu, Y., Park, J. S., Cao, J., Farhadi, A., & Choi, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02636 · Cited by 239 · cited 1×
“…et al., 2020; Bao et al., 2021), audio and speech processing (Oord et al., 2016; 2018; Dhari-wal et al., 2020; Baevski et al., 2020), biology (Alley et al., 2019; Rives et al., 2021), and even across multiple modalities (Das et al., 2017; Lu et al., 2019; Ramesh et al., 2021; Zellers et al., 2021).”
BackgroundCodeBLEU: a Method for Automatic Evaluation of Code Synthesis
Ren, S., Guo, D., Lu, S., Zhou, L., Liu, S., Tang, D., Sundaresan, N., Zhou, M., Blanco, A., & Ma, S. (2020) · arXiv · doi:10.48550/arxiv.2009.10297 · Cited by 214 · cited 2×
“For instance, Ren et al. (2020) finds that BLEU has problems capturing semantic features specific to code, and suggests several semantic modifications to the score.”
BackgroundGeneral Program Synthesis Benchmark Suite
Helmuth, T., & Spector, L. C. (2015) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/2739480.2754769 · Cited by 174 · cited 2×
“While evaluating code generation is well-studied (Xu et al., 2021; Helmuth & Spector, 2015; Pantridge et al., 2017), many existing metrics measure performance in tightly specified, constrained problem instances (e.g., string manipulation in FlashFill (Gul-wani, 2011)).”
BackgroundMethodFrey, C. B. (2019) · Princeton University Press eBooks · doi:10.1515/9780691191959 · Cited by 171
Bimodal Modelling of Source Code and Natural Language
Allamanis, M., Tarlow, D., Gordon, A. D., & Wei, Y. (2015) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 164
The Economics of Software Quality
Jones, C., & Bonsignour, O. (2011) · CERN Document Server (European Organization for Nuclear Research) · Cited by 158
In-IDE Code Generation from Natural Language: Promise and Challenges
Xu, F. F., Vasilescu, B., & Neubig, G. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487569 · Cited by 154 · cited 4×
“Most past studies of the impacts of code generation models consider performance on a closed set of tasks in a simulated environment (Xu et al., 2021).”
BackgroundMethodInfluentialThe Wrong Kind of AI? Artificial Intelligence and the Future of Labor Demand
Acemoğlu, D., & Restrepo, P. (2019) · National Bureau of Economic Research · doi:10.3386/w25682 · Cited by 148 · cited 1×
“Furthermore, as with many tools that substitute investments in capital for investments in labor (or increase the productivity of labor) (Frey, 2019; Acemoglu & Restrepo, 2020a;b), more sophisticated future code generation tools could potentially contribute to the displacement of some programmer or…”
BackgroundPyMT5: multi-mode translation of natural language and Python code with transformers
Clement, C. B., Drain, D., Timcheck, J., Svyatkovskiy, A., & Sundaresan, N. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.728 · Cited by 145
Structured Generative Models of Natural Source CodePreprint
Maddison, C. J., & Tarlow, D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1401.0514 · Cited by 142 · cited 1×
“Maddison & Tarlow (2014) improved on this setup by learning a state vector used to condition child node expansion.”
BackgroundMeasuring Coding Challenge Competence With APPS
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.09938 · Cited by 142 · cited 9×
“Most relevant to our evaluation work is the APPS (Hendrycks et al., 2021) benchmark for measuring functional correctness based on problems from the competitive programming website Codeforces.”
BackgroundMethodInfluentialCodeSearchNet Challenge: Evaluating the State of Semantic Code Search
Husain, H., Wang, H.-H., Gazit, T., Allamanis, M., & Brockschmidt, M. (2019) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.7908467 · Cited by 141
Contrastive Code Representation Learning
Jain, P., Jain, A., Zhang, T., Abbeel, P., Gonzalez, J., & Stoica, I. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.482 · Cited by 129 · cited 1×
“In fact, ContraCode (Jain et al., 2020) leveraged the large space of functionally correct programs to train a contrastive code model, which improved model performance on tasks like type inference.”
BackgroundBarone, A. V. M., & Sennrich, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.02275 · Cited by 123 · cited 1×
“Barone & Sennrich (2017) proposed a large training and evaluation dataset consisting of Python declarations, docstrings, and bodies scraped from GitHub.”
BackgroundGenerating High Fidelity Images with Subscale Pixel Networks and Multidimensional UpscalingPreprint
Menick, J., & Kalchbrenner, N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.01608 · Cited by 122 · cited 1×
“…Sutskever et al., 2014; Dai & Le, 2015; Peters et al., 2018; Radford et al., 2018; Devlin et al., 2018), computer vision (Van Oord et al., 2016; Menick & Kalchbrenner, 2018; Chen et al., 2020; Bao et al., 2021), audio and speech processing (Oord et al., 2016; 2018; Dhari-wal et al., 2020;…”
BackgroundWomen’s Participation in Open Source Software: A Survey of the Literature
Trinkenreich, B., Wiese, I., Sarma, A., Gerosa, M. A., & Steinmacher, I. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3510460 · Cited by 106 · cited 1×
“This could increase open-source developers’ incentive to maintain backward compatibility, which could pose challenges given that open-source projects are often under-resourced (Eghbal, 2020; Trinkenreich et al., 2021).”
BackgroundSPoC: Search-based Pseudocode to Code
Kulal, S., Pasupat, P., Chandra, K., Lee, M., Padon, O., Aiken, A., & Liang, P. (2019) · arXiv · doi:10.48550/arxiv.1906.04908 · Cited by 103 · cited 4×
“As a consequence, recent works in unsupervised code translation (Lachaux et al., 2020) and pseudocode-to-code translation (Kulal et al., 2019) have turned to functional correctness instead, where a sample is considered correct if it passes a set of unit tests.”
BackgroundMethodResultInfluentialYou Autocomplete Me: Poisoning Vulnerabilities in Neural Code CompletionPreprint
Schuster, R., Song, C., Tromer, E., & Shmatikov, V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.02220 · Cited by 87 · cited 2×
“Similarly, the public data should generally be treated as untrusted, as previous work (Goldblum et al., 2021; Schuster et al., 2020) has found that attackers may be able to corrupt training data to trigger specific model behaviors at runtime.”
BackgroundUnsupervised translation of programming languages
Lachaux, M.-A., Roziere, B., Chanussot, L., & Lample, G. (2020) · arXiv · doi:10.48550/arxiv.2006.03511 · Cited by 83 · cited 2×
“As a consequence, recent works in unsupervised code translation (Lachaux et al., 2020) and pseudocode-to-code translation (Kulal et al., 2019) have turned to functional correctness instead, where a sample is considered correct if it passes a set of unit tests.”
BackgroundBugFix: A learning-based tool to assist developers in fixing bugs
Jeffrey, D., Feng, M., Gupta, N., & Gupta, R. (2009) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2009.5090029 · Cited by 82 · cited 2×
“Early works used static or dynamic code analysis (Agrawal et al., 1995; Korel & Rilling, 1997), learned association rules (Jeffrey et al., 2009), and genetic programming (Goues et al., 2012) to debug faulty code.”
BackgroundMethodLatent Predictor Networks for Code GenerationPreprint
Ling, W., Grefenstette, E., Hermann, K. M., Kočiský, T., Senior, A., Wang, F., & Blunsom, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.06744 · Cited by 82 · cited 2×
“Latent Predictor Networks (Ling et al., 2016) showed that character-level language models could generate working code for implementing Magic the Gathering cards in an online arena, when aided with a latent mode that allows card attributes to be copied into code.”
BackgroundApplication of Dynamic Slicing in Program Debugging
Korel, B., & Rilling, J. (1997) · Cited by 67
Experiments with a Heuristic Compiler
Simon, H. A. (1963) · Journal of the ACM · doi:10.1145/321186.321192 · Cited by 65 · cited 1×
“More recently, language models have also fueled progress towards the longstanding challenge of program synthesis (Simon, 1963; Manna & Waldinger, 1971), spurred by the presence of code in large datasets (Husain et al., 2019; Gao et al., 2020) and the resulting programming capabilities of language…”
BackgroundGenerating bug-fixes using pretrained transformers
Drain, D., Wu, C. H., Svyatkovskiy, A., & Sundaresan, N. (2021) · MAPS@PLDI · doi:10.1145/3460945.3464951 · Cited by 49 · cited 2×
“More recent works (Tufano et al., 2019; Drain et al., 2021) considered bug-fixing as neural machine translation from buggy to correct programs.”
BackgroundWhat distinguishes great software engineers
Li, P. L., Ko, A. J., & Begel, A. (2020) · Empirical Software Engineering · doi:10.1007/s10664-019-09773-y · Cited by 48 · cited 1×
“…(BLS) classifies computer programmers and software developers separately, where developers are more highly paid than programmers, have more tasks indirectly related to writing and interacting with code, and, in the US, are projected to see greater demand over the next 10 years (Li et al., 2020).”
BackgroundKenton, Z., Everitt, T., Weidinger, L., Gabriel, I., Mikulik, V., & Irving, G. (2021) · arXiv · doi:10.48550/arxiv.2103.14659 · Cited by 45 · cited 1×
“In the literature, a model is defined informally as “intent aligned” with a user if (and only if) the model intends to do what the user wants (Chris-tiano, 2018; Kenton et al., 2021).”
MethodAutomatic programming: The open issue?
O’Neill, M., & Spector, L. C. (2019) · Genetic Programming and Evolvable Machines · doi:10.1007/s10710-019-09364-2 · Cited by 25 · cited 1×
“Indeed, the synthesis and automatic programming community (O’Neill & Spector, 2019) have recently called for principled benchmarks and grand challenge problems to be made in order to adopt a scientifically rigorous approach to compare synthesis methodologies against.”
Improving Neural Program Synthesis with Inferred Execution Traces
Shin, E. C. R., Polosukhin, I., & Song, D. X. (2018) · neural information processing systems · Cited by 23
Learning Compositional Neural Programs with Recursive Tree Search and Planning
Pierrot, T., Ligner, G., Reed, S., Sigaud, O., Perrin, N., Laterre, A., Kas, D., Beguir, K., & de Freitas, N. (2019) · arXiv · doi:10.48550/arxiv.1905.12941 · Cited by 20 · cited 2×
“More recent approaches like the Neural Program Interpreter (Reed & de Freitas, 2016; Shin et al., 2018; Pierrot et al., 2021) and”
MethodOn the difficulty of benchmarking inductive program synthesis methods
Pantridge, E., Helmuth, T., McPhee, N. F., & Spector, L. C. (2017) · Genetic and Evolutionary Computation Conference Companion (GECCO Companion) · doi:10.1145/3067695.3082533 · Cited by 16 · cited 4×
“While evaluating code generation is well-studied (Xu et al., 2021; Helmuth & Spector, 2015; Pantridge et al., 2017), many existing metrics measure performance in tightly specified, constrained problem instances (e.”
BackgroundMethodInfluential
Flagged first, then by citations
5 of the 96 references couldn’t be matched to a known paper.
Research works printed but not linked: 11
- Aye, G. A., Kim, S., and Li, H. Learning autocompletion from realworld datasets. 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) , pp. 131-139, 2021.
- Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. ArXiv , abs/2005.14165, 2020. · arXiv
- Carlini, N., Tram` er, F., Wallace, E., Jagielski, M., Herbert-V oss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., Oprea, A., and Raffel, C. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21) . USENIX Association, August 2021. URL https://www . usenix . org/conference/ usenixsecurity21/presentation/carliniextracting . https://www.usenix.org/conference/usenixsecurity21/presentation/carlini-extracting
- Crawford, K. The trouble with bias. NIPS 2017 Keynote , 2017. URL https://www . youtube . com/watch?v= fMym BKWQzk .
- Devlin, J., Uesato, J., Bhupatiraju, S., Singh, R., rahman Mohamed, A., and Kohli, P. Robustfill: Neural program learning under noisy i/o. In ICML , 2017.
- Eghbal, N. Working in public: the making and maintenance of open source software . Stripe Press, 2020.
- Goues, C. L., Dewey-Vogt, M., Forrest, S., and Weimer, W. A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each. 2012 34th International Conference on Software Engineering (ICSE) , pp. 3-13, 2012.
- Graves, A., Wayne, G., and Danihelka, I. Neural turing machines. arXiv preprint arXiv:1410.5401 , 2014. · arXiv
- Raffel, C., Shazeer, N. M., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv , abs/1910.10683, 2020. · arXiv
- Rokon, M. O. F., Islam, R., Darki, A., Papalexakis, E. E., and Faloutsos, M. Sourcefinder: Finding malware source-code from publicly available repositories in github. In 23rd International Symposium on Research in Attacks, Intrusions and Defenses (RAID 2020) , pp. 149-163, San Sebastian, October 2020. USENIX Association. ISBN 978-1-93913318-2. URL https://www . usenix . org/conference/ raid2020/presentation/omar .
- Yin, P. and Neubig, G. A syntactic neural model for generalpurpose code generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL) , pp. 440-450, 2017.
Cited by
Sorted by citations
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Self-citationA Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
“Large Language Models (LLMs) (Chen et al., 2021; Kasneci et al., 2023; Zhao et al., 2023a) have gained significant attention in recent years due to their remarkable capabilities in natural language processing tasks.”
BackgroundOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Self-citationPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
“In the HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) datasets, the model is given an English-language description of a few sentences and a small number of input-output examples, and the goal is to generate a short Python program, usually a single function.”
BackgroundMethodResultInfluentialSelf-citationChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
“, 2020), as well as the use of large language models (Chen et al., 2021; Austin et al., 2021).”
Background
Metadata
- Authors
- Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., & Zaremba, W.
- Affiliations
- OpenAI (United States) (US)
- Anthropic (United States) (US)
- Zipline International (United States) (US)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2107.03374
- OpenAlex
- W3177813494
- Cited by
- 1,436
- References
- 96
- Citations by year
- 2026: 121 · 2025: 261 · 2024: 376 · 2023: 551 · 2022: 93 · 2021: 34
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62472339, 62302375)
- China Postdoctoral Science Foundation (2023M723736)
- Ministry of Industry and Information Technology of the People's Republic of China
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Software › Logic, programming, and type systems
- Keywords
- large language models, code generation, LLVM intermediate representation, AI-generated code, AWS Lambda, performance comparison, memory usage, CPU utilization, garbage collection overhead
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
