OctoPack: Instruction Tuning Code Large Language Models

Probably not a substantial claim: may not pass S3 (load-bearing).

All instruction datasets provide similar boosts for code synthesis.

“All instruction datasets provide similar boosts for code synthesis (HUMANEVALSYNTHESIZE), which has been the focus of all prior work on code instruction models”

• cites Wang et al., 2023cNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Luo et al., 2023 sourceDisputed

  The paper's Section 4.5 says that models trained on the seed instruction data lag behind those trained on evolved data, even when sample or token counts are matched. Table 5 reports materially different pass@1 scores across these datasets, contrary to the claim that they provide similar boosts for code synthesis. (LLM, conf=0.87)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured
• cites Muennighoff et al., 2022bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Instruction tuning of large language models on natural language tasks improves their usability and general performance.

“Finetuning large language models (LLMs) on a variety of language tasks explained via instructions (instruction tuning) has been shown to improve model usability and general performance (Wei et al., 2022;Sanh et al., 2022;Min et al., 2022;Ouyang et al., 2022).”

• cites Wei et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Sanh et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Min et al., 2022 sourcePartly supported

  The paper’s instruction ablation reports that multi-task zero-shot models outperform a raw-LM baseline and that adding instructions improves MetaICL’s task performance (section “Number of meta-training tasks”; page unavailable). It does not establish the claimed improvement in model usability, and its strongest performance finding concerns combining instructions with MetaICL rather than instruction tuning alone. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

May combine two claims.

Python HumanEval is the most widely used code benchmark and many training datasets have been decontaminated for it.

“Python HumanEval is the most widely used code benchmark and many training datasets have already been decontaminated for it (Kocetkov et al., 2022).”

• cites Kocetkov et al., 2022 sourcePartly supported

  Section 4.1 (p. 10) identifies HumanEval as a Python code benchmark and says the authors removed files containing exact copies of its prompts from two training datasets. Section 4.2 (p. 11) reports results for those decontaminated datasets. The paper does not establish that HumanEval is the most widely used code benchmark or that many training datasets generally have been decontaminated for it. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

• AssumptionRests on an assumption: Python HumanEval is the most widely used code benchmark.

Classified as probably not a substantial claim.

For HUMANEVALEXPLAIN, they score explanations by having the same model regenerate code from its own explanation and executing the regenerated code, avoiding BLEU/ROUGE.

“Subsequently, the same model is tasked to regenerate the code given only its own explanation.The second step allows us to score this task via code execution and measure pass@k (Chen et al., 2021) instead of evaluating the explanation itself using heuristic-based metrics like BLEU (Papineni et al., 2002) or ROUGE (Lin, 2004) which have major limitations”

• cites Chen et al., 2021 sourceSupported
• cites Papineni et al., 2002Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Lin, 2004Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

They reproduced WizardCoder's HumanEval score and found it also performs well across other languages.

“We reproduce the HumanEval score of WizardCoder (Luo et al., 2023;Xu et al., 2023a) and find it to also perform well across other languages.”

• cites Luo et al., 2023 sourceSupported
• cites Xu et al., 2023aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

By chaining code explanation with code synthesis, they can evaluate explanation using execution-based pass@k, overcoming limitations of BLEU and other heuristics.

“By chaining code explanation with code synthesis, we can evaluate this task using the execution-based pass@k metric overcoming the major limitations of BLEU and other heuristics-based metrics (Reiter, 2018).”

• cites Reiter, 2018 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Prior work on evaluating code explanations relied on metrics such as METEOR or BLEU.

“Prior work on evaluating code explanations (Lu et al., 2021;Cui et al., 2022) has relied on metrics such as METEOR (Banerjee & Lavie, 2005) or BLEU (Papineni et al., 2002).”

• cites Lu et al., 2021Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Cui et al., 2022 sourceSupported
• cites Banerjee & Lavie, 2005Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Code repair is commonly evaluated on Quixbugs or Python bugs.

“Code repair is commonly evaluated on Quixbugs (Lin et al., 2017;Prenner & Robbes, 2021;Ye et al., 2021;Xia & Zhang, 2023;Jiang et al., 2023;Sobania et al., 2023) or Python bugs (He et al., 2022;Bradley et al., 2023).”

• cites Lin et al., 2017 sourceSupported
• cites He et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Bradley et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

CoEditor proposes an auto-editing task trained on 1650 Python commit history repositories.

“CoEditor (Wei et al., 2023) proposes an “auto-editing” task, trained on 1650 python commit history repositories.”

• cites Wei et al., 2023 sourcePartly supported

  The Introduction says the paper introduces a multi-round auto-editing task. Section 3.4 says its dataset draws on commit histories from 1,650 Python projects, but only 1,550 were used for training; 50 each were reserved for validation and testing. Thus, the claim overstates the number of training repositories. (LLM, conf=0.90)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Prior work has found that including raw code data during pretraining benefits some natural language tasks.

“Prior work has also found that including raw code data during pretraining benefits some natural language tasks (Muennighoff et al., 2023;Soldaini et al., 2024;Groeneveld et al., 2024).”

• cites Muennighoff et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Soldaini et al., 2024 sourceSupported
• cites Groeneveld et al., 2024 sourcePartly supported

  Section 2.2 (Pretraining Data: Dolma; page unavailable) lists GitHub code in the pretraining corpus, and Section 4.1 reports evaluation on natural-language tasks. Neither passage shows that including code, rather than omitting it, benefits those tasks; that effect is not supported by this paper's copy. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

This reward model could be used to guide a language model to generate code preferred by humans.

“Similar to prior work (Christiano et al., 2017;Ouyang et al., 2022), this reward model could then be used to guide a language model to generate code that is preferred by humans.”

• cites Christiano et al., 2017 sourcePartly supported

  The Introduction (p. 2) says the authors fit a reward function to human preferences and train an agent to optimize its predicted reward. That supports the proposed guiding mechanism, but the paper does not establish its use to guide a language model generating human-preferred code. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites Ouyang et al., 2022 sourceNot checked — no full text of the source

May combine two claims.

They introduced HUMANEVALPACK, expanding HumanEval to three coding tasks (repair, explanation, synthesis) across six languages (Python, JavaScript, Java, Go, C++, Rust).

“We further introduce HUMANEVALPACK, expanding the HumanEval benchmark to a total of 3 coding tasks (Code Repair, Code Explanation, Code Synthesis) across 6 languages (Python, JavaScript, Java, Go, C++, Rust).”

• AssumptionRests on an assumption: Prior code benchmarks commonly cover only code synthesis.
• AssumptionRests on an assumption: Users may want to use code models in all three input-output scenarios.

May combine two claims.

Instruction tuning StarCoder on COMMITPACK and other code instruction datasets achieved 46.2% pass@1 on HumanEval Python, state-of-the-art among models not trained on OpenAI outputs.

“We benchmark COMMITPACK against other natural and synthetic code instructions (xP3x, Self-Instruct, OASST) on the 16B parameter StarCoder model, and achieve state-of-the-art performance among models not trained on OpenAI outputs, on the HumanEval Python benchmark (46.2% pass@1).”

The quote only partly states this claim.

They created COMMITPACKFT, a filtered 2 GB variant of COMMITPACK containing high-quality code instructions for finetuning.

“we apply several strict filters to reduce the dataset to 2 gigabytes and 277 languages (COMMITPACKFT).”

May combine two claims.

Instruction tuning StarCoder on a filtered COMMITPACK variant and OASST yields OCTOCODER, which surpasses all other openly licensed models but falls short of GPT-4.

“Instruction tuning StarCoder (Li et al., 2023b) on a filtered variant of COMMITPACK and OASST leads to our best model, OCTOCODER, which surpasses all other openly licensed models (Figure 1), but falls short of the much larger GPT-4 (OpenAI, 2023).”

• cites Li et al., 2023bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites OpenAI, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

GPT-4 achieves 86.6% pass@1 on Python HumanEval code synthesis, close to maximum performance.

“GPT-4 is close to maximum performance on the code synthesis variant, notably with a pass@1 score of 86.6% on Python HumanEval.”

May combine two claims.

Instruction tuning on COMMITPACKFT is critical for the performance boost on code repair, whereas instruction tuning on only OASST or other variants results in significantly lower scores.

“COMMITPACKFT is critical for the performance boost on code repair (HUMANEVALFIX), where instruction tuning on only OASST or other variants results in a significantly lower score.”

May combine two claims.

Instruction tuning on COMMITPACKFT mixed with filtered OASST yields the best average score, an absolute 23% improvement over StarCoder.

“We achieve the best average score by instruction tuning on COMMITPACKFT mixed with our filtered OASST data yielding an absolute 23% improvement over StarCoder.”

May combine two claims.

Using n=1 instead of n=20 for GPT-4 changed HUMANEVALSYNTHESIZE Python pass@1 only from 75.0% to 75.2%, saving 20x cost.

“Using n = 1 instead of n = 20 for GPT-4 only changed scores from 75.0% to 75.2% pass@1 on HUMANEVALSYNTHESIZE Python while providing 20x cost savings.”

May combine two claims.

The authors manually added a bug to each of the 164 HumanEval solutions across all six languages, producing 984 total bugs.

“We manually add a bug to each of the 164 HumanEval solutions across all 6 languages (984 total bugs).”

May combine two claims.

OCTOGEEX, with 6 billion parameters, is the smallest benchmarked model but still outperforms all prior permissive Code LLMs.

“With just 6 billion parameters, OCTOGEEX is the smallest model benchmarked, but still outperforms all prior permissive Code LLMs.”

May combine two claims.

Models commonly write very short underspecified explanations or excessively long explanations that are cut off.

“They commonly write very short and thus underspecified explanations (e.g.BLOOMZ in Figure 35) or excessively long explanations that end up being cut off (e.g.StarChat-β in Figure 38).”

May combine two claims. The quote only partly states this claim.

BLOOMZ and InstructCodeT5+ occasionally generate code in explanations.

“BLOOMZ and InstructCodeT5+ also occasionally generate code.”

May combine two claims.

OCTOCODER performs weakest on Rust, the lowest-resource language among the benchmarked languages in StarCoder's pretraining data (1.2%).

“OCTOCODER performs weakest on Rust, which is the lowest resource language of StarCoder among the languages we benchmark (1.2% of pretraining data).”

The quote only partly states this claim.

OCTOCODER performs comparatively well across all languages on HUMANEVALFIX.

“OCTOCODER performs comparatively well across all languages.”

May combine two claims.

After instruction tuning StarCoder into OCTOCODER, the best performance is on Python, Java, and JavaScript, especially for HUMANEVALSYNTHESIZE.

“After instruction tuning StarCoder to produce OCTOCODER, we see the best performance among these three languages, especially for HUMANEVALSYNTHESIZE.”

May combine two claims.

Go and Rust are absent from BLOOMZ's instruction data, yet BLOOMZ performs much better than the random baseline of 0.0 for these languages across most tasks.

“Go and Rust are not contained in BLOOMZ's instruction data, yet it performs much better than the random baseline of 0.0 for these two languages across most tasks.”

May combine two claims.

Models commonly either regenerate the buggy function without changes or introduce new bugs on HUMANEVALFIX.

“They commonly regenerate the buggy function without making any change (e.g.WizardCoder in Figure 34) or they introduce new bugs (e.g.GPT-4 in Figure 33).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Pure code synthesis on HUMANEVALSYNTHESIZE is the easiest task for all models.

“Pure code synthesis on HUMANEVALSYN-THESIZE is the easiest task for all models.”

May combine two claims.

OCTOCODER achieves 46.2% pass@1 on the original Python split, a relative 38% improvement over StarCoder.

“With a pass@1 of 46.2% on the original Python split, OCTOCODER improves by a relative 38% over its base model, StarCoder.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Their coding problems derived from HumanEval are very specific.

“Further, our coding problems, derived from HumanEval, are very specific, such as keeping track of a bank account balance (see Figure 14).”

May combine two claims.

OCTOCODER and OCTOGEEX are the best permissively licensed Code LLMs available but are outperformed by closed-source models such as GPT-4.

“While our models, OCTOCODER and OCTOGEEX, are the best permissively licensed Code LLMs available, they are outperformed by closed-source models such as GPT-4.”

Probably not a substantial claim: may not pass S3 (load-bearing).

They analyzed commits on the Python subset of COMMITPACKFT, which has 56K samples.

“To gain a deeper understanding of the rich content within COMMITPACKFT, we analyze commits on its Python subset (56K samples).”

May combine two claims.

The filters reduce the dataset by a factor of around 1000, resulting in close to 2 GB across 277 languages.

“Our filters reduce the dataset by a factor of around 1000 resulting in close to 2 gigabytes across 277 languages.”

May combine two claims.

The commit pretraining format is unsuitable for HUMANEVALEXPLAIN, which limits its universality.

“However, this pretraining format is not suitable for HUMANEVALEXPLAIN limiting its universality.”

Probably not a substantial claim: may not pass S3 (load-bearing).

By producing 18 scores for 9 models, they take a step towards large-scale benchmarking of code models.

“By producing 18 scores (6 languages across 3 tasks) for 9 models, we take a step towards large-scale benchmarking of code models.”

Probably not a substantial claim: may not pass S3 (load-bearing).

About 49.3% of COMMITPACK repositories also appear in The Stack.

“Around half (49.3%) of them are also in The Stack.”

May combine two claims.

The formatting ablation improves HUMANEVALFIX but reduces performance on other tasks, yielding a worse average score of 35.3.

“While it seems to improve performance for HUMANEVALFIX compared to our default formatting (see Figure 18), it reduces performance on the other tasks leading to a worse average score of 35.3 in Table 12.”

Probably not a substantial claim: may not pass S3 (load-bearing).

COMMITPACK contains 1,934,255 repositories.

“COMMITPACK contains a total of 1,934,255 repositories.”

May combine two claims.

The Stack contains 3 terabytes of data, while COMMITPACK contains close to 4 terabytes.

“Therefore, The Stack only contains 3 terabytes of data, while COMMITPACK contains close to 4.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Docstring statistics are largely similar across languages except for Rust.

“Docstrings are largely the same across languages leading to similar statistics except for Rust.”

May combine two claims.

The Target Loss ablation achieves the best performance on HUMANEVALSYNTHESIZE but has worse average performance than COMMITPACKFT + OASST.

“While this leads to the best performance on HUMANEVALSYNTHESIZE, its average performance is worse compared to COMMITPACKFT + OASST, where the loss is computed over the full sequence.”

May combine two claims.

Manual selection performed better for OCTOGEEX, so it was used for OCTOGEEX.

“It performed better for OCTOGEEX, however, hence we used it for OCTOGEEX.”

May combine two claims.

Line diff finetuning outperforms commit format finetuning, which only provides gains on JavaScript and Java compared to no finetuning.

“It also outperforms finetuning using the commit format, which only provides gains on JavaScript and Java compared to no finetuning.”

May combine two claims.

WizardCoder solves only 11% of excess logic bugs in HUMANEVALFIX, about one-fourth its rate on value misuse bugs.

“WizardCoder is only able to solve 11% of excess logic bugs while solving about four times more bugs that relate to value misuse.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Those two new bugs cause the provided unit tests to fail.

“both of which evidently lead to the provided unit tests failing.”

Probably not a substantial claim: may not pass S3 (load-bearing).

OCTOCODER writes a correct solution to a HUMANEVALEXPLAIN Go problem given its own description.

“Given its own description OCTOCODER writes a correct solution to the problem.”

Probably not a substantial claim: may not pass S3 (load-bearing).

GPT-4 improves the efficiency of that Rust function by starting the inner loop from i+1 instead of 1.

“It additionally improves the efficiency of the function by starting the inner loop from 'i+1' instead of 1 like in the original solution.”

Probably not a substantial claim: may not pass S3 (load-bearing).

BLOOMZ's synthesized function based on its own description simply collects all integers between a and b.

“Thus, its synthesized function based on its own description simply collects all integers between a and b.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

GPT-4 writes a description for a HUMANEVALEXPLAIN Python problem that is significantly shorter than the docstring and within the character limit.

“GPT-4 writes a description that is significantly shorter than the docstring and thus within the character limit.”

Probably not a substantial claim: may not pass S3 (load-bearing).

GPT-4's solution to a HUMANEVALSYNTHESIZE Go problem includes an unused math import, causing a compilation error and test failure.

“GPT-4 includes an import that it does not use (math) which leads to a compilation error in Golang and thus the function does not pass the test cases.”

Probably not a substantial claim: may not pass S3 (load-bearing).

GPT-4 generates the perfect solution from its own description for that Python problem.

“It then generates the perfect solution from its own description.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

OCTOCODER's solution to that Go problem is more compact and efficient than the original solution.

“Its solution is more compact and efficient than the original solution.”

Probably not a substantial claim: may not pass S3 (load-bearing).

WizardCoder returns the same function without making a single change, leaving the bug in a HUMANEVALFIX Python problem.

“WizardCoder returns the same function without making a single change leaving the bug.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Diff Codegen 2B suggests an irrelevant diff modifying parts of the docstring.

“Diff Codegen 2B suggests an irrelevant diff modifying parts of the docstring.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

StarChat-β's description exceeds the 296-character limit and is cut off at the end.

“As the description exceeds the 296-character limit for the code, it is cut off at the end.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

This is one of the simplest bugs that most other models are able to fix.

“This is one of the simplest bugs that most other models are able to fix.”

Probably not a substantial claim: may not pass S3 (load-bearing).

OCTOCODER correctly modifies the 3 to 2 in 6 out of 20 generations for that problem.

“OCTOCODER correctly modifies the 3 to 2 in 6 out of n = 20 generations.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Diff Codegen 2B commonly outputs diffs that modify the docstring or an import statement and rarely addresses the actual bug.

“The model commonly outputs diffs that modify the docstring or an import statement and rarely addresses the actual bug.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

BLOOMZ misses in its description that the function generates only the even digits between a and b, in ascending order.

“BLOOMZ misses in its description that the function generates only the even digits between a and b, in ascending order.”

Probably not a substantial claim: may not pass S3 (load-bearing).

GPT-4 correctly fixes the missing +1 after m-n in a HUMANEVALFIX Python problem.

“GPT-4 correctly fixes that”

Probably not a substantial claim: may not pass S3 (load-bearing).

GPT-4 correctly fixes a HUMANEVALFIX Rust bug by adding an abs statement to compute absolute value.

“GPT-4 correctly fixes the bug by adding the necessary 'abs' statement to compute the absolute value.”

Probably not a substantial claim: may not pass S3 (load-bearing).

GPT-4 fails to catch that the test cases for a HUMANEVALFIX Python problem expect squaring, not cubing n.

“GPT-4 does not catch that the test cases expect the function to square, not cube n.”

Classified as probably not a substantial claim.

Prior work shows models can generalize to programming languages included in pretraining but absent during instruction tuning.

“Prior work has shown that models can generalize to languages included in pretraining, but absent during instruction tuning (Muennighoff et al., 2022b).”

• cites Muennighoff et al., 2022bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The xP3x subset contains natural language outputs, but many are short, causing a bias for short answers and low explanation scores when mixed with OASST.

“While the xP3x subset also contains samples with natural language output, many of its target outputs are short, which leads to models with a bias for short answers.This is impractical for the explanation task leading to the comparatively low score of mixing xP3x with OASST.”

Classified as probably not a substantial claim. May combine two claims.

The bugs were written so that the code still runs but produces an incorrect result causing at least one unit test to fail.

“Bugs are written such that the code still runs but produces an incorrect result leading to at least one unit test failing.”

Classified as probably not a substantial claim.

They removed any solution overlap of at least 20 characters from the explanation to prevent copying.

“To prevent models from copying the solution into the description, we remove any solution overlap of at least 20 characters from the description.”

May combine two claims.

They generated 20 samples per problem, which they state is enough for reliable pass@1 estimates.

“We generate n = 20 samples, which is enough to get reliable pass@1 estimates (Li et al., 2023b).”

• cites Li et al., 2023bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Pretrained StarCoder and the Self-Instruct variant perform poorly on code explanation because they are only conditioned to write code rather than natural language.

“The pretrained StarCoder model, as well as the Self-Instruct variant, perform poorly on code explanation (HUMANEVALEXPLAIN).This is because both models are only conditioned to write code instead of natural language.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

This decontamination validity may not hold for all models, such as GPT-4, so results should be interpreted carefully.

“However, this may not hold for all models (e.g.GPT-4), thus results should be interpreted carefully.”

• AssumptionRests on an assumption: Python HumanEval is the most widely used code benchmark.

Classified as probably not a substantial claim.

Prior work shows natural language performance after instruction tuning correlates with pretraining weight.

“Prior work has shown that the performance on natural languages after instruction tuning is correlated with the weight of these languages during pretraining (Muennighoff et al., 2022b).”

• cites Muennighoff et al., 2022bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The authors' GPT-4 score is significantly higher than prior scores, possibly due to API improvements, HumanEval contamination, or different prompting/evaluation.

“Our score for GPT-4 is significantly higher, possibly due to improvements made to the API by OpenAI, contamination of HumanEval in GPT-4 training, or slightly different prompting and evaluation.”

May combine two claims.

The authors' tuning of OCTOCODER and OCTOGEEX on many COMMITPACKFT languages improved generalization, reflected in HUMANEVALPACK new-language performance.

“To improve programming language generalization further, we tune OCTOCODER and OCTOGEEX on many languages from COMMITPACKFT, and this generalization improvement is reflected in the performance on HUMANEVALPACK's new languages.”

Classified as probably not a substantial claim.

The authors performed careful evaluation to ensure every generation was correctly processed.

“We perform very careful evaluation to ensure every generation is correctly processed.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

GPT-4 was originally found to score 67% on Python HumanEval and 81% in later work.

“GPT-4 was originally found to score 67% on Python HumanEval (OpenAI, 2023) and 81% in later work (Bubeck et al., 2023).”

• cites OpenAI, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Bubeck et al., 2023 sourceNot checked yet

  Checked before — its earlier verdict is held back until a new check runs.

May combine two claims.

Automatic translation approaches are error-prone and only translate tests, not solutions, which are needed for code explanation.

“These approaches are error-prone and only translate tests, not the actual solutions, which are needed for tasks like code explanation.”

May combine two claims.

Python bugs does not support code execution, limiting its utility.

“The latter does not support code execution, which limits its utility.”

Classified as probably not a substantial claim. May combine two claims.

Using data from powerful closed-source models provides a strong advantage but limits model use and has other limitations.

“Using data from powerful closed-source models provides a strong advantage, but limits the model use and has other limitations highlighted in §1.”

May combine two claims.

Quixbugs problems are generic functions, making them easy to solve and hard to decontaminate training data for.

“Further, the problems in Quixbugs are generic functions, such as bucket sort.This makes them easy to solve and hard to decontaminate training data for.”

May combine two claims.

PanGu-Coder2, WizardCoder, and InstructCodeT5+ all use the CodeAlpaca dataset, which is synthetically generated from OpenAI models.

“However, they all use the CodeAlpaca dataset (Chaudhary, 2023), which is synthetically generated from OpenAI models.”

• cites Chaudhary, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Quixbugs supports execution with unit tests but contains only 40 samples in Python and Java.

“While Quixbugs supports execution with unit tests, it only contains 40 samples in Python and Java.”

May combine two claims.

Their work expands CoEditor to more general coding tasks via instructions, more languages, and orders of magnitude more commit data.

“Our work expands this to more general coding tasks via instructions, more languages, and orders of magnitude more commit data.”

May combine two claims.

The filtered data provides more training signal per token and repeats pre-commit code less often.

“Thus, the filtered data carries significantly more signal per token with fewer repetitions of the code prior to the commit.”

• AssumptionRests on an assumption: The code before and after the commit are usually largely the same.

Probably not a substantial claim: may not pass S3 (load-bearing).

SANTACODERPACK may be bottlenecked by its small model size of 1.1B parameters.

“SANTACODERPACK may also be bottlenecked by its small model size of 1.1B parameters.”

Classified as probably not a substantial claim. May combine two claims.

Rust has type annotations for every function, so providing example function calls is less needed than for JavaScript, which lacks type annotations.

“Rust also has type annotations for every function so providing these examples is not as needed as it is for e.g.JavaScript which lacks type annotations.”

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

The Python performance gap is due to a multilingual tax, because SANTACODERPACK must accommodate three additional coding languages (Go, C++ and Rust).

“We hypothesize that this discrepancy is due to a multilingual tax, as SANTACODERPACK needs to accommodate three additional coding languages (Go, C++ and Rust).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Solving from docstrings is generally easier for models than from tests, as models can solve via pure code synthesis without looking at the buggy function.

“Solving from docstrings is generally easier for models than from tests, as models can also solve it via pure code synthesis without looking at the buggy function at all.”

May combine two claims.

pass@k does not require a ground truth explanation, while BLEU and METEOR do.

“While our pass@k formulation does not require a ground truth explanation, BLEU and METEOR do.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Using the prompting format from respective model authors and a simple intuitive prompt for models without a canonical prompt may put models without a canonical prompt recommendation at a slight disadvantage.

“However, this may put models without a canonical prompt recommendation (e.g.BLOOMZ, GPT-4) at a slight disadvantage.”

• AssumptionRests on an assumption: The prompting format can significantly impact performance.

Probably not a substantial claim: may not pass S3 (load-bearing).

One could train a reward model that learns the code after a commit is better given the code before and after.

“Thus, one could train a reward model that given the code before and after a commit, learns that the code afterward is better.”

Probably not a substantial claim: may not pass S4 (based).

Generating only the code diff can make inference significantly more efficient by avoiding repeated characters in output generation.

“Only generating the code diff corresponding to the necessary change can make inference significantly more efficient by avoiding repeated characters in the output generation.”

• AssumptionRests on an assumption: When fixing a code bug, usually only a small part of the code needs to change.

May combine two claims.

Rust scores highest on BLEU/METEOR likely because Rust docstrings contain no example function calls.

“Rust scores are the highest which is likely due to Rust docstrings containing no example function calls (Appendix H).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The greater stability of OCTOGEEX and OCTOCODER across bug types may be due to the diversity of COMMITPACKFT.

“possibly due to the diversity of COMMITPACKFT as displayed in Figure 2.”

Probably not a substantial claim: may not pass S4 (based).

The changed code after a commit generally represents a human-preferred version of the code.

“The changed code after a commit generally represents a human-preferred version of the code (else the code would not have been modified).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Recent work suggests LLM-based evaluation can be biased towards certain contexts.

“However, recent work (Wang et al., 2023b) suggests LLM-based evaluation can be biased towards certain contexts.”

• cites Wang et al., 2023bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Finetuning on the diff format may converge slower than the commit format because the diff format differs significantly from raw code seen during pretraining.

“However, finetuning on the diff format may converge slower than the commit format as the diff format significantly differs from the raw code seen during pretraining.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Adding line numbers diminishes efficiency slightly by adding additional input tokens.

“However, it diminishes efficiency slightly by adding additional input tokens that the model needs to process.”

Probably not a substantial claim: may not pass S4 (based).

Models may accidentally impair otherwise correct code by including a natural language explanation in their output.

“Models may accidentally impair otherwise correct code by e.g.including a natural language explanation in their output.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

BLOOMZ likely left out the even-digit constraint due to its bias for short and concise answers.

“BLOOMZ likely left out this part due to its bias for short and concise answers.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

StarChat-β has a bias for excessively long answers.

“StarChat-β has a bias for excessively long answers.”

Probably not a substantial claim: may not pass S3 (load-bearing).

This bias backfires for this task.

“However, this bias backfires for this task.”