CoChat
Sign Up
PreprintCited by 3Software Engineering ResearchTopic ModelingTeaching and Learning Programming

Does Few-Shot Learning Help LLM Performance in Code Synthesis?

Open access
Xu, Derek, Xie, Tong, Xia, Botao, Li, Haoyu, Bai, Yunsheng, Sun, Yizhou, Wang, Wei·2024·arXiv
DOI 10.48550/arxiv.2412.02906arXiv 2412.02906
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LLMs) have made significant strides at code generation through improved model design, training, and chain-of-thought. However, prompt-level optimizations remain an important yet under-explored aspect of LLMs for coding. This work focuses on the few-shot examples present in most code generation prompts, offering a systematic study on whether few-shot examples improve LLM's coding capabilities, which few-shot examples have the largest impact, and how to select impactful examples. Our work offers 2 approaches for selecting few-shot examples, a model-free method, CODEEXEMPLAR-FREE, and a model-based method, CODEEXEMPLAR-BASED. The 2 methods offer a trade-off between improved performance and reliance on training data and interpretability. Both methods significantly improve CodeLlama's coding ability across the popular HumanEval+ coding benchmark. In summary, our work provides valuable insights into how to pick few-shot examples in code generation prompts to improve LLM code generation capabilities.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2412.02906

Cites

72 of 72 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

15 of 26 cited sources checked · 10 supported · 3 flagged · accuracy not yet measured

  • Premisecitation

    State-of-the-art models show minimal performance differences between different algorithms.

    “Specifically, state-of-the-art models (Guo et al., 2024;Huang et al., 2023) show minimal performance differences between different algorithms.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Guo et al., 2024 source

      The paper's model comparisons report substantial rather than minimal performance differences. In the Instruction Tuning section, it describes DeepSeek-Coder's 9% and 11% accuracy improvements over a similarly sized CodeLlama model, and Table 3 reports differing benchmark scores across models. The paper compares models, not algorithms as such; no page number is available in our copy. (LLM, conf=0.84)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites Huang et al., 2023 source

      Section 4.2 (RQ1) says AgentCoder outperforms all baseline optimisation approaches, rather than finding minimal differences between algorithms. It reports 89.1% pass@1 for AgentCoder versus 63.0% for CodeCoT on MBPP-ET, and an 8.8-point difference in their average pass@1 scores. (LLM, conf=0.88)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
  • Premisecitation

    Existing techniques for code generation evaluate on predefined prompt templates with little to no modifications.

    “In fact, existing techniques evaluate on predefined prompt templates with little to no modifications (Austin et al., 2021;Liu et al., 2024b).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Austin et al., 2021 source

      Section 3 (p. 5) describes evaluating code synthesis with a specified few-shot prompt format. But Sections 4.2–4.3 (pp. 7–8) report varying the number of test assertions and the choice of prompt examples, so the claim that prompts received little to no modification overstates this paper's methods. The paper also does not establish that this is how existing techniques generally evaluate. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites Liu et al., 2024b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Premisecitation

    Longer prompts require much more compute costs.

    “While performance monotonically improves, longer prompts require much more compute costs (Dao et al., 2022).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Dao et al., 2022 source
  • Interpretationreasoning

    Existing ICL techniques for example selection do not work in our setting.

    “Hence, existing ICL techniques for example selection(Liu et al., 2021) does not work in our setting.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Liu et al., 2021 source
    • Rests on an assumption: Few-shot examples in code synthesis prompts are not in the same input-output space as in-context learning.
  • Findingcitation

    Larger models such as MISTRAL, LLAMA, and CODELLAMA exhibit much better zero-shot coding capabilities than T5-SMALL and T5-BASE.

    “Following neural scaling laws (Kaplan et al., 2020), larger models, such as MISTRAL, LLAMA, and CODELLAMA, exhibit much better zero-shot coding capabilities than T5-SMALL and T5-BASE.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Kaplan et al., 2020
  • May combine two claims. 23
  • Classified as probably not a substantial claim. 33
  • The quote only partly states this claim. 1

  • May combine two claims.

    Premisecitation

    Large pools of test cases can originate from human-generated documentation or GPT-generated unit tests.

    “In practice, large pools of test cases, Dj , can either originate from human generated documentation (Nassif et al., 2021;Lemieux et al., 2023) or GPTgenerated unit tests (Austin et al., 2021).”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites Nassif et al., 2021

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Lemieux et al., 2023 source
    • cites Austin et al., 2021 source

      Austin et al. do not describe GPT-generated unit tests as the source of their test cases. Section 2.1 (p. 4) says crowdworkers wrote three test cases for each MBPP problem; Section 3 (p. 5) describes the model generating code that was checked against those tests, not generating the tests. (LLM, conf=0.84)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation

    LLM in-context learning ability is shaped by which examples are included in the ICL prompt.

    “Inspired by existing work (Liu et al., 2021(Liu et al., , 2024c) ) that show LLM's in-context learning (ICL) ability is shaped by which examples are included in the ICL prompt”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Liu et al., 2021 source
    • cites Liu et al., 2024c

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Mistral is a 7B decoder-only model trained for language generation.

    “MISTRAL (Jiang et al., 2023), which is an 7B decoder-only model trained for language generation,”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Jiang et al., 2023 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    CodeLlama is more specialized in coding tasks.

    “Furthermore CODELLAMA is more specialized in coding tasks thus yielding slightly lower PERPLEX-ITY (TARGET) score compared to MISTRAL despite their comparable sizes.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Roziere et al., 2023 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The main experiments are performed on CodeLlama.

    “The main experiments are performed on CODELLAMA (Roziere et al., 2023).”
    • Also stated in: § Additional information · plain
      “Our experiments are primarily performed on CODELLAMA (Roziere et al., 2023),”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Roziere et al., 2023 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    HumanEval+ augments the original HumanEval dataset with GPT-generated test cases and further test cases from mutation genetic algorithms.

    “This dataset augments the original HumanEval (Chen et al., 2021) dataset with GPT-generated test cases and further test cases from mutation genetic algorithms.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Chen et al., 2021 source
    • cites Liu et al., 2024b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Premisecitation

    Foundation models learn more from data they cannot generate.

    “Inspired by recent work showing foundation models learn more from data they cannot generate (Shumailov et al., 2024)”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Shumailov et al., 2024 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    CodeLlama is a 7B decoder-only model trained on code synthesis.

    “which is a 7B decoder-only model trained on code synthesis.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Roziere et al., 2023 source

      The Abstract describes Code Llama as a family with 7B, 13B, 34B, and 70B variants, rather than a single 7B model. The Introduction describes training for code generation, and Additional information calls the models autoregressive transformers, consistent with the decoder-only description. The 7B description is accurate for a variant, but not for Code Llama as a whole. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    Both CODEEXEMPLAR-FREE and CODEEXAMPLAR-BASE significantly improve CODELLAMA's coding ability on the HUMANEVAL+ benchmark under fixed token constraints.

    “Both methods significantly improve CODELLAMA's coding ability across the popular HUMANEVAL+ coding benchmark.”
    • Also stated in: § 1 Introduction · plain
      “Both approaches substantially improve CODELLAMA's coding capabilities on the HUMANEVAL+ benchmark under fixed token constraints.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The choice of few-shot examples in the code generation prompt significantly affects LLM coding capability, confirmed on T5-SMALL, T5-BASE, MISTRAL, and CODELLAMA on the HUMANEVAL+ benchmark.

    “We confirm our hypothesis on several language models: T5-SMALL, T5-BASE, MISTRAL, and CODELLAMA on the HUMANEVAL+ benchmark.”
    • Also stated in: § 1 Introduction · boosted
      “We demonstrate that choice of few-shot examples in the LLM prompt has a significant effect on LLM coding capabilities across 5 different LLMs.”
    • Also stated in: § 1 Introduction · boosted
      “Given few-shot examples greatly affect LLM coding capability”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingreasoning

    CODEEXAMPLAR-BASE is a data-driven neural network trained on a dataset of code generation prompts and performs better than CODEEXEMPLAR-FREE.

    “the latter offers a better performing data-driven model.”
    • Also stated in: § 1 Introduction · plain
      “We propose a data-driven neural network, trained on a dataset of code generation prompts”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Both CODEEXEMPLAR-FREE and CODEEXAMPLAR-BASE improve CodeLlama's likelihood of generating ground truth code by up to 5.0 through solely optimizing the prompt.

    “Our experiments show both CODEEXEMPLAR-FREE and CODEEXAMPLAR-BASE improves CODELLAMA's likelihood of generating ground truth code by up to 5.0 through solely optimizing the prompt (Figure 5).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    CodeLlama yields slightly lower PERPLEXITY (TARGET) score compared to Mistral despite their comparable sizes.

    “Furthermore CODELLAMA is more specialized in coding tasks thus yielding slightly lower PERPLEX-ITY (TARGET) score compared to MISTRAL despite their comparable sizes.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Adding more few-shot examples to CODELLAMA significantly improves coding performance, then saturates around a log perplexity improvement of around 5.0 after 6 examples.

    “By adding more few-shot examples, CODELLAMA coding performance significantly improves then saturates around a log perplexity improvement of around 5.0 after 6 examples (Figure 2).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Both CODEEXEMPLAR-FREE and CODEEXAMPLAR-BASE improve CodeLlama's raw Pass@1 coding performance by 5.70 ± 2.17% and 5.05 ± 1.70% on average respectively through solely optimizing the prompt.

    “Not only does our approach improve the PERPLEXITY (TARGET) score, but both CODEEXEMPLAR-FREE and CODEEXAMPLAR-BASE improves CODEL-LAMA's raw Pass@1 coding performance by 5.70 ± 2.17% and 5.05 ± 1.70% on average respectively through solely optimizing the prompt (Figure 6).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    CODEEXEMPLAR-FREE and CODEEXAMPLAR-BASE boost model performance by selecting informative few-shot examples quantitatively and automatically.

    “As shown in Section 6, CODEEXEMPLAR-FREE and CODEEXAMPLAR-BASE boost model performance by selecting these informative fewshot examples quantitatively and automatically.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Most few-shot examples improve PERPLEXITY (TARGET), because the median test case improves PERPLEXITY (TARGET).

    “We find most examples improve PERPLEXITY (TARGET), because the median test case improves PERPLEXITY (TAR-GET).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Under the fixed N-shot setting, CODEEXEMPLAR-FREE extracts more effective prompts than both the original prompt (HumanEval) and randomly selecting few-shot examples (Random).

    “Specifically, under the fixed N -shot setting, CODEEXEMPLAR-FREE extracts more effective prompts than both the original prompt (Human Eval), randomly selecting few-shot examples (Random).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Larger models (Mistral and CodeLlama) have lower perplexity generating the test prompts compared to smaller models (T5-BASE and T5-SMALL).

    “As expected the larger models (MISTRAL and CODELLAMA) have lower perplexity generating the test prompts compared to the smaller models (T5-BASE and T5-SMALL).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Pass@1 scores correlate with PERPLEXITY (TARGET) scores, since successful prompts have higher mean perplexity.

    “Our analysis (Table 2) confirms Pass@1 scores correlate with the PERPLEXITY (TARGET) scores, since successful prompts have higher mean perplexity.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    In-context learning enables LLMs to adapt to new tasks by providing examples within the input prompt.

    “In-context learning enables LLMs to adapt to new tasks by providing examples within the input prompt (Brown, 2020).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Brown, 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    CODEEXEMPLAR-FREE is an interpretable, model-free, data-agnostic algorithm requiring no training.

    “The former offers an interpretable dataagnostic algorithm”
    • Also stated in: § 1 Introduction · plain
      “We propose an interpretable model-free algorithm, requiring no training”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Both approaches support arbitrary token cost constraints.

    “Both approaches support arbitrary token cost constraints.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given

    Both algorithms are gray-box: they do not require access to ground-truth weights of the model, only the logits for given input.

    “Both algorithms are gray-box: they do not require access to ground-truth weights of the model, only the logits for given input.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Study design and conductnone given

    We follow the intended usage of the LLMs.

    “We follow the intended usage of the LLMs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Both CODEEXEMPLAR-FREE and CODEEXAMPLAR-BASE meaningfully improve CodeLlama's coding capabilities.

    “We show that both CODEEXEMPLAR-FREE and CODEEXAMPLAR-BASE meaningfully improves CODELLAMA's coding capabilties.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Llama is an 8B decoder-only model trained for language generation.

    “and LLAMA (Dubey et al., 2024), which is a 8B decoder-only model also trained for language generation.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Dubey et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    All other experiments adopt out-of-distribution training, where prompts are split into train and test sets never seen in training.

    “All other experiments in this work adopt the harder out-of-distribution training, where the prompts are split in to train test sets that are never seen in training.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data

    Our prompt optimization techniques have practical application.

    “These results indicate that our prompt optimization techniques have practical application.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    This work tests on the HumanEval+ benchmark.

    “This work tests on the HUMANEVAL+ (Liu et al., 2024b) benchmark.”
    • Also stated in: § Additional information · plain
      “All experiments are conducted on the popular HU-MANEVAL+ (Liu et al., 2024b) benchmark.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Liu et al., 2024b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    There is a need to select the best N ≪ M few-shot examples from the larger pool of M many-shot examples.

    “motivating a need to select the best N ≪ M few-shot examples from the larger pool of M many-shot examples.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: LLM compute costs are determined by the prompt token length.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    We chose our settings by evaluating on the validation set.

    “We chose our settings by evaluating on the validation set.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisereasoning

    The Pass@1 metric is binary on a single coding problem.

    “Specifically, the Pass@1 metric is binary on a single coding problem.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The training dataset was split from HUMANEVAL+ into training and testing splits in an 80:20 ratio, ensuring prompts in the test set do not leak into the training set.

    “We collect the training dataset by splitting the HUMANEVAL+ dataset into the training and testing splits in a 80:20 split, ensuring prompts in the test set do not leak into the training set.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    Generation from scratch entails ensuring input-output pairs both describe ground truth code and maximize Pass@1.

    “generation from scratch entails ensuring input-output pairs both describe ground truth code and maximize Pass@1.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    If the LLM does not expect to read a certain few-shot example, then this example will have a large effect on the LLM's generated output.

    “Intuitively, if the LLM does not expect to read a certain few-shot example, then this example will have a large effect on the LLM's generated output.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data

    There is a need to rank candidate examples by their helpfulness.

    “motivating a need to rank candidate examples, f p , by their helpfulness.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationown data

    These results motivate a learning-based approach, CODEEXAMPLAR-BASE, which encodes semantic representations of one-shot examples to estimate PERPLEXITY (TARGET).

    “These results motivate for our learningbased approach, CODEEXAMPLAR-BASE, which encodes semantic representations of one-shot examples to estimate PERPLEXITY (TARGET).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    All experiments in this work adopt the harder setting, where the train and test set have disjunct natural language descriptions.

    “We highlight all the experiments in this work adopt the harder setting, where the train and test set have disjunct natural language descriptions .”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    T5-SMALL and T5-BASE are 60M and 220M encoder-decoder models trained on many different tasks.

    “We also explore T5-SMALL and T5-BASE (Raffel et al., 2020), which are 60M and 220M encoder-decoder models trained on many different tasks,”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Raffel et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    Unlike Pass@1, PERPLEXITY (TARGET) is a more granular evaluation metric that measures the effect of few-shot examples on individual problems.

    “Unlike Pass@1, PERPLEXITY (TARGET) is a more granular evaluation metric that measures the effect of few-shot examples on individual problems.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The large pool of examples is obtained from the HUMANEVAL+ benchmark to prevent data leakage.

    “However, because the HUMANEVAL+ benchmark (Liu et al., 2024b) already uses GPT to generate its test cases, to prevent data leakage, the large pool of examples is obtained from HUMANEVAL+ benchmark.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Liu et al., 2024b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Better dataset design that improves the prompt generalization of CODEEXAMPLAR-BASE can further improve its performance.

    “This indicates better dataset design which improves the prompt generalization of CODEEXAMPLAR-BASE can further improve its performance, encouraging development of larger coding datasets.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    This work focuses on code generation.

    “This work focuses on code generation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given

    CODEEXEMPLAR-FREE is a model-free algorithm with no hyperparameters.

    “Because CODEEXEMPLAR-FREE is a modelfree algorithm with no hyperparameters.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    This work does not tackle chain of thought prompting, only the original prompt in a zero-shot manner.

    “This work does not tackle chain of thought prompting, only the original prompt in a zero-shot manner.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    We follow the intended usage of the datasets.

    “We follow the intended usage of the datasets.”
    Quote in the paperNot a substantial claim
47 of 47

  • The authors state this as an assumption.

    All few-shot examples will improve performance by disambiguating the natural language prompt.

    “(1) all few-shot examples will improve performance by disambiguating the natural language prompt”

  • The authors state this as an assumption.

    Few-shot examples that the LLM cannot generate are more effective at disambiguating the natural language prompt.

    “(2) few-shot examples that the LLM cannot generate are more effective at disambiguating the natural language prompt.”

  • The authors state this as an assumption.

    Each example is added independently, where the (K+1)th best example is not dependent on which example is the Kth best.

    “our model design assumes each example is added independently, where the (K + 1)th best example is not dependent on which example is the Kth best.”

  • The paper relies on this without support.

    CODELLAMA is the best LLM among the five evaluated models.

    “on the best LLM among the above choices, CODELLAMA”

  • The paper relies on this without support.

    LLM compute costs are determined by the prompt token length.

    “As mentioned previously, LLM comput costs are determined by the prompt token length”

  • The paper relies on this without support.

    Few-shot examples in code synthesis prompts are not in the same input-output space as in-context learning.

    “Unlike in-context learning (ICL), few-shot examples in code synthesis prompts are not in the same input output space.”
6 of 6

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Unit-test allocation has conflicting proportions

    Major

    The evaluation setup gives a 60:20:20 split for the unit tests, but the additional information says test cases are divided 80:20 between evaluation and few-shot examples.

  • Prompt split is reported as both 80:20 and 60:20:20

    Major

    The methods section assigns 80% of prompts to training, while the evaluation setup assigns 60%. These cannot be the same prompt split.

  • An increment example leaves one element unchanged

    Minor

    The stated function increments every list element by 1, but this example maps the input element 2 to 2 rather than 3.

3 of 3

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. [31]31. Jia Li, Ge Li, Yongmin Li, and Zhi Jin. 2023. Struc- tured chain-of-thought prompting for code genera- tion. ACM Transactions on Software Engineering and Methodology.
  2. [48]48. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  3. [55]55. PMLR.
  4. [69]69. PMLR.

Cited by

Sorted by citations

Related

Authors
Xu, D., Xie, T., Xia, B., Li, H., Bai, Y., Sun, Y., & Wang, W.
Affiliations
  • University of California, Los Angeles (US)
  • University of Illinois Urbana-Champaign (US)
  • Nvidia (United Kingdom) (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2412.02906
arXiv
2412.02906
OpenAlex
W4405089172
Semantic Scholar
20a62371018946ac5e6ae5120a8baf9d17cfd5d0
Cited by
3
References
66
Citations by year
2026: 1 · 2025: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
Keywords
few-shot learning, code generation, large language models, prompt engineering, program synthesis, CodeLlama

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review