CoChat
Sign Up
Cited by 240Software Engineering ResearchTopic ModelingSoftware Engineering Techniques and Practices

Few-shot training LLMs for project-specific code-summarization

Open access
Ahmed, Toufique, Devanbu, Premkumar·2022·Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering
DOI 10.1145/3551349.3559555
Open this paper in CoChat — free accountOpen PDF

Abstract

Very large language models (LLMs), such as GPT-3 and Codex have achieved state-of-the-art performance on several natural-language tasks, and show great promise also for code. A particularly exciting aspect of LLMs is their knack for few-shot and zero-shot learning: they can learn to perform a task with very few examples. Few-shotting has particular synergies in software engineering, where there are a lot of phenomena (identifier names, APIs, terminology, coding patterns) that are known to be highly project-specific. However, project-specific data can be quite limited, especially early in the history of a project; thus the few-shot learning capacity of LLMs might be very relevant. In this paper, we investigate the use few-shot training with the very large GPT (Generative Pre-trained Transformer) Codex model, and find evidence suggesting that one can significantly surpass state-of-the-art models for code-summarization, leveraging project-specific training.

Integrity signals

Registry

Crossref · 10.1145/3551349.3559555

Cites

25 of 25 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

3 of 9 cited sources checked · 1 supported · 1 flagged · accuracy not yet measured

  • Premisecitation

    BLEU-4 improvements of more than +2 points are reasonable proxies for human-perceptible preference.

    “Roy et al. show that BLEU-4 improvements of more than +2 points are reasonable proxies for human-perceptible preference [18].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [18] 18. Devjeet Roy, Sarah Fakhoury, and Venera Arnaoudova. 2021. Reassessing auto- matic evaluation metrics for code summarization tasks. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1105-1116. source

      In Pairwise Model Comparisons (page unavailable), the paper reports that BLEU's disagreement with human assessment remains high above 2 points: 97.5% for differences of 2–5 points and over 75% in the two highest difference bins. The Implications section (page unavailable) recommends METEOR and chrF, rather than BLEU, as more reliable indicators of human judgment for differences larger than 2 points. (LLM, conf=0.90)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
  • Findingown data

    We found statistically significant improvement for 2 projects over cross-project Codex for same-project training.

    “We found statistically significant improvement for 2 projects (Table 2) over cross-project Codex for same-project training even though we improved for all 8 projects (2% to 46% improvement).”
    Quote in the paperSubstantial claim
  • Findingown data

    For cross-project few-shot training, we observed 1%-15% improvement for all six programming languages.

    “For cross-project setup, we observe 1%-15% improvement for all six programming languages (see Table 1).”
    Quote in the paperSubstantial claim
  • Findingown data

    We found very high statistically significant improvement for four languages in cross-project few-shot training.

    “We also found very high statistically significant improvement for four languages.”
    Quote in the paperSubstantial claim
  • Findingown data

    We failed to find any significant improvement for Python and PHP in cross-project few-shot training.

    “Though we failed to find any significant improvement for Python and PHP, Codex few-shot training still outperforms the traditional fine-tuned pre-trained models with 10 samples.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 11
  • May combine two claims. 13
  • The quote only partly states this claim. 1

  • Classified as probably not a substantial claim.

    Premisecitation

    The performance of deep learning models depends on the identifiers for the code summarization task.

    “Ahmed and Devanbu showed that the performance of the deep learning models depends on the identifiers for the code summarization task [2].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [2] 2. Toufique Ahmed and Premkumar Devanbu. 2022. Multilingual training for soft- ware engineering. In Proceedings of the 44th International Conference on Software Engineering. 1443-1455. source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Using too much data for few-shot or fine-tuning may cause catastrophic forgetting in the model.

    “Also, note that using too much data for few-shot or fine-tuning may cause catastrophic forgetting in the model [13].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [13] 13. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences 114, 13 (2017), 3521- 3526. source

      The Introduction (p. 1) explains that learning a new task sequentially can cause a network to forget an earlier one. Section 2.1 (p. 3) reports that forgetting worsened with more training time, but it does not establish that using too much data in few-shot learning or fine-tuning causes forgetting. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    We improved for all 8 projects in same-project training, with 2% to 46% improvement.

    “even though we improved for all 8 projects (2% to 46% improvement).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The model does continue generating tokens even after completing the summary.

    “However, the model does continue generating tokens even after completing the summary.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The overall improvements from few-shot training are statistically significant for both cross-project and same-project settings.

    “Observation 3. Though we did not observe statistically significant results for all programming languages and all projects, we observe overall statistically significant improvements.”
    • Also stated in: § 4.3 Are we getting statistically significant result? · plain
      “However, for both settings, we observe overall statistically significant improvements.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The majority of the summaries are less than 50 tokens.

    “The majority of the summaries are less than 50 tokens.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Zero-shot and one-shot training in Codex do not work for the code summarization task.

    “Observation 4. Zero-shot and one-shot training in Codex do not work for code summarization task.”
    • Also stated in: § 4.4 Zero-shot and one-shot training · boosted
      “However, we did try both zero-shot and one-shot training with Codex and achieved only 2.96 and 6.22 BLEU-4 on average; we omit details due to the convincingly bad performance.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    Same-project few-shot training improves the Codex model's performance for all 8 projects compared to cross-project few-shot training.

    “Observation 2. Same-project few-shot training improves the Codex model's performance for all 8 projects.”
    • Also stated in: § 4.2 Same-project few-shot · plain
      “Table 2 shows that we outperform all the models, even the Codex model with cross-project data for all the projects under consideration.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    We tried 5, 10, and 15 samples for few-shot training for 1000 test samples from the CodeXGLUE Java code summarization dataset and achieved 19.76, 21.88, and 21.46 BLEU-4, respectively.

    “We tried 5, 10, and 15 samples for few-shot training for 1000 test samples from the CodeXGLUE Java code summarization dataset and achieved 19.76, 21.88, and 21.46 BLEU-4, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Zero-shot and one-shot training with Codex achieved only 2.96 and 6.22 BLEU-4 on average.

    “However, we did try both zero-shot and one-shot training with Codex and achieved only 2.96 and 6.22 BLEU-4 on average; we omit details due to the convincingly bad performance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For both cross-project and same-project settings, we observe overall statistically significant improvements.

    “However, for both settings, we observe overall statistically significant improvements.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    We randomly chose 1000 examples from the test set of all six languages.

    “we randomly chose just 1000 examples from the test set of all six languages.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    CodeXGLUE is a properly deduplicated dataset and uses cross-project splits for training, testing, and dev set.

    “Note that CodeXGLUE is a properly deduplicated dataset and uses the cross-project splits for training, testing, and dev set [19].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [19] 19. Ensheng Shia, Yanlin Wangb, Lun Dub, Junjie Chenc, Shi Hanb, Hongyu Zhangd, Dongmei Zhangb, and Hongbin Suna. 2022. On the Evaluation of Neural Code Summarization. ICSE.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Vocabularies of a project are highly local, and functions from the same projects are likely to share the same set of identifiers.

    “Vocabularies of a project are highly local, and functions from the same projects are likely to share same set of identifiers [10,22].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [10] 10. Vincent J Hellendoorn and Premkumar Devanbu. 2017. Are deep neural networks the best choice for modeling source code?. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. 763-773. source
    • cites [22] 22. Zhaopeng Tu, Zhendong Su, and Premkumar Devanbu. 2014. On the localness of software. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. 269-280. source

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    We randomly chose ten samples from the training set for few-shot training with Codex.

    “We randomly chose ten samples from the training set for few-shot training with Codex.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    We weren't able to fully explore hyper-parameter tuning due to API access limits.

    “We weren't able to fully explore hyper-parameter turning due to API access limits.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Codex few-shot training is sample-efficient, achieving performance with 10 samples that surpasses fine-tuned models trained with thousands of samples.

    “This result suggests that LLMs like Codex are really sample-efficient.”
    • Also stated in: § 4.1 Cross-project few-shot · plain
      “All the baselines are fine-tuned with 24K-251K for each language, whereas the LLM outperforms all of them with just 10 samples!.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning

    None of the prior works reported the performance of few-shot training or investigated the effectiveness of same-project few-shot training.

    “None of the above works reported the performance of few-shot training or investigated the effectiveness of same-project few-shot training.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Few-shot learning has particular synergies in software engineering due to project-specific phenomena such as identifier names, APIs, terminology, and coding patterns.

    “Fewshotting has particular synergies in software engineering, where there are a lot of phenomena (identifier names, APIs, terminology, coding patterns) that are known to be highly project-specific.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisereasoning

    It is not clear how Codex-D is performing compared to other pre-trained models.

    “This is not clear how Codex-D is performing compared to other pre-trained models.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The model's performance in a zero- or one-shot setting is quite abysmal, offering suggestive evidence that the model hasn't just previously memorized the test data.

    “first, it's performance in a zero-or one-shot setting in most cases is quite abysmal.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Applying same-project data is promising and feasible because ten samples for a task like summarization can be generated soon after the development begins.

    “Applying same-project data is very promising and feasible because ten samples for a task like summarization can be generated soon after the development begins.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The model's conditioned generative ability improves with more training samples, suggesting that it is actually generalizing from the few-shots rather than just regurgitating an example it has seen before.

    “This suggests that the model's conditioned generative ability improves with more training samples; the prior that the model internally computes and uses to condition its comment generation (p(comments | code)) is gradually improving with more training samples, suggesting that it is actually generalizing from the few-shots, rather that just regurgiating an example it's seen before.”
    Quote in the paperSubstantial claim
23 of 23

  • The paper relies on this without support.

    Project-specific data, especially early in a project's history, may be limited in volume.

    “However, project-specific data can be quite limited, especially early in the history of a project; thus the few-shot learning capacity of LLMs might be very relevant.”
    “The sticking point here, however, is that project-specific data, especially early-on in a project's history, may be quite limited in volume.”
1 of 1

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The CodeXGLUE benchmark recommends BLEU-CN, and most newer models use this metric.

    “The CodeXGLUE benchmark recommends BLEU-CN, and most newer models [3,21,64] use this metric.”

    From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [3]

    The citation is attached to the claim that newer models use BLEU-CN. In §4.1 (page unavailable), this paper says it uses smoothed BLEU-4; its Background and Related Work section also names BLEU-4, not BLEU-CN, as the metric recommended by CodeXGLUE. (LLM, conf=0.86)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • ASAP defaults to three shots because related work and the authors' own experiments confirmed that more shots did not significantly improve performance.

    “A𝑆𝐴𝑃 defaults to three shots because related work [3,12] has shown, and our own experiments with A𝑆𝐴𝑃 confirmed, that more shots did not significantly improve performance.”

    From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [3]

    The Dataset section reports that 10 shots outperformed 15 in the authors’ code-summarization test, supporting the narrower point that adding shots did not always improve performance. It did not test three shots, and its statistical-significance analysis in Section 4.3 did not compare different shot counts. The source therefore does not establish that increasing ASAP’s three-shot default would yield no significant improvement. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • Large Language Models now typically outperform smaller pre-trained models on many problems.

    “However, Large Language Models (LLMs) now typically outperform smaller pre-trained models on many problems.”

    From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [3]

    Section 4.1 (page not supplied) reports that few-shot Codex outperformed several fine-tuned pre-trained models on code summarization across six programming languages. That supports an example of LLM superiority, but not the broader claim that LLMs typically outperform smaller pre-trained models on many problems. Section 2 also reports a code-summarization case in which an LLM did not outperform fine-tuned models. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Few-shot learning works so well with LLMs that it is unclear whether sufficient task-specific data can ever be gathered to train a customized model to rival their performance.

    “Few-shot learning works so well with LLMs that it is unclear whether sufficient task-specific data can ever be gathered to train a customized model to rival their performance [3,12].”

    From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [3]

    Section 4.1 reports that, for code summarization, few-shot Codex outperformed models fine-tuned on thousands of examples using just ten samples; the Conclusion reiterates this finding. The paper also says project-specific data may be limited, especially early in a project (Introduction). It does not say that it is unclear whether enough task-specific data could ever be gathered for a customized model to rival an LLM, so the citing sentence extends beyond its findings. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Ahmed & Devanbu report that LLMs can outperform pre-trained language models with a simple prompt consisting of just a few samples already in the same project.

    “Ahmed & Devanbu [3] report that LLMs can outperform pre-trained language models with a simple prompt consisting of just a few samples already in the same project”

    From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [3]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Same-project improvement disagrees with the results table

    Minor

    The prose reports a cross-project average of 21.65 BLEU-4 and a 12.56% improvement, while the table reports 21.74 and 12.09% for the same comparison. Both give 24.37 as the same-project average.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81.
  2. Ensheng Shia Yanlin Wangb Lun Dub Junjie Chenc Shi Hanb Hongyu Zhangd Dongmei Zhangb and Hongbin Suna. 2022. On the Evaluation of Neural Code Summarization. ICSE.
  3. Ilya Sutskever Oriol Vinyals and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems. 3104–3112.

Cited by

Sorted by citations

Related

Authors
Ahmed, T., & Devanbu, P.
Affiliations
  • University of California, Davis (US)
  • Computer Science, UC Davis, United States of America
ORCID
Venue
Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering
Publisher
ACM
Published
2022-10-10
Pages
1-5
DOI
10.1145/3551349.3559555
OpenAlex
W4313547549
Cited by
240
References
14
Citations by year
2026: 38 · 2025: 97 · 2024: 87 · 2023: 18
FWCI
45.14
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor, 87 d)
Locations
  • IEEE/ACM International Conference on Automated Software Engineering (ASE) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • NSF (National Science Foundation) (SHF-2107592, SHF-1414172)
  • UC Davis College of Engineering (Dean?s Distinguished Fellowship)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
Keywords
code summarization, few-shot learning, codices, GPT, large language models, software engineering

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review