The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index; 3 of the 9 sources its claims cite were checked for support.
Few-shot training LLMs for project-specific code-summarization
Open accessAbstract
Very large language models (LLMs), such as GPT-3 and Codex have achieved state-of-the-art performance on several natural-language tasks, and show great promise also for code. A particularly exciting aspect of LLMs is their knack for few-shot and zero-shot learning: they can learn to perform a task with very few examples. Few-shotting has particular synergies in software engineering, where there are a lot of phenomena (identifier names, APIs, terminology, coding patterns) that are known to be highly project-specific. However, project-specific data can be quite limited, especially early in the history of a project; thus the few-shot learning capacity of LLMs might be very relevant. In this paper, we investigate the use few-shot training with the very large GPT (Generative Pre-trained Transformer) Codex model, and find evidence suggesting that one can significantly surpass state-of-the-art models for code-summarization, leveraging project-specific training.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
12 Own result · 7 No basis given · 6 Own method · 5 Own interpretation · 2 No full text of the sources · 1 Supported · 1 Partly supported · 1 Not supported · 1 Sources not found · accuracy not yet measured
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Roy et al. show that BLEU-4 improvements of more than +2 points are reasonable proxies for human-perceptible preference [18].”
§ Few-shot training LLMs for project-specific code-summarization > 4.1 Cross-project few-shot
What the sources say
- Disputedcites [18] 18. Devjeet Roy, Sarah Fakhoury, and Venera Arnaoudova. 2021. Reassessing auto- matic evaluation metrics for code summarization tasks. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1105-1116. sourceExperimental — accuracy not yet measured
In Pairwise Model Comparisons (page unavailable), the paper reports that BLEU's disagreement with human assessment remains high above 2 points: 97.5% for differences of 2–5 points and over 75% in the two highest difference bins. The Implications section (page unavailable) recommends METEOR and chrF, rather than BLEU, as more reliable indicators of human judgment for differences larger than 2 points. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found statistically significant improvement for 2 projects (Table 2) over cross-project Codex for same-project training even though we improved for all 8 projects (2% to 46% improvement).”
§ Few-shot training LLMs for project-specific code-summarization > 4.3 Are we getting statistically significant result?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For cross-project setup, we observe 1%-15% improvement for all six programming languages (see Table 1).”
§ Few-shot training LLMs for project-specific code-summarization > 4.3 Are we getting statistically significant result?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also found very high statistically significant improvement for four languages.”
§ Few-shot training LLMs for project-specific code-summarization > 4.3 Are we getting statistically significant result?
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Though we failed to find any significant improvement for Python and PHP, Codex few-shot training still outperforms the traditional fine-tuned pre-trained models with 10 samples.”
§ Few-shot training LLMs for project-specific code-summarization > 4.3 Are we getting statistically significant result?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We found that we could safely fit 10-15 sequences in the prompt and ask the model to generate the comment for us.”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Observation 1.With 10 samples, Codex outperforms all finetuned foundation models CodeT5, CodeBERT, GraphCodeBERT, Polyglot CodeBERT, and PolyGlotGraphCodeBERT in all six programming languages, even though the fine-tuned models are trained with thousands of data.”
- Also stated in: § 4.1 Cross-project few-shot · hedged
“Table 1 suggests that Codex, few-shotted for code summarization, can outperform competitive models.”
- Also stated in: § 6 CONCLUSION · plain
“In this paper, we investigated the effectiveness of fewshot training for code summarization task and found that it can significantly outperform a fine-tuned model trained with thousands of samples with just ten samples.”
§ Few-shot training LLMs for project-specific code-summarization > 4.1 Cross-project few-shot
plaindirectn = 10- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4.1 Cross-project few-shot · hedged
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Second, the performance does smoothly improve, as expected, in most cases upto around 10 training samples embedded in the prompt.”
§ Few-shot training LLMs for project-specific code-summarization > 5 THREATS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“It's been well-known right from the outset that language modeling for code has to deal with project-specific phenomena [10,11,22].”
§ Few-shot training LLMs for project-specific code-summarization > 1 INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites [10] 10. Vincent J Hellendoorn and Premkumar Devanbu. 2017. Are deep neural networks the best choice for modeling source code?. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. 763-773. source
- Not checked — no full text of the sourcecites [11] 11. Abram Hindle, Earl T Barr, Zhendong Su, Mark Gabel, and Premkumar Devanbu. 2012. On the naturalness of software. In2012 34th International Conference on Software Engineering (ICSE). source
- Not checked — no full text of the sourcecites [22] 22. Zhaopeng Tu, Zhendong Su, and Premkumar Devanbu. 2014. On the localness of software. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. 269-280. source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We sorted the functions according to the creation date and ensured that only historical data was used for few-shot training to prevent data leakage from future samples.”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 11
- It may state two separate claims as one. 13
- The quote only partly states this claim. 1
Every source the paper cites for this claim was checked and supports it.
In the paper
“Ahmed and Devanbu showed that the performance of the deep learning models depends on the identifiers for the code summarization task [2].”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
What the sources say
- Supportedcites [2] 2. Toufique Ahmed and Premkumar Devanbu. 2022. Multilingual training for soft- ware engineering. In Proceedings of the 44th International Conference on Software Engineering. 1443-1455. source
Section 2.1 compares code-summarization models fine-tuned with identifiers retained or replaced by a generic token. It reports that abstracting keywords, separators, and operators has a smaller impact on BLEU-4 performance, supporting the claim that performance depends on identifiers. (LLM, conf=0.88)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Also, note that using too much data for few-shot or fine-tuning may cause catastrophic forgetting in the model [13].”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
What the sources say
- Partly supportedcites [13] 13. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences 114, 13 (2017), 3521- 3526. sourceExperimental — accuracy not yet measured
The Introduction (p. 1) explains that learning a new task sequentially can cause a network to forget an earlier one. Section 2.1 (p. 3) reports that forgetting worsened with more training time, but it does not establish that using too much data in few-shot learning or fine-tuning causes forgetting. (LLM, conf=0.84)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“even though we improved for all 8 projects (2% to 46% improvement).”
§ Few-shot training LLMs for project-specific code-summarization > 4.3 Are we getting statistically significant result?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, the model does continue generating tokens even after completing the summary.”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Observation 3. Though we did not observe statistically significant results for all programming languages and all projects, we observe overall statistically significant improvements.”
- Also stated in: § 4.3 Are we getting statistically significant result? · plain
“However, for both settings, we observe overall statistically significant improvements.”
§ Few-shot training LLMs for project-specific code-summarization > 4.3 Are we getting statistically significant result?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 4.3 Are we getting statistically significant result? · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The majority of the summaries are less than 50 tokens.”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Observation 4. Zero-shot and one-shot training in Codex do not work for code summarization task.”
- Also stated in: § 4.4 Zero-shot and one-shot training · boosted
“However, we did try both zero-shot and one-shot training with Codex and achieved only 2.96 and 6.22 BLEU-4 on average; we omit details due to the convincingly bad performance.”
§ Few-shot training LLMs for project-specific code-summarization > Observation 4.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 4.4 Zero-shot and one-shot training · boosted
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Observation 2. Same-project few-shot training improves the Codex model's performance for all 8 projects.”
- Also stated in: § 4.2 Same-project few-shot · plain
“Table 2 shows that we outperform all the models, even the Codex model with cross-project data for all the projects under consideration.”
§ Few-shot training LLMs for project-specific code-summarization > 4.2 Same-project few-shot
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
- Also stated in: § 4.2 Same-project few-shot · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We tried 5, 10, and 15 samples for few-shot training for 1000 test samples from the CodeXGLUE Java code summarization dataset and achieved 19.76, 21.88, and 21.46 BLEU-4, respectively.”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
plaincorrelationaln = 1,000- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we did try both zero-shot and one-shot training with Codex and achieved only 2.96 and 6.22 BLEU-4 on average; we omit details due to the convincingly bad performance.”
§ Few-shot training LLMs for project-specific code-summarization > 4.4 Zero-shot and one-shot training
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, for both settings, we observe overall statistically significant improvements.”
§ Few-shot training LLMs for project-specific code-summarization > 4.3 Are we getting statistically significant result?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“we randomly chose just 1000 examples from the test set of all six languages.”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Note that CodeXGLUE is a properly deduplicated dataset and uses the cross-project splits for training, testing, and dev set [19].”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
What the sources say
- Not matchedcites [19] 19. Ensheng Shia, Yanlin Wangb, Lun Dub, Junjie Chenc, Shi Hanb, Hongyu Zhangd, Dongmei Zhangb, and Hongbin Suna. 2022. On the Evaluation of Neural Code Summarization. ICSE.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“Vocabularies of a project are highly local, and functions from the same projects are likely to share same set of identifiers [10,22].”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
What the sources say
- Not checked — no full text of the sourcecites [10] 10. Vincent J Hellendoorn and Premkumar Devanbu. 2017. Are deep neural networks the best choice for modeling source code?. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. 763-773. source
- Not checked — no full text of the sourcecites [22] 22. Zhaopeng Tu, Zhendong Su, and Premkumar Devanbu. 2014. On the localness of software. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. 269-280. source
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We randomly chose ten samples from the training set for few-shot training with Codex.”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
plainn = 10- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We weren't able to fully explore hyper-parameter turning due to API access limits.”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This result suggests that LLMs like Codex are really sample-efficient.”
- Also stated in: § 4.1 Cross-project few-shot · plain
“All the baselines are fine-tuned with 24K-251K for each language, whereas the LLM outperforms all of them with just 10 samples!.”
§ Few-shot training LLMs for project-specific code-summarization > 4.1 Cross-project few-shot
plaindirectn = 10- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 4.1 Cross-project few-shot · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“None of the above works reported the performance of few-shot training or investigated the effectiveness of same-project few-shot training.”
§ Few-shot training LLMs for project-specific code-summarization > 2 BACKGROUND AND RELATED WORK
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Fewshotting has particular synergies in software engineering, where there are a lot of phenomena (identifier names, APIs, terminology, coding patterns) that are known to be highly project-specific.”
§ Few-shot training LLMs for project-specific code-summarization > Abstract
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This is not clear how Codex-D is performing compared to other pre-trained models.”
§ Few-shot training LLMs for project-specific code-summarization > 2 BACKGROUND AND RELATED WORK
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“first, it's performance in a zero-or one-shot setting in most cases is quite abysmal.”
§ Few-shot training LLMs for project-specific code-summarization > 5 THREATS
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Applying same-project data is very promising and feasible because ten samples for a task like summarization can be generated soon after the development begins.”
§ Few-shot training LLMs for project-specific code-summarization > 6 CONCLUSION
plainn = 10- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This suggests that the model's conditioned generative ability improves with more training samples; the prior that the model internally computes and uses to condition its comment generation (p(comments | code)) is gradually improving with more training samples, suggesting that it is actually generalizing from the few-shots, rather that just regurgiating an example it's seen before.”
§ Few-shot training LLMs for project-specific code-summarization > 5 THREATS
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper relies on this without support.
Project-specific data, especially early in a project's history, may be limited in volume.
“However, project-specific data can be quite limited, especially early in the history of a project; thus the few-shot learning capacity of LLMs might be very relevant.”
“The sticking point here, however, is that project-specific data, especially early-on in a project's history, may be quite limited in volume.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“The CodeXGLUE benchmark recommends BLEU-CN, and most newer models [3,21,64] use this metric.”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [3]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A𝑆𝐴𝑃 defaults to three shots because related work [3,12] has shown, and our own experiments with A𝑆𝐴𝑃 confirmed, that more shots did not significantly improve performance.”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, Large Language Models (LLMs) now typically outperform smaller pre-trained models on many problems.”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Few-shot learning works so well with LLMs that it is unclear whether sufficient task-specific data can ever be gathered to train a customized model to rival their performance [3,12].”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [3]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Ahmed & Devanbu [3] report that LLMs can outperform pre-trained language models with a simple prompt consisting of just a few samples already in the same project”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [3]
The cited source says what the claim attributes to it.
In the citing paper
“Prior work has shown that selecting few-shots from the same projects substantially improves performance [3].”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [3]
The cited source says what the claim attributes to it.
In the citing paper
“We note that Ahmed & Devanbu didn't use IR to select few-shot samples and consistently achieved better results with same-project few-shot learning [3].”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Few-shot training LLMs for project-specific code-summarization > 4.2 Same-project few-shot
“The performance went up from 21.65 BLEU-4 to 24.37 BLEU-4 (12.56% improvement) for the Codex models, which exhibits the effectiveness of few-shot training.”
§ Few-shot training LLMs for project-specific code-summarization > 6 CONCLUSION
“| Average | | | | 19.81 | 20.12 | | 19.67 | 19.94 | 20.39 | | 21.74 | 24.37 | 12.09% | <0.01 |”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Unified Pre-training for Program Understanding and Generation
Ahmad, W., Chakraborty, S., Ray, B., & Chang, K.-W. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.211 · Cited by 642
On the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Dévanbu, P. (2012) · doi:10.5555/2337223.2337322 · Cited by 580
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.04664 · Cited by 536
Towards automatically generating summary comments for Java methods
Sridhara, G., Hill, E., Muppaneni, D., Pollock, L., & Vijay-Shanker, K. (2010) · Proceedings of the IEEE/ACM international conference on Automated software engineering · doi:10.1145/1858996.1859006 · Cited by 514
CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
Husain, H., Wu, H.-H., Gazit, T., Allamanis, M., & Brockschmidt, M. (2019) · arXiv · doi:10.48550/arxiv.1909.09436 · Cited by 483
GraphCodeBERT: Pre-training Code Representations with Data Flow
Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Liu, S., Zhou, L., Duan, N., Svyatkovskiy, A., Fu, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.08366 · Cited by 480
On the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Devanbu, P. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227135 · Cited by 425
Self-citationMeasuring Program Comprehension: A Large-Scale Field Study with Professionals
Xia, X., Bao, L., Lo, D., Xing, Z., Hassan, A. E., & Li, S. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2734091 · Cited by 326
Are deep neural networks the best choice for modeling source code?
Hellendoorn, V. J., & Devanbu, P. (2017) · Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering · doi:10.1145/3106237.3106290 · Cited by 292
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · arXiv · doi:10.48550/arxiv.2002.08155 · Cited by 243
Tu, Z., Su, Z., & Devanbu, P. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635875 · Cited by 238 · cited 2×
“Vocabularies of a project are highly local, and functions from the same projects are likely to share same set of identifiers [10, 22].”
BackgroundInCoder: A Generative Model for Code Infilling and Synthesis
Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, W.-T., Zettlemoyer, L., & Lewis, M. (2022) · arXiv · doi:10.48550/arxiv.2204.05999 · Cited by 187 · cited 1×
“[9] introduce a large language model, InCoder, and try zero-shot training on CodeXGLUE Python dataset.”
MethodReassessing automatic evaluation metrics for code summarization tasks
Roy, D., Fakhoury, S., & Arnaoudova, V. (2021) · Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3468264.3468588 · Cited by 106
Multilingual training for software engineering
Ahmed, T., & Devanbu, P. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510049 · Cited by 57
Self-citationMeasuring program comprehension
Xia, X., Bao, L., Lo, D., Xing, Z., Hassan, A. E., & Li, S. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3182538 · Cited by 22
Research works printed but not linked: 2
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81.
- Ensheng Shia Yanlin Wangb Lun Dub Junjie Chenc Shi Hanb Hongyu Zhangd Dongmei Zhangb and Hongbin Suna. 2022. On the Evaluation of Neural Code Summarization. ICSE.
Cited by
Sorted by citations
Evaluating Large Language Models in Class-Level Code Generation
Du, X., Liu, M., Wang, K., Wang, H., Jun-wei, L., Chen, Y., Feng, J., Sha, C., Peng, X., & Lou, Y. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639219 · Cited by 271
“These include code generation [46, 54, 62, 72], code summarization [12, 51] and various software maintenance tasks, including vulnerability detection [58, 61, 73], test generation [68, 69], and program repair [26, 36, 37, 63].”
MethodRefining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues
Liu, Y., Le-Cong, T., Widyasari, R., Tantithamthavorn, C., Li, L., Le, X.-B. D., & Lo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3643674 · Cited by 149
Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models
Ahmed, T., Ghosh, S., Bansal, C., Zimmermann, T., Zhang, X., & Rajmohan, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00149 · Cited by 148
“Several researchers have addressed code generation [8], [36], docstring generation [8], [61], and code repair [62], [63] problems.”
Self-citationCacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving
Liu, Y., Li, H., Cheng, Y., Ray, S., Huang, Y., Zhang, Q., Du, K., Yao, J., Lu, S., Ananthanarayanan, G., et al. (2024) · Proceedings of the ACM SIGCOMM 2024 Conference · doi:10.1145/3651890.3672274 · Cited by 118
Towards an understanding of large language models in software engineering tasks
Zheng, Z., Ning, K., Zhong, Q., Chen, J., Chen, W., Guo, L., Wang, W., & Wang, Y. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10602-0 · Cited by 212
“Other works assess the capabilities of LLMs in code summarization from different perspectives, such as causal reasoning abilities (Liu et al. 2023d), effectiveness with minimal training (Ahmed and Devanbu 2022), and interpretability (Leinonen et al. 2023a).”
Metadata
- Authors
- Ahmed, T., & Devanbu, P.
- Affiliations
- University of California, Davis (US)
- Computer Science, UC Davis, United States of America
- Venue
- Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering
- Publisher
- ACM
- Published
- 2022-10-10
- Pages
- 1-5
- OpenAlex
- W4313547549
- Cited by
- 240
- References
- 14
- Citations by year
- 2026: 38 · 2025: 97 · 2024: 87 · 2023: 18
- FWCI
- 45.14
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- IEEE/ACM International Conference on Automated Software Engineering (ASE) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- NSF (National Science Foundation) (SHF-2107592, SHF-1414172)
- UC Davis College of Engineering (Dean?s Distinguished Fellowship)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- code summarization, few-shot learning, codices, GPT, large language models, software engineering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
