The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index; 3 of the 9 sources its claims cite were checked for support.
Few-shot training LLMs for project-specific code-summarization
Open accessAbstract
Very large language models (LLMs), such as GPT-3 and Codex have achieved state-of-the-art performance on several natural-language tasks, and show great promise also for code. A particularly exciting aspect of LLMs is their knack for few-shot and zero-shot learning: they can learn to perform a task with very few examples. Few-shotting has particular synergies in software engineering, where there are a lot of phenomena (identifier names, APIs, terminology, coding patterns) that are known to be highly project-specific. However, project-specific data can be quite limited, especially early in the history of a project; thus the few-shot learning capacity of LLMs might be very relevant. In this paper, we investigate the use few-shot training with the very large GPT (Generative Pre-trained Transformer) Codex model, and find evidence suggesting that one can significantly surpass state-of-the-art models for code-summarization, leveraging project-specific training.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
12 Own result · 7 No basis given · 6 Own method · 5 Own interpretation · 2 No full text of the sources · 1 Supported · 1 Partly supported · 1 Not supported · 1 Sources not found · accuracy not yet measured
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Roy et al. show that BLEU-4 improvements of more than +2 points are reasonable proxies for human-perceptible preference [18].”
§ Few-shot training LLMs for project-specific code-summarization > 4.1 Cross-project few-shot
What the sources say
- Disputedcites [18] 18. Devjeet Roy, Sarah Fakhoury, and Venera Arnaoudova. 2021. Reassessing auto- matic evaluation metrics for code summarization tasks. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1105-1116. sourceExperimental — accuracy not yet measured
In Pairwise Model Comparisons (page unavailable), the paper reports that BLEU's disagreement with human assessment remains high above 2 points: 97.5% for differences of 2–5 points and over 75% in the two highest difference bins. The Implications section (page unavailable) recommends METEOR and chrF, rather than BLEU, as more reliable indicators of human judgment for differences larger than 2 points. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found statistically significant improvement for 2 projects (Table 2) over cross-project Codex for same-project training even though we improved for all 8 projects (2% to 46% improvement).”
§ Few-shot training LLMs for project-specific code-summarization > 4.3 Are we getting statistically significant result?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For cross-project setup, we observe 1%-15% improvement for all six programming languages (see Table 1).”
§ Few-shot training LLMs for project-specific code-summarization > 4.3 Are we getting statistically significant result?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also found very high statistically significant improvement for four languages.”
§ Few-shot training LLMs for project-specific code-summarization > 4.3 Are we getting statistically significant result?
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Though we failed to find any significant improvement for Python and PHP, Codex few-shot training still outperforms the traditional fine-tuned pre-trained models with 10 samples.”
§ Few-shot training LLMs for project-specific code-summarization > 4.3 Are we getting statistically significant result?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We found that we could safely fit 10-15 sequences in the prompt and ask the model to generate the comment for us.”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Observation 1.With 10 samples, Codex outperforms all finetuned foundation models CodeT5, CodeBERT, GraphCodeBERT, Polyglot CodeBERT, and PolyGlotGraphCodeBERT in all six programming languages, even though the fine-tuned models are trained with thousands of data.”
- Also stated in: § 4.1 Cross-project few-shot · hedged
“Table 1 suggests that Codex, few-shotted for code summarization, can outperform competitive models.”
- Also stated in: § 6 CONCLUSION · plain
“In this paper, we investigated the effectiveness of fewshot training for code summarization task and found that it can significantly outperform a fine-tuned model trained with thousands of samples with just ten samples.”
§ Few-shot training LLMs for project-specific code-summarization > 4.1 Cross-project few-shot
plaindirectn = 10- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4.1 Cross-project few-shot · hedged
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Second, the performance does smoothly improve, as expected, in most cases upto around 10 training samples embedded in the prompt.”
§ Few-shot training LLMs for project-specific code-summarization > 5 THREATS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“It's been well-known right from the outset that language modeling for code has to deal with project-specific phenomena [10,11,22].”
§ Few-shot training LLMs for project-specific code-summarization > 1 INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites [10] 10. Vincent J Hellendoorn and Premkumar Devanbu. 2017. Are deep neural networks the best choice for modeling source code?. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. 763-773. source
- Not checked — no full text of the sourcecites [11] 11. Abram Hindle, Earl T Barr, Zhendong Su, Mark Gabel, and Premkumar Devanbu. 2012. On the naturalness of software. In2012 34th International Conference on Software Engineering (ICSE). source
- Not checked — no full text of the sourcecites [22] 22. Zhaopeng Tu, Zhendong Su, and Premkumar Devanbu. 2014. On the localness of software. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. 269-280. source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We sorted the functions according to the creation date and ensured that only historical data was used for few-shot training to prevent data leakage from future samples.”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 11
- It may state two separate claims as one. 13
- The quote only partly states this claim. 1
Every source the paper cites for this claim was checked and supports it.
In the paper
“Ahmed and Devanbu showed that the performance of the deep learning models depends on the identifiers for the code summarization task [2].”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
What the sources say
- Supportedcites [2] 2. Toufique Ahmed and Premkumar Devanbu. 2022. Multilingual training for soft- ware engineering. In Proceedings of the 44th International Conference on Software Engineering. 1443-1455. source
Section 2.1 compares code-summarization models fine-tuned with identifiers retained or replaced by a generic token. It reports that abstracting keywords, separators, and operators has a smaller impact on BLEU-4 performance, supporting the claim that performance depends on identifiers. (LLM, conf=0.88)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Also, note that using too much data for few-shot or fine-tuning may cause catastrophic forgetting in the model [13].”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
What the sources say
- Partly supportedcites [13] 13. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences 114, 13 (2017), 3521- 3526. sourceExperimental — accuracy not yet measured
The Introduction (p. 1) explains that learning a new task sequentially can cause a network to forget an earlier one. Section 2.1 (p. 3) reports that forgetting worsened with more training time, but it does not establish that using too much data in few-shot learning or fine-tuning causes forgetting. (LLM, conf=0.84)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“even though we improved for all 8 projects (2% to 46% improvement).”
§ Few-shot training LLMs for project-specific code-summarization > 4.3 Are we getting statistically significant result?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, the model does continue generating tokens even after completing the summary.”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Observation 3. Though we did not observe statistically significant results for all programming languages and all projects, we observe overall statistically significant improvements.”
- Also stated in: § 4.3 Are we getting statistically significant result? · plain
“However, for both settings, we observe overall statistically significant improvements.”
§ Few-shot training LLMs for project-specific code-summarization > 4.3 Are we getting statistically significant result?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 4.3 Are we getting statistically significant result? · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The majority of the summaries are less than 50 tokens.”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Observation 4. Zero-shot and one-shot training in Codex do not work for code summarization task.”
- Also stated in: § 4.4 Zero-shot and one-shot training · boosted
“However, we did try both zero-shot and one-shot training with Codex and achieved only 2.96 and 6.22 BLEU-4 on average; we omit details due to the convincingly bad performance.”
§ Few-shot training LLMs for project-specific code-summarization > Observation 4.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 4.4 Zero-shot and one-shot training · boosted
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Observation 2. Same-project few-shot training improves the Codex model's performance for all 8 projects.”
- Also stated in: § 4.2 Same-project few-shot · plain
“Table 2 shows that we outperform all the models, even the Codex model with cross-project data for all the projects under consideration.”
§ Few-shot training LLMs for project-specific code-summarization > 4.2 Same-project few-shot
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
- Also stated in: § 4.2 Same-project few-shot · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We tried 5, 10, and 15 samples for few-shot training for 1000 test samples from the CodeXGLUE Java code summarization dataset and achieved 19.76, 21.88, and 21.46 BLEU-4, respectively.”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
plaincorrelationaln = 1,000- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we did try both zero-shot and one-shot training with Codex and achieved only 2.96 and 6.22 BLEU-4 on average; we omit details due to the convincingly bad performance.”
§ Few-shot training LLMs for project-specific code-summarization > 4.4 Zero-shot and one-shot training
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, for both settings, we observe overall statistically significant improvements.”
§ Few-shot training LLMs for project-specific code-summarization > 4.3 Are we getting statistically significant result?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“we randomly chose just 1000 examples from the test set of all six languages.”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Note that CodeXGLUE is a properly deduplicated dataset and uses the cross-project splits for training, testing, and dev set [19].”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
What the sources say
- Not matchedcites [19] 19. Ensheng Shia, Yanlin Wangb, Lun Dub, Junjie Chenc, Shi Hanb, Hongyu Zhangd, Dongmei Zhangb, and Hongbin Suna. 2022. On the Evaluation of Neural Code Summarization. ICSE.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“Vocabularies of a project are highly local, and functions from the same projects are likely to share same set of identifiers [10,22].”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
What the sources say
- Not checked — no full text of the sourcecites [10] 10. Vincent J Hellendoorn and Premkumar Devanbu. 2017. Are deep neural networks the best choice for modeling source code?. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. 763-773. source
- Not checked — no full text of the sourcecites [22] 22. Zhaopeng Tu, Zhendong Su, and Premkumar Devanbu. 2014. On the localness of software. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. 269-280. source
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We randomly chose ten samples from the training set for few-shot training with Codex.”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
plainn = 10- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We weren't able to fully explore hyper-parameter turning due to API access limits.”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This result suggests that LLMs like Codex are really sample-efficient.”
- Also stated in: § 4.1 Cross-project few-shot · plain
“All the baselines are fine-tuned with 24K-251K for each language, whereas the LLM outperforms all of them with just 10 samples!.”
§ Few-shot training LLMs for project-specific code-summarization > 4.1 Cross-project few-shot
plaindirectn = 10- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 4.1 Cross-project few-shot · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“None of the above works reported the performance of few-shot training or investigated the effectiveness of same-project few-shot training.”
§ Few-shot training LLMs for project-specific code-summarization > 2 BACKGROUND AND RELATED WORK
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Fewshotting has particular synergies in software engineering, where there are a lot of phenomena (identifier names, APIs, terminology, coding patterns) that are known to be highly project-specific.”
§ Few-shot training LLMs for project-specific code-summarization > Abstract
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This is not clear how Codex-D is performing compared to other pre-trained models.”
§ Few-shot training LLMs for project-specific code-summarization > 2 BACKGROUND AND RELATED WORK
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“first, it's performance in a zero-or one-shot setting in most cases is quite abysmal.”
§ Few-shot training LLMs for project-specific code-summarization > 5 THREATS
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Applying same-project data is very promising and feasible because ten samples for a task like summarization can be generated soon after the development begins.”
§ Few-shot training LLMs for project-specific code-summarization > 6 CONCLUSION
plainn = 10- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This suggests that the model's conditioned generative ability improves with more training samples; the prior that the model internally computes and uses to condition its comment generation (p(comments | code)) is gradually improving with more training samples, suggesting that it is actually generalizing from the few-shots, rather that just regurgiating an example it's seen before.”
§ Few-shot training LLMs for project-specific code-summarization > 5 THREATS
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper relies on this without support.
Project-specific data, especially early in a project's history, may be limited in volume.
“However, project-specific data can be quite limited, especially early in the history of a project; thus the few-shot learning capacity of LLMs might be very relevant.”
“The sticking point here, however, is that project-specific data, especially early-on in a project's history, may be quite limited in volume.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“The CodeXGLUE benchmark recommends BLEU-CN, and most newer models [3,21,64] use this metric.”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [3]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A𝑆𝐴𝑃 defaults to three shots because related work [3,12] has shown, and our own experiments with A𝑆𝐴𝑃 confirmed, that more shots did not significantly improve performance.”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, Large Language Models (LLMs) now typically outperform smaller pre-trained models on many problems.”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Few-shot learning works so well with LLMs that it is unclear whether sufficient task-specific data can ever be gathered to train a customized model to rival their performance [3,12].”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [3]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Ahmed & Devanbu [3] report that LLMs can outperform pre-trained language models with a simple prompt consisting of just a few samples already in the same project”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [3]
The cited source says what the claim attributes to it.
In the citing paper
“Prior work has shown that selecting few-shots from the same projects substantially improves performance [3].”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [3]
The cited source says what the claim attributes to it.
In the citing paper
“We note that Ahmed & Devanbu didn't use IR to select few-shot samples and consistently achieved better results with same-project few-shot learning [3].”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Few-shot training LLMs for project-specific code-summarization > 4.2 Same-project few-shot
“The performance went up from 21.65 BLEU-4 to 24.37 BLEU-4 (12.56% improvement) for the Codex models, which exhibits the effectiveness of few-shot training.”
§ Few-shot training LLMs for project-specific code-summarization > 6 CONCLUSION
“| Average | | | | 19.81 | 20.12 | | 19.67 | 19.94 | 20.39 | | 21.74 | 24.37 | 12.09% | <0.01 |”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495
Research works printed but not linked: 2
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81.
- Ensheng Shia Yanlin Wangb Lun Dub Junjie Chenc Shi Hanb Hongyu Zhangd Dongmei Zhangb and Hongbin Suna. 2022. On the Evaluation of Neural Code Summarization. ICSE.
Cited by
Sorted by citations
Evaluating Large Language Models in Class-Level Code Generation
Du, X., Liu, M., Wang, K., Wang, H., Jun-wei, L., Chen, Y., Feng, J., Sha, C., Peng, X., & Lou, Y. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639219 · Cited by 271
“These include code generation [46, 54, 62, 72], code summarization [12, 51] and various software maintenance tasks, including vulnerability detection [58, 61, 73], test generation [68, 69], and program repair [26, 36, 37, 63].”
MethodRefining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues
Liu, Y., Le-Cong, T., Widyasari, R., Tantithamthavorn, C., Li, L., Le, X.-B. D., & Lo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3643674 · Cited by 149
Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models
Ahmed, T., Ghosh, S., Bansal, C., Zimmermann, T., Zhang, X., & Rajmohan, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00149 · Cited by 148
“Several researchers have addressed code generation [8], [36], docstring generation [8], [61], and code repair [62], [63] problems.”
Self-citationCacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving
Liu, Y., Li, H., Cheng, Y., Ray, S., Huang, Y., Zhang, Q., Du, K., Yao, J., Lu, S., Ananthanarayanan, G., et al. (2024) · Proceedings of the ACM SIGCOMM 2024 Conference · doi:10.1145/3651890.3672274 · Cited by 118
Towards an understanding of large language models in software engineering tasks
Zheng, Z., Ning, K., Zhong, Q., Chen, J., Chen, W., Guo, L., Wang, W., & Wang, Y. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10602-0 · Cited by 212
“Other works assess the capabilities of LLMs in code summarization from different perspectives, such as causal reasoning abilities (Liu et al. 2023d), effectiveness with minimal training (Ahmed and Devanbu 2022), and interpretability (Leinonen et al. 2023a).”
What Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs?
Gao, S., Wen, X.-C., Gao, C., Wang, W., Zhang, H., & Lyu, M. R. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00109 · Cited by 106
“Due to the space limitation, we only present the performance on EM and CB and the results on other metrics can be found in our replication package.”
Exploring and Unleashing the Power of Large Language Models in Automated Code Translation
Yang, Z., Liu, F., Yu, Z., Keung, J. W., Li, J., Liu, S., Hong, Y., Ma, X., Jin, Z., & Li, G. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660778 · Cited by 104
“…Large Language Models (LLMs), pre-trained on billions of text/code tokens, bypass the need for re-training/fine-tuning but demonstrate the powerful generality of various code-related tasks, such as code generation [7, 11, 23, 25, 28, 52], program repair [12, 48], and code summarization [2, 14].”
BackgroundAutomatic Semantic Augmentation of Language Model Prompts (for Code Summarization)
Ahmed, T., Pai, K. S., Devanbu, P., & Barr, E. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639183 · Cited by 112
“We now discuss our dataset, models, and methodology.”
BackgroundMethodInfluentialSelf-citationGenerative AI for self-adaptive systems: State of the art and research roadmap
Li, J., Zhang, M., Li, N., Weyns, D., Jin, Z., & Tei, K. (2024) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3686803 · Cited by 77
Generative Artificial Intelligence for Software Engineering - a Research Agenda
NguyenDuc, A., Cabrero-Daniel, B., Arora, C., Przybylek, A., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K.-K., et al. (2023) · doi:10.2139/ssrn.4622517 · Cited by 71
Large Language Models and Simple, Stupid Bugs
Jesse, K., Ahmed, T., Dévanbu, P., & Morgan, E. (2023) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr59073.2023.00082 · Cited by 83
“[6], [9] apply Codex in various settings: automatic program repair (APR) [5], security vulnerability prediction [22], HackerRank challenges [21], code summarization [6], and incident management [9].”
BackgroundSelf-citationLLM for Test Script Generation and Migration: Challenges, Capabilities, and Opportunities
Yu, S., Fang, C., Ling, Y., Wu, C., & Chen, Z. (2023) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs60937.2023.00029 · Cited by 100
“The investigation also identifies limitations, including issues related to context memory, API usage randomness, and human effort requirements, and thus provides valuable insights for future research and development in the field of mobile app test script generation and migration.”
BackgroundNuances are the Key: Unlocking ChatGPT to Find Failure-Inducing Tests with Differential Prompting
Li, T.-O., Zong, W., Wang, Y., Tian, H., Wang, Y., Cheung, S.-C., & Kramer, J. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00089 · Cited by 70
“There are also other works studying ChatGPT’s (or other LLMs’) capabilities for various software engineering tasks, including bug repair [25], [3], fuzzing [22], code generation [51], [52], code summarization [53], software testing education [54], and vulnerability detection [55], [56].”
BackgroundOn the use of large language models in model-driven engineering
Rocco, J. D., Ruscio, D. D., Sipio, C. D., Nguyen, P. T., & Rubei, R. (2025) · Software & Systems Modeling · doi:10.1007/s10270-025-01263-8 · Cited by 73
“– Code Summarization: LLMs can be employed to automatically summarize codebases, functions, or methods [4].”
On the Effectiveness of Large Language Models in Domain-Specific Code Generation
Gu, X., Chen, M., Lin, Y., Hu, Y., Zhang, H., Wan, C., Wei, Z., Xu, Y., & Wang, J. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3697012 · Cited by 92
“For the chain-of-thought models, we do not count the tokens occupied by knowledge states in the overall length.”
Source Code Summarization in the Era of Large Language Models
Sun, W., Miao, Y., Li, Y., Zhang, H., Fang, C., Liu, Y., Deng, G., Liu, Y., & Chen, Z. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00034 · Cited by 157
Formal requirements engineering and large language models: A two-way roadmap
Ferrari, A., & Spoletini, P. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107697 · Cited by 71
Bhatia, S., Gandhi, T., Kumar, D., & Jalote, P. (2024) · Proceedings of the 1st International Workshop on Large Language Models for Code · doi:10.1145/3643795.3648396 · Cited by 55
A Comparative Analysis of Large Language Models for Code Documentation Generation
Dvivedi, S. S., Vijay, V., Pujari, S. L. R., Lodh, S., & Kumar, D. (2024) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1145/3664646.3664765 · Cited by 67
SpecGen: Automated Generation of Formal Program Specifications via Large Language Models
Ma, L., Liu, S., Li, Y., Xie, X., & Bu, L. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00129 · Cited by 179
Better Patching Using LLM Prompting, via Self-Consistency
Ahmed, T., & Dévanbu, P. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00065 · Cited by 56
Self-citationShi, J., Yang, Z., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708525 · Cited by 43
Trustworthy and Synergistic Artificial Intelligence for Software Engineering: Vision and Roadmaps
Lo, D. (2023) · IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00010 · Cited by 72
“This is because practitioners may find their trust dwindling if they do not witness an increase in favorable outcomes that is commensurate with an increased effort in scrutinizing an AI4SE solution’s recommendations.”
BackgroundApplicability of large language models and generative models for legal case judgement summarization
Deroy, A., Ghosh, K., & Ghosh, S. (2024) · Artificial Intelligence and Law · doi:10.1007/s10506-024-09411-z · Cited by 48
“A recent work [22] on code summarization using GPT used zero-shot and few-shot capabilities of LLMs for project-specific code summarization.”
MethodAljedaani, W., Habib, A., Aljohani, A., Eler, M., & Feng, Y. (2024) · Proceedings of the 21st International Web for All Conference · doi:10.1145/3677846.3677854 · Cited by 42
LLM-Powered Static Binary Taint Analysis
Liu, P., Sun, C. P., Zheng, Y., Feng, X., Qin, C., Wang, Y., Xu, Z., Li, Z., Di, P., Jiang, Y., et al. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3711816 · Cited by 96
State of Practice: LLMs in Software Engineering and Software Architecture
Jahić, J., & Sami, A. (2024) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c63560.2024.00059 · Cited by 55
“These challenges range from program repair [10] [11] [12] to code summarization [13] [14].”
BackgroundMultilingual Code Co-evolution using Large Language Models
Zhang, J., Nie, P., Li, J. J., & Gligoric, M. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616350 · Cited by 50
Can LLMs Replace Manual Annotation of Software Engineering Artifacts?
Ahmed, T., Dévanbu, P., Treude, C., & Pradel, M. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00086 · Cited by 105
“Third, advances in prompt engineering, such as the impressive abilities of LLMs at in-context learning [17], [18], suggest that with a few illustrative examples, LLMs can execute fairly complex software engineering tasks.”
Self-citationLLM-enabled Intent-driven Service Configuration for Next Generation Networks
Mekrache, A., & Ksentini, A. (2024) · IEEE International Conference on Network Softwarization (NetSoft) · doi:10.1109/netsoft60951.2024.10588881 · Cited by 50
“Few-shot learning has been shown to be effective for a variety of tasks, including code summarization [14], question answering [15], and machine translation [16].”
BackgroundA study on prompt design, advantages and limitations of ChatGPT for deep learning program repair
Cao, J., Li, M., Wen, M., & Cheung, S.-C. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00492-x · Cited by 136
“With the breakthrough of LLMs, studies have proposed to explore how LLMs can be used to assist developers in a variety of tasks, such as code generation [35, 49], program repair [13, 20, 21, 29, 33, 40, 48], and code summarization [2, 49].”
Prompt Engineering or Fine-Tuning: An Empirical Assessment of LLMs for Code
Shin, J., Tang, C. J., Mohati, T., Nayebi, M., Wang, S., & Hemmati, H. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00082 · Cited by 165
“We select three typical automated code-related software engineering tasks for the experiments, i.e., Code summarization: [45]–[47]The model generates a short natural language summary from a source code snippet (SC-to-NL), Code generation: [48]–[50]The model generates the corresponding code snippet…”
Can LLMs Reason Like Humans? Assessing Theory of Mind Reasoning in LLMs for Open-Ended Questions
Amirizaniani, M., Martin, E., Sivachenko, M., Mashhadi, A., & Shah, C. S. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679832 · Cited by 78
“Despite the proficiency of models such as Zephyr-7B, Llama2-Chat-13B, and GPT-4 in structured tasks like summarization [3, 17, 48], question answering [1, 47], and language translation [15, 29], they exhibit deficiencies in ToM reasoning with open-ended questions.”
Bias Testing and Mitigation in LLM-based Code Generation
Huang, D., Zhang, J. M., Bu, Q., Xie, X., Chen, J., & Cui, H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3724117 · Cited by 128
“Biases in automatically generated code for adult income prediction will lead to disparities in salary offers during recruitment, severely affect-Bias ing an individual’s financial stability and career progression.”
Type-migrating C-to-Rust translation using a large language model
Hong, J., & Ryu, S. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10573-2 · Cited by 31
From Commit Message Generation to History-Aware Commit Message Completion
Eliseeva, A., Sokolov, Y., Bogomolov, E., Golubev, Y., Dig, D., & Bryksin, T. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00078 · Cited by 36
“Note that many previous works used comparable or even smaller datasets when experimenting with LLMs [38], [41], [43], [44].”
ResultX-Lifecycle Learning for Cloud Incident Management using LLMs
Goel, D., Husain, F., Singh, A., Ghosh, S., Parayil, A., Bansal, C., Zhang, X., & Rajmohan, S. (2024) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3663529.3663861 · Cited by 33
Text2VRScene: Exploring the Framework of Automated Text-driven Generation System for VR Experience
Yin, Z., Wang, Y., Papatheodorou, T., & Hui, P. (2024) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr58804.2024.00090 · Cited by 29
“In addition to generating natural language responses, LLMs can be equipped with automatic techniques such as code generation [9,15], program repair [15,21], and code summarization [1].”
MethodNashaat, M., & Miller, J. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3428324 · Cited by 30
“Moreover, large language models have been applied to code summarization using few-shot and zero-shot learning [47], [48].”
MethodOn the Effectiveness of LLM-as-a-Judge for Code Generation and Summarization
Crupi, G., Tufano, R., Velasco, A., Mastropaolo, A., Poshyvanyk, D., & Bavota, G. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3586082 · Cited by 55
Systematic Literature Review of Prompt Engineering Patterns in Software Engineering
Sasaki, Y., Washizaki, H., Li, J., Sander, D., Yoshioka, N., & Fukazawa, Y. (2024) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00096 · Cited by 27
“• PA. Learning Approach: Seven papers address the pattern to provide the context necessary for their purpose to LLMs [9]- [15].”
BackgroundDetecting Code Comment Inconsistencies using LLM and Program Analysis
Zhang, Y. (2024) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3663529.3664458 · Cited by 25
Esale: Enhancing Code-Summary Alignment Learning for Source Code Summarization
Fang, C., Sun, W., Chen, Y., Chen, X., Wei, Z., Zhang, Q., You, Y., Luo, B., Liu, Y., & Chen, Z. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3422274 · Cited by 25
Kumar, J., & Chimalakonda, S. (2024) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3661167.3661210 · Cited by 24
LLM-Based Chatbots for Mining Software Repositories: Challenges and Opportunities
Abedu, S., Abdellatif, A., & Shihab, E. (2024) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3661167.3661218 · Cited by 28
“Prior studies on using LLMs for software engineering tasks mainly focus on program repairs [13, 20], code generation [18, 47, 51], code reviews [26, 27, 48] and code summarization [3, 14, 45].”
BackgroundMethodInterleaving Static Analysis and LLM Prompting
Chapman, P. J., Rubio-González, C., & Thakur, A. (2024) · State Of the Art in Java Program Analysis · doi:10.1145/3652588.3663317 · Cited by 19
Lv, S., Peng, L., Wu, W., Yao, Y., Jiao, S., & Hu, W. (2025) · Materials Genome Engineering Advances · doi:10.1002/mgea.70013 · Cited by 18
LUNA: A Model-Based Universal Analysis Framework for Large Language Models
Song, D., Xie, X., Song, J., Zhu, D., Huang, Y., Juefei-Xu, F., & Ma, L. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3411928 · Cited by 33
“Particularly, abstract model-wise metrics assess the intrinsic properties of the model regardless of subject trustworthiness perspectives, such as the stability of the model and the degree of well-fitting to the distribution of training data.”
BackgroundSarda, K., Namrud, Z., Litoiu, M., Shwartz, L., & Watts, I. (2024) · Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering · doi:10.1145/3663529.3663855 · Cited by 19
“Moreover, numerous scholars have conducted research on code generation [9, 85], creating docstrings [3, 9], and repairing code [15, 36].”
A review of automatic source code summarization
Zhang, X., Hou, X., Qiao, X., & Song, W. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10553-6 · Cited by 18
GPTuner: An LLM-Based Database Tuning System
Lao, J., Wang, Y., Li, Y., Wang, J., Zhang, Y., Cheng, Z., Chen, W., Tang, M., & Wang, J. (2025) · ACM SIGMOD Record · doi:10.1145/3733620.3733641 · Cited by 17
Generative AI for Software Development: A Family of Studies on Code Generation
Dakhel, A. M., Nikanjam, A., Khomh, F., Desmarais, M. C., & Washizaki, H. (2024) · Springer eBooks · doi:10.1007/978-3-031-55642-5_7 · Cited by 18
Reasoning and Planning with Large Language Models in Code Development
Ding, H., Fan, Z., Guehring, I., Gupta, G. P., Ha, W., Huan, J., Liu, L. Q., Omidvar-Tehrani, B., Wang, S., & Zhou, H. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671452 · Cited by 23
Giannilias, T., Papadakis, A. E., Nikolaou, N., & Zahariadis, T. B. (2025) · Future Internet · doi:10.3390/fi17050207 · Cited by 16
Harrath, Y., Adohinzin, O., Kaabi, J., & Saathoff, M. (2025) · Computers · doi:10.3390/computers14090374 · Cited by 18
“GPT models, particularly GPT-3, demonstrated the capability of short-cut learning, where the model could perform tasks with minimal examples, showcasing an unprecedented level of adapt-ability [124,125].”
LLMSmartSec: Smart Contract Security Auditing with LLM and Annotated Control Flow Graph
(2024) · doi:10.1109/blockchain62396.2024.00064 · Cited by 13
“Numerous studies [12], [13], [14] explore LLMs for code analysis, demonstrating that LLMs can efficiently and accurately handle large volumes of complex code.”
BackgroundUsing LLMs to Customize the UI of Webpages
(2023) · doi:10.1145/3586182.3616671 · Cited by 13
ChatGPT Incorrectness Detection in Software Reviews
Tanzil, M. H., Khan, J. Y., & Uddin, G. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639194 · Cited by 23
RMCBench: Benchmarking Large Language Models' Resistance to Malicious Code
Chen, J., Zhong, Q., Wang, Y., Ning, K., Liu, Y., Xu, Z., Zhao, Z., Chen, T., & Zheng, Z. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695480 · Cited by 22
“This process can be achieved through code summarization [12] from malicious code, which is the task used to extract textual descriptions from code.”
BackgroundMethodCalibration of Large Language Models on Code Summarization
Virk, Y., Dévanbu, P., & Ahmed, T. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729400 · Cited by 26
“…et al. 2020; Nashid et al. 2023; Su et al. 2024; Su and McMillan 2024]; with such powerful LLMs, in-context learning techniques e.g few-shotting [Ahmed and Devanbu 2022a; Brown et al. 2020], chain-of-thought [Sun et al. 2024], and semantic augmentation [Ahmed et al. 2024], attain state-of-the…”
InfluentialSelf-citationContextual Predictive Mutation Testing
Jain, K., Alon, U., Groce, A., & Goues, C. L. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616289 · Cited by 11
“This is consistent with other results showing that projects have distinct vocabulary and style, making cross project prediction difficult for many tasks [3, 13].”
Xiao, X., Chen, P., Qi, B., Zhao, H., Liang, J., Tong, J., & Wang, H. (2025) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2025.111585 · Cited by 12
CoDefeater: Using LLMs To Find Defeaters in Assurance Cases
Gohar, U., Hunter, M. C., Lutz, R. R., & Cohen, M. B. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695296 · Cited by 12
Fine-Tuning Large Language Models for Task Specific Data
Ramesh, R., Akarsh Thejasvi Raju, M., Reddy, H. V., & Sandeep Varma, N. (2024) · International Conference on Networking, Embedded and Wireless Systems (ICNEWS) · doi:10.1109/icnews60873.2024.10730913 · Cited by 10
“[10] III.”
Context-aware prompting for LLM-based program repair
Li, Y., Cai, M., Chen, J., Xu, Y., Huang, L., & Li, J. K. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00512-w · Cited by 14
“…large language models (LLMs) have achieved remarkable performances in software engineering tasks (Brown et al. 2020; Chen et al. 2021; Fried et al. 2023; Guo et al. 2022; Ahmed and Devanbu 2022; Wang et al. 2023; Xu et al. 2022) due to code naturalness (Ray et al. 2016; Hindle et al. 2016).”
UISGPT: Automated Mobile UI Design Smell Detection with Large Language Models
Yang, B., & Li, S. (2024) · Electronics · doi:10.3390/electronics13163127 · Cited by 10
A Language Model of Java Methods with Train/Test Deduplication
Su, C., Bansal, A., Jain, V., Ghanavati, S., & McMillan, C. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3613090 · Cited by 15
“…in several areas of software engineering have indicated how the “fine-tuned LLM” strategy is likely to achieve state-of-the-art results when foundation models are further trained with task-specific examples in problems such as code generation [34], code summarization [1], and clone detection [29].”
BackgroundClassEval-T: Evaluating Large Language Models in Class-Level Code Translation
Xue, P., Wu, L., Yang, Z., Wang, C., Li, X., Zhang, Y., Li, J., Jin, R., Pei, Y., Shen, Z., et al. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728940 · Cited by 15
Supply Chain Network Extraction and Entity Classification Leveraging Large Language Models
Liu, T., & Meidani, H. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10826017 · Cited by 10
“Unlike traditional NLP techniques, LLMs can perform zero-shot or few-shot learning, enabling them to generalize across various industries or contexts without requiring extensive retraining (Ahmed and Devanbu, 2022).”
LLM-Based Misconfiguration Detection for AWS Serverless Computing
Wen, J., Chen, Z.-P., Zhu, Z., Sarro, F., Liu, Y., Ping, H.-D., & Wang, S.-G. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3745766 · Cited by 17
Fault Localization from the Semantic Code Search Perspective
Qin, Y., Wang, S., Lei, Y., Zhang, Z., Lin, B., Peng, X., Ma, J., Chen, L., & Mao, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3757915 · Cited by 13
“” With these multi-granularity queries, the subsequent CS process is able to pinpoint the bug-related methods from the codebase.”
Fixing Large Language Models' Specification Misunderstanding for Better Code Generation
Tian, Z., Chen, J.-J., & Zhang, X.-Y. (2025) · International Conference on Software Engineering · doi:10.1109/icse55347.2025.00108
Stoica, B. A., Sethi, U., Su, Y., Zhou, C., Lu, S., Mace, J., Musuvathi, M., & Nath, S. (2024) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3694715.3695971 · Cited by 8
“The rise of LLMs has brought opportunities to software engineering research, with an emergent set of papers applying LLMs to code generation [46, 53, 66], testing [12, 38, 41], repair [24, 33, 64], analysis [13, 43, 44, 54], summarization [1, 2], and documentation [60].”
Vulnerabilities Introduced by LLMs Through Code Suggestions
Panichella, S. (2024) · Springer eBooks · doi:10.1007/978-3-031-54827-7_9 · Cited by 10
Tinnes, C., Welter, A., & Apel, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00112 · Cited by 13
“Most notably, the use of large language models (LLMs) has become state-of-the-art for the auto-completion of source code [18, 77, 5, 29, 6, 75].”
Prompt engineering and its implications on the energy consumption of Large Language Models
Rubei, R., Moussaid, A., Sipio, C. D., & Ruscio, D. D. (2025) · IEEE/ACM International Workshop on Green and Sustainable Software (GREENS) · doi:10.1109/greens66463.2025.00014 · Cited by 30
“We fix the number of examples equal to five for the few-shots technique since it obtains adequate accuracy with limited token size [31].”
An Overview on Large Language Models
Dakhel, A. M., Nikanjam, A., Khomh, F., Desmarais, M. C., & Washizaki, H. (2024) · Springer eBooks · doi:10.1007/978-3-031-55642-5_1 · Cited by 11
Learning to Generate Structured Code Summaries From Hybrid Code Context
Zhou, Z., Li, M., Yu, H., Fan, G., Yang, P., & Huang, Z. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3439562 · Cited by 10
“…are typically trained using auto-regressive objective for token generation, and can generate code summaries either in a zero-shot manner [22] or through in-context learning without any parameter update [23], [24], i.e., by providing several demonstrations of training examples in the input prompt.”
BackgroundContext-aware code summary generation
Su, C., Bansal, A., Huang, Y., Li, T. J.-J., & McMillan, C. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112580 · Cited by 17
Enhancing Vulnerability Detection via Inter-procedural Semantic Completion
Wu, B., Liu, C., Li, Z., Cao, Y., Sun, J., & Lin, S. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728912 · Cited by 12
“Since LLMs have demonstrated inspiring performance in code understanding and summarization [13, 42], we utilize LLMs to summarize the called functions by designing appropriate prompts with various prompting strategies to capture essential behaviors related to vulnerabilities within the called…”
Experience Report: Identifying common misconceptions and errors of novice programmers with ChatGPT
Fwa, H. L. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1145/3639474.3640059 · Cited by 12
Salerno, F., Al-Kaswan, A., & Izadi, M. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00080 · Cited by 8
“Use cases for fine-tuning include adapting a model to a specific project [33], organization, or a different field [34].”
Hybrid API Migration: A Marriage of Small API Mapping Models and Large Language Models
Zhou, B., Wang, X., Xu, S., Yao, Y., Pan, M., Xu, F., & Ma, X. (2023) · Asia-Pacific Symposium on Internetware · doi:10.1145/3609437.3609466 · Cited by 10
“Recently, large language models (LLMs) [55] like GPT-3 [7] and PaLM [14] exhibit some surprising emergent abilities [50] and can perform various software engineering tasks [2, 40, 43] very well without fine-tuning.”
BackgroundSparseCoder: Advancing source code analysis with sparse attention and learned token pruning
Yang, X., Jakubowski, M. H., Li, K., Yu, H., & Menzies, T. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10558-1 · Cited by 8
“Our contributions can be concluded as follows: • Improved model efficiency : By integrating a learned to-ken pruning algorithm into Transformer with sparse attention, SparseCoder can adaptively prune unessential tokens layer-wised and significantly reduce the model’s inference overhead.”
Automated Test Transfer across Android Apps using Large Language Models
Beyzaei, B., Talebipour, S., Rafiei, G., Medvidović, N., & Malek, S. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728975 · Cited by 10
TURAN, T., & Küçüksille, E. U. (2025) · IEEE Access · doi:10.1109/access.2025.3556725 · Cited by 9
“GPT-3/4 [48], [49], Hugging Face’s Transformers [50], [51], [52], Tensor-flow [53], and PyTextRank [54] have been seen as the most frequently used libraries for summarization in the literature.”
Resource-Efficient & Effective Code Summarization
Afrin, S., Call, J., Nguyen, K.-N., Chaparro, O., & Mastropaolo, A. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00032 · Cited by 11
“Ahmed et al. [9] found that few-shot prompting, which involves providing the model with few examples for generation tasks, significantly improves Codex’s performance in code summarization, outperforming smaller pre-trained models like CodeT5.”
Analysis of Student-LLM Interaction in a Software Engineering Project
Naman, A., Shariffdeen, R., Wang, G., Rasnayaka, S., & Iyer, G. N. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00019 · Cited by 7
“In software engineering, LLMs particularly excel in tasks like code summarization [4], test generation [5], program analysis [6], code review [7], bug fixing [8], and code generation [9].”
Interleaving static analysis and LLM prompting with applications to error specification inference
Chapman, P. J., Rubio-González, C., & Thakur, A. (2025) · International Journal on Software Tools for Technology Transfer · doi:10.1007/s10009-025-00780-7 · Cited by 8
Enhanced Prompting Framework for Code Summarization with Large Language Models
Fang, M., Yuan, X., Li, Y., Li, H., Fang, C., & Du, J. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728949 · Cited by 6
On the Challenges of Using Large Language Models for NCL Code Generation
Moraes, D. D. S., da Costa, P. B., Busson, A. J. G., Boaro, J. M. C., Neto, C. D. S. S., & Colcher, S. (2023) · Brazilian Symposium on Multimedia and the Web · doi:10.5753/webmedia_estendido.2023.236175 · Cited by 9
“According to Ahmed and Devanbu [1], this method does not require weight adjustments.”
MethodRevisiting file context for source code summarization
Su, C., Bansal, A., & McMillan, C. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00460-x · Cited by 9
Alsofyani, M., & Wang, L. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07237-3 · Cited by 5
MiSum: Multi-modality Heterogeneous Code Graph Learning for Multi-intent Binary Code Summarization
Zhu, K., Tian, Z., Wang, S., Chen, W., Dong, Z., Leng, M., & Mao, X. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715780 · Cited by 6
Can Generative AI Create Accessible Websites?
Panchanadikar, R., Bhosekar, M. S., & Dixon, E. (2025) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663547.3759755 · Cited by 6
CrashEventLLM: Predicting System Crashes with Large Language Models
Mudgal, P., Arbab, B., & Kumar, S. S. (2024) · International Conference on Information Technology and Computing (ICITCOM) · doi:10.1109/icitcom62788.2024.10762255 · Cited by 8
“In particular, due to being pre-trained on text, logs, and code corpora, LLMs have also shown promising effectiveness in software engineering tasks, such as software comprehension [10], [11], code generation and bug resolving [12]–[15], log analysis [16]–[21], and failure mode classification [22].”
Assessing Human Viewpoints in Theory of Mind for Large Language Models in Open-Ended Questioning
Amirizaniani, M. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3680273 · Cited by 5
Harnessing the Power of LLMs: LLM Summarization for Human-Centric DAST Reports
Thool, A., & Brown, C. (2024) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc60511.2024.00014 · Cited by 7
“LLMs have been applied to automate various aspects of software development, such as code summarization [37].”
BackgroundMethodHow Propense Are Large Language Models at Producing Code Smells? A Benchmarking Study
Velasco, A., Rodríguez-Cárdenas, D., Alif, L. R., Palacio, D. N., & Poshyvanyk, D. (2025) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1109/icse-nier66352.2025.00025 · Cited by 12
“Large Language Models ( LLMs ) have been used to automate multiple software engineering (SE) tasks, including code completion [1]–[3], code summarization [4], program repair [5], clone detection [6], and assertion generation [7].”
Batista, S. S., Branco, B., Castro, O., & Avelino, G. A. (2024) · Brazilian Symposium on Software Quality (SBQS) · doi:10.1145/3701625.3701673 · Cited by 6
Metadata
- Authors
- Ahmed, T., & Devanbu, P.
- Affiliations
- University of California, Davis (US)
- Computer Science, UC Davis, United States of America
- Venue
- Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering
- Publisher
- ACM
- Published
- 2022-10-10
- Pages
- 1-5
- OpenAlex
- W4313547549
- Cited by
- 240
- References
- 14
- Citations by year
- 2026: 38 · 2025: 97 · 2024: 87 · 2023: 18
- FWCI
- 45.14
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- IEEE/ACM International Conference on Automated Software Engineering (ASE) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- NSF (National Science Foundation) (SHF-2107592, SHF-1414172)
- UC Davis College of Engineering (Dean?s Distinguished Fellowship)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- code summarization, few-shot learning, codices, GPT, large language models, software engineering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
