The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index; 3 of the 9 sources its claims cite were checked for support.
Few-shot training LLMs for project-specific code-summarization
Open accessAbstract
Very large language models (LLMs), such as GPT-3 and Codex have achieved state-of-the-art performance on several natural-language tasks, and show great promise also for code. A particularly exciting aspect of LLMs is their knack for few-shot and zero-shot learning: they can learn to perform a task with very few examples. Few-shotting has particular synergies in software engineering, where there are a lot of phenomena (identifier names, APIs, terminology, coding patterns) that are known to be highly project-specific. However, project-specific data can be quite limited, especially early in the history of a project; thus the few-shot learning capacity of LLMs might be very relevant. In this paper, we investigate the use few-shot training with the very large GPT (Generative Pre-trained Transformer) Codex model, and find evidence suggesting that one can significantly surpass state-of-the-art models for code-summarization, leveraging project-specific training.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
12 Own result · 7 No basis given · 6 Own method · 5 Own interpretation · 2 No full text of the sources · 1 Supported · 1 Partly supported · 1 Not supported · 1 Sources not found · accuracy not yet measured
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Roy et al. show that BLEU-4 improvements of more than +2 points are reasonable proxies for human-perceptible preference [18].”
§ Few-shot training LLMs for project-specific code-summarization > 4.1 Cross-project few-shot
What the sources say
- Disputedcites [18] 18. Devjeet Roy, Sarah Fakhoury, and Venera Arnaoudova. 2021. Reassessing auto- matic evaluation metrics for code summarization tasks. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1105-1116. sourceExperimental — accuracy not yet measured
In Pairwise Model Comparisons (page unavailable), the paper reports that BLEU's disagreement with human assessment remains high above 2 points: 97.5% for differences of 2–5 points and over 75% in the two highest difference bins. The Implications section (page unavailable) recommends METEOR and chrF, rather than BLEU, as more reliable indicators of human judgment for differences larger than 2 points. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found statistically significant improvement for 2 projects (Table 2) over cross-project Codex for same-project training even though we improved for all 8 projects (2% to 46% improvement).”
§ Few-shot training LLMs for project-specific code-summarization > 4.3 Are we getting statistically significant result?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For cross-project setup, we observe 1%-15% improvement for all six programming languages (see Table 1).”
§ Few-shot training LLMs for project-specific code-summarization > 4.3 Are we getting statistically significant result?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also found very high statistically significant improvement for four languages.”
§ Few-shot training LLMs for project-specific code-summarization > 4.3 Are we getting statistically significant result?
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Though we failed to find any significant improvement for Python and PHP, Codex few-shot training still outperforms the traditional fine-tuned pre-trained models with 10 samples.”
§ Few-shot training LLMs for project-specific code-summarization > 4.3 Are we getting statistically significant result?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We found that we could safely fit 10-15 sequences in the prompt and ask the model to generate the comment for us.”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Observation 1.With 10 samples, Codex outperforms all finetuned foundation models CodeT5, CodeBERT, GraphCodeBERT, Polyglot CodeBERT, and PolyGlotGraphCodeBERT in all six programming languages, even though the fine-tuned models are trained with thousands of data.”
- Also stated in: § 4.1 Cross-project few-shot · hedged
“Table 1 suggests that Codex, few-shotted for code summarization, can outperform competitive models.”
- Also stated in: § 6 CONCLUSION · plain
“In this paper, we investigated the effectiveness of fewshot training for code summarization task and found that it can significantly outperform a fine-tuned model trained with thousands of samples with just ten samples.”
§ Few-shot training LLMs for project-specific code-summarization > 4.1 Cross-project few-shot
plaindirectn = 10- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4.1 Cross-project few-shot · hedged
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Second, the performance does smoothly improve, as expected, in most cases upto around 10 training samples embedded in the prompt.”
§ Few-shot training LLMs for project-specific code-summarization > 5 THREATS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“It's been well-known right from the outset that language modeling for code has to deal with project-specific phenomena [10,11,22].”
§ Few-shot training LLMs for project-specific code-summarization > 1 INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites [10] 10. Vincent J Hellendoorn and Premkumar Devanbu. 2017. Are deep neural networks the best choice for modeling source code?. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. 763-773. source
- Not checked — no full text of the sourcecites [11] 11. Abram Hindle, Earl T Barr, Zhendong Su, Mark Gabel, and Premkumar Devanbu. 2012. On the naturalness of software. In2012 34th International Conference on Software Engineering (ICSE). source
- Not checked — no full text of the sourcecites [22] 22. Zhaopeng Tu, Zhendong Su, and Premkumar Devanbu. 2014. On the localness of software. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. 269-280. source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We sorted the functions according to the creation date and ensured that only historical data was used for few-shot training to prevent data leakage from future samples.”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 11
- It may state two separate claims as one. 13
- The quote only partly states this claim. 1
Every source the paper cites for this claim was checked and supports it.
In the paper
“Ahmed and Devanbu showed that the performance of the deep learning models depends on the identifiers for the code summarization task [2].”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
What the sources say
- Supportedcites [2] 2. Toufique Ahmed and Premkumar Devanbu. 2022. Multilingual training for soft- ware engineering. In Proceedings of the 44th International Conference on Software Engineering. 1443-1455. source
Section 2.1 compares code-summarization models fine-tuned with identifiers retained or replaced by a generic token. It reports that abstracting keywords, separators, and operators has a smaller impact on BLEU-4 performance, supporting the claim that performance depends on identifiers. (LLM, conf=0.88)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Also, note that using too much data for few-shot or fine-tuning may cause catastrophic forgetting in the model [13].”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
What the sources say
- Partly supportedcites [13] 13. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences 114, 13 (2017), 3521- 3526. sourceExperimental — accuracy not yet measured
The Introduction (p. 1) explains that learning a new task sequentially can cause a network to forget an earlier one. Section 2.1 (p. 3) reports that forgetting worsened with more training time, but it does not establish that using too much data in few-shot learning or fine-tuning causes forgetting. (LLM, conf=0.84)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“even though we improved for all 8 projects (2% to 46% improvement).”
§ Few-shot training LLMs for project-specific code-summarization > 4.3 Are we getting statistically significant result?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, the model does continue generating tokens even after completing the summary.”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Observation 3. Though we did not observe statistically significant results for all programming languages and all projects, we observe overall statistically significant improvements.”
- Also stated in: § 4.3 Are we getting statistically significant result? · plain
“However, for both settings, we observe overall statistically significant improvements.”
§ Few-shot training LLMs for project-specific code-summarization > 4.3 Are we getting statistically significant result?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 4.3 Are we getting statistically significant result? · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The majority of the summaries are less than 50 tokens.”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Observation 4. Zero-shot and one-shot training in Codex do not work for code summarization task.”
- Also stated in: § 4.4 Zero-shot and one-shot training · boosted
“However, we did try both zero-shot and one-shot training with Codex and achieved only 2.96 and 6.22 BLEU-4 on average; we omit details due to the convincingly bad performance.”
§ Few-shot training LLMs for project-specific code-summarization > Observation 4.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 4.4 Zero-shot and one-shot training · boosted
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Observation 2. Same-project few-shot training improves the Codex model's performance for all 8 projects.”
- Also stated in: § 4.2 Same-project few-shot · plain
“Table 2 shows that we outperform all the models, even the Codex model with cross-project data for all the projects under consideration.”
§ Few-shot training LLMs for project-specific code-summarization > 4.2 Same-project few-shot
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
- Also stated in: § 4.2 Same-project few-shot · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We tried 5, 10, and 15 samples for few-shot training for 1000 test samples from the CodeXGLUE Java code summarization dataset and achieved 19.76, 21.88, and 21.46 BLEU-4, respectively.”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
plaincorrelationaln = 1,000- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we did try both zero-shot and one-shot training with Codex and achieved only 2.96 and 6.22 BLEU-4 on average; we omit details due to the convincingly bad performance.”
§ Few-shot training LLMs for project-specific code-summarization > 4.4 Zero-shot and one-shot training
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, for both settings, we observe overall statistically significant improvements.”
§ Few-shot training LLMs for project-specific code-summarization > 4.3 Are we getting statistically significant result?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“we randomly chose just 1000 examples from the test set of all six languages.”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Note that CodeXGLUE is a properly deduplicated dataset and uses the cross-project splits for training, testing, and dev set [19].”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
What the sources say
- Not matchedcites [19] 19. Ensheng Shia, Yanlin Wangb, Lun Dub, Junjie Chenc, Shi Hanb, Hongyu Zhangd, Dongmei Zhangb, and Hongbin Suna. 2022. On the Evaluation of Neural Code Summarization. ICSE.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“Vocabularies of a project are highly local, and functions from the same projects are likely to share same set of identifiers [10,22].”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
What the sources say
- Not checked — no full text of the sourcecites [10] 10. Vincent J Hellendoorn and Premkumar Devanbu. 2017. Are deep neural networks the best choice for modeling source code?. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. 763-773. source
- Not checked — no full text of the sourcecites [22] 22. Zhaopeng Tu, Zhendong Su, and Premkumar Devanbu. 2014. On the localness of software. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. 269-280. source
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We randomly chose ten samples from the training set for few-shot training with Codex.”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
plainn = 10- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We weren't able to fully explore hyper-parameter turning due to API access limits.”
§ Few-shot training LLMs for project-specific code-summarization > Dataset
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This result suggests that LLMs like Codex are really sample-efficient.”
- Also stated in: § 4.1 Cross-project few-shot · plain
“All the baselines are fine-tuned with 24K-251K for each language, whereas the LLM outperforms all of them with just 10 samples!.”
§ Few-shot training LLMs for project-specific code-summarization > 4.1 Cross-project few-shot
plaindirectn = 10- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 4.1 Cross-project few-shot · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“None of the above works reported the performance of few-shot training or investigated the effectiveness of same-project few-shot training.”
§ Few-shot training LLMs for project-specific code-summarization > 2 BACKGROUND AND RELATED WORK
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Fewshotting has particular synergies in software engineering, where there are a lot of phenomena (identifier names, APIs, terminology, coding patterns) that are known to be highly project-specific.”
§ Few-shot training LLMs for project-specific code-summarization > Abstract
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This is not clear how Codex-D is performing compared to other pre-trained models.”
§ Few-shot training LLMs for project-specific code-summarization > 2 BACKGROUND AND RELATED WORK
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“first, it's performance in a zero-or one-shot setting in most cases is quite abysmal.”
§ Few-shot training LLMs for project-specific code-summarization > 5 THREATS
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Applying same-project data is very promising and feasible because ten samples for a task like summarization can be generated soon after the development begins.”
§ Few-shot training LLMs for project-specific code-summarization > 6 CONCLUSION
plainn = 10- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This suggests that the model's conditioned generative ability improves with more training samples; the prior that the model internally computes and uses to condition its comment generation (p(comments | code)) is gradually improving with more training samples, suggesting that it is actually generalizing from the few-shots, rather that just regurgiating an example it's seen before.”
§ Few-shot training LLMs for project-specific code-summarization > 5 THREATS
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper relies on this without support.
Project-specific data, especially early in a project's history, may be limited in volume.
“However, project-specific data can be quite limited, especially early in the history of a project; thus the few-shot learning capacity of LLMs might be very relevant.”
“The sticking point here, however, is that project-specific data, especially early-on in a project's history, may be quite limited in volume.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“The CodeXGLUE benchmark recommends BLEU-CN, and most newer models [3,21,64] use this metric.”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [3]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A𝑆𝐴𝑃 defaults to three shots because related work [3,12] has shown, and our own experiments with A𝑆𝐴𝑃 confirmed, that more shots did not significantly improve performance.”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, Large Language Models (LLMs) now typically outperform smaller pre-trained models on many problems.”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Few-shot learning works so well with LLMs that it is unclear whether sufficient task-specific data can ever be gathered to train a customized model to rival their performance [3,12].”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [3]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Ahmed & Devanbu [3] report that LLMs can outperform pre-trained language models with a simple prompt consisting of just a few samples already in the same project”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [3]
The cited source says what the claim attributes to it.
In the citing paper
“Prior work has shown that selecting few-shots from the same projects substantially improves performance [3].”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [3]
The cited source says what the claim attributes to it.
In the citing paper
“We note that Ahmed & Devanbu didn't use IR to select few-shot samples and consistently achieved better results with same-project few-shot learning [3].”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Few-shot training LLMs for project-specific code-summarization > 4.2 Same-project few-shot
“The performance went up from 21.65 BLEU-4 to 24.37 BLEU-4 (12.56% improvement) for the Codex models, which exhibits the effectiveness of few-shot training.”
§ Few-shot training LLMs for project-specific code-summarization > 6 CONCLUSION
“| Average | | | | 19.81 | 20.12 | | 19.67 | 19.94 | 20.39 | | 21.74 | 24.37 | 12.09% | <0.01 |”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495
Research works printed but not linked: 2
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81.
- Ensheng Shia Yanlin Wangb Lun Dub Junjie Chenc Shi Hanb Hongyu Zhangd Dongmei Zhangb and Hongbin Suna. 2022. On the Evaluation of Neural Code Summarization. ICSE.
Cited by
Sorted by citations
Items 101–200 of 233
Fault Localization via Fine-Tuning Large Language Models with Mutation Generated Stack Traces
Jambigi, N., Bogacz, B., Mueller, M., Bach, T., & Felderer, M. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10988982 · Cited by 6
From Retrieval to Response: Tracing the Impact of Embedding Quality in RAG Systems
Amaral, O., Guo, Y., & Tang, Q. (2025) · IEEE Access · doi:10.1109/access.2025.3644595 · Cited by 4
“To date, RAG has quickly become one of the most used techniques across a wide range of applications (e.g., search tools, question answering, summarization, and automated assistants) [1], [2], [3], [4], [5], [6], [7].”
Exploring Large Language Models for Method Name Prediction
Qian, H., Xu, T., Ding, Z., Liu, W., & Zhu, S. (2024) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs62785.2024.00028 · Cited by 6
Exploring the impact of code review factors on the code review comment generation
Lu, J., Li, Z., Shen, C., Li, Y., & Zuo, C. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00469-2 · Cited by 5
Deep Learning-based Code Reviews: A Paradigm Shift or a Double-Edged Sword?
Tufano, R., Martin-Lopez, A., Tayeb, A. J., Dabić, O., Haiduc, S., & Bavota, G. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00060 · Cited by 4
Coding Historical Causes of Death Data with Large Language Models
Pedersen, B.-R., Islam, M., Kristoffersen, D. T., Bongo, L. A., Garrett, E., Reid, A., & Sommerseth, H. L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73741-1_3 · Cited by 7
Towards a Natural Language Interface for Flexible Multi-Agent Task Assignment
Brawer, J., Bishop, K., Hayes, B., & Roncone, A. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v2i1.27665 · Cited by 6
Exploring the Effectiveness of Abstract Syntax Tree Patterns for Algorithm Recognition
Neumüller, D., Sihler, F., Straub, R., & Tichy, M. (2024) · International Conference on Code Quality (ICCQ) · doi:10.1109/iccq60895.2024.10576984 · Cited by 4
An Analysis of LLM Fine-Tuning and Few-Shot Learning for Flaky Test Detection and Classification
More, R., & Bradbury, J. S. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10988995 · Cited by 6
Exploration of Open Large Language Models for eDiscovery
Pai, S., Lahiri, S., Kumar, U., Baksi, K. D., Soba, E., Suesserman, M., Pudota, N., Foster, J., Bowen, E., & Bhattacharya, S. (2023) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2023.nllp-1.17 · Cited by 4
Giabbanelli, P. J., Beverley, J., Dávid, I., & Tolk, A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5365069 · Cited by 4
Evaluating LLMs for Arabic Code Summarization: Challenges and Insights from GPT-4
Aljohani, A., Alharbi, R., Alkhaldi, A., & Aljedaani, W. (2025) · International Conference on Data Science and Machine Learning Applications (CDMA) · doi:10.1109/cdma61895.2025.00017 · Cited by 3
Chen, W.-C., & Chen, S.-Y. (2026) · IEEE Access · doi:10.1109/access.2026.3658502 · Cited by 3
Evaluation of the Choice of LLM in a Multi-Agent Solution for GUI-Test Generation
Tomic, S., Alégroth, E., & Isaac, M. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10989038 · Cited by 3
How well can LLMs grade essays in Arabic?
Ghazawi, R., & Simpson, E. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100449 · Cited by 9
AdvFusion: Adapter-based Knowledge Transfer for Code Summarization on Code Language Models
Saberi, I., Esmaeili, A., Fard, F. H., & Chen, F. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00059 · Cited by 9
Comprehending C codes with LLMs: Effective comment generation through retrieval and reasoning
Majumdar, S., Deshpande, A., Das, P. P., & Chakrabarti, P. P. (2025) · Pattern Recognition Letters · doi:10.1016/j.patrec.2025.10.007 · Cited by 4
Evaluating LLMs in Persian News Summarization
VarastehNezhad, A., Tavasoli, R., Masumi, M., Majd, S. S., & Shamsfard, M. (2024) · International Conference on Information and Knowledge Technology (IKT) · doi:10.1109/ikt65497.2024.10892758 · Cited by 3
LLM-Assisted Requirements Engineering in Agile MDD: Industry Insights and Validation
Spijkman, T., Molenkamp, B., Beudeker, S., Overbeek, S. J., & Dalpiaz, F. (2025) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re63999.2025.00043 · Cited by 3
Dandamudi, R., & Rodríguez-Pérez, G. (2024) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1145/3691621.3694939 · Cited by 3
Leveraging Large Language Models to Enhance Domain Expert Inclusion in Data Science Workflows
Shih, J. Y., Mohanty, V., Katsis, Y., & Subramonyam, H. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3651115 · Cited by 3
Do Current Language Models Support Code Intelligence for R Programming Language?
Zhao, Z., & Fard, F. H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3735635 · Cited by 8
Cheng, Y., Zhang, T., Shar, L. K., Yang, S., Dong, C., Lo, D., Lv, S., Shi, Z., & Sun, L. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3599581 · Cited by 7
iiPCS: Intent-Based In-Context Learning for Project-Specific Code Summarization
Wang, Y., Liu, X., Lu, X., & Zhou, A. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650776 · Cited by 2
Evaluation of the Code Generated By Large Language Models: The State of the Art
Ying, Z., Towey, D., & Zhang, Y. (2025) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac65507.2025.00064 · Cited by 3
Zhao, H., Zhang, L., Lian, K., Sun, F., Chen, B., Liu, Y., Wu, Z., Zhang, Y., & Yang, M. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00082 · Cited by 2
Integrating Chemistry Knowledge in Large Language Models Via Prompt EngineeringPreprint
Liu, H., Yin, H., Luo, Z., & Wang, X. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4790854 · Cited by 4
Guaki, G. S., & Genove, G. P. (2025) · AIP conference proceedings · doi:10.1063/5.0262017 · Cited by 2
Steer Your Model: Secure Code Generation With Contrastive Decoding
Huang, L., Yan, M. L., Yin, T., Sun, W., Liu, Z., Zhang, H., & Lo, D. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3650127 · Cited by 4
RealWeb: A Benchmark for Universal Instruction Following in Realistic Web Services Navigation
Zhang, B., Xiong, S., Sui, D., Xu, Y. D., Tu, Z., & Chu, D. (2024) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws62655.2024.00056 · Cited by 2
Dang, Y.-T., Le-Cong, T., Nguyen, P.-T., Bui, A. M. T., Nguyen, P. T., Le, X.-B. D., & Thắng, H. Q. (2024) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3661167.3661168 · Cited by 3
Banitaan, S., Daoud, M., Alquran, H., & Akour, M. A. (2026) · Information · doi:10.3390/info17010073 · Cited by 5
Improving Evidence-Based Tech Hiring with GitHub-Supported Resume Matching
Vaishampayan, S., Gulzar, M. A., & Brown, C. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00081 · Cited by 2
SwiftEval: Developing a Language-Specific Benchmark for LLM-generated Code Evaluation
Petrukha, I., Kurliak, Y., & Stulova, N. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00015 · Cited by 3
PRECOS: Project-specific Retrieval for Better Code Summarization
Zhu, T., Li, Z., Zhang, T. Y., Pan, M., & Li, X. (2024) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58944.2024.00051 · Cited by 1
De-duplicating Silent Compiler Bugs via Deep Semantic Representation
Chen, J., Fan, X., Yang, C., Liu, S., & Sun, J. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729375 · Cited by 1
Comparison of language models for wine sentiment analysis
Yang, C., & Cao, J. (2025) · Journal of Wine Economics · doi:10.1017/jwe.2025.10091 · Cited by 1
CLCoSum: Curriculum Learning-Based Code Summarization for Code Language Models
He, H., Wang, J., Cao, L., & Cai, Y. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00035 · Cited by 1
An empirical analysis of vulnerability detection tools for solidity smart contracts
Salzano, F., Antenucci, C. K., Scalabrino, S., Rosa, G., Oliveto, R., & Pareschi, R. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10867-7 · Cited by 3
LLMs as Evaluators: A Novel Approach to Commit Message Quality Assessment
Kumar, A., Sankar, S., Haiduc, S., Das, P. P., & Chakrabarti, P. P. (2025) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1109/icse-nier66352.2025.00028 · Cited by 2
Automated Code Comments Generation Using Large Language Models: Empirical Evaluation of T5 and BART
Ghale, D. P., & Dabbagh, M. (2025) · IEEE Access · doi:10.1109/access.2025.3597601 · Cited by 1
Prompt Learning for Source Code Summarization
Xu, T., Fang, C., Qian, H., Feng, X., & Sun, W. (2025) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs65678.2025.00028 · Cited by 2
RepoMasterEval: Evaluating Code Completion via Real-World Repositories
Wu, Q., Peng, C., Gao, P., Hu, R., Gan, H., Jiang, B., Tang, J., Deng, Z., Guan, Z., Gao, C., et al. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00304 · Cited by 6
Zuo, Y., Lan, P., & Liao, Z. (2024) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec65559.2024.00030 · Cited by 1
Generation and Visualization of BPMN from Legal Documents
Wilsen, W., Dewandaru, A., & Candra, M. Z. C. (2025) · IEEE International Conference on Data and Software Engineering (ICoDSE) · doi:10.1109/icodse68111.2025.11351687 · Cited by 1
Comprehensive Evaluation of LLMs in HPC Code Performance Optimization
Cui, B., Ramesh, T., Hernandez, O. R., et al. (2025) · Proceedings - International Workshops on Parallel Processing · doi:10.1145/3750720.3757280 · Cited by 1
KeyTitle: towards better bug report title generation by keywords planning
Meng, Q., Zou, W., Cai, B., & Zhang, J. (2024) · Software Quality Journal · doi:10.1007/s11219-024-09695-z · Cited by 1
RA-Gen: A Controllable Code Generation Framework Using ReAct for Multi-Agent Task Execution
Liu, A. Q., Li, H., Wang, B., Yang, A., & Li, H. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228706 · Cited by 1
AI Assistant Book based on Question Generation using Generative Artificial Intelligence
Byeon, G., Choi, K., & Yu, S. (2023) · The Journal of Korean Institute of Information Technology · doi:10.14801/jkiit.2023.21.12.217 · Cited by 1
Reinforcement Learning vs Supervised Learning: A tug of war to generate refactored code accurately
Palit, I., & Sharma, T. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3756681.3756971 · Cited by 8
Security Analysis of Large Language Models on API Misuse Programming Repair
Zhang, R., Qiao, Z., & Yu, Y. (2024) · International Journal of Intelligent Systems · doi:10.1155/2024/7135765 · Cited by 1
Giabbanelli, P. J., Beverley, J. R., Dávid, I., & Tolk, A. (2025) · Winter Simulation Conference · doi:10.1109/wsc68292.2025.11338852 · Cited by 1
Large Language Models for Named Entity Recognition (NER) of Skills in Job Postings in German
Käser, J., Hanne, T., & Dornberger, R. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-64836-6_3 · Cited by 2
Data race detection via few-shot parameter-efficient fine-tuning
Shen, Y., Peng, M., Zhang, F., & Wu, Q. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112289 · Cited by 1
An exploratory study on fine-tuning large language models for secure code generation
Li, J., Rabbi, F., Cheng, C., Sangalay, A., Tian, Y., & Yang, J. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10803-9 · Cited by 9
Amasanti, G., & Jahić, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04403-7_10 · Cited by 1
Icing on the Cake: Automatic Code Summarization at Ericsson
Sridhara, G., Roychowdhury, S., Soman, S., Ranjani, H. G., & Britto, R. (2024) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58944.2024.00073 · Cited by 2
Enhancing Security Awareness and Education for LLMs
Panichella, S. (2024) · Springer eBooks · doi:10.1007/978-3-031-54827-7_18 · Cited by 1
Semantic similarity loss for neural source code summarization
Su, C., & McMillan, C. (2024) · Journal of Software Evolution and Process · doi:10.1002/smr.2706 · Cited by 1
Adapting Online Customer Reviews for Blind Users: A Case Study of Restaurant Reviews
Sunkara, M., Nayak, A. K., Kalari, S., Prakash, Y., Jayarathna, S., Lee, H.-N., & Ashok, V. G. (2025) · International Web for All Conference (W4A) · doi:10.1145/3744257.3744276 · Cited by 2
Issue Retrieval and Verification Enhanced Supplementary Code Comment Generation
Zou, Y., Zhao, X., Pan, X., & Xie, B. (2025) · Asia-Pacific Symposium on Internetware · doi:10.1145/3755881.3755897 · Cited by 1
Vision Paper: Proof-Carrying Code Completions
Kamran, P. S., Dévanbu, P., & Stanford, C. (2024) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1145/3691621.3694932 · Cited by 5
Snell, Q. O., Westhoff, C., Westhoff, J., Low, E., Hanson, C. L., & Tass, E. S. (2025) · Journal of Medical Internet Research · doi:10.2196/74094 · Cited by 1
Hallucinations in LLM-Based Code Summarization: Unveiling, Detection, and Mitigation
Wan, G.-X., Feng, Y.-N., Wan, Y., Chu, Z.-Y., Bi, Z., Han, J.-X., Zhao, Z., Zhang, H.-Y., Yuan, P.-P., Shi, X.-H., et al. (2026) · Proc. ACM Softw. Eng. · doi:10.1145/3808139
Learning to Navigate (L2N): A Human-Centered Framework for Local Code Search
Leon, S., Zhang, Z., Tadepalli, P. V., & Kuttal, S. K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31048-4_21
Ensuring Code Integrity in the Era of AI-Assisted Software Development
Pilone, A. (2025) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme64153.2025.00095
AutoReSpec: A Framework for Generating Specification using Large Language Models
Ayon, R. S., & Ahmed, S. (2026) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1145/3793655.3793731 · Cited by 6
Unlocking Large Language Models for Project Scheduling
Zhang, Y., Li, X., Teng, Y., Shen, Q., & Bai, S. (2025) · International Conference on Construction and Real Estate Management (ICCREM) · doi:10.1061/9780784485910.053 · Cited by 1
A Systematic Literature Review of Parameter-Efficient Fine-Tuning for Large Code Models
Afrin, S., Haque, M. Z., & Mastropaolo, A. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3796522 · Cited by 12
Automated Prompt Generation for Code Intelligence: An Empirical study and Experience in WeChat
Ji, K., Fu, S., Gao, C., Chen, Y., Yang, Z., Wang, C. J., & Deng, Y. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00285
Agent4cs: A Multi-agent System for Code Summarization in Large Hierarchical Codebases
Tang, Y., Sarıkayak, E., Tuncel, D., Zhang, J. M., & Runkler, T. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39395-1_24 · Cited by 1
Wu, P. Y., Chou, H.-W., & Chen, K.-Y. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3659424
Wang, R., Chang, C., Chen, X., Tian, Y., & Huang, R. (2026) · Automated Software Engineering · doi:10.1007/s10515-026-00604-1
A Comprehensive Evaluation of Large Language Models for Method Name PredictionPreprint
Qian, H., Xu, T., Ding, Z., Liu, W., Zhu, S., Yin, J., Feng, X., Fang, C., Zhao, J., & Sun, W. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4886577
VulKG-RAG: Graph-Enhanced Retrieval for Hierarchical Vulnerability Detection
Jiang, M., Chen, X., & Lu, Y. (2026) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd68734.2026.11582592
A Lightweight Sequence-to-Sequence Transformer Model for Software Documentation Generation
Sampavy, K., Vasanthapriyan, S., & Kayalvizhy, T. (2026) · International Conference on Advanced Network Technologies and Applications (APAN) · doi:10.1109/apan70967.2026.11709782
Leveraging Design-Aware Context in Large Language Models for Code Comment Generation
Mitra, A., Majumdar, S., Mukhopadhyay, A., Das, P. P., Clough, P. D., & Chakrabarti, P. P. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3786582.3786825
Evidence-based Hiring: A GitHub-Centric Approach to Candidate Assessment
Kumar, G. K., Reddy, B. D., Reddy, S. V., Macharla, A., & Ramu, S. C. (2025) · International Conference on IoT Based Control Networks and Intelligent Systems (ICICNIS) · doi:10.1109/icicnis66685.2025.11315719
Data-Efficient Tabular Classification with Transformer-Based Small Language Models
Haddad-Neto, M., Silva, D., CALIARI, Í. P., & Bragança, H. L. D. S. (2025) · Brazilian Symposium on Multimedia and the Web · doi:10.5753/webmedia.2025.15193
A Two-Stage Framework Integrating Prompt Learning and Fine-Tuning for Code Summarization
Sun, X., Lv, S., Wan, W., Qin, Y., & Hu, G. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04549-2_24
Why Personalizing Deep Learning-Based Code Completion Tools Matters
Giagnorio, A., Martin-Lopez, A., & Bavota, G. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3725732
From brittle to robust: Improving LLM annotations for SE optimization
Senthilkumar, L., & Menzies, T. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10823-5
Can Large Language Models Improve SE Active Learning via Warm Starts?
Senthilkumar, L., & Menzies, T. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3796511 · Cited by 10
Automatic Generation Method of GIM Model Audit Rule Based on LLM
Lin, L., Xu, Z., & Xiao, H. (2025) · doi:10.1109/icaiem66060.2025.11139288
Kumar, J., Tanguturi, M. R., Janapati, L. S., & Chimalakonda, S. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.113010
Leveraging Large Language Models for Scalable Smart Contracts in Blockchain IoT Systems
Zaidi, T., Aftab, M. U., & Usama, M. (2026) · International Conference on Computing, Mathematics and Engineering Technologies (iCoMET) · doi:10.1109/icomet69771.2026.11591635
From Human to Machine Refactoring: Assessing GPT-4’s Impact on Python Class Quality and Readability
Midolo, A., Tramontana, E., & Penta, M. D. (2026) · doi:10.1145/3793655.3793733
Avaliando o Papel do GPT-4o na Aprendizagem de Engenharia de Software: Um Relato de Experiência
Branco, B. M. D. C. C., Neto, A. L. D. M., Avelino, G. A., & Machado, V. P. (2025) · Simpósio Brasileiro de Engenharia de Software (SBES) · doi:10.5753/sbes.2025.10716 · Cited by 1
Metadata
- Authors
- Ahmed, T., & Devanbu, P.
- Affiliations
- University of California, Davis (US)
- Computer Science, UC Davis, United States of America
- Venue
- Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering
- Publisher
- ACM
- Published
- 2022-10-10
- Pages
- 1-5
- OpenAlex
- W4313547549
- Cited by
- 240
- References
- 14
- Citations by year
- 2026: 38 · 2025: 97 · 2024: 87 · 2023: 18
- FWCI
- 45.14
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- IEEE/ACM International Conference on Automated Software Engineering (ASE) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- NSF (National Science Foundation) (SHF-2107592, SHF-1414172)
- UC Davis College of Engineering (Dean?s Distinguished Fellowship)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- code summarization, few-shot learning, codices, GPT, large language models, software engineering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
