Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)
Open accessThe DOI matches its Crossref record; 80 of the 80 references it cites were found in a registry or an index; 33 of the 57 sources its claims cite were checked for support.
Abstract
Large Language Models (LLM) are a new class of computation engines, "programmed" via prompt engineering. Researchers are still learning how to best "program" these LLMs to help developers. We start with the intuition that developers tend to consciously and unconsciously collect semantics facts, from the code, while working. Mostly these are shallow, simple facts arising from a quick read. For a function, such facts might include parameter and local variable names, return expressions, simple pre- and post-conditions, and basic control and data flow, etc.
Overview
CoChat Summary
## TL;DR
The paper proposes ASAP (Automatic Semantic Augmentation of Prompts), which adds automatically extracted semantic facts from code into few-shot prompts for LLM-based code summarization and code completion. On CodeSearchNet/CodeXGLUE-style evaluations, the augmented prompts generally improved performance, including statistically significant BLEU gains across six languages and improvements in zero-shot line completion.
## Key findings
- On code summarization with `code-davinci-002`, ASAP improved BLEU-4 over BM25 few-shot prompting in all six languages: Java 22.87→25.41 (+11.11%), Python 21.78→24.26 (+11.39%), Ruby 17.21→19.62 (+14.00%), JavaScript 23.27→25.36 (+8.98%), Go 22.67→24.51 (+8.12%), and PHP 28.15→32.73 (+16.27%); overall 22.66→25.32 (+11.74%), with p < 0.01 in each case.
- In same-project summarization, ASAP outperformed vanilla BM25 few-shot learning in 14 of 16 BLEU comparisons (87.5%).
- For line completion on 9,292 Java and 6,550 Python samples, ASAP improved Exact Match from 17.97 to 19.01 overall (+5.79%, p < 0.01) and Edit Similarity from 53.01 to 55.72 overall (+5.11%, p < 0.01).
- ASAP also improved other metrics overall: BLEU-DC and ROUGE-L showed significant gains, while METEOR improved on average but was not significant for 4 of 6 languages.
## Methods
- The paper evaluates few-shot prompting for code summarization on the CodeSearchNet dataset, using 1,000 test samples per language across six languages: Java, Python, Ruby, JavaScript, Go, and PHP; for same-project experiments, it used project-specific subsets and sorted them by creation date to avoid leakage.
- The core method, ASAP, augments each retrieved exemplar and the target code with automatically derived semantic facts: repository name/path, tagged identifiers, and a data flow graph; exemplars are retrieved with BM25 from the training set.
- Main analysis used OpenAI models (`code-davinci-002` primarily, plus `text-davinci-003` and `gpt-3.5-turbo`), with 3-shot prompts, temperature 0, and statistical testing via one-sided pairwise Wilcoxon signed-rank tests (with B-H correction where reported); line completion was evaluated with Exact Match and Edit Similarity.
## Limitations
- The paper states it did not conduct a user study, so the metric gains may not translate directly into improved developer performance.
- It could not directly check whether the model had seen test data during training because the training dataset is inaccessible; this is a limitation stated in the paper.
- Evident from the text: the main experiments were constrained by OpenAI API rate limits and cost, so many analyses used only 1,000 samples per language, and some deeper analyses were restricted mainly to Java and Python.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
17 Valid · 8 Not valid · 11 Not checked · 81 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“In most cases, we see improvements of over 2.0 BLEU, the required threshold for human perception noted by Roy et al. [57].”
§ Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) > 4.2 A𝑆𝐴𝑃 Prompt Enhancement
What the sources say
- Disputedcites [57] 57. Devjeet Roy, Sarah Fakhoury, and Venera Arnaoudova. 2021. Reassessing auto- matic evaluation metrics for code summarization tasks. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1105-1116. sourceExperimental — accuracy not yet measured
In “Pairwise Model Comparisons” (page not provided), Roy et al. say human assessors perceived differences even when metric scores differed by less than 2 points. They describe 2 points as a boundary below which automatic metrics are unreliable, not a required BLEU threshold for human perception. The paper does not assess ASAP’s improvements. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“De-duplication is key: Code duplication in machine learning models can deceptively inflate performance metrics a lot, when compared to de-duplicated datasets [7,46,59].”
§ Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) > 3.1 Dataset
What the sources say
- Supportedcites [7] 7. Miltiadis Allamanis. 2019. The adverse effects of code duplication in machine learning models of code. In Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software. 143-153. source
The Introduction says duplication can cause evaluations to overestimate the performance users actually observe, with observed performance up to 50% worse than reported results. Section 5 adds that end-user performance is often significantly worse than evaluation metrics suggest, supporting the claim that inflation can be substantial. (LLM, conf=0.93)
- Partly supportedcites [46] 46. Cristina V Lopes, Petr Maj, Pedro Martins, Vaibhav Saini, Di Yang, Jakub Zitny, Hitesh Sajnani, and Jan Vitek. 2017. DéjàVu: a map of code duplicates on GitHub. Proceedings of the ACM on Programming Languages 1, OOPSLA (2017), 1-28. sourceExperimental — accuracy not yet measured
Our copy of DéjàVu supports only the general, software-engineering-level point: the Conclusions (Sec. 7) state that 'Code duplication can severely skew the conclusions of those studies' and that the 'assumption of diversity of projects in those datasets may be compromised', and the Introduction (Sec. 1) warns that reuse and file copying 'at scale, may bias results of research'; Sec. 2 adds that duplicate projects in a dataset create a 'lack of diversity ... [that] might lead to incorrect observations' (crediting Nagappan et al. 2013). None of these passages concern machine learning models, nor any comparison of performance metrics on duplicated versus de-duplicated datasets — the paper measures file- and project-level clone rates in a 4.5M-project GitHub corpus (Java, C++, Python, JavaScript), not model accuracy. The ML-specific inflation claim attributed here is therefore not supported by this source; only the background point that duplication distorts dataset validity is. (LLM, conf=0.72)
- Partly supportedcites [59] 59. Ensheng Shi, Yanlin Wang, Lun Du, Junjie Chen, Shi Han, Hongyu Zhang, Dong- mei Zhang, and Hongbin Sun. 2023. On the evaluation of neural code summariza- tion. In Proceedings of the 44th International Conference on Software Engineering. 1597-1608. sourceExperimental — accuracy not yet measured
Our copy supports the direction and significance of the claim: the Abstract (p. 1) and CCS Concepts (p. 2) list duplication ratios among dataset characteristics that 'have a significant impact on model evaluation', and RQ3 (p. 5) states that TL-CodeSum contains 20% exact code duplication while Funcom and CodeSearchNet were deduplicated, motivating experiments at different duplication ratios, with Section 4.1 (p. 5) reporting that most models score lower on the deduplicated TLC Dedup variant. What the source does not say is the claim's strong framing: it hedges that 'ignoring code duplication may result in model performance and generalization ability not being comprehensively evaluated' rather than asserting metrics are 'deceptively inflated a lot', and the explicit overestimation-by-duplication conclusion is credited to another work in Related Work (p. 10) — Allamanis et al. — not reported as this paper's own finding. The magnitude ('a lot') is therefore not established by our copy for this source. (LLM, conf=0.72)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“the 2-BLEU threshold noted by Roy et al. [57], below which BLEU results are unreliable predictors of human preference.”
§ Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) > 1 INTRODUCTION
What the sources say
- Supportedcites [57] 57. Devjeet Roy, Sarah Fakhoury, and Venera Arnaoudova. 2021. Reassessing auto- matic evaluation metrics for code summarization tasks. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1105-1116. source
In Pairwise Model Comparisons, the paper says that when two code-summarization approaches differ by no more than 2 metric points, automatic metrics cannot reliably distinguish their quality in agreement with human ratings. This includes BLEU. The threshold concerns the difference between BLEU scores, not an absolute BLEU score below 2. (LLM, conf=0.90)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Roy et al. [57] provide arguments, grounded on human subject study that for code summarization (our central task), that a gain of 2.0 or more BLEU is more likely to correspond with human perception of improvement.”
§ Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) > 3.5 Metrics
What the sources say
- Partly supportedcites [57] 57. Devjeet Roy, Sarah Fakhoury, and Venera Arnaoudova. 2021. Reassessing auto- matic evaluation metrics for code summarization tasks. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1105-1116. sourceExperimental — accuracy not yet measured
The Pairwise Model Comparisons section reports that metric differences of 2 points or less do not consistently align with human judgments, supporting the claim’s general point. But it does not establish 2.0 BLEU as a threshold for perceived improvement: the threshold varies by metric, and BLEU’s disagreement with human assessment is described as erratic even at larger differences. The Implications section recommends METEOR and chrF over BLEU as indicators of human judgment. (LLM, conf=0.88)
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“For the tests that we used (Wilcoxon Signed-rank test), we found that the needed sample size was always below the sample size we used for our primary studies, viz., 1000.”
§ Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) > Sample Size Analysis:
What the sources say
- Could not verifycites [19] 19. Franz Faul, Edgar Erdfelder, Axel Buchner, and Albert-Georg Lang. 2009. Sta- tistical power analyses using G* Power 3.1: Tests for correlation and regression analyses. Behavior research methods 41, 4 (2009), 1149-1160. sourceExperimental — accuracy not yet measured
The paper’s sample-size examples concern correlation and regression analyses (Effect size measure; 5.1 Logistic Regression; page numbers unavailable). Searches for Wilcoxon signed-rank tests and the cited studies’ sample size of 1,000 found no passage addressing that comparison. (LLM, conf=0.77)
- Could not verifycites [20] 20. Franz Faul, Edgar Erdfelder, Albert-Georg Lang, and Axel Buchner. 2007. G* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior research methods 39, 2 (2007), 175-191. sourceExperimental — accuracy not yet measured
The A Priori Power Analyses section (page unavailable) explains how required sample size is calculated, but it does not report the citing authors’ calculations. Searches for Wilcoxon signed-rank tests and a sample size of 1,000 found no passage establishing that the required size was always below 1,000. (LLM, conf=0.77)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
36 more claims
Create a free account to see all 117 claims and to verify the 9 references still open.
- It may state two separate claims as one. 50
- Probably not a substantial claim. 45
- The quote only partly states this claim. 4
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“ASAP [171] incorporates exemplar tuples, consisting of input code, function definitions, analysis results, and corresponding comments, into prompts to yield better results.”
From Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024) · cites this paper as [171]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193146 · Cited by 65,971
Statistical power analyses using G∗Power 3.1: tests for correlation and regression analyses
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009) · Behavior Research Methods · doi:10.3758/brm.41.4.1149 · Cited by 37,615
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
The probabilistic relevance framework: BM25 and beyond
Robertson, S., & Zaragoza, H. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000019 · Cited by 3,374 · cited 1×
“BM25 [55] is a frequency-based retrieval method which improves upon TF-IDF [54].”
MethodBERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824 · cited 5×
“The CodeXGLUE benchmark recommends BLEU-CN, and most newer models [3, 21, 64] use this metric.”
BackgroundMethodInfluentialLarge Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
Proceedings of the 40th Annual Meeting on Association for Computational Linguistics
Isabelle, P. (2002) · Cited by 1,635
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495 · cited 4×
“Pre-trained transformer-based [62] models such as CodeBERT [21], PLBART [2], andCodeT5[64]havebeen extensivelyused ontheCodeXGLUE[47] code summarization dataset, resulting in significant improvements.”
BackgroundMethodInfluentialWang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
BLEURT: Learning Robust Metrics for Text Generation
Sellam, T., Das, D., & Parikh, A. P. (2020) · arXiv · doi:10.48550/arxiv.2004.04696 · Cited by 1,005 · cited 1×
“A 𝑆𝐴𝑃 defaults to three shots because related work [3, 12] has shown, and our own experiments with A 𝑆𝐴𝑃 confirmed, that more shots did not significantly improve performance.”
ResultBLEURT: Learning Robust Metrics for Text Generation
Sellam, T., Das, D., & Parikh, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.704 · Cited by 1,005
ChatGPT: Jack of all trades, master of none
Kocoń, J., Cichecki, I., Kaszyca, O., Kochanek, M., Szydło, D., Baran, J., Bielaniewicz, J., Gruza, M., Janz, A., Kanclerz, K., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101861 · Cited by 758 · cited 1×
“This underperformance by the chat model is consistent with the findings by Kocon et al. [39].”
ResultMurray, G., Renals, S. J., Carletta, J., & Moore, J. D. (2005) · Meeting of the Association for Computational Linguistics · Cited by 716
Summarizing Source Code using a Neural Attention Model
Iyer, S., Konstas, I., Cheung, A., & Zettlemoyer, L. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1195 · Cited by 701 · cited 3×
“The model is designed to allow a window of approximately 4K tokens; this limits the number of few-shot samples.”
BackgroundMethodVanderwende, L., Daumé, H., & Kirchhoff, K. (2013) · North American Chapter of the Association for Computational Linguistics · Cited by 685
Unified Pre-training for Program Understanding and Generation
Ahmad, W., Chakraborty, S., Ray, B., & Chang, K.-W. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.211 · Cited by 642 · cited 3×
“Pre-trained transformer-based [62] models such as CodeBERT [21], PLBART [2], andCodeT5[64]havebeen extensivelyused ontheCodeXGLUE[47] code summarization dataset, resulting in significant improvements.”
BackgroundMethodHu, X., Li, G., Xia, X., Lo, D., & Jin, Z. (2018) · Proceedings of the 26th Conference on Program Comprehension · doi:10.1145/3196321.3196334 · Cited by 621 · cited 1×
“We report data on two variants: BLEU-CN, which uses a kind of Laplacian smoothing [2, 3, 8, 21, 33, 47, 64] and BLEU-DC, which uses newer smoothing methods [29, 65].”
MethodA systematic evaluation of large language models of code
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534862 · Cited by 540 · cited 3×
“Several models such as Polycoder [67] and Codegen [49] perform reasonably well, and due to their few-shot learning or prompting, they can be applied to a wide set of problems.”
BackgroundMethodResultCodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.04664 · Cited by 536 · cited 5×
“Although LLMs are not yet so widely used for code summarization, they are extensively used for code generation [14, 49, 67] and program repair [5, 18, 35, 36].”
BackgroundInfluentialTowards automatically generating summary comments for Java methods
Sridhara, G., Hill, E., Muppaneni, D., Pollock, L., & Vijay-Shanker, K. (2010) · Proceedings of the IEEE/ACM international conference on Automated software engineering · doi:10.1145/1858996.1859006 · Cited by 514 · cited 1×
“Initially, template-based approaches were popular [17, 26, 27, 56, 61]; however, creating a list of templates with good coverage is very challenging.”
CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
Husain, H., Wu, H.-H., Gazit, T., Allamanis, M., & Brockschmidt, M. (2019) · arXiv · doi:10.48550/arxiv.1909.09436 · Cited by 483 · cited 2×
“Finally, we discuss code summarization as a sample problem to demonstrate A 𝑆𝐴𝑃 ’s effectiveness.”
MethodGraphCodeBERT: Pre-training Code Representations with Data Flow
Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Liu, S., Zhou, L., Duan, N., Svyatkovskiy, A., Fu, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.08366 · Cited by 480
Using Machine Learning for Vulnerability Detection and Classification
Alon, U., Brody, S., Levy, O., & Yahav, E. (2018) · arXiv (Cornell University) · doi:10.4230/oasics.slate.2021.14 · Cited by 467
CodeT5+: Open Code Large Language Models for Code Understanding and Generation
Wang, Y., Le, H., Gotmare, A. D., Bui, N. D. Q., Li, J., & Hoi, S. C. H. (2023) · arXiv · doi:10.48550/arxiv.2305.07922 · Cited by 459
Reasoning with Language Model Prompting: A SurveyPreprint
Qiao, S.-F., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S.-M., Tan, C.-Q., Huang, F., & Chen, H.-J. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.09597 · Cited by 459
CodeT5+: Open Code Large Language Models for Code Understanding and Generation
Wang, Y., Le, H., Gotmare, A., Bui, N., Li, J., & Hoi, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.68 · Cited by 459
Towards Reasoning in Large Language Models: A Survey
Huang, J., & Chang, K. C.-C. (2022) · arXiv · doi:10.48550/arxiv.2212.10403 · Cited by 428
On the Use of Automated Text Summarization Techniques for Summarizing Source Code
Haiduc, S., Aponte, J., Moreno, L., & Marcus, A. (2010) · 2010 17th Working Conference on Reverse Engineering · doi:10.1109/wcre.2010.13 · Cited by 408 · cited 1×
“We now discuss our dataset, models, and methodology.”
MethodA Transformer-based Approach for Source Code Summarization
Ahmad, W., Chakraborty, S., Ray, B., & Chang, K.-W. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.449 · Cited by 378 · cited 1×
“To induce the model to generate steps, and finally a summary, we framed the problem as chain of thought, and included few-shot samples containing both intermediate steps (“lemmas") and final comments.”
MethodCodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., & Xiong, C. (2022) · arXiv · doi:10.48550/arxiv.2203.13474 · Cited by 330
Proceedings of the 44th International Conference on Software Engineering
(2022) · doi:10.1145/3510003 · Cited by 326
CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models
Lemieux, C., Inala, J. P., Lahiri, S. K., & Sen, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00085 · Cited by 306
(2004) · doi:10.3115/1220355.1220427 · Cited by 294 · cited 1×
“Various smoothing techniques [13, 23, 44] have been used, to reduce Sentence BLEU’s sensitivity to sparse 𝑛 -gram matches, and better align it with human quality assessment.”
MethodA Neural Model for Generating Natural Language Summaries of Program Subroutines
LeClair, A., Jiang, S., & McMillan, C. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00087 · Cited by 294 · cited 2×
“Several models such as Polycoder [67] and Codegen [49] perform reasonably well, and due to their few-shot learning or prompting, they can be applied to a wide set of problems.”
MethodSupporting program comprehension with source code summarization
Haiduc, S., Aponte, J., & Marcus, A. (2010) · Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 2 · doi:10.1145/1810295.1810335 · Cited by 291 · cited 2×
“Initially, template-based approaches were popular [17, 26, 27, 56, 61]; however, creating a list of templates with good coverage is very challenging.”
BackgroundThe adverse effects of code duplication in machine learning models of code
Allamanis, M. (2019) · Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software · doi:10.1145/3359591.3359735 · Cited by 284 · cited 1×
“De-duplication is key: Code duplication in machine learning models can deceptively inflate performance metrics a lot, when compared to de-duplicated datasets [7, 46, 59].”
ResultRetrieval-based neural source code summarization
Zhang, J., Wang, X., Zhang, H., Sun, H., & Liu, X. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380383 · Cited by 275 · cited 1×
“Some have combined previous approaches, such as template-based and retrieval-based approaches, using neural models [69], and have reported promising results.”
The relevance of software documentation, tools and technologies: a survey
Forward, A., & Lethbridge, T. C. (2002) · Proceedings of the 2002 ACM symposium on Document engineering · doi:10.1145/585058.585065 · Cited by 268 · cited 1×
“We now discuss our dataset, models, and methodology.”
MethodSummarizing Source Code with Transferred API Knowledge
Hu, X., Li, G., Xia, X., Lo, D., Lu, S., & Jin, Z. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/314 · Cited by 263 · cited 2×
“TL-CodeSum [30] is a relatively smaller dataset, with around 87K samples, but it does include duplicates, which may result in high performance estimates that may not generalize.”
BackgroundAutomated Repair of Programs from Large Language Models
Fan, Z., Gao, X., Mirchev, M., Roychoudhury, A., & Tan, S. H. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00128 · Cited by 256
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · arXiv · doi:10.48550/arxiv.2002.08155 · Cited by 243
Few-shot training LLMs for project-specific code-summarization
Ahmed, T., & Devanbu, P. (2022) · Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3551349.3559555 · Cited by 240 · cited 10×
“We now discuss our dataset, models, and methodology.”
BackgroundMethodInfluentialSelf-citationLarge Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction
Kang, S., Yoon, J., & Yoo, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00194 · Cited by 204
Impact of Code Language Models on Automated Program Repair
Jiang, N., Liu, K., Lutellier, T., & Tan, L. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00125 · Cited by 202
Improving automated source code summarization via an eye-tracking study of programmers
Rodeghero, P., McMillan, C., McBurney, P. W., Bosch, N., & D'Mello, S. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568247 · Cited by 198 · cited 2×
“Initially, template-based approaches were popular [17, 26, 27, 56, 61]; however, creating a list of templates with good coverage is very challenging.”
BackgroundA Systematic Comparison of Smoothing Techniques for Sentence-Level BLEU
Chen, B., & Cherry, C. (2014) · Proceedings of the Ninth Workshop on Statistical Machine Translation · doi:10.3115/v1/w14-3346 · Cited by 191 · cited 1×
“We include waiting periods between attempts, following OpenAI’s recommendations.”
MethodDéjàVu: a map of code duplicates on GitHub
Lopes, C. V., Maj, P., Martins, P., Saini, V., Yang, D., Zitny, J., Sajnani, H., & Vitek, J. (2017) · Proceedings of the ACM on Programming Languages · doi:10.1145/3133908 · Cited by 183 · cited 1×
“De-duplication is key: Code duplication in machine learning models can deceptively inflate performance metrics a lot, when compared to de-duplicated datasets [7, 46, 59].”
ResultRetrieval-Based Prompt Selection for Code-Related Few-Shot Learning
Nashid, N., Sintaha, M., & Mesbah, A. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00205 · Cited by 178
Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models
Ahmed, T., Ghosh, S., Bansal, C., Zimmermann, T., Zhang, X., & Rajmohan, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00149 · Cited by 148
Self-citationEvaluating source code summarization techniques: Replication and expansion
Eddy, B. P., Robinson, J. A., Kraft, N. A., & Carver, J. C. (2013) · 2013 21st International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2013.6613829 · Cited by 144 · cited 1×
“We now discuss our dataset, models, and methodology.”
Method(2021) · doi:10.1145/3510003.3510203 · Cited by 139
Repair Is Nearly Generation: Multilingual Program Repair with LLMs
Joshi, H., Cambronero Sanchez, J., Gulwani, S., Le, V., Verbruggen, G., & Radiček, I. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i4.25642 · Cited by 126
Semantic similarity metrics for evaluating source code summarization
Haque, S., Eberhart, Z., Bansal, A., & McMillan, C. (2022) · Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension · doi:10.1145/3524610.3527909 · Cited by 116 · cited 2×
“However, for up to 2% of the randomly chosen samples in each experiment, we didn’t get good results; either the prompt didn’t fit into the model’s 5 Please see Section 7 for the rationale. window, or the model mysteriously generated an empty string.”
BackgroundResultReassessing automatic evaluation metrics for code summarization tasks
Roy, D., Fakhoury, S., & Arnaoudova, V. (2021) · Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3468264.3468588 · Cited by 106 · cited 5×
“In most cases, we see improvements of over 2.0 BLEU, the required threshold for human perception noted by Roy et al. [57].”
BackgroundMethodInfluential(2023) · doi:10.1145/3510003.3510060 · Cited by 88 · cited 1×
“BERT is one of the earliest pre-trained language models [15]; it was pre-trained on two self-supervised tasks: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP).”
Software documentation: how much is enough?
Briand, L. C. (2003) · Seventh European Conference onSoftware Maintenance and Reengineering, 2003. Proceedings. · doi:10.1109/csmr.2003.1192406 · Cited by 69
Multilingual training for software engineering
Ahmed, T., & Devanbu, P. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510049 · Cited by 57 · cited 3×
“These models use different representations and sources of information to perform well in code summarization.”
MethodSelf-citationCode Generation as a Dual Task of Code Summarization
Wei, B., Li, G., Xia, X., Fu, Z., & Jin, Z. G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.05923 · Cited by 53
Chatgpt: Jack of All Trades, Master of NonePreprint
Kocoń, J., Cichecki, I., Kaszyca, O., et al. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4372889 · Cited by 52
Code to comment "translation": data, metrics, baselining & evaluation
Gros, D., Sezhiyan, H., Devanbu, P., & Yu, Z. (2020) · Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3324884.3416546 · Cited by 43
Self-citationBetter Patching Using LLM Prompting, via Self-Consistency
Ahmed, T., & Devanbu, P. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00065 · Cited by 41 · cited 1×
“Sadly, one can’t directly check this since the training dataset is not accessible.”
BackgroundSelf-citationA Transformer-based Approach for Source Code SummarizationPreprint
Ahmad, W. U., Chakraborty, S., Ray, B., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00653 · Cited by 37
NUBIA: NeUral Based Interchangeability Assessor for Text Generation
Kane, H., Kocyigit, M. Y., Abdalla, A., Ajanoh, P., & Coulibali, M. (2020) · arXiv · doi:10.48550/arxiv.2004.14667 · Cited by 32
Repository-Level Prompt Generation for Large Language Models of Code
Shrivastava, D., Larochelle, H., & Tarlow, D. (2022) · arXiv · doi:10.48550/arxiv.2206.12839 · Cited by 27 · cited 1×
“Various smoothing techniques [13, 23, 44] have been used, to reduce Sentence BLEU’s sensitivity to sparse 𝑛 -gram matches, and better align it with human quality assessment.”
MethodCode Generation Tools (Almost) for Free? A Study of Few-Shot, Pre-Trained Language Models on Code
Bareiß, P., Souza, B., d’Amorim, M., & Pradel, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.01335 · Cited by 27
Jigsaw: Large Language Models meet Program SynthesisPreprint
Jain, N., Vaidyanath, S., Iyer, A. S., Natarajan, N., Parthasarathy, S., Rajamani, S. K., & Sharma, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.02969 · Cited by 19
Code to Comment "Translation": Data, Metrics, Baselining & EvaluationPreprint
Gros, D., Sezhiyan, H., Dévanbu, P., & Yu, Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.01410 · Cited by 16
Repair Is Nearly Generation: Multilingual Program Repair with LLMsPreprint
Joshi, H., Cambronero, J., Gulwani, S., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.11640 · Cited by 15
Evaluating commit message generation: to BLEU or not to BLEU?
Dey, S., Vinayakarao, V., Gupta, M., & Dechu, S. (2022) · Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results · doi:10.1145/3510455.3512790 · Cited by 9 · cited 2×
“However, for up to 2% of the randomly chosen samples in each experiment, we didn’t get good results; either the prompt didn’t fit into the model’s 5 Please see Section 7 for the rationale. window, or the model mysteriously generated an empty string.”
BackgroundMethodGenerate-and-Retrieve: use your predictions to improve retrieval for semantic parsing
Zemlyanskiy, Y., de Jong, M., Ainslie, J., Pasupat, P., Shaw, P., Qiu, L., Sanghai, S., & Sha, F. (2022) · arXiv · doi:10.48550/arxiv.2209.14899 · Cited by 8 · cited 1×
“We will leave its consideration to future research.”
BackgroundThe relevance of software documentation, tools and technologies
Forward, A., & Lethbridge, T. C. (2002) · doi:10.1145/585064.585065 · Cited by 4
Research works printed but not linked: 3
- Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2018. code2seq: Generating sequences from structured representations of code. arXiv preprint arXiv:1808.01400 (2018). · arXiv
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74--81.
- Juan Ramos et al. 2003. Using tf-idf to determine word relevance in document queries. In Proceedings of the first instructional conference on machine learning, Vol. 242. Citeseer, 29--48.
Cited by
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695988 · Cited by 1,029
Retrieval-Augmented Generation for AI-Generated Content: A Survey
Zhao, P., Zhang, H., Yu, Q., Wang, Z., Geng, Y., Fu, F., Yang, L., Zhang, W., Jiang, J., & Cui, B. (2024) · arXiv · doi:10.48550/arxiv.2402.19473 · Cited by 240
“ASAP [203] add exemplar tuples to the prompt for better results.”
Towards an understanding of large language models in software engineering tasks
Zheng, Z., Ning, K., Zhong, Q., Chen, J., Chen, W., Guo, L., Wang, W., & Wang, Y. (2025) · Empirical Software Engineering · doi:10.1007/s10664-024-10602-0 · Cited by 116
Generative AI for self-adaptive systems: State of the art and research roadmap
Li, J., Zhang, M., Li, N., Weyns, D., Jin, Z., & Tei, K. (2024) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3686803 · Cited by 77
Prompt Engineering in Clinical Practice: Tutorial for Clinicians
Liu, J., Liu, F., Wang, C., & Liu, S. (2025) · Journal of Medical Internet Research · doi:10.2196/72644 · Cited by 76
Metadata
- Authors
- Ahmed, T., Pai, K. S., Devanbu, P., & Barr, E.
- Affiliations
- University of California, Davis (US)
- UC Davis, Davis, California, USA
- University College London (GB)
- Venue
- Proceedings of the IEEE/ACM 46th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2024-04-12
- Pages
- 1-13
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4394745253
- Semantic Scholar
- 0ff8c04c8bdbf93b39b49582c9195cf3fc894d03
- Cited by
- 112
- References
- 39
- Citations by year
- 2026: 33 · 2025: 57 · 2024: 22
- FWCI
- 42.65
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- UCL Discovery (University College London) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- NSF (National Science Foundation) (CCF (SHF-MEDIUM) No. 2107592)
- Intelligence Advanced Research Projects Agency (IARPA) (W911NF20C0038)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Software › Model-Driven Software Engineering Techniques
- Keywords
- semantic augmentation, prompt engineering, large language models, code summarization, program comprehension
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.