The DOI matches its Crossref record; 79 of the 79 references it cites were found in a registry or an index; 31 of the 57 sources its claims cite were checked for support.
Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)
Open accessAbstract
Large Language Models (LLM) are a new class of computation engines, "programmed" via prompt engineering. Researchers are still learning how to best "program" these LLMs to help developers. We start with the intuition that developers tend to consciously and unconsciously collect semantics facts, from the code, while working. Mostly these are shallow, simple facts arising from a quick read. For a function, such facts might include parameter and local variable names, return expressions, simple pre- and post-conditions, and basic control and data flow, etc.
Overview
CoChat Summary
## TL;DR
The paper proposes ASAP (Automatic Semantic Augmentation of Prompts), which adds automatically extracted semantic facts from code into few-shot prompts for LLM-based code summarization and code completion. On CodeSearchNet/CodeXGLUE-style evaluations, the augmented prompts generally improved performance, including statistically significant BLEU gains across six languages and improvements in zero-shot line completion.
## Key findings
- On code summarization with `code-davinci-002`, ASAP improved BLEU-4 over BM25 few-shot prompting in all six languages: Java 22.87→25.41 (+11.11%), Python 21.78→24.26 (+11.39%), Ruby 17.21→19.62 (+14.00%), JavaScript 23.27→25.36 (+8.98%), Go 22.67→24.51 (+8.12%), and PHP 28.15→32.73 (+16.27%); overall 22.66→25.32 (+11.74%), with p < 0.01 in each case.
- In same-project summarization, ASAP outperformed vanilla BM25 few-shot learning in 14 of 16 BLEU comparisons (87.5%).
- For line completion on 9,292 Java and 6,550 Python samples, ASAP improved Exact Match from 17.97 to 19.01 overall (+5.79%, p < 0.01) and Edit Similarity from 53.01 to 55.72 overall (+5.11%, p < 0.01).
- ASAP also improved other metrics overall: BLEU-DC and ROUGE-L showed significant gains, while METEOR improved on average but was not significant for 4 of 6 languages.
## Methods
- The paper evaluates few-shot prompting for code summarization on the CodeSearchNet dataset, using 1,000 test samples per language across six languages: Java, Python, Ruby, JavaScript, Go, and PHP; for same-project experiments, it used project-specific subsets and sorted them by creation date to avoid leakage.
- The core method, ASAP, augments each retrieved exemplar and the target code with automatically derived semantic facts: repository name/path, tagged identifiers, and a data flow graph; exemplars are retrieved with BM25 from the training set.
- Main analysis used OpenAI models (`code-davinci-002` primarily, plus `text-davinci-003` and `gpt-3.5-turbo`), with 3-shot prompts, temperature 0, and statistical testing via one-sided pairwise Wilcoxon signed-rank tests (with B-H correction where reported); line completion was evaluated with Exact Match and Edit Similarity.
## Limitations
- The paper states it did not conduct a user study, so the metric gains may not translate directly into improved developer performance.
- It could not directly check whether the model had seen test data during training because the training dataset is inaccessible; this is a limitation stated in the paper.
- Evident from the text: the main experiments were constrained by OpenAI API rate limits and cost, so many analyses used only 1,000 samples per language, and some deeper analyses were restricted mainly to Java and Python.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
17 Valid · 8 Not valid · 11 Not checked · 81 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“In most cases, we see improvements of over 2.0 BLEU, the required threshold for human perception noted by Roy et al. [57].”
§ Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) > 4.2 A𝑆𝐴𝑃 Prompt Enhancement
What the sources say
- Disputedcites [57] 57. Devjeet Roy, Sarah Fakhoury, and Venera Arnaoudova. 2021. Reassessing auto- matic evaluation metrics for code summarization tasks. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1105-1116. sourceExperimental — accuracy not yet measured
In “Pairwise Model Comparisons” (page not provided), Roy et al. say human assessors perceived differences even when metric scores differed by less than 2 points. They describe 2 points as a boundary below which automatic metrics are unreliable, not a required BLEU threshold for human perception. The paper does not assess ASAP’s improvements. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“De-duplication is key: Code duplication in machine learning models can deceptively inflate performance metrics a lot, when compared to de-duplicated datasets [7,46,59].”
§ Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) > 3.1 Dataset
What the sources say
- Supportedcites [7] 7. Miltiadis Allamanis. 2019. The adverse effects of code duplication in machine learning models of code. In Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software. 143-153. source
The Introduction says duplication can cause evaluations to overestimate the performance users actually observe, with observed performance up to 50% worse than reported results. Section 5 adds that end-user performance is often significantly worse than evaluation metrics suggest, supporting the claim that inflation can be substantial. (LLM, conf=0.93)
- Not checked yetcites [46] 46. Cristina V Lopes, Petr Maj, Pedro Martins, Vaibhav Saini, Di Yang, Jakub Zitny, Hitesh Sajnani, and Jan Vitek. 2017. DéjàVu: a map of code duplicates on GitHub. Proceedings of the ACM on Programming Languages 1, OOPSLA (2017), 1-28. source
Checked before — its earlier verdict is held back until a new check runs.
- Not checked yetcites [59] 59. Ensheng Shi, Yanlin Wang, Lun Du, Junjie Chen, Shi Han, Hongyu Zhang, Dong- mei Zhang, and Hongbin Sun. 2023. On the evaluation of neural code summariza- tion. In Proceedings of the 44th International Conference on Software Engineering. 1597-1608. source
Checked before — its earlier verdict is held back until a new check runs.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“the 2-BLEU threshold noted by Roy et al. [57], below which BLEU results are unreliable predictors of human preference.”
§ Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) > 1 INTRODUCTION
What the sources say
- Supportedcites [57] 57. Devjeet Roy, Sarah Fakhoury, and Venera Arnaoudova. 2021. Reassessing auto- matic evaluation metrics for code summarization tasks. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1105-1116. source
In Pairwise Model Comparisons, the paper says that when two code-summarization approaches differ by no more than 2 metric points, automatic metrics cannot reliably distinguish their quality in agreement with human ratings. This includes BLEU. The threshold concerns the difference between BLEU scores, not an absolute BLEU score below 2. (LLM, conf=0.90)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Roy et al. [57] provide arguments, grounded on human subject study that for code summarization (our central task), that a gain of 2.0 or more BLEU is more likely to correspond with human perception of improvement.”
§ Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) > 3.5 Metrics
What the sources say
- Partly supportedcites [57] 57. Devjeet Roy, Sarah Fakhoury, and Venera Arnaoudova. 2021. Reassessing auto- matic evaluation metrics for code summarization tasks. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1105-1116. sourceExperimental — accuracy not yet measured
The Pairwise Model Comparisons section reports that metric differences of 2 points or less do not consistently align with human judgments, supporting the claim’s general point. But it does not establish 2.0 BLEU as a threshold for perceived improvement: the threshold varies by metric, and BLEU’s disagreement with human assessment is described as erratic even at larger differences. The Implications section recommends METEOR and chrF over BLEU as indicators of human judgment. (LLM, conf=0.88)
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“For the tests that we used (Wilcoxon Signed-rank test), we found that the needed sample size was always below the sample size we used for our primary studies, viz., 1000.”
§ Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) > Sample Size Analysis:
What the sources say
- Could not verifycites [19] 19. Franz Faul, Edgar Erdfelder, Axel Buchner, and Albert-Georg Lang. 2009. Sta- tistical power analyses using G* Power 3.1: Tests for correlation and regression analyses. Behavior research methods 41, 4 (2009), 1149-1160. sourceExperimental — accuracy not yet measured
The paper’s sample-size examples concern correlation and regression analyses (Effect size measure; 5.1 Logistic Regression; page numbers unavailable). Searches for Wilcoxon signed-rank tests and the cited studies’ sample size of 1,000 found no passage addressing that comparison. (LLM, conf=0.77)
- Could not verifycites [20] 20. Franz Faul, Edgar Erdfelder, Albert-Georg Lang, and Axel Buchner. 2007. G* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior research methods 39, 2 (2007), 175-191. sourceExperimental — accuracy not yet measured
The A Priori Power Analyses section (page unavailable) explains how required sample size is calculated, but it does not report the citing authors’ calculations. Searches for Wilcoxon signed-rank tests and a sample size of 1,000 found no passage establishing that the required size was always below 1,000. (LLM, conf=0.77)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
36 more claims
Create a free account to see all 117 claims and to verify the 11 references still open.
- It may state two separate claims as one. 50
- Probably not a substantial claim. 45
- The quote only partly states this claim. 4
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“ASAP [171] incorporates exemplar tuples, consisting of input code, function definitions, analysis results, and corresponding comments, into prompts to yield better results.”
From Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024) · cites this paper as [171]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193146 · Cited by 65,971
Statistical power analyses using G∗Power 3.1: tests for correlation and regression analyses
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009) · Behavior Research Methods · doi:10.3758/brm.41.4.1149 · Cited by 37,615
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
The probabilistic relevance framework: BM25 and beyond
Robertson, S., & Zaragoza, H. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000019 · Cited by 3,374 · cited 1×
“BM25 [55] is a frequency-based retrieval method which improves upon TF-IDF [54].”
Method
Research works printed but not linked: 3
- Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2018. code2seq: Generating sequences from structured representations of code. arXiv preprint arXiv:1808.01400 (2018). · arXiv
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74--81.
- Juan Ramos et al. 2003. Using tf-idf to determine word relevance in document queries. In Proceedings of the first instructional conference on machine learning, Vol. 242. Citeseer, 29--48.
Cited by
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695988 · Cited by 1,029
Retrieval-Augmented Generation for AI-Generated Content: A Survey
Zhao, P., Zhang, H., Yu, Q., Wang, Z., Geng, Y., Fu, F., Yang, L., Zhang, W., Jiang, J., & Cui, B. (2024) · arXiv · doi:10.48550/arxiv.2402.19473 · Cited by 240
“ASAP [203] add exemplar tuples to the prompt for better results.”
Towards an understanding of large language models in software engineering tasks
Zheng, Z., Ning, K., Zhong, Q., Chen, J., Chen, W., Guo, L., Wang, W., & Wang, Y. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10602-0 · Cited by 283
Generative AI for self-adaptive systems: State of the art and research roadmap
Li, J., Zhang, M., Li, N., Weyns, D., Jin, Z., & Tei, K. (2024) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3686803 · Cited by 77
Prompt Engineering in Clinical Practice: Tutorial for Clinicians
Liu, J., Liu, F., Wang, C., & Liu, S. (2025) · Journal of Medical Internet Research · doi:10.2196/72644 · Cited by 76
Metadata
- Authors
- Ahmed, T., Pai, K. S., Devanbu, P., & Barr, E.
- Affiliations
- University of California, Davis (US)
- UC Davis, Davis, California, USA
- University College London (GB)
- Venue
- Proceedings of the IEEE/ACM 46th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2024-04-12
- Pages
- 1-13
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4394745253
- Semantic Scholar
- 0ff8c04c8bdbf93b39b49582c9195cf3fc894d03
- Cited by
- 112
- References
- 39
- Citations by year
- 2026: 33 · 2025: 57 · 2024: 22
- FWCI
- 42.65
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- UCL Discovery (University College London) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- NSF (National Science Foundation) (CCF (SHF-MEDIUM) No. 2107592)
- Intelligence Advanced Research Projects Agency (IARPA) (W911NF20C0038)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Software › Model-Driven Software Engineering Techniques
- Keywords
- semantic augmentation, prompt engineering, large language models, code summarization, program comprehension
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization). https://app.cochat.ai/papers/10.1145/3597503.3639183
- The paper
- Ahmed, T., Pai, K. S., Devanbu, P., & Barr, E. (2024). Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization). Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. https://doi.org/10.1145/3597503.3639183
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.