Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 79 of the 79 references it cites were found in a registry or an index; 31 of the 57 sources its claims cite were checked for support.

Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)

Open access
Ahmed, Toufique, Pai, Kunal Suresh, Devanbu, Premkumar, Barr, Earl · 2024 · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering

Abstract

Large Language Models (LLM) are a new class of computation engines, "programmed" via prompt engineering. Researchers are still learning how to best "program" these LLMs to help developers. We start with the intuition that developers tend to consciously and unconsciously collect semantics facts, from the code, while working. Mostly these are shallow, simple facts arising from a quick read. For a function, such facts might include parameter and local variable names, return expressions, simple pre- and post-conditions, and basic control and data flow, etc.

Overview

CoChat Summary

## TL;DR

The paper proposes ASAP (Automatic Semantic Augmentation of Prompts), which adds automatically extracted semantic facts from code into few-shot prompts for LLM-based code summarization and code completion. On CodeSearchNet/CodeXGLUE-style evaluations, the augmented prompts generally improved performance, including statistically significant BLEU gains across six languages and improvements in zero-shot line completion.

## Key findings

  • On code summarization with `code-davinci-002`, ASAP improved BLEU-4 over BM25 few-shot prompting in all six languages: Java 22.87→25.41 (+11.11%), Python 21.78→24.26 (+11.39%), Ruby 17.21→19.62 (+14.00%), JavaScript 23.27→25.36 (+8.98%), Go 22.67→24.51 (+8.12%), and PHP 28.15→32.73 (+16.27%); overall 22.66→25.32 (+11.74%), with p < 0.01 in each case.
  • In same-project summarization, ASAP outperformed vanilla BM25 few-shot learning in 14 of 16 BLEU comparisons (87.5%).
  • For line completion on 9,292 Java and 6,550 Python samples, ASAP improved Exact Match from 17.97 to 19.01 overall (+5.79%, p < 0.01) and Edit Similarity from 53.01 to 55.72 overall (+5.11%, p < 0.01).
  • ASAP also improved other metrics overall: BLEU-DC and ROUGE-L showed significant gains, while METEOR improved on average but was not significant for 4 of 6 languages.

## Methods

  • The paper evaluates few-shot prompting for code summarization on the CodeSearchNet dataset, using 1,000 test samples per language across six languages: Java, Python, Ruby, JavaScript, Go, and PHP; for same-project experiments, it used project-specific subsets and sorted them by creation date to avoid leakage.
  • The core method, ASAP, augments each retrieved exemplar and the target code with automatically derived semantic facts: repository name/path, tagged identifiers, and a data flow graph; exemplars are retrieved with BM25 from the training set.
  • Main analysis used OpenAI models (`code-davinci-002` primarily, plus `text-davinci-003` and `gpt-3.5-turbo`), with 3-shot prompts, temperature 0, and statistical testing via one-sided pairwise Wilcoxon signed-rank tests (with B-H correction where reported); line completion was evaluated with Exact Match and Edit Similarity.

## Limitations

  • The paper states it did not conduct a user study, so the metric gains may not translate directly into improved developer performance.
  • It could not directly check whether the model had seen test data during training because the training dataset is inaccessible; this is a limitation stated in the paper.
  • Evident from the text: the main experiments were constrained by OpenAI API rate limits and cost, so many analyses used only 1,000 samples per language, and some deeper analyses were restricted mainly to Java and Python.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3597503.3639183

Cites

79 of 79 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Different steps of A 𝑆𝐴𝑃 . (1) Input code and (2) Pool of samples are given the BM25 engine, which matches the given input code against the pool and (3) retrieves bestmatching samples, viz. 3 input+output pairs. These examples are processed by A 𝑆𝐴𝑃 to produce a prompt (4) including 3 exemplars. Each exemplar includes a function definition, the results of analyzing that definition, and its associated comment; the input code is finally appended, along with its analysis product. Exemplar details are in Figure 2. The final prompt is sent via API call (5) to the GPT-3.x model; the returned output, e.g. , summary (6) is returned by GPT-3x.
Figure 1 Different steps of A 𝑆𝐴𝑃 . (1) Input code and (2) Pool of samples are given the BM25 engine, which matches the given input code against the pool and (3) retrieves bestmatching samples, viz. 3 input+output pairs. These examples are processed by A 𝑆𝐴𝑃 to produce a prompt (4) including 3 exemplars. Each exemplar includes a function definition, the results of analyzing that definition, and its associated comment; the input code is finally appended, along with its analysis product. Exemplar details are in Figure 2. The final prompt is sent via API call (5) to the GPT-3.x model; the returned output, e.g. , summary (6) is returned by GPT-3x.CC-BY · From the PDF
Figure 2 Components of an A 𝑆𝐴𝑃 Exemplar. Source Code and Output Comment are extracted from the retrieved pool sample. The Repo info is derived from the source code using GitHub; the Dataflow Info and tagged Identifiers with labels is obtained from an analysis using Treesitter.
Figure 2 Components of an A 𝑆𝐴𝑃 Exemplar. Source Code and Output Comment are extracted from the retrieved pool sample. The Repo info is derived from the source code using GitHub; the Dataflow Info and tagged Identifiers with labels is obtained from an analysis using Treesitter.CC-BY · From the PDF

Verification

Claims the paper makes

17 Valid · 8 Not valid · 11 Not checked · 81 Own results and interpretations

36 more claims

Create a free account to see all 117 claims and to verify the 11 references still open.

See all claims — free account
  • It may state two separate claims as one. 50
  • Probably not a substantial claim. 45
  • The quote only partly states this claim. 4

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 39 references

Research works printed but not linked: 3

  1. Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2018. code2seq: Generating sequences from structured representations of code. arXiv preprint arXiv:1808.01400 (2018). · arXiv
  2. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74--81.
  3. Juan Ramos et al. 2003. Using tf-idf to determine word relevance in document queries. In Proceedings of the first instructional conference on machine learning, Vol. 242. Citeseer, 29--48.

Cited by

See all 112

Metadata

Authors
Ahmed, T., Pai, K. S., Devanbu, P., & Barr, E.
Affiliations
  • University of California, Davis (US)
  • UC Davis, Davis, California, USA
  • University College London (GB)
Venue
Proceedings of the IEEE/ACM 46th International Conference on Software Engineering
Publisher
ACM
Published
2024-04-12
Pages
1-13
ISSN
0270-5257 (ISSN-L)
OpenAlex
W4394745253
Cited by
112
References
39
Citations by year
2026: 33 · 2025: 57 · 2024: 22
FWCI
42.65
Type
conference paper
Language
en
Open access
Locations
  • ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • UCL Discovery (University College London) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Funders
  • NSF (National Science Foundation) (CCF (SHF-MEDIUM) No. 2107592)
  • Intelligence Advanced Research Projects Agency (IARPA) (W911NF20C0038)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Software › Model-Driven Software Engineering Techniques
Keywords
semantic augmentation, prompt engineering, large language models, code summarization, program comprehension

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization). https://app.cochat.ai/papers/10.1145/3597503.3639183
The paper
Ahmed, T., Pai, K. S., Devanbu, P., & Barr, E. (2024). Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization). Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. https://doi.org/10.1145/3597503.3639183

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.