CoChat

Post-Incorporating Code Structural Knowledge Into Pretrained Models via ICL for Code Translation

Open access
Du, Yali, Sun, Hui, Li, Ming · 2025 · IEEE Transactions on Software Engineering

The DOI matches its Crossref record; 92 of the 112 references it cites were found in a registry or an index, and 15 are not checked yet; 7 of the 41 sources its claims cite were checked for support.

Abstract

Code translation migrates codebases across programming languages. Recently, large language models (LLMs) have achieved significant advancements in software mining. However, handling the syntactic structure of source code remains a challenge. Classic syntax-aware methods depend on intricate model architectures and loss functions, rendering their integration into LLM training resource-intensive. This paper employs in-context learning (ICL), which directly integrates task exemplars into the input context, to post-incorporate code structural knowledge into pre-trained LLMs. We revisit exemplar selection in ICL from an information-theoretic perspective, proposing that list-wise selection based on information coverage is more precise and general objective than traditional methods based on combine similarity and diversity. To address the challenges of quantifying information coverage, we introduce a surrogate measure, Coverage of Abstract Syntax Tree (CAST), measuring maximum subtree coverage between ASTs of test source code and exemplars. Furthermore, we formulate the NP-hard CAST maximization for exemplar selection and prove that it is a standard submodular maximization problem. Therefore, we propose a greedy algorithm for CAST submodular maximization, which theoretically guarantees a (1 − 1/e)-approximate solution in polynomial time complexity. Our method is the first training-free and model-agnostic approach to post-incorporate code structural knowledge into existing LLMs at test time. Experimental results show that our method significantly improves LLMs performance in code translation and reveals two meaningful insights: 1) Code structural knowledge can be effectively post-incorporated into pre-trained LLMs during inference, despite being overlooked during training; 2) Scaling up model size or training data does not lead to the emergence of code structural knowledge, underscoring the necessity of explicitly considering code syntactic structure.

Verified 2026-11-19 · 15 references not checked yet

Overview

CoChat Summary

## TL;DR

The paper proposes CAST, a training-free, model-agnostic in-context learning retrieval method that post-incorporates code structural knowledge into pretrained LLMs for code translation. Across code translation benchmarks, CAST-F generally outperforms similarity- and diversity-based exemplar selection, and the paper reports that scaling model size alone does not make structural knowledge emerge sufficiently.

## Key findings

  • On 5-shot code translation with Qwen2-7B, CAST-F achieved an average CA of 64.28%, higher than Random (61.28%), LD (62.33%), AST-ED (63.12%), BM25 (62.09%), CodeBERT (61.91%), and Diversity (62.68%).
  • For Gemma2-27B, CAST-F reached 70.03% average CA, exceeding Random (68.81%) and Diversity (68.13%); for Qwen2.5-72B, CAST-F reached 71.33% average CA versus Diversity at 70.19%.
  • On AVATAR with Qwen2-7B, CAST-F achieved the best reported computational accuracy in the table: 47.79% for Java→Python and 50.40% for Python→Java, and the paper says it also gave the best overall CodeBLEU scores.
  • For code summarization on CodeXGLUE, CAST-F achieved the highest BLEU and Rouge-L scores across nearly all languages and metrics; for example, on Java it scored 2.42 BLEU / 9.90 Rouge-L, and on Ruby 6.71 BLEU / 14.47 Rouge-L.
  • In the failure analysis for Python→Java 20-shot translation, CAST-F had the lowest Wrong Answer rate (26.41%) and the highest Accept rate (58.10%).

## Methods

  • The paper formulates code translation as ICL with exemplar selection and proposes CAST (Coverage of AST), a surrogate score for information coverage based on the ratio of maximum covered subtrees between the test code AST and exemplar ASTs.
  • It uses tree-sitter to extract ASTs, keeps only node types, computes subtree fingerprints by hashing, builds a co-occurrence matrix between test subtrees and database samples, and selects exemplars by maximizing CAST.
  • CAST maximization is shown to be NP-hard but a non-negative, monotone, submodular maximization problem; the paper uses a greedy algorithm with a theoretical (1 - 1/e) approximation guarantee and evaluates it on 948 parallel functions, focusing on the 568 functions with unit tests, plus AVATAR and CodeXGLUE code summarization benchmarks.

## Limitations

  • The paper explicitly states that improving CAST may require considering more advanced code structure beyond AST, such as CFG and DFG, and suggests a feedback loop during inference as future work.
  • The threat analysis states a possible internal validity threat from LLM data leakage due to training/testing overlap, though it says prior work suggests the threat is limited.
  • It also notes an external validity threat from dataset quality and an external validity threat from implementation differences across compared techniques; these are mitigated by using state-of-the-art models and a cleaned, widely used dataset, but the threat itself is still stated.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/tse.2025.3605768

Cites

92 of 112 references verified0 citation errors · 5 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

6 Valid · 8 Not valid · 21 Not checked · 126 Own results and interpretations

67 more claims

Create a free account to see all 161 claims and to verify the 14 references still open.

See all claims — free account
  • It may state two separate claims as one. 64
  • Probably not a substantial claim. 32

Create a free account to see these claims.

Citations

Cites

Show all 112 references

Cited by

See all 5

Metadata

Authors
Du, Y., Sun, H., & Li, M.
Affiliations
Nanjing University (CN)
Venue
IEEE Transactions on Software Engineering
Published
2025-11
Volume
51
Issue
11
Pages
3038-3055
ISSN
  • 0098-5589 (ISSN-L)
  • 1939-3520
  • 2326-3881
OpenAlex
W4413967102
Cited by
0
References
86
FWCI
0.00
Type
journal article
Language
en
Open access
Locations
  • IEEE Transactions on Software Engineering (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Natural Science Foundation of China (62076121)
  • Major Program (JD) of Hubei Province (2023BAA024)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
program translation, in-context learning, abstract syntax tree, exemplar selection, information coverage, submodular maximization, greedy approximation, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Post-Incorporating Code Structural Knowledge Into Pretrained Models via ICL for Code Translation. https://app.cochat.ai/papers/10.1109/tse.2025.3605768
The paper
Du, Y., Sun, H., & Li, M. (2025). Post-Incorporating Code Structural Knowledge Into Pretrained Models via ICL for Code Translation. IEEE Transactions on Software Engineering. https://doi.org/10.1109/tse.2025.3605768

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.