CoChat

The DOI matches its Crossref record; 153 of the 153 references it cites were found in a registry or an index; 19 of the 58 sources its claims cite were checked for support.

Exploring the Capabilities of LLMs for Code-Change-Related Tasks

Open access
Fan, Lishui, Liu, Jiakun, Liu, Zhongxin, Lo, David, Xia, Xin, Li, Shanping · 2025 · ACM Transactions on Software Engineering and Methodology

Abstract

Developers deal with code-change-related tasks daily, e.g., reviewing code. Pre-trained code and code-change-oriented models have been adapted to help developers with such tasks. Recently, large language models (LLMs) have shown their effectiveness in code-related tasks. However, existing LLMs for code focus on general code syntax and semantics rather than the differences between two code versions. Thus, it is an open question how LLMs perform on code-change-related tasks. To answer this question, we conduct an empirical study using \(>\) 1B parameters LLMs on three code-change-related tasks, i.e., code review generation, commit message generation, and just-in-time comment update, with in-context learning (ICL) and parameter-efficient fine-tuning (PEFT, including LoRA and prefix-tuning). We observe that the performance of LLMs is poor without examples and generally improves with examples, but more examples do not always lead to better performance. LLMs tuned with LoRA have comparable performance to the state-of-the-art small pre-trained models. Larger models are not always better, but Llama 2 and Code Llama families are always the best. The best LLMs outperform small pre-trained models on the code changes that only modify comments and perform comparably on other code changes. We suggest future work should focus more on guiding LLMs to learn the knowledge specific to the changes related to code rather than comments for code-change-related tasks.

Overview

CoChat Summary

## TL;DR

This paper empirically studies how >1B-parameter LLMs perform on three code-change-related generation tasks: code review generation (CRG), commit message generation (CMG), and just-in-time comment update (JITCU). The main takeaway is that LLMs can be promising, especially with LoRA fine-tuning and in the Code Llama / Llama 2 families, but performance depends strongly on examples, input format, and the type of code change.

## Key findings

  • Zero-shot is weak, but one example helps a lot. The paper reports that performance is poor without examples, and with one example the performance “generally drastically improves”; however, more examples do not always help because of context-length limits and truncation.
  • LoRA beats prefix-tuning. On CRG and CMG, the paper gives examples where Code Llama-13b with LoRA reaches 5.61 BLEU on CRG and 12.52 BLEU on CMG, versus 1.17 BLEU and 2.54 BLEU with prefix-tuning, respectively.
  • LLM-PEFTs can match or beat small pre-trained models in some settings. The paper reports that LLM-PEFTs outperform fully fine-tuned CodeT5 on all tasks, and can be comparable to CCT5; for example, on JITCU, the best LLM-PEFT improves ACC by 18.47% over CCT5, while on CRG the best LLM-PEFT is 8.3% better than CCT5.
  • Input format and code-change type matter. For LLM-ICLs, using diff usually works better than using two code snippets, especially on JITCU; for code-change categories, the paper reports that LLM-PEFTs do best on doc-only changes and are comparable to small models on other categories.

## Methods

  • The study evaluates four open-source LLM families—CodeGen, InCoder, Llama 2, and Code Llama—plus two small pre-trained models: CodeT5 and CCT5.
  • It studies three tasks: CRG, CMG, and JITCU. For CRG it uses 138,127 diff-review pairs; for JITCU it uses 98,622 comment update instances from 1,496 Java projects; for CMG it uses the MCMD dataset with 5 languages and 450,000 commits per language.
  • The main analysis compares ICL, PEFT (LoRA and prefix-tuning), and full fine-tuning; it uses BLEU-4 for CRG, B-Norm/BLEU for CMG, and GLEU + ACC for JITCU, with paired bootstrap resampling with 1000 resamples for significance testing.

## Limitations

  • The paper states a selection-of-models threat and a potential data leakage threat; it argues its zero-shot results reduce concern about memorization, but this does not eliminate leakage risk.
  • The paper states a limitation of automated evaluation metrics (BLEU-4, ACC, GLEU), noting they may not fully reflect human judgment; it partly addresses this with a human evaluation on CMG.
  • Evident from the text: the study only uses open-source LLMs and excludes closed-source models because their parameters are inaccessible, so the findings do not cover proprietary LLMs.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3709358

Cites

153 of 153 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

11 Valid · 9 Not valid · 15 Not checked · 167 Own results and interpretations

108 more claims

Create a free account to see all 202 claims and to verify the 27 references still open.

See all claims — free account
  • Probably not a substantial claim. 51
  • It may state two separate claims as one. 57
  • The quote only partly states this claim. 4

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 113 references

Research works printed but not linked: 7

  1. Michael E. Fagan. 2022. Design and code inspections to reduce errors in program development (reprint). In Software Pioneers. Manfred Broy and Ernst Denert (Eds.), Springer, Berlin, 575–607.
  2. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations (ICLR ’22), 1–26.
  3. 10.1145/3324884.3416581 · DOI · Paper page
  4. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations (ICLR ’23), 1–25.
  5. 10.5555/3455716.3455856 · doi:10.5555/3455716.3455856
  6. Di Wu, Wasi Uddin Ahmad, Dejiao Zhang, Murali Krishna Ramanathan, and Xiaofei Ma. 2024. Repoformer: Selective retrieval for repository-level code completion. In The Forty-first International Conference on Machine Learning (ICML ’24), 1–25.
  7. Mae Zeid, Khaled El-Bahnasy, and S. E. Abu-Youssef. 2022. An efficient optimized framework for analyzing the performance of breast cancer using machine learning algorithms. Journal of Theoretical and Applied Information Technology 100, 14 (2022), 5165–5178.

Cited by

Metadata

Authors
Fan, L., Liu, J., Liu, Z., Lo, D., Xia, X., & Li, S.
Affiliations
  • Zhejiang University (CN)
  • Singapore Management University (SG)
Venue
ACM Transactions on Software Engineering and Methodology
Published
2025-07
Volume
34
Issue
6
Pages
1-36
ISSN
  • 1049-331X (ISSN-L)
  • 1557-7392
OpenAlex
W4405703871
Cited by
41
References
113
Citations by year
2026: 17 · 2025: 22
FWCI
15.46
Type
journal article
Language
en
Open access
Locations
  • ACM Transactions on Software Engineering and Methodology (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
  • National Natural Science Foundation of China (62202420)
  • National Research Foundation, under its Investigatorship (NRF-NRFI08-2022-0002)
  • National Research Foundation, Singapore
  • Zhejiang University Education Foundation Qizhen Scholar Foundation
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
Keywords
large language models, code amendments, commit message generation, in-context learning, parameter-efficient fine-tuning, LoRa, prefix tuning, Llama 2, CodeLlama, developer productivity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.