The DOI matches its Crossref record; 153 of the 153 references it cites were found in a registry or an index; 19 of the 58 sources its claims cite were checked for support.
Exploring the Capabilities of LLMs for Code-Change-Related Tasks
Open accessAbstract
Developers deal with code-change-related tasks daily, e.g., reviewing code. Pre-trained code and code-change-oriented models have been adapted to help developers with such tasks. Recently, large language models (LLMs) have shown their effectiveness in code-related tasks. However, existing LLMs for code focus on general code syntax and semantics rather than the differences between two code versions. Thus, it is an open question how LLMs perform on code-change-related tasks. To answer this question, we conduct an empirical study using \(>\) 1B parameters LLMs on three code-change-related tasks, i.e., code review generation, commit message generation, and just-in-time comment update, with in-context learning (ICL) and parameter-efficient fine-tuning (PEFT, including LoRA and prefix-tuning). We observe that the performance of LLMs is poor without examples and generally improves with examples, but more examples do not always lead to better performance. LLMs tuned with LoRA have comparable performance to the state-of-the-art small pre-trained models. Larger models are not always better, but Llama 2 and Code Llama families are always the best. The best LLMs outperform small pre-trained models on the code changes that only modify comments and perform comparably on other code changes. We suggest future work should focus more on guiding LLMs to learn the knowledge specific to the changes related to code rather than comments for code-change-related tasks.
Overview
CoChat Summary
## TL;DR
This paper empirically studies how >1B-parameter LLMs perform on three code-change-related generation tasks: code review generation (CRG), commit message generation (CMG), and just-in-time comment update (JITCU). The main takeaway is that LLMs can be promising, especially with LoRA fine-tuning and in the Code Llama / Llama 2 families, but performance depends strongly on examples, input format, and the type of code change.
## Key findings
- Zero-shot is weak, but one example helps a lot. The paper reports that performance is poor without examples, and with one example the performance “generally drastically improves”; however, more examples do not always help because of context-length limits and truncation.
- LoRA beats prefix-tuning. On CRG and CMG, the paper gives examples where Code Llama-13b with LoRA reaches 5.61 BLEU on CRG and 12.52 BLEU on CMG, versus 1.17 BLEU and 2.54 BLEU with prefix-tuning, respectively.
- LLM-PEFTs can match or beat small pre-trained models in some settings. The paper reports that LLM-PEFTs outperform fully fine-tuned CodeT5 on all tasks, and can be comparable to CCT5; for example, on JITCU, the best LLM-PEFT improves ACC by 18.47% over CCT5, while on CRG the best LLM-PEFT is 8.3% better than CCT5.
- Input format and code-change type matter. For LLM-ICLs, using diff usually works better than using two code snippets, especially on JITCU; for code-change categories, the paper reports that LLM-PEFTs do best on doc-only changes and are comparable to small models on other categories.
## Methods
- The study evaluates four open-source LLM families—CodeGen, InCoder, Llama 2, and Code Llama—plus two small pre-trained models: CodeT5 and CCT5.
- It studies three tasks: CRG, CMG, and JITCU. For CRG it uses 138,127 diff-review pairs; for JITCU it uses 98,622 comment update instances from 1,496 Java projects; for CMG it uses the MCMD dataset with 5 languages and 450,000 commits per language.
- The main analysis compares ICL, PEFT (LoRA and prefix-tuning), and full fine-tuning; it uses BLEU-4 for CRG, B-Norm/BLEU for CMG, and GLEU + ACC for JITCU, with paired bootstrap resampling with 1000 resamples for significance testing.
## Limitations
- The paper states a selection-of-models threat and a potential data leakage threat; it argues its zero-shot results reduce concern about memorization, but this does not eliminate leakage risk.
- The paper states a limitation of automated evaluation metrics (BLEU-4, ACC, GLEU), noting they may not fully reflect human judgment; it partly addresses this with a human evaluation on CMG.
- Evident from the text: the study only uses open-source LLMs and excludes closed-source models because their parameters are inaccessible, so the findings do not cover proprietary LLMs.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
11 Valid · 9 Not valid · 15 Not checked · 167 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“CCT5 is the state-of-the-art code-change-oriented pre-trained models.”
What the sources say
- Supportedcites [39] [39] Bo Lin, Shangwen Wang, Zhongxin Liu, Yepang Liu, Xin Xia, and Xiaoguang Mao. CCT5: A code-change-oriented pre-trained model. In Satish Chandra, Kelly Blincoe, and Paolo Tonella, editors, Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, San Francisco, CA, USA, December 3-9, 2023 , pages 1509-1521. ACM, 2023. - source
The Introduction identifies CCT5 as a code-change-oriented pre-trained model and reports that it achieved state-of-the-art performance on three code-change-related tasks and two code-review tasks. That supports the citing sentence’s characterization in the context of the paper’s evaluations. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“It is pre-trained with 39.6 GB of code change data collected from 35K popular GitHub repositories in six programming languages.”
What the sources say
- Supportedcites [39] [39] Bo Lin, Shangwen Wang, Zhongxin Liu, Yepang Liu, Xin Xia, and Xiaoguang Mao. CCT5: A code-change-oriented pre-trained model. In Satish Chandra, Kelly Blincoe, and Paolo Tonella, editors, Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, San Francisco, CA, USA, December 3-9, 2023 , pages 1509-1521. ACM, 2023. - source
Section 3.1 says the pre-training data came from popular GitHub projects in six programming languages. Section 3.2 reports 35,530 projects, and Table 1 lists a total pre-training dataset size of 39.6 GB. “35K” is a reasonable rounding of the reported project count. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For LLM-PEFTs, we find no significant difference among the three tasks with different input formats.”
- Also stated in: § 3.4 RQ4: How Do LLMs Perform With Different Input Formats on Code-Change-Related Tasks? · plain
“For LLM-PEFTs, there is no significant performance difference between different input formats.”
- Also stated in: § 3.4 RQ4: How Do LLMs Perform With Different Input Formats on Code-Change-Related Tasks? · plain
“Finding 9 : When using LLM-PEFT, the input format does not significantly affect the performance of the LLM.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3.4 RQ4: How Do LLMs Perform With Different Input Formats on Code-Change-Related Tasks? · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“LLM-PEFTs outperform the fully fine-tuned small models on the code changes that only revised documents in code (i.e., code comment), and on other code changes that perform code featuring code refactoring or modify both code and documentation, LLM-PEFTs perform comparably to fully fine-tuned small models.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results show that the commit messages generated by LLM-PEFT are the most expressive and concise without losing much adequacy.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
108 more claims
Create a free account to see all 202 claims and to verify the 27 references still open.
- Probably not a substantial claim. 51
- It may state two separate claims as one. 57
- The quote only partly states this claim. 4
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.1 Human Evaluation
“The average scores achieved by LLM-PEFTs for adequacy, conciseness, and expressiveness are 3.29, 3.78, and 4.22, respectively.”
§ 4.1 Human Evaluation
“| LLM-PEFT | 3.29 | 3.78 | 4.23 |”
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Interrater reliability: The kappa statistic
McHugh, M. L. (2012) · Biochemia Medica · doi:10.11613/bm.2012.031 · Cited by 19,877 · cited 1×
“For the models reported with two parameter sizes, we use both of them in our work.”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“Furthermore, this category can be divided into two subcategories: ① Feature code change represents the code-only code changes where the functional logic is modified.”
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473 · cited 2×
“Publication date: July 2024.”
Proceedings of the 40th Annual Meeting on Association for Computational Linguistics
Isabelle, P. (2002) · Cited by 1,635
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495 · cited 1×
“The difference in model size and training data may make Bloom (or other similar models) perform better than small pre-trained models.”
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495
Research works printed but not linked: 7
- Michael E. Fagan. 2022. Design and code inspections to reduce errors in program development (reprint). In Software Pioneers. Manfred Broy and Ernst Denert (Eds.), Springer, Berlin, 575–607.
- Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations (ICLR ’22), 1–26.
- 10.1145/3324884.3416581 · DOI · Paper page
- Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations (ICLR ’23), 1–25.
- 10.5555/3455716.3455856 · doi:10.5555/3455716.3455856
- Di Wu, Wasi Uddin Ahmad, Dejiao Zhang, Murali Krishna Ramanathan, and Xiaofei Ma. 2024. Repoformer: Selective retrieval for repository-level code completion. In The Forty-first International Conference on Machine Learning (ICML ’24), 1–25.
- Mae Zeid, Khaled El-Bahnasy, and S. E. Abu-Youssef. 2022. An efficient optimized framework for analyzing the performance of breast cancer using machine learning algorithms. Journal of Theoretical and Applied Information Technology 100, 14 (2022), 5165–5178.
Cited by
On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub
Watanabe, M., Li, H., Kashiwa, Y., Reid, B., Iida, H., & Hassan, A. E. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3798166 · Cited by 34
BitsAI-CR: Automated Code Review via LLM in Practice
Sun, T., Xu, J., Li, Y., Yan, Z., Zhang, G., Xie, L., Geng, L., Wang, Z., Chen, Y., Lin, Q., et al. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3728552 · Cited by 57
Yang, X., Zhu, W., Pacheco, M., Zhou, J., Wang, S., Hu, X., & Liu, K. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715738 · Cited by 26
Intention is All you Need: Refining your Code from your Intention
Guo, Q., Xie, X., Liu, S., Hu, M., Li, X., & Bu, L. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00191 · Cited by 21
“Recent works [17], [18], [19], [20], [21], [22], [23] have been exploring the feasibility of using large language models (LLMs) for code refinement.”
Augmenting Large Language Models with Static Code Analysis for Automated Code Quality Improvements
Abtahi, S. M., & Azim, A. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00017 · Cited by 19
“Studies suggest that LLMs can enhance code-change tasks by improving the understanding of code modifications [18].”
Resource-Efficient & Effective Code Summarization
Afrin, S., Call, J., Nguyen, K.-N., Chaparro, O., & Mastropaolo, A. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00032 · Cited by 21
“This additional capability can be harnessed even in the context of QLoRA training, as demonstrated in prior work [57, 58].”
Automatically Fixing Dependency Breaking Changes
Fruntke, L., & Krinke, J. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729366 · Cited by 27
“This approach is supported by ongoing research on using diffs for code review tasks [18].”
Chang, J., Zhou, X., Wang, L., Lo, D., & Li, B. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3668601 · Cited by 28
“We fine-tune each p θ l separately using Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method that has been shown effective in software debugging tasks [39], [40], [41].”
Self-citationNazih, W., Abuhmed, T., Alharbi, M., & El–Sappagh, S. (2025) · PLoS ONE · doi:10.1371/journal.pone.0332134 · Cited by 6
“For this study, a model with at least 1B parameters is considered an LLM; otherwise, it is considered an LM [60,61].”
A Reflection on Change Classification in the Era of Large Language Models
Kim, S., Shivaji, S., & Whitehead, J. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3539566 · Cited by 3
“The use of LLMs to automatically create commit review comments has been explored by several teams, and could be adapted to explain a bug-introducing change [29], [30], [31], [32].”
iCodeReviewer: Improving Secure Code Review with Mixture of Prompts
Peng, Y. F., Kim, K., Meng, L., & Liu, K. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00264 · Cited by 9
“For example, prior studies evaluated the performance of their approaches on the code review comment generation task with BLEU and ROUGE-L [6], [31].”
Tang, J., Li, Y., Yu, L., Lu, J., Huang, Z., Zhang, F., & Zuo, C. (2025) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre66568.2025.00033 · Cited by 6
“Recent advancements in Large Language Models (LLMs) have demonstrated significant potential in code comprehension and generation [11], [12], leading to the development of various approaches for automating the code review process *Corresponding author.”
Integrating RAG and LLM for Automated Code Review in Practice
Yang, L., & Chen, Y. (2025) · International Symposium on Computer and Information Processing Technology (ISCIPT) · doi:10.1109/iscipt67144.2025.11265531 · Cited by 2
“More recent efforts have concentrated on optimizing hyperparameters, refining fine-tuning techniques, and developing effective prompt designs [12, 13, 14, 15].”
Bulhakov, V., d’Aloisio, G., Sipio, C. D., et al. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04190-6_21 · Cited by 2
VerLog: Enhancing Release Note Generation for Android Apps using Large Language Models
Guo, J., Yang, H., & Cai, H. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728961 · Cited by 8
“Future work should focus on expanding language support, integrating VerLog into CI/CD pipelines, and exploring personalization based on user roles.”
DiSCo: Towards Decompiling EVM Bytecode to Source Code using Large Language Models
Su, X., Liang, H., Wu, H., Niu, B., Xu, F., & Zhong, S. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729373 · Cited by 13
Botello, J. G., Frew, L., Padilla, J. J., & Weigle, M. C. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10826069 · Cited by 3
“Fan et al. [13] used LLMs to summarize code changes and update comments within code.”
No More Hidden Pitfalls? Exposing Smart Contract Bad Practices with LLM-Powered Hybrid Analysis
Li, X., Li, Z., Li, W., Zhang, Y., & Wang, X. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3795692 · Cited by 15
Code-DiTing: Automatic Evaluation of Code Generation without References or Test Cases
Yang, G., Zhou, Y., Chen, X., Zheng, W., Hu, X., Zhou, X., Lo, D., & Chen, T. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00021 · Cited by 13
“Their application has shown remarkable potential across various software engineering tasks [4], [5], particularly in code generation [6]–[8].”
Self-citationToo Noisy To Learn: Enhancing Data Quality for Code Review Comment Generation
Liu, C., Lin, H. Y., & Thongtanunam, P. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00043 · Cited by 29
“Furthermore, Fan et al. [49] found that fine-tuning more recent large language models like Llama2 and CodeLlama sometimes did not outperform CodeReviewer, with BLEU-4 score differences ranging from -1.28 to 0.42.”
InfluentialContext Conquers Parameters: Outperforming Proprietary Llm in Commit Message Generation
Imani, A., Ahmed, I., & Moshirpour, M. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00048 · Cited by 20
“The cautions are designed to warn the LLM about different fine-grained statement types (C3), such as Javadocs, method declarations, etc., to avoid confusion between documentation and code changes [57].”
A Systematic Literature Review of Code Smell Detection Tools for JavaScript Systems
Souza, S., Ribeiro, F., Fernandes, E., et al. (2026) · SBC Reviews on Computer Science · doi:10.5753/reviews.2026.6187
“JavaScript developers could benefit from AI-assisted code review and refactoring suggestions, as developers of other languages already benefit (Amaral et al ., 2025; Fan et al ., 2025; Han et al ., 2022; Lu et al ., 2023).”
Can LLMs Make Software Testing Greener? An Empirical Study on JUnit Test Energy Reengineering
Liu, X., & Zaidman, A. (2026) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3803437.3805600
Enhancing Feature Quality in JIT‐SDP: An Empirical Study of Branch Dependency Mining
Chen, Z., & Song, L. (2026) · Journal of Software Evolution and Process · doi:10.1002/smr.70155
One Size Does Fit All: Exploring Model Fusion for Software Engineering Tasks
Qiu, Y., Qin, Y., Geng, M., Wang, S., & Dong, D. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808198
Concept Labeling and Code Similarity Aware Programming Knowledge TracingPreprint
Li, X., Sun, Q., Yang, H., Wu, J. H., Ouyang, Y., Li, J., & Rong, W. (2026) · Research Square · doi:10.21203/rs.3.rs-9701000/v1
UntCC: Untangling Composite Commits Using Structural and Semantic Information
Jin, Y., Yang, L., Zhang, H., Li, G., Lyu, J., Liu, B., Zhou, X., Kuang, H., & Dong, L. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3692448
Code Language Models for Security Patch Management: How Far are We?
Lin, X., Cao, S., Yu, L., Sun, X., Xiao, F., Xue, L., Wu, C., Ren, K., & Lo, D. (2026) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2026.3690594
Self-citationHe, L., Huang, P., Shi, T., Li, J., Chen, Q., & Pan, J. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11461298
R$$^2$$ComSync: improving code-comment synchronization with in-context learning and reranking
Yang, Z., Lin, H., Yu, X., Keung, J., Liu, S., Chan, P. Y. P., Sun, Y., & Zhang, F. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10800-4 · Cited by 2
“In recent years, Large Language Models (LLMs) have been applied in various code intel - ligence scenarios, such as code generation (Jiang et al. 2024 ; Li et al. 2024 ), code translation (Yang et al. 2024 ; Xue et al. 2025a , b), and program repair (Xia et al. 2023 ; Xue et al. 2024 ), owing to their powerful generali…”
InfluentialA Systematic Literature Review of Parameter-Efficient Fine-Tuning for Large Code Models
Afrin, S., Haque, M. Z., & Mastropaolo, A. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3796522 · Cited by 19
“Additionally, we noted that tasks such as Code Review and Code Review Generation have received increasing attention from researchers [32, 88, 168].”
InfluentialAn Empirical Study on the Characteristics of Reusable Code Clones
Yu, C., Noei, S., Zhang, H., & Zou, Y. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3793251 · Cited by 5
An Empirical Study of Parameter-Efficient Fine-Tuning in Code Change Learning and Beyond
Liu, S., Keung, J., Jin, Z. G., Yang, Z., Liu, F., & Zhang, H. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3637335 · Cited by 2
“Recently, more and more studies have proved its effectiveness, especially when the given examples resemble the target problem [27], [39].”
FGit: Fault-Guided Fine-Tuning for Code Generation
Fan, L., Liu, Z., Wang, H., Bao, L., Xia, X., & Li, S. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00114 · Cited by 5
Self-citationA Feedback-Enhanced Effort-Aware Just in Time Defect Prediction Pipeline
AlGhussein, H., Mustafa, A., Talib, M. W. A., Azzeh, M., & Nassif, A. B. (2025) · International Conference on Developments in eSystems Engineering (DeSE) · doi:10.1109/dese68208.2025.11367918
“…the use of pre-trained transformer models, utilized the code semantics and commit history to enhance the accuracy of prediction and generalization [5]. demonstrated that code pre-trained models, fine-tuned with Lo-RA, perform strongly on various code-change-related tasks involving commit-level…”
Towards Trustworthy AI-Assisted Software Development
Haque, E. A. (2025) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc65237.2025.00062 · Cited by 1
“My prior work found that developers use AI for learning new technologies and best practices [1], expecting guidance on proper software engineering principles, but recent empirical evidence suggests that LLMs struggle with fundamental concepts like refactoring patterns [6].”
Can Llms Update Api Documentation?
Lee, S., Heo, J., & Dearstyne, K. R. (2025) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme64153.2025.00048
“Additionally, studies have investigated the use of LLMs for automatically generating code comments and commit messages [12] or for generating documentation from source code [13].”
Metadata
- Authors
- Fan, L., Liu, J., Liu, Z., Lo, D., Xia, X., & Li, S.
- Affiliations
- Zhejiang University (CN)
- Singapore Management University (SG)
- Venue
- ACM Transactions on Software Engineering and Methodology
- Published
- 2025-07
- Volume
- 34
- Issue
- 6
- Pages
- 1-36
- ISSN
- 1049-331X (ISSN-L)
- 1557-7392
- OpenAlex
- W4405703871
- Semantic Scholar
- b94ed0c8571d20e8403e4d3b95dacbda04f78a41
- Cited by
- 41
- References
- 113
- Citations by year
- 2026: 17 · 2025: 22
- FWCI
- 15.46
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Transactions on Software Engineering and Methodology (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62202420)
- National Research Foundation, under its Investigatorship (NRF-NRFI08-2022-0002)
- National Research Foundation, Singapore
- Zhejiang University Education Foundation Qizhen Scholar Foundation
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- large language models, code amendments, commit message generation, in-context learning, parameter-efficient fine-tuning, LoRa, prefix tuning, Llama 2, CodeLlama, developer productivity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.