Exploring the Capabilities of LLMs for Code-Change-Related Tasks
Open accessThe DOI matches its Crossref record; 153 of the 153 references it cites were found in a registry or an index; 19 of the 58 sources its claims cite were checked for support.
Abstract
Developers deal with code-change-related tasks daily, e.g., reviewing code. Pre-trained code and code-change-oriented models have been adapted to help developers with such tasks. Recently, large language models (LLMs) have shown their effectiveness in code-related tasks. However, existing LLMs for code focus on general code syntax and semantics rather than the differences between two code versions. Thus, it is an open question how LLMs perform on code-change-related tasks. To answer this question, we conduct an empirical study using \(>\) 1B parameters LLMs on three code-change-related tasks, i.e., code review generation, commit message generation, and just-in-time comment update, with in-context learning (ICL) and parameter-efficient fine-tuning (PEFT, including LoRA and prefix-tuning). We observe that the performance of LLMs is poor without examples and generally improves with examples, but more examples do not always lead to better performance. LLMs tuned with LoRA have comparable performance to the state-of-the-art small pre-trained models. Larger models are not always better, but Llama 2 and Code Llama families are always the best. The best LLMs outperform small pre-trained models on the code changes that only modify comments and perform comparably on other code changes. We suggest future work should focus more on guiding LLMs to learn the knowledge specific to the changes related to code rather than comments for code-change-related tasks.
Overview
CoChat Summary
## TL;DR
This paper empirically studies how >1B-parameter LLMs perform on three code-change-related generation tasks: code review generation (CRG), commit message generation (CMG), and just-in-time comment update (JITCU). The main takeaway is that LLMs can be promising, especially with LoRA fine-tuning and in the Code Llama / Llama 2 families, but performance depends strongly on examples, input format, and the type of code change.
## Key findings
- Zero-shot is weak, but one example helps a lot. The paper reports that performance is poor without examples, and with one example the performance “generally drastically improves”; however, more examples do not always help because of context-length limits and truncation.
- LoRA beats prefix-tuning. On CRG and CMG, the paper gives examples where Code Llama-13b with LoRA reaches 5.61 BLEU on CRG and 12.52 BLEU on CMG, versus 1.17 BLEU and 2.54 BLEU with prefix-tuning, respectively.
- LLM-PEFTs can match or beat small pre-trained models in some settings. The paper reports that LLM-PEFTs outperform fully fine-tuned CodeT5 on all tasks, and can be comparable to CCT5; for example, on JITCU, the best LLM-PEFT improves ACC by 18.47% over CCT5, while on CRG the best LLM-PEFT is 8.3% better than CCT5.
- Input format and code-change type matter. For LLM-ICLs, using diff usually works better than using two code snippets, especially on JITCU; for code-change categories, the paper reports that LLM-PEFTs do best on doc-only changes and are comparable to small models on other categories.
## Methods
- The study evaluates four open-source LLM families—CodeGen, InCoder, Llama 2, and Code Llama—plus two small pre-trained models: CodeT5 and CCT5.
- It studies three tasks: CRG, CMG, and JITCU. For CRG it uses 138,127 diff-review pairs; for JITCU it uses 98,622 comment update instances from 1,496 Java projects; for CMG it uses the MCMD dataset with 5 languages and 450,000 commits per language.
- The main analysis compares ICL, PEFT (LoRA and prefix-tuning), and full fine-tuning; it uses BLEU-4 for CRG, B-Norm/BLEU for CMG, and GLEU + ACC for JITCU, with paired bootstrap resampling with 1000 resamples for significance testing.
## Limitations
- The paper states a selection-of-models threat and a potential data leakage threat; it argues its zero-shot results reduce concern about memorization, but this does not eliminate leakage risk.
- The paper states a limitation of automated evaluation metrics (BLEU-4, ACC, GLEU), noting they may not fully reflect human judgment; it partly addresses this with a human evaluation on CMG.
- Evident from the text: the study only uses open-source LLMs and excludes closed-source models because their parameters are inaccessible, so the findings do not cover proprietary LLMs.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
11 Valid · 9 Not valid · 15 Not checked · 167 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“CCT5 is the state-of-the-art code-change-oriented pre-trained models.”
What the sources say
- Supportedcites [39] [39] Bo Lin, Shangwen Wang, Zhongxin Liu, Yepang Liu, Xin Xia, and Xiaoguang Mao. CCT5: A code-change-oriented pre-trained model. In Satish Chandra, Kelly Blincoe, and Paolo Tonella, editors, Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, San Francisco, CA, USA, December 3-9, 2023 , pages 1509-1521. ACM, 2023. - source
The Introduction identifies CCT5 as a code-change-oriented pre-trained model and reports that it achieved state-of-the-art performance on three code-change-related tasks and two code-review tasks. That supports the citing sentence’s characterization in the context of the paper’s evaluations. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“It is pre-trained with 39.6 GB of code change data collected from 35K popular GitHub repositories in six programming languages.”
What the sources say
- Supportedcites [39] [39] Bo Lin, Shangwen Wang, Zhongxin Liu, Yepang Liu, Xin Xia, and Xiaoguang Mao. CCT5: A code-change-oriented pre-trained model. In Satish Chandra, Kelly Blincoe, and Paolo Tonella, editors, Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, San Francisco, CA, USA, December 3-9, 2023 , pages 1509-1521. ACM, 2023. - source
Section 3.1 says the pre-training data came from popular GitHub projects in six programming languages. Section 3.2 reports 35,530 projects, and Table 1 lists a total pre-training dataset size of 39.6 GB. “35K” is a reasonable rounding of the reported project count. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For LLM-PEFTs, we find no significant difference among the three tasks with different input formats.”
- Also stated in: § 3.4 RQ4: How Do LLMs Perform With Different Input Formats on Code-Change-Related Tasks? · plain
“For LLM-PEFTs, there is no significant performance difference between different input formats.”
- Also stated in: § 3.4 RQ4: How Do LLMs Perform With Different Input Formats on Code-Change-Related Tasks? · plain
“Finding 9 : When using LLM-PEFT, the input format does not significantly affect the performance of the LLM.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3.4 RQ4: How Do LLMs Perform With Different Input Formats on Code-Change-Related Tasks? · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“LLM-PEFTs outperform the fully fine-tuned small models on the code changes that only revised documents in code (i.e., code comment), and on other code changes that perform code featuring code refactoring or modify both code and documentation, LLM-PEFTs perform comparably to fully fine-tuned small models.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results show that the commit messages generated by LLM-PEFT are the most expressive and concise without losing much adequacy.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
108 more claims
Create a free account to see all 202 claims and to verify the 27 references still open.
- Probably not a substantial claim. 51
- It may state two separate claims as one. 57
- The quote only partly states this claim. 4
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.1 Human Evaluation
“The average scores achieved by LLM-PEFTs for adequacy, conciseness, and expressiveness are 3.29, 3.78, and 4.22, respectively.”
§ 4.1 Human Evaluation
“| LLM-PEFT | 3.29 | 3.78 | 4.23 |”
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Interrater reliability: The kappa statistic
McHugh, M. L. (2012) · Biochemia Medica · doi:10.11613/bm.2012.031 · Cited by 19,877 · cited 1×
“For the models reported with two parameter sizes, we use both of them in our work.”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“Furthermore, this category can be divided into two subcategories: ① Feature code change represents the code-only code changes where the functional logic is modified.”
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473 · cited 2×
“Publication date: July 2024.”
Proceedings of the 40th Annual Meeting on Association for Computational Linguistics
Isabelle, P. (2002) · Cited by 1,635
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495 · cited 1×
“The difference in model size and training data may make Bloom (or other similar models) perform better than small pre-trained models.”
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264
Parameter-efficient fine-tuning of large-scale pre-trained language models
Ding, N., Qin, Y., Yang, G., Wei, F., Yang, Z., Su, Y., Hu, S., Chen, Y., Chan, C.-M., Chen, W., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00626-4 · Cited by 1,191 · cited 1×
“Similar phenomena are also reported by prior studies [8, 10, 12].”
Self-citationYadav, S., & Shukla, S. (2016) · 2016 IEEE 6th International Conference on Advanced Computing (IACC) · doi:10.1109/iacc.2016.25 · Cited by 773 · cited 1×
“We exclude the variants that are further trained on Python corpus (too specific) or instruction database (different capability) as they are not suitable for our task.”
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.759 · Cited by 755 · cited 1×
“…} msgs_to_rerender.push(msg); - msg.alerted=event.flags.indexOf("has_alert_word")!==-1; - msg.mentioned=event.flags.indexOf("mentioned")!==-1|| - event.flags.indexOf("wildcard_mentioned")!==-1; + message_store.set_message_booleans(msg,event.flags); condense.un_cache_message_content_height(msg.id);”
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · arXiv · doi:10.48550/arxiv.2202.12837 · Cited by 752
Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning
Raschka, S. (2018) · arXiv · doi:10.48550/arxiv.1811.12808 · Cited by 735 · cited 1×
“We exclude the variants that are further trained on Python corpus (too specific) or instruction database (different capability) as they are not suitable for our task.”
CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
Le, H., Wang, Y., Gotmare, A. D., Savarese, S., & Hoi, S. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2207.01780 · Cited by 614
Change Distilling:Tree Differencing for Fine-Grained Source Code Change Extraction
Fluri, B., Wursch, M., PInzger, M., & Gall, H. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.70731 · Cited by 607 · cited 1×
“After the launch of a project, developers constantly change code to introduce new features and maintain existing code (e.g., performing refactoring and fixing bugs) [11, 14, 16].”
Liu, C., Lowe, R., Serban, I. V., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.08023 · Cited by 603
Code Llama: Open Foundation Models for Code
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12950 · Cited by 531 · cited 6×
“1 For example, Meta released the Code Llama models [67], which are initialized from the Llama 2 models [73] and trained on 500B tokens from a code-heavy dataset.”
InfluentialSelf-citationUsing an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487
CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
Husain, H., Wu, H.-H., Gazit, T., Allamanis, M., & Brockschmidt, M. (2019) · arXiv · doi:10.48550/arxiv.1909.09436 · Cited by 483
Crosslingual Generalization through Multitask Finetuning
Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Biderman, S., Le Scao, T., Bari, M. S., Shen, S., Yong, Z. X., Schoelkopf, H., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.891 · Cited by 361 · cited 1×
“For example, the Bloom model [58] contains 176B parameters, 789 times more than CCT5 (223M), is pre-trained on 363B natural language tokens, and significantly outperforms all the models with less than 1B parameters on natural language processing tasks.”
Proceedings of the 44th International Conference on Software Engineering
(2022) · doi:10.1145/3510003 · Cited by 326
GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow
Black, S., Leo, G., Wang, P., Leahy, C., & Biderman, S. (2021) · Zenodo · doi:10.5281/zenodo.5297715 · Cited by 324
Automatically generating commit messages from diffs using neural machine translation
Jiang, S.-Y., Armaly, A., & McMillan, C. (2017) · International Conference on Automated Software Engineering · doi:10.48550/arxiv.1708.09492 · Cited by 300
Cross-Task Generalization via Natural Language Crowdsourcing Instructions
Mishra, S., Khashabi, D., Baral, C., & Hajishirzi, H. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.244 · Cited by 269 · cited 1×
“Publication date: July 2024.”
Cross-Task Generalization via Natural Language Crowdsourcing Instructions
Mishra, S., Khashabi, D., Baral, C., & Hajishirzi, H. (2021) · arXiv · doi:10.48550/arxiv.2104.08773 · Cited by 269
StarCoder: may the source be with you!
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.06161 · Cited by 261
Automatically generating commit messages from diffs using neural machine translation
Jiang, S., Armaly, A., & McMillan, C. (2017) · 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2017.8115626 · Cited by 255 · cited 1×
“To mitigate this threat, we design specific selection criteria for model choice, as demonstrated in Section 2.2.”
InfluentialNeural-machine-translation-based commit message generation: how far are we?
Liu, Z., Xia, X., Hassan, A. E., Lo, D., Xing, Z., & Wang, X. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238190 · Cited by 247 · cited 2×
“We present the approaches we would like to explore to utilize these models (to answer RQ1, RQ2, and RQ3) in Section 2.3.”
InfluentialSelf-citationStudying just-in-time defect prediction using cross-project models
Kamei, Y., Fukushima, T., McIntosh, S., Yamashita, K., Ubayashi, N., & Hassan, A. E. (2016) · Empirical Software Engineering · doi:10.1007/s10664-015-9400-x · Cited by 236 · cited 1×
“To deal with code-change-related tasks, prior studies have proposed a series of machine-learning-based [29, 31, 65] and deep-learning-based approaches [23, 40, 51].”
Automatically documenting program changes
Buse, R. P. L., & Weimer, W. R. (2010) · Proceedings of the IEEE/ACM international conference on Automated software engineering · doi:10.1145/1858996.1859005 · Cited by 219 · cited 1×
“We consider three emerging code-change-related tasks: code review generation [15], commit message generation [6], and just-in-time comment update [51].”
Automating code review activities by large-scale pre-training
Li, Z., Lu, S., Guo, D., Duan, N., Jannu, S., Jenks, G., Majumder, D., Green, J., Svyatkovskiy, A., Fu, S., et al. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549081 · Cited by 196 · cited 3×
“Nevertheless, during our exploration of the ICL process, we find that the selected LMs exhibit poor performance in zero-shot scenarios, indicating that there is no evidence of LMs memorizing the dataset.”
InfluentialAutomatic Prompt Optimization with “Gradient Descent” and Beam Search
Pryzant, R., Iter, D., Li, J., Lee, Y., Zhu, C., & Zeng, M. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.494 · Cited by 191
Automatic Prompt Optimization with "Gradient Descent" and Beam Search
Pryzant, R., Iter, D., Li, J., Lee, Y. T., Zhu, C., & Zeng, M. (2023) · arXiv · doi:10.48550/arxiv.2305.03495 · Cited by 191 · cited 1×
“Following prior studies [32, 51], we also use paired bootstrap resampling with 1000 resamples [32] to perform statistical significance tests for each evaluation metric.”
InCoder: A Generative Model for Code Infilling and Synthesis
Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, W.-T., Zettlemoyer, L., & Lewis, M. (2022) · arXiv · doi:10.48550/arxiv.2204.05999 · Cited by 187 · cited 3×
“In this work, we would like to fill this gap and investigate: How do LLMs perform on To answer this question, we select representative and popular >1B parameters LLMs including InCoder [17], CodeGen [59], Llama 2 [73], and Code Llama [67].”
Peng, B., Li, C., He, P., Galley, M., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2304.03277 · Cited by 183
DeepSeek-Coder: When the Large Language Model Meets Programming - The Rise of Code Intelligence
Guo, D., Zhu, Q., Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y. K., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.14196 · Cited by 180
CC2Vec: distributed representations of code changes
Hoang, T., Kang, H. J., Lo, D., & Lawall, J. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380361 · Cited by 174 · cited 2×
“Call message.set_message_booleans() in update_messages().”
Self-citationRepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation
Zhang, F., Chen, B., Zhang, Y., Keung, J., Liu, J., Zan, D., Mao, Y., Lou, J.-G., & Chen, W. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.151 · Cited by 173
Self-citationOn the Effectiveness of Large Language Models in Domain-Specific Code Generation
Gu, X., Chen, M., Lin, Y., Hu, Y., Zhang, H., Wan, C., Wei, Z., Xu, Y., & Wang, J. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3697012 · Cited by 171 · cited 1×
“Diffs and code snippets are the most common formats to represent code changes.”
Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering
(2020) · doi:10.1145/3324884 · Cited by 163
Self-citationUsing information fragments to answer the questions developers ask
Fritz, T., & Murphy, G. C. (2010) · Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1 · doi:10.1145/1806799.1806828 · Cited by 163 · cited 1×
“For example, in their daily work, developers need to understand existing code changes in code repositories [18], update the comments accompanied with the changed code [22], write commit messages [49], and review the code changed by other developers [33].”
CC2Vec: Distributed Representations of Code Changes
Hoang, T., Kang, H. J., Lawall, J., & Lo, D. (2020) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · doi:10.48550/arxiv.2003.05620 · Cited by 157
Self-citationGeng, M., Wang, S., Dong, D., Wang, H., Li, G., Jin, Z., Mao, X., & Liao, X. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3608134 · Cited by 155 · cited 2×
“In the realm of prompt engineering, In-Context Learning (ICL) is recognized as one of the most typical and effective approaches [20, 34, 56].”
Large Language Models for Test-Free Fault Localization
Yang, A. Z. H., Le Goues, C., Martins, R., & Hellendoorn, V. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623342 · Cited by 144
Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study
Guo, Q., Cao, J., Xie, X., Liu, S., Li, X., Chen, B., & Peng, X. (2023) · arXiv · doi:10.48550/arxiv.2309.08221 · Cited by 140
Large Language Models for Code: Security Hardening and Adversarial Testing
He, J., & Vechev, M. (2023) · Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3576915.3623175 · Cited by 138
A Neural Architecture for Generating Natural Language Descriptions from Source Code Changes
Loyola, P., Marrese-Taylor, E., & Matsuo, Y. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p17-2045 · Cited by 128
CodeRL: Mastering Code Generation Through Pretrained Models and Deep Reinforcement Learning
Le, H., Wang, Y., Gotmare, A. D., Savarese, S., & Hoi, S. C. H. (2022) · neural information processing systems · doi:10.52202/068431-1549 · Cited by 121
WizardCoder: Empowering Code Large Language Models with Evol-Instruct
Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., & Jiang, D. (2023) · arXiv · doi:10.48550/arxiv.2306.08568 · Cited by 120
Few-shot Learning with Multilingual Generative Language Models
Lin, X. V., Mihaylov, T., Artetxe, M., Wang, T., Chen, S., Simig, D., Ott, M., Goyal, N., Bhosale, S., Du, J., et al. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.616 · Cited by 118
Large Language Models Meet NL2Code: A Survey
Zan, D., Chen, B., Zhang, F., Lu, D., Wu, B., Guan, B., Yongji, W., & Lou, J.-G. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.411 · Cited by 114 · cited 1×
“For example, when using LLM with ICL, the best-performing Code Llama outperforms the best-performing Llama 2 and CodeGen across tasks.”
Lu, J., Yu, L., Li, X., Yang, L., & Zuo, C. (2023) · 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre59848.2023.00026 · Cited by 112 · cited 3×
“Researchers pre-trained a large amount of code change data with code-change-oriented objectives [37, 39, 54, 84, 86].”
Chain-of-Thought in Neural Code Generation: From and for Lightweight Language Models
Yang, G., Zhou, Y., Chen, X., Zhang, X., Zhuo, T. Y., & Chen, T. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3440503 · Cited by 112
CoRReCT: code reviewer recommendation in GitHub based on cross-project and technology experience
Rahman, M. M., Roy, C. K., & Collins, J. A. (2016) · Proceedings of the 38th International Conference on Software Engineering Companion · doi:10.1145/2889160.2889244 · Cited by 109 · cited 1×
“To deal with code-change-related tasks, prior studies have proposed a series of machine-learning-based [29, 31, 65] and deep-learning-based approaches [23, 40, 51].”
What Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs?
Gao, S., Wen, X.-C., Gao, C., Wang, W., Zhang, H., & Lyu, M. R. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00109 · Cited by 106 · cited 1×
“Following prior studies [2, 37, 84], we use the base model with 223M parameters.”
2016 IEEE 6th International Conference on Advanced Computing (IACC)
(2016) · doi:10.1109/iacc37532.2016 · Cited by 101
Impact of Peer Code Review on Peer Impression Formation: A Survey
Bosu, A., & Carver, J. C. (2013) · 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2013.23 · Cited by 100 · cited 1×
“These code-change-related tasks are important for project maintenance but can cost significant efforts and slow down the development process [4, 41].”
ATOM: Commit Message Generation Based on Abstract Syntax Tree and Hybrid Ranking
Liu, S., Gao, C., Chen, S., Nie, L. Y., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3038681 · Cited by 93 · cited 1×
“Tasks Related to Code Changes”
Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning
Lialin, V., Deshpande, V., Yao, X., & Rumshisky, A. (2023) · arXiv · doi:10.48550/arxiv.2303.15647 · Cited by 88
CoditT5: Pretraining for Source Code and Natural Language Editing
Zhang, J., Panthaplackel, S., Nie, P., Li, J. J., & Gligoric, M. (2022) · Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3551349.3556955 · Cited by 86 · cited 3×
“Nevertheless, during our exploration of the ICL process, we find that the selected LMs exhibit poor performance in zero-shot scenarios, indicating that there is no evidence of LMs memorizing the dataset.”
InfluentialQilin-Med: Multi-stage Knowledge Injection Advanced Medical Large Language ModelPreprint
Ye, Q., Liu, J., Chong, D., Zhou, P., Hua, Y., Liu, F., Cao, M., Wang, Z., Cheng, X., Zhu, L., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.09089 · Cited by 78
Self-citationHey! are you committing tangled changes?
Kirinuki, H., Higo, Y., Hotta, K., & Kusumoto, S. (2014) · International Conference on Program Comprehension · doi:10.1145/2597008.2597798 · Cited by 74
Few-shot Learning with Multilingual Language Models
Lin, X. V., Mihaylov, T., Artetxe, M., Wang, T., Chen, S., Simig, D., Ott, M., Goyal, N., Bhosale, S., Du, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.10668 · Cited by 74
CCT5: A Code-Change-Oriented Pre-trained Model
Lin, B., Wang, S., Liu, Z., Liu, Y., Xia, X., & Mao, X. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3616339 · Cited by 69 · cited 5×
“Tasks Related to Code Changes”
InfluentialSelf-citationScale Efficiently: Insights from Pre-training and Fine-tuning Transformers
Tay, Y., Dehghani, M., Rao, J., Fedus, W., Abnar, S., Chung, H. W., Narang, S., Yogatama, D., Vaswani, A., & Metzler, D. (2021) · arXiv · doi:10.48550/arxiv.2109.10686 · Cited by 63 · cited 1×
“To make a fair comparison between models, following the prior studies [19, 66, 79, 81, 83] that considered the non-trivial costs of fine-tuning LLMs, we sample 16,000, 2,000 and 2,000 samples from the original dataset (or each sub-dataset in MCMD [70]) as the training, validation, and test sets for…”
FIRA: fi ne-grained g ra ph-based code change representation for automated commit message generation
Dong, J., Lou, Y., Zhu, Q., Sun, Z., Li, Z., Zhang, W., & Hao, D. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510069 · Cited by 61 · cited 1×
“Tasks Related to Code Changes”
SantaCoder: don't reach for the stars!
Allal, L. B., Li, R., Kocetkov, D., Mou, C., Akiki, C., Ferrandis, C. M., Muennighoff, N., Mishra, M., Gu, A., Dey, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.03988 · Cited by 61
Chang, K., Wang, K., Yang, N., Wang, Y., Jin, D., Zhu, W., Chen, Z., Li, C., Yan, H., Zhou, Y., et al. (2024) · Proceedings of the 61st ACM/IEEE Design Automation Conference · doi:10.1145/3649329.3657356 · Cited by 60 · cited 1×
“Publication date: July 2024.”
Revisiting Parameter-Efficient Tuning: Are We Really There Yet?
Chen, G., Liu, F., Meng, Z., & Liang, S. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.168 · Cited by 60 · cited 1×
“We present all the experiment results of LLM-ICLs on CMG in Appendix A.”
Radiology-GPT: A Large Language Model for Radiology
Liu, Z., Zhong, A., Li, Y., Yang, L., Ju, C., Wu, Z., Ma, C., Shu, P., Chen, C., Kim, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.08666 · Cited by 59 · cited 1×
“It is widely used on domain-specific tasks and has proven to be effective [48, 54, 80].”
Self-citationDelving into Parameter-Efficient Fine-Tuning in Code Change Learning: An Empirical Study
Liu, S., Keung, J., Yang, Z., Liu, F., Zhou, Q., & Liao, Y. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00055 · Cited by 58 · cited 2×
“To address this limitation, we take the commit message generation task as an example and conduct a human evaluation.”
Xu, L., Xie, H., Qin, S. J., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.12148 · Cited by 58
CORE: Automating Review Recommendation for Code Changes
Siow, J. K., Gao, C., Fan, L., Chen, S., & Liu, Y. (2020) · 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner48275.2020.9054794 · Cited by 56 · cited 1×
“Tasks Related to Code Changes”
Self-citationAutomating just-in-time comment updating
Liu, Z., Xia, X., Yan, M., & Li, S. (2020) · Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3324884.3416581 · Cited by 56 · cited 4×
“We consider three emerging code-change-related tasks: code review generation [15], commit message generation [6], and just-in-time comment update [51].”
InfluentialSelf-citationUncovering the Causes of Emotions in Software Developer Communication Using Zero-shot LLMs
Imran, M. M., Chatterjee, P., & Damevski, K. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639223 · Cited by 55 · cited 1×
“Finding 14 : LLM-PEFTs can outperform the fully fine-tuned small models on the doc-only code changes and achieve comparable performance to fully fine-tuned small models on other code change types.”
Assess and Summarize: Improve Outage Understanding with Large Language Models
Jin, P., Zhang, S., Ma, M., Li, H., Kang, Y., Li, L., Liu, Y., Qiao, B., Zhang, C., Zhao, P., et al. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3613891 · Cited by 50 · cited 1×
“In Section 3.4, we investigate the impact of input formats on the effectiveness of LLM-ICLs, and find that LLM-ICLs with diffs as the input format outperforms those representing code changes as code snippets in most of the settings, especially on the JITCUP tasks.”
InfluentialDistilled GPT for source code summarization
Su, C.-Y., & McMillan, C. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00421-4 · Cited by 49 · cited 1×
“In Section 3.4, we investigate the impact of input formats on the effectiveness of LLM-ICLs, and find that LLM-ICLs with diffs as the input format outperforms those representing code changes as code snippets in most of the settings, especially on the JITCUP tasks.”
InfluentialPYEVOLVE: Automating Frequent Code Changes in Python ML Systems
Dilhara, M., Dig, D., & Ketkar, A. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00091 · Cited by 46 · cited 1×
“After the launch of a project, developers constantly change code to introduce new features and maintain existing code (e.g., performing refactoring and fixing bugs) [11, 14, 16].”
RACE: Retrieval-augmented Commit Message Generation
Shi, E., Wang, Y., Tao, W., Du, L., Zhang, H., Han, S., Zhang, D., & Sun, H. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.372 · Cited by 42 · cited 1×
“We exclude the variants that are further trained on Python corpus (too specific) or instruction database (different capability) as they are not suitable for our task.”
Reasoning and Planning with Large Language Models in Code Development
Ding, H., Fan, Z., Guehring, I., Gupta, G. P., Ha, W., Huan, J., Liu, L. Q., Omidvar-Tehrani, B., Wang, S., & Zhou, H. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671452 · Cited by 40 · cited 1×
“In Section 3.4, we investigate the impact of input formats on the effectiveness of LLM-ICLs, and find that LLM-ICLs with diffs as the input format outperforms those representing code changes as code snippets in most of the settings, especially on the JITCUP tasks.”
InfluentialCommitBART: A Large Pre-trained Model for GitHub CommitsPreprint
Liu, S., Li, Y., Xie, X., & Liu, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.08100 · Cited by 38 · cited 1×
“Tasks Related to Code Changes”
Kumar, J., & Chimalakonda, S. (2024) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3661167.3661210 · Cited by 37 · cited 1×
“Publication date: July 2024.”
One Model for All Domains: Collaborative Domain-Prefix Tuning for Cross-Domain NER
Chen, X., Li, L., Qiao, S., Zhang, N., Tan, C., Jiang, Y., Huang, F., & Chen, H. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/559 · Cited by 35
Proceedings of the 2026 34th IEEE/ACM International Conference on Program Comprehension
(2026) · doi:10.1145/3794763 · Cited by 34
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
Tay, Y., Dehghani, M., Abnar, S., Chung, H., Fedus, W., Rao, J., Narang, S., Tran, V., Yogatama, D., & Metzler, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.825 · Cited by 34
CCBERT: Self-Supervised Code Change Representation Learning
Zhou, X., Xu, B., Han, D., Yang, Z., He, J., & Lo, D. (2023) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58846.2023.00028 · Cited by 34 · cited 3×
“Nevertheless, during our exploration of the ICL process, we find that the selected LMs exhibit poor performance in zero-shot scenarios, indicating that there is no evidence of LMs memorizing the dataset.”
Self-citationJust-In-Time Obsolete Comment Detection and Update
Liu, Z., Xia, X., Lo, D., Yan, M., & Li, S. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3138909 · Cited by 33 · cited 1×
“Finally, we present the implementation of the work in Section 2.6.”
InfluentialSelf-citationCorrelating Automated and Human Evaluation of Code Documentation Generation Quality
Hu, X., Chen, Q., Wang, H., Xia, X., Lo, D., & Zimmermann, T. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3502853 · Cited by 33 · cited 1×
“Table 16 shows the best-performing models from small pre-trained models, LLM-ICLs and LLM-PEFTs selected to answer this research question.”
Self-citationAutomated Comment Update: How Far are We?
Lin, B., Wang, S., Liu, K., Mao, X., & Bissyandé, T. F. (2021) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc52881.2021.00013 · Cited by 33 · cited 1×
“Following that, we present the design of input in Section 2.4 (to answer RQ4) and the design of analyzing the impact of code changes types in Section 2.5 (to answer RQ5).”
InfluentialA large-scale empirical study of commit message generation: models, datasets and evaluation
Tao, W., Wang, Y., Shi, E., Du, L., Han, S., Zhang, H., Zhang, D., & Zhang, W. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10219-1 · Cited by 33 · cited 6×
“Following prior studies [39, 70], we ask evaluators to rate each generated message from the following three dimensions, with each dimension scored on a scale of 1 to 5 (1: poor, 2: marginal, 3: acceptable, 4: good, 5: excellent): (1) Adequacy : The extent to which the generated commit message…”
InfluentialLarge Language Model-Aware In-Context Learning for Code Generation
Li, J., Li, G., Tao, C., Li, J., Zhang, H., Liu, F., & Jin, Z. (2023) · arXiv · doi:10.48550/arxiv.2310.09748 · Cited by 32
Large Language Model-Aware In-Context Learning for Code Generation
Li, J., Tao, C., Li, J., Li, G., Jin, Z., Zhang, H., Fang, Z., & Liu, F. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715908 · Cited by 32
Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generation
Yuan, Z., Liu, J., Zi, Q., Liu, M., Peng, X., & Lou, Y. (2023) · arXiv · doi:10.48550/arxiv.2308.01240 · Cited by 31 · cited 1×
“We exclude the variants that are further trained on Python corpus (too specific) or instruction database (different capability) as they are not suitable for our task.”
Self-citationParameter-Efficient Finetuning of Transformers for Source CodePreprint
Ayupov, S., & Chirkova, N. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.05901 · Cited by 29
Predictive Comment Updating With Heuristics and AST-Path-Based Neural Learning: A Two-Phase Approach
Lin, B., Wang, S., Liu, Z., Xia, X., & Mao, X. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3185458 · Cited by 27
Self-citationProceedings of the 22nd International Conference on Program Comprehension
(2014) · doi:10.1145/2597008 · Cited by 24
Zhang, M., Yuan, B., Li, H., et al. (2024) · Journal of Artificial Intelligence General science (JAIGS) ISSN 3006-4023 · doi:10.60087/jaigs.v5i1.200 · Cited by 24 · cited 1×
“Diffs and code snippets are the most common formats to represent code changes.”
Classifying software maintenance
Lin, I.-H., & Gustafson, D. A. (1988) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.1988.10169 · Cited by 23 · cited 2×
“These code-change-related tasks are important for project maintenance but can cost significant efforts and slow down the development process [4, 41].”
International Joint Conference on Neural Networks (IJCNN 2005)
(2004) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.836605 · Cited by 23
MFTCoder: Boosting Code LLMs with Multitask Fine-Tuning
Liu, B., Chen, C., Gong, Z., Liao, C., Wang, H., Lei, Z., Liang, M., Chen, D., Shen, M., Zhou, H., et al. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671609 · Cited by 23
Low-Resources Project-Specific Code Summarization
Xie, R., Hu, T., Ye, W., & Zhang, S. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556909 · Cited by 23 · cited 1×
“Publication date: July 2024.”
Zhao, L., Zheng, F., Zeng, W., He, K., Xu, W., Jiang, H., Wu, W., & Wu, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.357 · Cited by 23
Non-Autoregressive Line-Level Code Completion
Liu, F., Fu, Z., Li, G., Jin, Z., Liu, H., Hao, Y., & Zhang, L. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3649594 · Cited by 22 · cited 1×
“Table 16 shows the best-performing models from small pre-trained models, LLM-ICLs and LLM-PEFTs selected to answer this research question.”
Code review for newcomers: is it different?
Kovalenko, V., & Bacchelli, A. (2018) · Proceedings of the 11th International Workshop on Cooperative and Human Aspects of Software Engineering · doi:10.1145/3195836.3195842 · Cited by 18 · cited 1×
“For example, in their daily work, developers need to understand existing code changes in code repositories [18], update the comments accompanied with the changed code [22], write commit messages [49], and review the code changed by other developers [33].”
ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languages
Chai, Y., Wang, S., Pang, C., Sun, Y., Tian, H., & Wu, H. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.676 · Cited by 18
ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languages
Chai, Y., Wang, S., Pang, C., Sun, Y., Tian, H., & Wu, H. (2022) · arXiv · doi:10.48550/arxiv.2212.06742 · Cited by 18
HatCUP: hybrid analysis and attention based just-in-time comment updating
Zhu, H., He, X., & Xu, L. (2022) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3524610.3527901 · Cited by 16 · cited 1×
“Tasks Related to Code Changes”
AceCoder: Utilizing Existing Code to Enhance Code GenerationPreprint
Li, J., Zhao, Y., Li, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.17780 · Cited by 16
ATOM: Commit Message Generation Based on Abstract Syntax Tree and Hybrid RankingPreprint
Liu, S., Gao, C., Chen, S., Nie, L. Y., & Liu, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.02972 · Cited by 14
SparseCoder: Identifier-Aware Sparse Transformer for File- Level Code Summarization
Wang, Y., Huang, Y., Guo, D., Zhang, H., & Zheng, Z. (2024) · 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00068 · Cited by 13 · cited 1×
“Publication date: July 2024.”
Using In-Context Learning to Improve Dialogue Safety
Meade, N., Gella, S., Hazarika, D., Gupta, P., Jin, D., Reddy, S., Liu, Y., & Hakkani-Tur, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.796 · Cited by 13
Tufano, R. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion58688.2023.00053 · Cited by 13 · cited 2×
“Following prior studies [2, 37, 84], we use the base model with 223M parameters.”
How Many Demonstrations Do You Need for In-context Learning?
Chen, J., Chen, L., Zhu, C., & Zhou, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.745 · Cited by 12 · cited 1×
“…} msgs_to_rerender.push(msg); - msg.alerted=event.flags.indexOf("has_alert_word")!==-1; - msg.mentioned=event.flags.indexOf("mentioned")!==-1|| - event.flags.indexOf("wildcard_mentioned")!==-1; + message_store.set_message_booleans(msg,event.flags); condense.un_cache_message_content_height(msg.id);”
Inducer-tuning: Connecting Prefix-tuning and Adapter-tuning
Chen, Y., Hazarika, D., Namazifar, M., Liu, Y., Jin, D., & Hakkani-Tür, D. Z. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.50 · Cited by 12
Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language ModelsPreprint
Weyssow, M., Zhou, X., Kim, K., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.10462 · Cited by 12
IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generators
Paul, I., Glavaš, G., & Gurevych, I. (2024) · arXiv · doi:10.48550/arxiv.2403.03894 · Cited by 12
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
Tay, Y., Dehghani, M., Abnar, S., Chung, H. W., Fedus, W., Rao, J., Narang, S., Tran, V. Q., Yogatama, D., & Metzler, D. (2022) · arXiv · doi:10.48550/arxiv.2207.10551 · Cited by 11 · cited 1×
“To make a fair comparison between models, following the prior studies [19, 66, 79, 81, 83] that considered the non-trivial costs of fine-tuning LLMs, we sample 16,000, 2,000 and 2,000 samples from the original dataset (or each sub-dataset in MCMD [70]) as the training, validation, and test sets for…”
Li, L., Liang, B., Chen, L., & Zhang, X. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107527 · Cited by 9
Conference on Software Maintenance-1987
(1986) · Computer · doi:10.1109/mc.1986.1663325 · Cited by 8
Refactoring operations Grounded in manual code changes
Eilertsen, A. M. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377812.3381395 · Cited by 8 · cited 1×
“After the launch of a project, developers constantly change code to introduce new features and maintain existing code (e.g., performing refactoring and fixing bugs) [11, 14, 16].”
APT: Adaptive Prefix-Tuning on Pretrained Models for Code Intelligence
Wu, Y., Yu, Y., Yuan, Z., Huang, S., & Cai, B. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651243 · Cited by 6 · cited 1×
“Publication date: July 2024.”
Inducer-tuning: Connecting Prefix-tuning and Adapter-tuning
Chen, Y., Hazarika, D., Namazifar, M., Liu, Y., Jin, D., & Hakkani-Tür, D. Z. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2210.14469 · Cited by 5
Deep Just-In-Time Consistent Comment Update via Source Code Changes
Guo, S., Xu, X., Li, H., & Chen, R. (2022) · 2022 IEEE 13th International Symposium on Parallel Architectures, Algorithms and Programming (PAAP) · doi:10.1109/paap56126.2022.10010469 · Cited by 3 · cited 1×
“For example, in their daily work, developers need to understand existing code changes in code repositories [18], update the comments accompanied with the changed code [22], write commit messages [49], and review the code changed by other developers [33].”
Deep Just-In-Time Inconsistency Detection Between Comments and Source Code
Panthaplackel, S., Li, J. J., Gligoric, M., & Mooney, R. J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i1.16119 · Cited by 2 · cited 1×
“To perform our empirical study, we consider three emerging code-change-related tasks that have been popularly researched in recent years, namely, code review generation [37, 69], commit message generation [13, 45], and just-in-time comment update [60, 87].”
Delving into Parameter-Efficient Fine-Tuning in Code Change Learning: An Empirical StudyPreprint
Liu, S., Keung, J., Yang, Z., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.06247 · Cited by 2
Revisiting Parameter-Efficient Tuning: Are We Really There Yet?
Chen, G., Liu, F., Meng, Z., et al. (2022) · arXiv (Cornell University) · doi:10.17863/cam.93587 · Cited by 2
What Makes Good In-context Demonstrations for Code Intelligence Tasks with LLMs?
Gao, S., Wen, X.-C., Gao, C., Wang, W., Zhang, H., & Lyu, M. R. (2023) · arXiv
Research works printed but not linked: 7
- Michael E. Fagan. 2022. Design and code inspections to reduce errors in program development (reprint). In Software Pioneers. Manfred Broy and Ernst Denert (Eds.), Springer, Berlin, 575–607.
- Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations (ICLR ’22), 1–26.
- 10.1145/3324884.3416581 · DOI · Paper page
- Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations (ICLR ’23), 1–25.
- 10.5555/3455716.3455856 · doi:10.5555/3455716.3455856
- Di Wu, Wasi Uddin Ahmad, Dejiao Zhang, Murali Krishna Ramanathan, and Xiaofei Ma. 2024. Repoformer: Selective retrieval for repository-level code completion. In The Forty-first International Conference on Machine Learning (ICML ’24), 1–25.
- Mae Zeid, Khaled El-Bahnasy, and S. E. Abu-Youssef. 2022. An efficient optimized framework for analyzing the performance of breast cancer using machine learning algorithms. Journal of Theoretical and Applied Information Technology 100, 14 (2022), 5165–5178.
Cited by
On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub
Watanabe, M., Li, H., Kashiwa, Y., Reid, B., Iida, H., & Hassan, A. E. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3798166 · Cited by 34
BitsAI-CR: Automated Code Review via LLM in Practice
Sun, T., Xu, J., Li, Y., Yan, Z., Zhang, G., Xie, L., Geng, L., Wang, Z., Chen, Y., Lin, Q., et al. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3728552 · Cited by 57
Yang, X., Zhu, W., Pacheco, M., Zhou, J., Wang, S., Hu, X., & Liu, K. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715738 · Cited by 26
Intention is All you Need: Refining your Code from your Intention
Guo, Q., Xie, X., Liu, S., Hu, M., Li, X., & Bu, L. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00191 · Cited by 21
“Recent works [17], [18], [19], [20], [21], [22], [23] have been exploring the feasibility of using large language models (LLMs) for code refinement.”
Augmenting Large Language Models with Static Code Analysis for Automated Code Quality Improvements
Abtahi, S. M., & Azim, A. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00017 · Cited by 19
“Studies suggest that LLMs can enhance code-change tasks by improving the understanding of code modifications [18].”
Metadata
- Authors
- Fan, L., Liu, J., Liu, Z., Lo, D., Xia, X., & Li, S.
- Affiliations
- Zhejiang University (CN)
- Singapore Management University (SG)
- Venue
- ACM Transactions on Software Engineering and Methodology
- Published
- 2025-07
- Volume
- 34
- Issue
- 6
- Pages
- 1-36
- ISSN
- 1049-331X (ISSN-L)
- 1557-7392
- OpenAlex
- W4405703871
- Semantic Scholar
- b94ed0c8571d20e8403e4d3b95dacbda04f78a41
- Cited by
- 41
- References
- 113
- Citations by year
- 2026: 17 · 2025: 22
- FWCI
- 15.46
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Transactions on Software Engineering and Methodology (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62202420)
- National Research Foundation, under its Investigatorship (NRF-NRFI08-2022-0002)
- National Research Foundation, Singapore
- Zhejiang University Education Foundation Qizhen Scholar Foundation
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- large language models, code amendments, commit message generation, in-context learning, parameter-efficient fine-tuning, LoRa, prefix tuning, Llama 2, CodeLlama, developer productivity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.