No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Pre-trained models have been shown effective in many code intelligence tasks. These models are pre-trained on large-scale unlabeled corpus and then fine-tuned in downstream tasks. However, as the inputs to pre-training and downstream tasks are in different forms, it is hard to fully explore the knowledge of pre-trained models. Besides, the performance of fine-tuning strongly relies on the amount of downstream data, while in practice, the scenarios with scarce data are common. Recent studies in the natural language processing (NLP) field show that prompt tuning, a new paradigm for tuning, alleviates the above issues and achieves promising results in various NLP tasks. In prompt tuning, the prompts inserted during tuning provide task-specific knowledge, which is especially beneficial for tasks with relatively scarce data. In this paper, we empirically evaluate the usage and effect of prompt tuning in code intelligence tasks. We conduct prompt tuning on popular pre-trained models CodeBERT and CodeT5 and experiment with three code intelligence tasks including defect prediction, code summarization, and code translation. Our experimental results show that prompt tuning consistently outperforms fine-tuning in all three tasks. In addition, prompt tuning shows great potential in low-resource scenarios, e.g., improving the BLEU scores of fine-tuning by more than 26% on average for code summarization. Our results suggest that instead of fine-tuning, we could adapt prompt tuning for code intelligence tasks to achieve better performance, especially when lacking task-specific data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 1×
“Hard prompt is also known as discrete prompt since each token in the prompts is meaningful and understandable [13, 30].”
BackgroundThe Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824
Research works printed but not linked: 35
- 10.1145/3338906.3340458 · DOI · Paper page
- Yitian Chai Hongyu Zhang Beijun Shen and Xiaodong Gu. 2022. Cross-Domain Deep Code Search with Few-Shot Meta Learning. arXiv preprint arXiv:2201.00150. Yitian Chai Hongyu Zhang Beijun Shen and Xiaodong Gu. 2022. Cross-Domain Deep Code Search with Few-Shot Meta Learning. arXiv preprint arXiv:2201.00150. · DOI
- Ahmed Elnaggar Wei Ding Llion Jones Tom Gibbs Tamas Feher Christoph Angerer Silvia Severini Florian Matthes and Burkhard Rost. 2021. CodeTrans: Towards Cracking the Language of Silicon’s Code Through Self-Supervised Deep Learning and High Performance Computing. arXiv preprint arXiv:2104.02443. Ahmed Elnaggar Wei Ding Llion Jones Tom Gibbs Tamas Feher Christoph Angerer Silvia Severini Florian Matthes and Burkhard Rost. 2021. CodeTrans: Towards Cracking the Language of Silicon’s Code Through Self-Supervised Deep Learning and High Performance Computing. arXiv preprint arXiv:2104.02443. · arXiv
- 10.1145/3106237.3106256 · DOI
- 10.1016/j.neunet.2021.04.019 · DOI
- 10.1145/3180155.3180167 · DOI · Paper page
- Yuxian Gu , Xu Han , Zhiyuan Liu , and Minlie Huang . 2021 . Ppt: Pre-trained prompt tuning for few-shot learning. arXiv preprint arXiv:2109.04332. Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. 2021. Ppt: Pre-trained prompt tuning for few-shot learning. arXiv preprint arXiv:2109.04332. · arXiv
- Daya Guo , Shuo Ren , Shuai Lu , Zhangyin Feng , Duyu Tang , Shujie Liu , Long Zhou , Nan Duan , Alexey Svyatkovskiy , and Shengyu Fu . 2020 . Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:2009.08366. Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, and Shengyu Fu. 2020. Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:2009.08366. · arXiv
- 10.1109/CVPR.2019.00494 · DOI · Paper page
- 10.18653/v1/2021.eacl-main.316 · DOI · Paper page
- 10.18653/v1/P16-1195 · DOI · Paper page
- Aditya Kanade , Petros Maniatis , Gogul Balakrishnan , and Kensen Shi . 2020 . Learning and evaluating contextual embedding of source code . In International Conference on Machine Learning. 5110–5121 . Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020. Learning and evaluating contextual embedding of source code. In International Conference on Machine Learning. 5110–5121.
- 10.1109/ICPC.2017.24 · DOI · Paper page
- 10.1145/3387904.3389268 · DOI · Paper page
- 10.1145/3360588 · DOI · Paper page
- Zhen Li , Deqing Zou , Shouhuai Xu , Zhaoxuan Chen , Yawei Zhu , and Hai Jin . 2021 . Vuldeelocator: a deep learning-based fine-grained vulnerability detector . IEEE Transactions on Dependable and Secure Computing. Zhen Li, Deqing Zou, Shouhuai Xu, Zhaoxuan Chen, Yawei Zhu, and Hai Jin. 2021. Vuldeelocator: a deep learning-based fine-grained vulnerability detector. IEEE Transactions on Dependable and Secure Computing.
- Pengfei Liu Xipeng Qiu and Xuanjing Huang. 2016. Recurrent neural network for text classification with multi-task learning. arXiv preprint arXiv:1605.05101. Pengfei Liu Xipeng Qiu and Xuanjing Huang. 2016. Recurrent neural network for text classification with multi-task learning. arXiv preprint arXiv:1605.05101. · arXiv
- Shangqing Liu , Yu Chen , Xiaofei Xie , Jing Kai Siow , and Yang Liu . 2020 . Retrieval-Augmented Generation for Code Summarization via Hybrid GNN . In International Conference on Learning Representations. Shangqing Liu, Yu Chen, Xiaofei Xie, Jing Kai Siow, and Yang Liu. 2020. Retrieval-Augmented Generation for Code Summarization via Hybrid GNN. In International Conference on Learning Representations.
- Shangqing Liu , Cuiyun Gao , Sen Chen , Nie Lun Yiu , and Yang Liu . 2020 . ATOM: Commit message generation based on abstract syntax tree and hybrid ranking . IEEE Transactions on Software Engineering. Shangqing Liu, Cuiyun Gao, Sen Chen, Nie Lun Yiu, and Yang Liu. 2020. ATOM: Commit message generation based on abstract syntax tree and hybrid ranking. IEEE Transactions on Software Engineering.
- 10.1109/ICSE43902.2021.00041 · DOI · Paper page
- Sewon Min Mike Lewis Hannaneh Hajishirzi and Luke Zettlemoyer. 2021. Noisy channel language model prompting for few-shot text classification. arXiv preprint arXiv:2108.04106. Sewon Min Mike Lewis Hannaneh Hajishirzi and Luke Zettlemoyer. 2021. Noisy channel language model prompting for few-shot text classification. arXiv preprint arXiv:2108.04106. · DOI
- 10.1109/TKDE.2009.191 · DOI · Paper page
- Fabio Petroni Tim Rocktäschel Patrick Lewis Anton Bakhtin Yuxiang Wu Alexander H Miller and Sebastian Riedel. 2019. Language models as knowledge bases? arXiv preprint arXiv:1909.01066. Fabio Petroni Tim Rocktäschel Patrick Lewis Anton Bakhtin Yuxiang Wu Alexander H Miller and Sebastian Riedel. 2019. Language models as knowledge bases? arXiv preprint arXiv:1909.01066. · DOI · Paper page
- Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , and Ilya Sutskever . 2019. Language models are unsupervised multitask learners. OpenAI blog, 1, 8 ( 2019 ), 9. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1, 8 (2019), 9.
- Baptiste Roziere , Marie-Anne Lachaux , Lowik Chanussot , and Guillaume Lample . 2020 . Unsupervised translation of programming languages . Advances in Neural Information Processing Systems , 33 (2020), 20601 – 20611 . Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. 2020. Unsupervised translation of programming languages. Advances in Neural Information Processing Systems, 33 (2020), 20601–20611.
- Timo Schick and Hinrich Schütze. 2020. It’s not just size that matters: Small language models are also few-shot learners. arXiv preprint arXiv:2009.07118. Timo Schick and Hinrich Schütze. 2020. It’s not just size that matters: Small language models are also few-shot learners. arXiv preprint arXiv:2009.07118. · DOI · Paper page
- 10.1609/aaai.v35i11.17155 · DOI
- Zhensu Sun Li Li Yan Liu and Xiaoning Du. 2022. On the Importance of Building High-quality Training Datasets for Neural Code Search. arXiv preprint arXiv:2202.06649. Zhensu Sun Li Li Yan Liu and Xiaoning Du. 2022. On the Importance of Building High-quality Training Datasets for Neural Code Search. arXiv preprint arXiv:2202.06649. · arXiv
- Maria Tsimpoukelli , Jacob Menick , Serkan Cabi , SM Eslami , Oriol Vinyals , and Felix Hill . 2021. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34 ( 2021 ). Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. 2021. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34 (2021).
- 10.1145/3238147.3238206 · DOI · Paper page
- Wenhan Wang , Ge Li , Sijie Shen , Xin Xia , and Zhi Jin . 2020 . Modular Tree Network for Source Code Representation Learning . ACM Transactions on Software Engineering and Methodology (TOSEM) , 29 , 4 (2020), 1 – 23 . Wenhan Wang, Ge Li, Sijie Shen, Xin Xia, and Zhi Jin. 2020. Modular Tree Network for Source Code Representation Learning. ACM Transactions on Software Engineering and Methodology (TOSEM), 29, 4 (2020), 1–23. · DOI
- Weizhe Yuan , Graham Neubig , and Pengfei Liu . 2021 . Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34 (2021). Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34 (2021).
- 10.1109/ICSE.2019.00086 · DOI · Paper page
- Ningyu Zhang Luoqiu Li Xiang Chen Shumin Deng Zhen Bi Chuanqi Tan Fei Huang and Huajun Chen. 2021. Differentiable prompt makes pre-trained language models better few-shot learners. arXiv preprint arXiv:2108.13161. Ningyu Zhang Luoqiu Li Xiang Chen Shumin Deng Zhen Bi Chuanqi Tan Fei Huang and Huajun Chen. 2021. Differentiable prompt makes pre-trained language models better few-shot learners. arXiv preprint arXiv:2108.13161. · arXiv
- Yaqin Zhou , Shangqing Liu , Jingkai Siow , Xiaoning Du , and Yang Liu . 2019 . Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. Advances in neural information processing systems, 32 (2019). Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. 2019. Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. Advances in neural information processing systems, 32 (2019).
Cited by
Fuzz4ALL: Universal Fuzzing with Large Language Models
Xia, C. S., Paltenghi, M., Le Tian, J., Pradel, M., & Zhang, L. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639121 · Cited by 229
Log Parsing with Prompt-based Few-shot Learning
Le, V.-H., & Zhang, H. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00204 · Cited by 129
Self-citationWhat Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs?
Gao, S., Wen, X.-C., Gao, C., Wang, W., Zhang, H., & Lyu, M. R. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00109 · Cited by 106
Self-citationCCT5: A Code-Change-Oriented Pre-trained Model
Lin, B., Wang, S., Liu, Z., Liu, Y., Xia, X., & Mao, X. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3616339 · Cited by 69
“Such inconsistency makes the knowledge learned by pre-trained models hard to be fully exploited, easily leading to sub-optimal results for down-stream tasks [40, 67].”
BackgroundRepository-Level Prompt Generation for Large Language Models of Code
Shrivastava, D., Larochelle, H., & Tarlow, D. (2022) · arXiv · doi:10.48550/arxiv.2206.12839 · Cited by 27
Metadata
- Authors
- Wang, C., Yang, Y., Gao, C., Peng, Y., Zhang, H., & Lyu, M. R.
- Venue
- Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2022-11-07
- Pages
- 382-394
- OpenAlex
- W4288055447
- Cited by
- 146
- References
- 28
- FWCI
- 29.56
- Type
- conference paper
- Language
- en
- Open access
- green
- Funders
- National Natural Science Foundation of China (62002084)
- Stable support plan for colleges and universities in Shenzhen (GXWD2020 1230155427003-20200730101839009)
- Major Key Project of PCL (PCL2022A03, PCL2021A02, PCL2021A09)
- Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies (2022B1212010005)
- Research Grants Council of the Hong Kong Special Administrative Region, China (CUHK 14210920)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence. https://app.cochat.ai/papers/10.1145/3540250.3549113
- The paper
- Wang, C., Yang, Y., Gao, C., Peng, Y., Zhang, H., & Lyu, M. R. (2022). No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence. Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. https://doi.org/10.1145/3540250.3549113
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.