CoChat

No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence

Open access
Wang, Chaozheng, Yang, Yuanhang, Gao, Cuiyun, Peng, Yun, Zhang, Hongyu, Lyu, Michael R. · 2022 · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
Cited by 146

The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.

Abstract

Pre-trained models have been shown effective in many code intelligence tasks. These models are pre-trained on large-scale unlabeled corpus and then fine-tuned in downstream tasks. However, as the inputs to pre-training and downstream tasks are in different forms, it is hard to fully explore the knowledge of pre-trained models. Besides, the performance of fine-tuning strongly relies on the amount of downstream data, while in practice, the scenarios with scarce data are common. Recent studies in the natural language processing (NLP) field show that prompt tuning, a new paradigm for tuning, alleviates the above issues and achieves promising results in various NLP tasks. In prompt tuning, the prompts inserted during tuning provide task-specific knowledge, which is especially beneficial for tasks with relatively scarce data. In this paper, we empirically evaluate the usage and effect of prompt tuning in code intelligence tasks. We conduct prompt tuning on popular pre-trained models CodeBERT and CodeT5 and experiment with three code intelligence tasks including defect prediction, code summarization, and code translation. Our experimental results show that prompt tuning consistently outperforms fine-tuning in all three tasks. In addition, prompt tuning shows great potential in low-resource scenarios, e.g., improving the BLEU scores of fine-tuning by more than 26% on average for code summarization. Our results suggest that instead of fine-tuning, we could adapt prompt tuning for code intelligence tasks to achieve better performance, especially when lacking task-specific data.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3540250.3549113

Cites

28 of 28 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 28 references

Research works printed but not linked: 35

  1. 10.1145/3338906.3340458 · DOI · Paper page
  2. Yitian Chai Hongyu Zhang Beijun Shen and Xiaodong Gu. 2022. Cross-Domain Deep Code Search with Few-Shot Meta Learning. arXiv preprint arXiv:2201.00150. Yitian Chai Hongyu Zhang Beijun Shen and Xiaodong Gu. 2022. Cross-Domain Deep Code Search with Few-Shot Meta Learning. arXiv preprint arXiv:2201.00150. · DOI
  3. Ahmed Elnaggar Wei Ding Llion Jones Tom Gibbs Tamas Feher Christoph Angerer Silvia Severini Florian Matthes and Burkhard Rost. 2021. CodeTrans: Towards Cracking the Language of Silicon’s Code Through Self-Supervised Deep Learning and High Performance Computing. arXiv preprint arXiv:2104.02443. Ahmed Elnaggar Wei Ding Llion Jones Tom Gibbs Tamas Feher Christoph Angerer Silvia Severini Florian Matthes and Burkhard Rost. 2021. CodeTrans: Towards Cracking the Language of Silicon’s Code Through Self-Supervised Deep Learning and High Performance Computing. arXiv preprint arXiv:2104.02443. · arXiv
  4. 10.1145/3106237.3106256 · DOI
  5. 10.1016/j.neunet.2021.04.019 · DOI
  6. 10.1145/3180155.3180167 · DOI · Paper page
  7. Yuxian Gu , Xu Han , Zhiyuan Liu , and Minlie Huang . 2021 . Ppt: Pre-trained prompt tuning for few-shot learning. arXiv preprint arXiv:2109.04332. Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. 2021. Ppt: Pre-trained prompt tuning for few-shot learning. arXiv preprint arXiv:2109.04332. · arXiv
  8. Daya Guo , Shuo Ren , Shuai Lu , Zhangyin Feng , Duyu Tang , Shujie Liu , Long Zhou , Nan Duan , Alexey Svyatkovskiy , and Shengyu Fu . 2020 . Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:2009.08366. Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, and Shengyu Fu. 2020. Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:2009.08366. · arXiv
  9. 10.1109/CVPR.2019.00494 · DOI · Paper page
  10. 10.18653/v1/2021.eacl-main.316 · DOI · Paper page
  11. 10.18653/v1/P16-1195 · DOI · Paper page
  12. Aditya Kanade , Petros Maniatis , Gogul Balakrishnan , and Kensen Shi . 2020 . Learning and evaluating contextual embedding of source code . In International Conference on Machine Learning. 5110–5121 . Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020. Learning and evaluating contextual embedding of source code. In International Conference on Machine Learning. 5110–5121.
  13. 10.1109/ICPC.2017.24 · DOI · Paper page
  14. 10.1145/3387904.3389268 · DOI · Paper page
  15. 10.1145/3360588 · DOI · Paper page
  16. Zhen Li , Deqing Zou , Shouhuai Xu , Zhaoxuan Chen , Yawei Zhu , and Hai Jin . 2021 . Vuldeelocator: a deep learning-based fine-grained vulnerability detector . IEEE Transactions on Dependable and Secure Computing. Zhen Li, Deqing Zou, Shouhuai Xu, Zhaoxuan Chen, Yawei Zhu, and Hai Jin. 2021. Vuldeelocator: a deep learning-based fine-grained vulnerability detector. IEEE Transactions on Dependable and Secure Computing.
  17. Pengfei Liu Xipeng Qiu and Xuanjing Huang. 2016. Recurrent neural network for text classification with multi-task learning. arXiv preprint arXiv:1605.05101. Pengfei Liu Xipeng Qiu and Xuanjing Huang. 2016. Recurrent neural network for text classification with multi-task learning. arXiv preprint arXiv:1605.05101. · arXiv
  18. Shangqing Liu , Yu Chen , Xiaofei Xie , Jing Kai Siow , and Yang Liu . 2020 . Retrieval-Augmented Generation for Code Summarization via Hybrid GNN . In International Conference on Learning Representations. Shangqing Liu, Yu Chen, Xiaofei Xie, Jing Kai Siow, and Yang Liu. 2020. Retrieval-Augmented Generation for Code Summarization via Hybrid GNN. In International Conference on Learning Representations.
  19. Shangqing Liu , Cuiyun Gao , Sen Chen , Nie Lun Yiu , and Yang Liu . 2020 . ATOM: Commit message generation based on abstract syntax tree and hybrid ranking . IEEE Transactions on Software Engineering. Shangqing Liu, Cuiyun Gao, Sen Chen, Nie Lun Yiu, and Yang Liu. 2020. ATOM: Commit message generation based on abstract syntax tree and hybrid ranking. IEEE Transactions on Software Engineering.
  20. 10.1109/ICSE43902.2021.00041 · DOI · Paper page
  21. Sewon Min Mike Lewis Hannaneh Hajishirzi and Luke Zettlemoyer. 2021. Noisy channel language model prompting for few-shot text classification. arXiv preprint arXiv:2108.04106. Sewon Min Mike Lewis Hannaneh Hajishirzi and Luke Zettlemoyer. 2021. Noisy channel language model prompting for few-shot text classification. arXiv preprint arXiv:2108.04106. · DOI
  22. 10.1109/TKDE.2009.191 · DOI · Paper page
  23. Fabio Petroni Tim Rocktäschel Patrick Lewis Anton Bakhtin Yuxiang Wu Alexander H Miller and Sebastian Riedel. 2019. Language models as knowledge bases? arXiv preprint arXiv:1909.01066. Fabio Petroni Tim Rocktäschel Patrick Lewis Anton Bakhtin Yuxiang Wu Alexander H Miller and Sebastian Riedel. 2019. Language models as knowledge bases? arXiv preprint arXiv:1909.01066. · DOI · Paper page
  24. Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , and Ilya Sutskever . 2019. Language models are unsupervised multitask learners. OpenAI blog, 1, 8 ( 2019 ), 9. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1, 8 (2019), 9.
  25. Baptiste Roziere , Marie-Anne Lachaux , Lowik Chanussot , and Guillaume Lample . 2020 . Unsupervised translation of programming languages . Advances in Neural Information Processing Systems , 33 (2020), 20601 – 20611 . Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. 2020. Unsupervised translation of programming languages. Advances in Neural Information Processing Systems, 33 (2020), 20601–20611.
  26. Timo Schick and Hinrich Schütze. 2020. It’s not just size that matters: Small language models are also few-shot learners. arXiv preprint arXiv:2009.07118. Timo Schick and Hinrich Schütze. 2020. It’s not just size that matters: Small language models are also few-shot learners. arXiv preprint arXiv:2009.07118. · DOI · Paper page
  27. 10.1609/aaai.v35i11.17155 · DOI
  28. Zhensu Sun Li Li Yan Liu and Xiaoning Du. 2022. On the Importance of Building High-quality Training Datasets for Neural Code Search. arXiv preprint arXiv:2202.06649. Zhensu Sun Li Li Yan Liu and Xiaoning Du. 2022. On the Importance of Building High-quality Training Datasets for Neural Code Search. arXiv preprint arXiv:2202.06649. · arXiv
  29. Maria Tsimpoukelli , Jacob Menick , Serkan Cabi , SM Eslami , Oriol Vinyals , and Felix Hill . 2021. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34 ( 2021 ). Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. 2021. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34 (2021).
  30. 10.1145/3238147.3238206 · DOI · Paper page
  31. Wenhan Wang , Ge Li , Sijie Shen , Xin Xia , and Zhi Jin . 2020 . Modular Tree Network for Source Code Representation Learning . ACM Transactions on Software Engineering and Methodology (TOSEM) , 29 , 4 (2020), 1 – 23 . Wenhan Wang, Ge Li, Sijie Shen, Xin Xia, and Zhi Jin. 2020. Modular Tree Network for Source Code Representation Learning. ACM Transactions on Software Engineering and Methodology (TOSEM), 29, 4 (2020), 1–23. · DOI
  32. Weizhe Yuan , Graham Neubig , and Pengfei Liu . 2021 . Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34 (2021). Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34 (2021).
  33. 10.1109/ICSE.2019.00086 · DOI · Paper page
  34. Ningyu Zhang Luoqiu Li Xiang Chen Shumin Deng Zhen Bi Chuanqi Tan Fei Huang and Huajun Chen. 2021. Differentiable prompt makes pre-trained language models better few-shot learners. arXiv preprint arXiv:2108.13161. Ningyu Zhang Luoqiu Li Xiang Chen Shumin Deng Zhen Bi Chuanqi Tan Fei Huang and Huajun Chen. 2021. Differentiable prompt makes pre-trained language models better few-shot learners. arXiv preprint arXiv:2108.13161. · arXiv
  35. Yaqin Zhou , Shangqing Liu , Jingkai Siow , Xiaoning Du , and Yang Liu . 2019 . Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. Advances in neural information processing systems, 32 (2019). Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. 2019. Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. Advances in neural information processing systems, 32 (2019).

Cited by

See all 146

Metadata

Authors
Wang, C., Yang, Y., Gao, C., Peng, Y., Zhang, H., & Lyu, M. R.
Venue
Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
Publisher
ACM
Published
2022-11-07
Pages
382-394
OpenAlex
W4288055447
Cited by
146
References
28
FWCI
29.56
Type
conference paper
Language
en
Open access
green
Funders
  • National Natural Science Foundation of China (62002084)
  • Stable support plan for colleges and universities in Shenzhen (GXWD2020 1230155427003-20200730101839009)
  • Major Key Project of PCL (PCL2022A03, PCL2021A02, PCL2021A09)
  • Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies (2022B1212010005)
  • Research Grants Council of the Hong Kong Special Administrative Region, China (CUHK 14210920)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence. https://app.cochat.ai/papers/10.1145/3540250.3549113
The paper
Wang, C., Yang, Y., Gao, C., Peng, Y., Zhang, H., & Lyu, M. R. (2022). No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence. Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. https://doi.org/10.1145/3540250.3549113

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.