CoChat
Sign Up
Cited by 103Software Engineering ResearchTopic ModelingInformation Retrieval and Search Behavior

Self-Supervised Contrastive Learning for Code Retrieval and Summarization via Semantic-Preserving Transformations

Open access
Bui, Nghi D. Q., Yu, Yijun, Jiang, Lingxiao·2021·Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval
DOI 10.1145/3404835.3462840
Open this paper in CoChat — free accountOpen PDF

Abstract

We propose Corder, a self-supervised contrastive learning framework for source code model. Corder is designed to alleviate the need of labeled data for code retrieval and code summarization tasks. The pre-trained model of Corder can be used in two ways: (1) it can produce vector representation of code which can be applied to code retrieval tasks that do not have labeled data; (2) it can be used in a fine-tuning process for tasks that might still require label data such as code summarization. The key innovation is that we train the source code model by asking it to recognize similar and dissimilar code snippets through acontrastive learning objective. To do so, we use a set of semantic-preserving transformation operators to generate code snippets that are syntactically diverse but semantically equivalent. Through extensive experiments, we have shown that the code models pretrained by Corder substantially outperform the other baselines for code-to-code retrieval, text-to-code retrieval, and code-to-text summarization tasks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3404835.3462840

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 62

  1. [2][2] Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2020. A transformer-based approach for source code summarization. arXiv preprint arXiv:2005.00653 (2020). - · arXiv
  2. [3][3] Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. 2018. Learning to Represent Programs with Graphs. In ICLR . -
  3. [4][4] Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2019. code2seq: Generating Sequences from Structured Representations of Code. In ICLR . -
  4. [5][5] Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. 2020. Structural language models of code. In International Conference on Machine Learning . PMLR, 245-256. -
  5. [6][6] Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. 2019. Code2Vec: Learning Distributed Representations of Code. In POPL . 40:1-40:29. -
  6. [7][7] Antonio Valerio Miceli Barone and Rico Sennrich. 2017. A Parallel Corpus of Python Functions and Documentation Strings for Automated Code Documentation and Code Generation. In Proceedings of the Eighth International Joint Conference on Natural Language Processing, IJCNLP 2017, Taipei, Taiwan, November 27 - December 1, 2017, Volume 2: Short Papers , Greg Kondrak and Taro Watanabe (Eds.). Asian Federation of Natural Language Processing, 314-319. https://www.aclweb.org/anthology/I17-2053/ -
  7. [8][8] Marc Brockschmidt, Miltiadis Allamanis, Alexander L. Gaunt, and Oleksandr Polozov. 2019. Generative Code Modeling with Graphs. In 7th ICLR . -
  8. [10][10] Nghi DQ Bui, Lingxiao Jiang, and Yijun Yu. 2017. Cross-language learning for program classification using bilateral tree-based convolutional neural networks. arXiv preprint arXiv:1710.06159 (2017). - · arXiv
  9. [11][11] Nghi DQ Bui, Yijun Yu, and Lingxiao Jiang. 2019. Bilateral dependency neural networks for cross-language algorithm classification. In 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER) . IEEE, 422-433. -
  10. [12][12] Nghi DQ Bui, Yijun Yu, and Lingxiao Jiang. 2019. SAR: learning cross-language API mappings with little knowledge. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering . 796-806. -
  11. [13][13] Nghi DQ Bui, Yijun Yu, and Lingxiao Jiang. 2020. InferCode: Self-Supervised Learning of Code Representations by Predicting Subtrees. arXiv e-prints (2020), arXiv-2012. -
  12. [14][14] Nghi DQ Bui, Yijun Yu, and Lingxiao Jiang. 2020. TreeCaps: Tree-Based Capsule Networks for Source Code Processing. arXiv preprint arXiv:2009.09777 (2020). - · arXiv
  13. [15][15] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 (2020). - · arXiv
  14. [16][16] Xinyun Chen, Chang Liu, and Dawn Song. 2018. Tree-to-tree neural networks for program translation. In NeurIPS . 2547-2557. -
  15. [17][17] Zimin Chen and Martin Monperrus. 2019. A literature study of embeddings on source code. arXiv preprint arXiv:1904.03061 (2019). - · arXiv
  16. [18][18] Xiao Cheng, Haoyu Wang, Jiayi Hua, Guoai Xu, and Yulei Sui. 2021. DeepWukong: Statically detecting software vulnerabilities using deep graph neural network. ACMTransactions on Software Engineering and Methodology (TOSEM) 30, 3 (2021), 1-33. -
  17. [20][20] Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364 (2017). - · arXiv
  18. [21][21] George E Dahl, Jack W Stokes, Li Deng, and Dong Yu. 2013. Large-scale malware classification using random projections and neural networks. In IEEE International Conference on Acoustics, Speech and Signal Processing . 3422-3426. -
  19. [23][23] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 (2020). - · arXiv
  20. [24][24] Patrick Fernandes, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Structured Neural Summarization. In 7th ICLR . -
  21. [26][26] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. 2018. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07728 (2018). - · arXiv
  22. [27][27] Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. 2018. Deep code search. In 40th ICSE . 933-944. -
  23. [28][28] Xiaodong Gu, Hongyu Zhang, Dongmei Zhang, and Sunghun Kim. 2017. DeepAM: Migrate APIs with Multi-modal Sequence to Sequence Learning. In IJCAI (Melbourne, Australia). 3675-3681. -
  24. [29][29] Rahul Gupta, Aditya Kanade, and Shirish Shevade. 2019. Neural Attribution for Semantic Bug-Localization in Student Programs. In NeurIPS . 11861-11871. -
  25. [31][31] Geoffrey E Hinton, Simon Osindero, and Yee-Whye Teh. 2006. A fast learning algorithm for deep belief nets. Neural computation 18, 7 (2006), 1527-1554. -
  26. [33][33] Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436 (2019). - · arXiv
  27. [34][34] Yasir Hussain, Zhiqiu Huang, Yu Zhou, and Senzhang Wang. 2020. Deep transfer learning for source code modeling. International Journal of Software Engineering and Knowledge Engineering 30, 05 (2020), 649-668. -
  28. [36][36] Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing source code using a neural attention model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 2073-2083. -
  29. [37][37] Darryl Jarman, Jeffrey Berry, Riley Smith, Ferdian Thung, and David Lo. 2021. Legion: Massively Composing Rankers for Improved Bug Localization at Adobe. IEEE Transactions on Software Engineering (2021). -
  30. [40][40] Dahun Kim, Donghyeon Cho, and In So Kweon. 2019. Self-supervised video representation learning with space-time cubic puzzles. In AAAI , Vol. 33. 85458552. -
  31. [42][42] Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In NeurIPS . 32943302. -
  32. [43][43] Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. arXiv preprint arXiv:1701.02810 (2017). - · arXiv
  33. [44][44] Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the ACL on interactive poster and demonstration sessions . Association for Computational Linguistics, 177-180. -
  34. [45][45] Lingpeng Kong, Cyprien de Masson d'Autume, Wang Ling, Lei Yu, Zihang Dai, and Dani Yogatama. 2019. A mutual information maximization perspective of language representation learning. arXiv preprint arXiv:1910.08350 (2019). - · arXiv
  35. [46][46] Bruno Korbar, Du Tran, and Lorenzo Torresani. 2018. Cooperative learning of audio and video models from self-supervised synchronization. In NeurIPS . 7763-7774. -
  36. [48][48] Jian Li, Pinjia He, Jieming Zhu, and Michael R Lyu. 2017. Software defect prediction via convolutional neural network. In IEEE QRS . 318-328. -
  37. [49][49] Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. 2016. Gated Graph Sequence Neural Networks. In ICLR . -
  38. [50][50] Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. arXiv preprint arXiv:1803.02893 (2018). - · arXiv
  39. [51][51] Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research 9, Nov (2008), 2579-2605. -
  40. [52][52] Aravindh Mahendran, James Thewlis, and Andrea Vedaldi. 2018. Cross pixel optical-flow similarity for self-supervised learning. In Asian Conference on Computer Vision . 99-116. -
  41. [54][54] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In NeurIPS . 3111-3119. -
  42. [55][55] Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. 2016. Convolutional neural networks over tree structures for programming language processing. In AAAI . -
  43. [56][56] R. Nix and J. Zhang. 2017. Classification of Android apps and malware using deep neural networks. In International Joint Conference on Neural Networks . 1871-1878. -
  44. [57][57] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018). - · arXiv
  45. [58][58] Michael Pradel and Koushik Sen. 2018. DeepBugs: A learning approach to namebased bug detection. ACM on Programming Languages 2, OOPSLA (2018), 147. -
  46. [59][59] Md Rabin, Rafiqul Islam, Nghi DQ Bui, Yijun Yu, Lingxiao Jiang, and Mohammad Amin Alipour. 2020. On the Generalizability of Neural Program Analyzers with respect to Semantic-Preserving Program Transformations. arXiv preprint arXiv:2008.01566 (2020). - · arXiv
  47. [60][60] Saksham Sachdev, Hongyu Li, Sifei Luan, Seohyun Kim, Koushik Sen, and Satish Chandra. 2018. Retrieval on Source Code: A Neural Code Search. In 2nd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages (Philadelphia, PA, USA). 31-41. https://doi.org/10.1145/3211346.3211353 - · DOI
  48. [62][62] Tian Shi, Liuqing Li, Ping Wang, and Chandan K Reddy. 2020. A Simple and Effective Self-Supervised Contrastive Learning Framework for Aspect Detection. arXiv preprint arXiv:2009.09107 (2020). - · arXiv
  49. [63][63] Yulei Sui, Xiao Cheng, Guanqin Zhang, and Haoyu Wang. 2020. Flow2Vec: valueflow-based precise code embedding. Proceedings of the ACM on Programming Languages 4, OOPSLA (2020), 1-27. -
  50. [64][64] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems . 31043112. -
  51. [67][67] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems . 5998-6008. -
  52. [68][68] Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, Jian Wu, and Philip S. Yu. 2018. Improving Automatic Source Code Summarization via Deep Reinforcement Learning. In 33rd ASE (Montpellier, France). New York, NY, USA, 397-407. https://doi.org/10.1145/3238147.3238206 - · DOI
  53. [69][69] Hong Wang, Xin Wang, Wenhan Xiong, Mo Yu, Xiaoxiao Guo, Shiyu Chang, and William Yang Wang. 2019. Self-supervised learning for contextualized extractive summarization. arXiv preprint arXiv:1906.04466 (2019). - · arXiv
  54. [70][70] Haoye Wang, Xin Xia, David Lo, John Grundy, and Xinyu Wang. 2021. Automatic Solution Summarization for Crash Bugs. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . IEEE, 1286-1297. -
  55. [71][71] Jiawei Wu, Xin Wang, and William Yang Wang. 2019. Self-supervised dialogue learning. arXiv preprint arXiv:1907.00448 (2019). - · arXiv
  56. [72][72] Xinli Yang, David Lo, Xin Xia, Yun Zhang, and Jianling Sun. 2015. Deep Learning for Just-in-Time Defect Prediction. In IEEE QRS . 17-26. -
  57. [73][73] Michihiro Yasunaga and Percy Liang. 2020. Graph-based, Self-Supervised Program Repair from Diagnostic Feedback. arXiv preprint arXiv:2005.10636 (2020). - · arXiv
  58. [74][74] Huangzhao Zhang, Zhuo Li, Ge Li, Lei Ma, Yang Liu, and Zhi Jin. 2020. Generating Adversarial Examples for Holding Robustness of Source Code Processing Models. In 34th AAAI . -
  59. [75][75] Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Kaixuan Wang, and Xudong Liu. 2019. A novel neural source code representation based on abstract syntax tree. In 41st ICSE . 783-794. -
  60. [76][76] Richard Zhang, Phillip Isola, and Alexei A Efros. 2016. Colorful image colorization. In European conference on computer vision . 649-666. -
  61. [77][77] Han Zhao, Zhengdong Lu, and Pascal Poupart. 2015. Self-adaptive hierarchical sentence model. arXiv preprint arXiv:1504.05070 (2015). - · arXiv
  62. [78][78] Yaqin Zhou, Shangqing Liu, Jing Kai Siow, Xiaoning Du, and Yang Liu. 2019. Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks. In NeurIPS . 10197-10207.

Cited by

Sorted by citations

Authors
Bui, N. D. Q., Yu, Y., & Jiang, L.
Affiliations
  • Huawei Technologies (China) (CN)
  • Singapore Management University (SG)
Venue
Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval
Publisher
ACM
Published
2021-07-11
Pages
511-521
DOI
10.1145/3404835.3462840
OpenAlex
W3155146092
Cited by
103
References
6
Citations by year
2026: 8 · 2025: 21 · 2024: 23 · 2023: 30 · 2022: 21
FWCI
15.34
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
self-supervised contrastive learning, code search, code summarization, semantics-preserving transformations, code embeddings

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review