CoChat
Sign Up
PreprintCited by 129Software Engineering ResearchTopic ModelingTeaching and Learning Programming

Contrastive Code Representation Learning

Open access
Jain, Paras, Jain, Ajay, Zhang, Tianjun, Abbeel, Pieter, Gonzalez, Joseph E., Stoica, Ion·2020·arXiv
DOI 10.48550/arxiv.2007.04973arXiv 2007.04973
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent work learns contextual representations of source code by reconstructing tokens from their context. For downstream semantic understanding tasks like summarizing code in English, these representations should ideally capture program functionality. However, we show that the popular reconstruction-based BERT model is sensitive to source code edits, even when the edits preserve semantics. We propose ContraCode: a contrastive pre-training task that learns code functionality, not form. ContraCode pre-trains a neural network to identify functionally similar variants of a program among many non-equivalent distractors. We scalably generate these variants using an automated source-to-source compiler as a form of data augmentation. Contrastive pre-training improves JavaScript summarization and TypeScript type inference accuracy by 2% to 13%. We also propose a new zero-shot JavaScript code clone detection dataset, showing that ContraCode is both more robust and semantically meaningful. On it, we outperform RoBERTa by 39% AUROC in an adversarial setting and up to 5% on natural code.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2007.04973

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 55

  1. Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2020. A transformer- based approach for source code summarization. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics (ACL).
  2. Miltiadis Allamanis. 2019. The adverse effects of code duplication in machine learning models of code. In Proceedings of the 2019 ACM SIGPLAN Interna- tional Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, On- ward! 2019, page 143-153, New York, NY, USA. Association for Computing Machinery. · DOI · Paper page
  3. Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and Charles Sutton. 2018. A survey of machine learning for big code and naturalness. ACM Com- puting Surveys (CSUR), 51(4):81.
  4. Miltiadis Allamanis, Earl T. Barr, Soline Ducousso, and Zheng Gao. 2020. Typilus: Neural type hints. In Programming Language Design and Implemen- tation (PLDI).
  5. Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016. A convolutional attention network for ex- treme summarization of source code. In Interna- tional Conference on Machine Learning (ICML).
  6. Uri Alon, Meital Zilberstein, Omer Levy, and Eran Ya- hav. 2019b. code2vec: Learning distributed repre- sentations of code. Proceedings of the ACM on Pro- gramming Languages.
  7. Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. 2019. A theoretical analysis of contrastive unsuper- vised representation learning. In Proceedings of the 36th International Conference on Machine Learn- ing, volume 97 of Proceedings of Machine Learning Research, pages 5628-5637. PMLR.
  8. Sorav Bansal and Alex Aiken. 2006. Automatic gener- ation of peephole superoptimizers. In Proceedings of the 12th International Conference on Architec- tural Support for Programming Languages and Op- erating Systems, ASPLOS XII, page 394-403, New York, NY, USA. Association for Computing Ma- chinery. · DOI
  9. Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hoefler. 2018. Neural code comprehension: A learnable representation of code semantics. In NeurIPS.
  10. Samuel Benton, Ali Ghanbari, and Lingming Zhang. 2019. Defexts: A curated dataset of reproducible real-world bugs for modern jvm languages. In 2019 IEEE/ACM 41st International Conference on Soft- ware Engineering: Companion Proceedings (ICSE- Companion), pages 47-50. IEEE.
  11. Pavol Bielik and Martin Vechev. 2020. Adversarial ro- bustness for code. CoRR.
  12. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas- try, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cum- mings, Matthias Plappert, Fotios Chantzis, Eliza- beth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, An- drew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating large language models trained on code. CoRR, abs/2107.03374. · arXiv
  13. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020a. A simple framework for contrastive learning of visual representations. In International Conference on Macnine Learning.
  14. Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaim- ing He. 2020b. Improved baselines with mo- mentum contrastive learning. arXiv preprint arXiv:2003.04297. · arXiv
  15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Annual Conference of the North Amer- ican Chapter of the Association for Computational Linguistics.
  16. Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. 2010. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research, 11(Feb):625-660.
  17. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xi- aocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. CodeBERT: A pre-trained model for programming and natural lan- guages. arXiv preprint arXiv:2002.08155. · arXiv
  18. Rudolf Ferenc, Zoltán Tóth, Gergely Ladányi, István Siket, and Tibor Gyimóthy. 2018. A public unified bug dataset for Java. In Proceedings of the 14th International Conference on Predictive Models and Data Analytics in Software Engineering, pages 12- 21.
  19. James Fogarty, Ryan S. Baker, and Scott E. Hudson. 2005. Case studies in the use of roc curve analysis for sensor-based estimates in human computer inter- action. In Proceedings of Graphics Interface 2005, GI '05, page 129-136, Waterloo, CAN. Canadian Human-Computer Communications Society.
  20. Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Confer- ence on Computer Vision and Pattern Recognition (CVPR'06), volume 2, pages 1735-1742. IEEE.
  21. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2019. Momentum contrast for un- supervised visual representation learning. arXiv preprint arXiv:1911.05722. · arXiv
  22. Vincent J Hellendoorn, Christian Bird, Earl T Barr, and Miltiadis Allamanis. 2018. Deep learning type in- ference. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Confer- ence and Symposium on the Foundations of Software Engineering, pages 152-162.
  23. Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, S. M. Ali Eslami, and Aaron van den Oord. 2019. Data-efficient image recognition with contrastive predictive coding. In International Conference on Machine Learning.
  24. Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. 2019. Using self-supervised learn- ing can improve model robustness and uncertainty. Advances in Neural Information Processing Systems (NeurIPS).
  25. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirec- tional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991. · arXiv
  26. Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Code- SearchNet challenge: Evaluating the state of seman- tic code search. arXiv preprint arXiv:1909.09436. · arXiv
  27. Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. 2019. Adversarial examples are not bugs, they are features. In Advances in Neural Informa- tion Processing Systems, volume 32, pages 125-136.
  28. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing source code using a neural attention model. In Proceedings of the 54th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 2073-2083.
  29. Rajeev Joshi, Greg Nelson, and Keith Randall. 2002. Denali: A goal-directed superoptimizer. In Pro- ceedings of the ACM SIGPLAN 2002 Conference on Programming Language Design and Implemen- tation, PLDI '02, page 304-314, New York, NY, USA. Association for Computing Machinery. · DOI
  30. Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020. Pre-trained contextual em- bedding of source code. ArXiv, abs/2001.00059. · arXiv
  31. Miryung Kim, Thomas Zimmermann, and Nachiap- pan Nagappan. 2012. A field study of refactor- ing challenges and benefits. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, pages 1- 11.
  32. James Koppel, Varot Premtoon, and Armando Solar- Lezama. 2018. One tool, many languages: Language-parametric transformation with incre- mental parametric syntax. Proc. ACM Program. Lang., 2(OOPSLA). · DOI
  33. Taku Kudo. 2018. Subword regularization: Improv- ing neural network translation models with multiple subword candidates. In Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66-75.
  34. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv:1907.11692. · arXiv
  35. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579-2605.
  36. Henry Massalin. 1987. Superoptimizer: A look at the smallest program. In Proceedings of the Second In- ternational Conference on Architectual Support for Programming Languages and Operating Systems, ASPLOS II, page 122-126, Washington, DC, USA. IEEE Computer Society Press. · DOI · Paper page
  37. Charith Mendis, Alex Renda, Saman Amarasinghe, and Michael Carbin. 2019. Ithemal: Accurate, portable and fast basic block throughput estimation using deep neural networks. In International Conference on Machine Learning, pages 4505-4515.
  38. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their compositional- ity. In Advances in neural information processing systems, pages 3111-3119.
  39. Dana Movshovitz-Attias and William Cohen. 2013. Natural language models for predicting program- ming comments. In Proceedings of the 51st Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 35-40.
  40. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive pre- dictive coding. arXiv preprint arXiv:1807.03748. · arXiv
  41. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Pas- sos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830.
  42. Michael Pradel, Georgios Gousios, Jason Liu, and Satish Chandra. 2019. Typewriter: Neural type pre- diction with search-based validation. arXiv preprint arXiv:1912.03768. · arXiv
  43. Michael Pradel and Koushik Sen. 2018. Deepbugs: A learning approach to name-based bug detection. Proceedings of the ACM on Programming Lan- guages, 2(OOPSLA):1-25.
  44. Mike Schuster and Kuldip Paliwal. 1997. Bidirectional recurrent neural networks. Signal Processing, IEEE Transactions on, 45:2673 -2681. · DOI · Paper page
  45. Roei Schuster, Congzheng Song, Eran Tromer, and Vi- taly Shmatikov. 2021. You autocomplete me: Poi- soning vulnerabilities in neural code completion. In 30th {USENIX} Security Symposium ({USENIX} Security 21).
  46. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. · arXiv
  47. Jeffrey Svajlenko, Judith F. Islam, Iman Keivanloo, Chanchal K. Roy, and Mohammad Mamun Mia. 2014. Towards a big data curated benchmark of inter-project code clones. In Proceedings of the 2014 IEEE International Conference on Soft- ware Maintenance and Evolution, ICSME '14, page 476-480, USA. IEEE Computer Society. · DOI · Paper page
  48. Wilson L Taylor. 1953. "Cloze procedure": A new tool for measuring readability. Journalism Quar- terly, 30(4):415-433.
  49. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, pages 5998-6008.
  50. Ke Wang and Mihai Christodorescu. 2019. COSET: A benchmark for evaluating neural program embed- dings. arXiv preprint arXiv:1905.11445. · arXiv
  51. Ke Wang and Zhendong Su. 2019. Learning blended, precise semantic program embeddings. ArXiv.
  52. Martin White, Michele Tufano, Christopher Vendome, and Denys Poshyvanyk. 2016. Deep learning code fragments for code clone detection. In 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 87-98. IEEE.
  53. Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. 2018. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3733-3742.
  54. Noam Yefet, Uri Alon, and Eran Yahav. 2019. Adver- sarial examples for models of code. arXiv preprint arXiv:1910.07517. · arXiv
  55. Noam Yefet, Uri Alon, and Eran Yahav. 2020. Ad- versarial examples for models of code. Pro- ceedings of the ACM on Programming Languages, 4(OOPSLA):1-30.

Cited by

Sorted by citations

Authors
Jain, P., Jain, A., Zhang, T., Abbeel, P., Gonzalez, J. E., & Stoica, I.
Affiliations
University of California, Berkeley (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Pages
5954-5971
DOI
10.48550/arxiv.2007.04973
arXiv
2007.04973
OpenAlex
W3040955294
Cited by
129
References
0
Citations by year
2026: 6 · 2025: 24 · 2024: 25 · 2023: 34 · 2022: 26 · 2021: 12 · 2020: 2
FWCI
20.11
Type
conference paper
Language
en
Open access
Licence
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (1730628, 1752814)
  • Nvidia
  • Amazon Web Services
  • VMware
  • Futurewei Technologies
  • Ant Financial Services Group
  • Directorate for Computer and Information Science and Engineering
  • Office of Naval Research
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
Keywords
contrastive learning, code representation learning, code clone detection, adversarial robustness, source-to-source compiler, data augmentation, code summarization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review