CoChat

CoditT5: Pretraining for Source Code and Natural Language Editing

Open access
Zhang, Jiyang, Panthaplackel, Sheena, Nie, Pengyu, Li, Junyi Jessy, Gligoric, Milos · 2022 · Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering
Cited by 86

The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.

Abstract

Pretrained language models have been shown to be effective in many software-related generation tasks; however, they are not well-suited for editing tasks as they are not designed to reason about edits. To address this, we propose a novel pretraining objective which explicitly models edits and use it to build CoditT5, a large language model for software-related editing tasks that is pretrained on large amounts of source code and natural language comments. We fine-tune it on various downstream editing tasks, including comment updating, bug fixing, and automated code review. By outperforming standard generation-based models, we demonstrate the generalizability of our approach and its suitability for editing tasks. We also show how a standard generation model and our edit-based model can complement one another through simple reranking strategies, with which we achieve state-of-the-art performance for the three downstream editing tasks.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3551349.3556955

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 12 references

Research works printed but not linked: 51

  1. Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Unified Pre-training for Program Understanding and Generation. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2655–2668.
  2. Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations.
  3. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. 65–72.
  4. Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. 2012. An Empirical Investigation of Statistical Significance in NLP. In Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. 995–1005.
  5. Avrim Blum and Tom Mitchell. 1998. Combining labeled and unlabeled data with co-training. In Computational Learning Theory. 92–100. · DOI · Paper page
  6. Shaked Brody, Uri Alon, and Eran Yahav. 2020. A structural model for contextual code changes. International Conference on Object-Oriented Programming, Systems, Languages, and Applications 4(2020), 1–28.
  7. Tom Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared D Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems. 1877–1901.
  8. Saikat Chakraborty and Baishakhi Ray. 2021. On Multi-Modal Learning of Editing Source Code. In Automated Software Engineering. 443–455. · DOI · Paper page
  9. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374(2021). · arXiv
  10. Zimin Chen, Vincent J Hellendoorn, Pascal Lamblin, Petros Maniatis, Pierre-Antoine Manzagol, Daniel Tarlow, and Subhodeep Moitra. 2021. PLUR: A unifying, graph-based view of program learning, understanding, and repair. In Advances in Neural Information Processing Systems. 23089–23101.
  11. Colin Clement Dawn Drain Jonathan Timcheck Alexey Svyatkovskiy and Neel Sundaresan. 2020. PyMT5: multi-mode translation of natural language and Python code with transformers. In Empirical Methods in Natural Language Processing. 9052–9065. · DOI · Paper page
  12. Yangruibo Ding Baishakhi Ray Premkumar Devanbu and Vincent J Hellendoorn. 2020. Patching as Translation: the Data and the Metaphor. In Automated Software Engineering. 275–286. · DOI
  13. 10.1145/3460945.3464951 · DOI · Paper page
  14. Angela Fan, Mike Lewis, and Yann Dauphin. 2019. Strategies for Structuring Story Generation. In Annual Meeting of the Association for Computational Linguistics. 2650–2660.
  15. 10.1145/3468264.3468553 · DOI
  16. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor OK Li. 2016. Incorporating Copying Mechanism in Sequence-to-Sequence Learning. In Annual Meeting of the Association for Computational Linguistics. 1631–1640.
  17. Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. UniXcoder: Unified Cross-Modal Pre-training for Code Representation. In Annual Meeting of the Association for Computational Linguistics. 7212–7225.
  18. Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, LIU Shujie, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, 2020. GraphCodeBERT: Pre-training Code Representations with Data Flow. In International Conference on Learning Representations.
  19. Tatsunori B Hashimoto Kelvin Guu Yonatan Oren and Percy S Liang. 2018. A retrieve-and-edit framework for predicting structured outputs. In Advances in Neural Information Processing Systems.
  20. Sepp Hochreiter, and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural computation 9 (1997), 1735–1780.
  21. Robert Iv, Alexandre Passos, Sameer Singh, and Ming-Wei Chang. 2022. FRUIT: Faithfully Reflecting Updated Information in Text. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 3670–3686.
  22. Reno Kriz João Sedoc Marianna Apidianaki Carolina Zheng Gaurav Kumar Eleni Miltsakaki and Chris Callison-Burch. 2019. Complexity-Weighted Loss and Diverse Reranking for Sentence Simplification. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 3137–3147. · DOI · Paper page
  23. Alexander LeClair, Aakash Bansal, and Collin McMillan. 2021. Ensemble Models for Neural Source Code Summarization of Subroutines. In International Conference on Software Maintenance and Evolution. 286–297.
  24. Jia Li, Yongmin Li, Ge Li, Xing Hu, Xin Xia, and Zhi Jin. 2021. Editsum: A retrieve-and-edit framework for source code summarization. In Automated Software Engineering. 155–166.
  25. Zhiyu Li Shuai Lu Daya Guo Nan Duan Shailesh Jannu Grant Jenks Deep Majumder Jared Green Alexey Svyatkovskiy Shengyu Fu 2022. CodeReviewer: Pre-Training for Automating Code Review Activities. arXiv preprint arXiv:2203.09095(2022). · arXiv
  26. 10.1109/ICPC52881.2021.00013 · DOI
  27. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692(2019). · arXiv
  28. 10.1109/TSE.2021.3138909 · DOI · Paper page
  29. 10.1609/aaai.v32i1.11430 · DOI · Paper page
  30. 10.1109/ICSE43902.2021.00041 · DOI · Paper page
  31. Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. Ground truth for grammatical error correction metrics. In Annual Meeting of the Association for Computational Linguistics and International Joint Conference on Natural Language Processing. 588–593.
  32. Graham Neubig, Makoto Morishita, and Satoshi Nakamura. 2015. Neural Reranking Improves Subjective Quality of Machine Translation: NAIST at WAT2015. In Workshop on Asian Translation. 35–41.
  33. 10.1145/3510003.3510096 · DOI · Paper page
  34. 10.1609/aaai.v35i15.17606 · DOI
  35. 10.1609/aaai.v35i1.16119 · DOI · Paper page
  36. 10.18653/v1/2020.acl-main.168 · DOI · Paper page
  37. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics. 311–318.
  38. 10.1609/aaai.v30i1.10347 · DOI · Paper page
  39. 10.1017/S1351324997001502 · DOI · Paper page
  40. 10.1145/3387940.3392181 · DOI · Paper page
  41. 10.1109/ICSE.2019.00021 · DOI · Paper page
  42. 10.1145/3510003.3510621 · DOI · Paper page
  43. Rosalia Tufano, Luca Pascarella, Michele Tufanoy, Denys Poshyvanykz, and Gabriele Bavota. 2021. Towards Automating Code Review Activities. In International Conference on Software Engineering. 163–174.
  44. Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Łukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. 5998–6008.
  45. Oriol Vinyals Meire Fortunato and Navdeep Jaitly. 2015. Pointer networks. Advances in Neural Information Processing Systems.
  46. Xin Wang Yasheng Wang Fei Mi Pingyi Zhou Yao Wan Xiao Liu Li Li Hao Wu Jin Liu and Xin Jiang. 2021. SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representation. arXiv preprint arXiv:2108.04556(2021). · arXiv
  47. 10.1162/tacl_a_00107 · DOI · Paper page
  48. Ziyu Yao, Frank F. Xu, Pengcheng Yin, Huan Sun, and Graham Neubig. 2021. Learning Structural Edits via Incremental Tree Transformations. In International Conference on Learning Representations.
  49. Pengcheng Yin, Graham Neubig, Miltiadis Allamanis, Marc Brockschmidt, and Alexander L Gaunt. 2018. Learning to Represent Edits. In International Conference on Learning Representations.
  50. 10.1609/aaai.v32i1.11915 · DOI
  51. 10.1109/TKDE.2005.186 · DOI · Paper page

Cited by

Metadata

Authors
Zhang, J., Panthaplackel, S., Nie, P., Li, J. J., & Gligoric, M.
Venue
Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering
Publisher
ACM
Published
2022-10-10
Pages
1-12
OpenAlex
W4313547604
Cited by
86
References
12
FWCI
17.45
Type
conference paper
Language
en
Open access
bronze
Funders
NSF (National Science Foundation) (CCF-1652517, CCF-2107291, IIS-2107524, IIS-2145479)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.