CoChat
Sign Up
Cited by 141Software Engineering ResearchTopic ModelingInformation Retrieval and Search Behavior

Retrieval Augmented Code Generation and Summarization

Open access
Parvez, Md Rizwan, Ahmad, Wasi, Chakraborty, Saikat, Ray, Baishakhi, Chang, Kai-Wei·2021·Findings of the Association for Computational Linguistics: EMNLP 2021
DOI 10.18653/v1/2021.findings-emnlp.232
Open this paper in CoChat — free accountOpen PDF

Abstract

Software developers write a lot of source code and documentation during software development.Intrinsically, developers often recall parts of source code or code summaries that they had written in the past while implementing software or documenting them.To mimic developers' code or summary generation behavior, we propose a retrieval augmented framework, REDCODER, that retrieves relevant code or summaries from a retrieval database and provides them as a supplement to code generation or summarization models.REDCODER has a couple of uniqueness.First, it extends the state-of-the-art dense retrieval technique to search for relevant code or summaries.Second, it can work with retrieval databases that include unimodal (only code or natural language description) or bimodal instances (code-description pairs).We conduct experiments and extensive analysis on two benchmark datasets of code generation and summarization in Java and Python, and the promising results endorse the effectiveness of our proposed retrieval augmented framework.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2021.findings-emnlp.232

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Joint modeling of retrieval and generation has been extended to code generation by incorporating retrieved documents or code examples into the generation process.

    “This approach has also been extended to code generation by incorporating retrieved documents or code examples into the generation process (Rizwan Parvez et al., 2021;Zhou et al., 2022;Lu et al., 2022;Zan et al., 2022).”

    From RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation (2023) · cites this paper as Rizwan Parvez et al., 2021

    The Introduction and Section 2.1 describe REDCODER retrieving relevant code and adding it to the input used for code generation. But the Introduction calls it a two-step process, and the Code Retrieval and Others section contrasts it with a method that optimizes a joint objective. The source supports retrieval-augmented code generation, not joint optimization of retrieval and generation. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 52

  1. Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2020. A transformer-based ap- proach for source code summarization. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4998-5007, Online. Association for Computational Linguistics. · DOI
  2. Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Unified pre-training for program understanding and generation. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics.
  3. Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and Charles Sutton. 2018. A survey of machine learning for big code and naturalness. ACM Com- puting Surveys (CSUR), 51(4):1-37.
  4. Miltiadis Allamanis, Hao Peng, and Charles A. Sut- ton. 2016. A convolutional attention network for extreme summarization of source code. In Pro- ceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 2091- 2100. JMLR.org.
  5. Achmad Arwan, Siti Rochimah, and Rizky Januar Akbar. 2015. Source code retrieval on stackover- flow using lda. In 2015 3rd International Confer- ence on Information and Communication Technol- ogy (ICoICT), pages 295-299. IEEE.
  6. Joel Brandt, Mira Dontcheva, Marcos Weskamp, and Scott R Klemmer. 2010. Example-centric program- ming: integrating web search into the development environment. In Proceedings of the SIGCHI Con- ference on Human Factors in Computing Systems, pages 513-522.
  7. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre- training text encoders as discriminators rather than generators. In International Conference on Learn- ing Representations.
  8. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020a. CodeBERT: A pre-trained model for programming and natural languages. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2020, pages 1536-1547, Online. Association for Compu- tational Linguistics.
  9. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020b. CodeBERT: A pre-trained model for programming and natural languages. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2020, pages 1536-1547, Online. Association for Compu- tational Linguistics. · DOI · Paper page
  10. Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ie, and Diego Garcia-Olano. 2019. Learning dense repre- sentations for entity retrieval. In Proceedings of the 23rd Conference on Computational Natural Lan- guage Learning (CoNLL), pages 528-537, Hong Kong, China. Association for Computational Lin- guistics. · DOI
  11. Xiaodong Gu, Hongyu Zhang, Dongmei Zhang, and Sunghun Kim. 2016. Deep api learning. In Proceed- ings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 631-642.
  12. Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Jian Yin, Daxin Jiang, et al. 2021. Graphcodebert: Pre- training code representations with data flow. In International Conference on Learning Representa- tions.
  13. Daya Guo, Duyu Tang, Nan Duan, Ming Zhou, and Jian Yin. 2019. Coupling retrieval and meta- learning for context-dependent semantic parsing. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 855- 866, Florence, Italy. Association for Computational Linguistics. · DOI
  14. Ruiqi Guo, Sanjiv Kumar, Krzysztof Choromanski, and David Simcha. 2016. Quantization based fast inner product search. In Artificial Intelligence and Statis- tics, pages 482-490.
  15. PMLR.
  16. Jacob Harer, Chris Reale, and Peter Chin. 2019. Tree- transformer: A transformer-based method for cor- rection of tree-structured data. arXiv preprint arXiv:1908.00449. · arXiv
  17. Tatsunori B Hashimoto, Kelvin Guu, Yonatan Oren, and Percy S Liang. 2018. A retrieve-and-edit frame- work for predicting structured outputs. In Ad- vances in Neural Information Processing Systems, volume 31.
  18. Shirley Anugrah Hayati, Raphael Olivier, Pravalika Av- varu, Pengcheng Yin, Anthony Tomasic, and Gra- ham Neubig. 2018. Retrieval-based neural code gen- eration. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 925-930, Brussels, Belgium. Association for Computational Linguistics. · DOI
  19. Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2018a. Deep code comment generation. In Proceedings of the 26th Conference on Program Comprehension, page 200-210, New York, NY, USA. Association for Computing Machinery. · DOI
  20. Xing Hu, Ge Li, Xin Xia, David Lo, Shuai Lu, and Zhi Jin. 2018b. Summarizing source code with trans- ferred api knowledge. In Proceedings of the Twenty- Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 2269-2275. Interna- tional Joint Conferences on Artificial Intelligence Organization. · DOI
  21. Michael Hucka and Matthew J Graham. 2018. Soft- ware search is not a science, even among scientists: A survey of how scientists and engineers find soft- ware. Journal of Systems and Software, 141:171- 191.
  22. Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020. Poly-encoders: Architec- tures and pre-training strategies for fast and accurate multi-sentence scoring. In International Conference on Learning Representations.
  23. Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Code- searchnet challenge: Evaluating the state of seman- tic code search. arXiv preprint arXiv:1909.09436. · arXiv
  24. Srinivasan Iyer, Alvin Cheung, and Luke Zettlemoyer. 2019. Learning programmatic idioms for scalable semantic parsing. In Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP- IJCNLP), pages 5426-5435, Hong Kong, China. As- sociation for Computational Linguistics. · DOI
  25. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing source code using a neural attention model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2073-2083, Berlin, Germany. Association for Computational Linguistics. · DOI
  26. Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the As- sociation for Computational Linguistics: Main Vol- ume, pages 874-880, Online. Association for Com- putational Linguistics.
  27. Alexander LeClair, Siyuan Jiang, and Collin McMil- lan. 2019. A neural model for generating natural language summaries of program subroutines. In Proceedings of the 41st International Conference on Software Engineering, page 795-806. IEEE Press. · DOI
  28. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge- intensive nlp tasks. In Advances in Neural Infor- mation Processing Systems, volume 33, pages 9459- 9474.
  29. Hongwei Li, Zhenchang Xing, Xin Peng, and Wenyun Zhao. 2013. What help do developers seek, when and how? In 2013 20th working conference on re- verse engineering (WCRE), pages 142-151. IEEE.
  30. Yuding Liang and Kenny Qili Zhu. 2018. Automatic generation of text descriptive comments for code blocks. In Thirty-Second AAAI Conference on Ar- tificial Intelligence, pages 5229-5236.
  31. Chin-Yew Lin and Franz Josef Och. 2004. ORANGE: a method for evaluating automatic evaluation met- rics for machine translation. In COLING 2004: Pro- ceedings of the 20th International Conference on Computational Linguistics, pages 501-507, Geneva, Switzerland. COLING.
  32. Shangqing Liu, Yu Chen, Xiaofei Xie, Jing Kai Siow, and Yang Liu. 2021. Retrieval-augmented genera- tion for code summarization via hybrid {gnn}. In International Conference on Learning Representa- tions.
  33. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664. · arXiv
  34. CD Manning, P Raghavan, and H Schütze. 2008. Xml retrieval. In Introduction to Information Retrieval. Cambridze University Press.
  35. Seyed Mehdi Nasehi, Jonathan Sillito, Frank Maurer, and Chris Burns. 2012. What makes a good code example?: A study of programming q&a in stack- overflow. In 2012 28th IEEE International Confer- ence on Software Maintenance (ICSM), pages 25-34. IEEE.
  36. Md Rizwan Parvez, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2018. Building language mod- els for text with named entities. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2373-2383, Melbourne, Australia. Associa- tion for Computational Linguistics. · DOI
  37. Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017. Abstract syntax networks for code generation and semantic parsing. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1139- 1149, Vancouver, Canada. Association for Computa- tional Linguistics. · DOI
  38. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  39. Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for auto- matic evaluation of code synthesis. arXiv preprint arXiv:2009.10297. · arXiv
  40. Caitlin Sadowski, Kathryn T Stolee, and Sebastian El- baum. 2015. How developers search for code: a case study. In Proceedings of the 2015 10th joint meeting on foundations of software engineering, pages 191- 201.
  41. Yusuke Shido, Yasuaki Kobayashi, Akihiro Yamamoto, Atsushi Miyamoto, and Tadayuki Matsumura. 2019. Automatic source code summarization with ex- tended tree-lstm. In International Joint Conference on Neural Networks, IJCNN 2019 Budapest, Hun- gary, July 14-19, 2019, pages 1-8. IEEE. · DOI
  42. Susan Elliott Sim, Medha Umarji, Sukanya Ratano- tayanon, and Cristina V Lopes. 2011. How well do search engines support code retrieval on the web? ACM Transactions on Software Engineering and Methodology (TOSEM), 21(1):1-25.
  43. Kathryn T Stolee, Sebastian Elbaum, and Daniel Do- bos. 2014. Solving the search for source code. ACM Transactions on Software Engineering and Method- ology (TOSEM), 23(3):1-45.
  44. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Sys- tems, volume 27, pages 3104-3112.
  45. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30, pages 5998-6008.
  46. Bolin Wei, Ge Li, Xin Xia, Zhiyi Fu, and Zhi Jin. 2019. Code generation as a dual task of code summariza- tion. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Ad- vances in Neural Information Processing Systems 32, pages 6563-6573.
  47. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-art natural language process- ing. In Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Asso- ciation for Computational Linguistics. · DOI · Paper page
  48. Xin Xia, Lingfeng Bao, David Lo, Pavneet Singh Kochhar, Ahmed E Hassan, and Zhenchang Xing. 2017. What do developers search for on the web? Empirical Software Engineering, 22(6):3149-3185.
  49. Frank F. Xu, Zhengbao Jiang, Pengcheng Yin, Bogdan Vasilescu, and Graham Neubig. 2020. Incorporating external knowledge through pre-training for natural language to code generation. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 6045-6052, Online. As- sociation for Computational Linguistics. · DOI
  50. Wen-tau Yih, Kristina Toutanova, John C Platt, and Christopher Meek. 2011. Learning discriminative projections for text similarity measures. In Proceed- ings of the fifteenth conference on computational nat- ural language learning, pages 247-256.
  51. Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for general-purpose code generation. In Proceedings of the 55th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 440-450, Vancouver, Canada. Association for Computational Linguistics. · DOI · Paper page
  52. Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, and Xudong Liu. 2020. Retrieval-based neural source code summarization. In 2020 IEEE/ACM 42nd International Conference on Software Engi- neering (ICSE), pages 1385-1397. IEEE.

Cited by

Sorted by citations

Authors
Parvez, M. R., Ahmad, W., Chakraborty, S., Ray, B., & Chang, K.-W.
Affiliations
  • University of California, Los Angeles (US)
  • Columbia University (US)
ORCID
Venue
Findings of the Association for Computational Linguistics: EMNLP 2021
Publisher
Association for Computational Linguistics
Published
2021
Pages
2719-2734
DOI
10.18653/v1/2021.findings-emnlp.232
OpenAlex
W3194195934
Cited by
141
References
9
Citations by year
2026: 29 · 2025: 52 · 2024: 32 · 2023: 24 · 2022: 4
FWCI
12.44
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (2040961, 1920462, 1845893, 2107405)
  • VMware
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
retrieval-augmented generation, code generation, code summarization, Java, Python

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review