CoChat
Sign Up
PreprintCited by 12Software Engineering ResearchTopic ModelingTeaching and Learning Programming

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generators

Open access
Paul, Indraneil, Glavaš, Goran, Gurevych, Iryna·2024·arXiv
DOI 10.48550/arxiv.2403.03894arXiv 2403.03894
Open this paper in CoChat — free accountOpen PDF

Abstract

Code understanding and generation have fast become some of the most popular applications of language models (LMs). Nonetheless, research on multilingual aspects of Code-LMs (i.e., LMs for code generation) such as cross-lingual transfer between different programming languages, language-specific data augmentation, and post-hoc LM adaptation, alongside exploitation of data sources other than the original textual content, has been much sparser than for their natural language counterparts. In particular, most mainstream Code-LMs have been pre-trained on source code files alone. In this work, we investigate the prospect of leveraging readily available compiler intermediate representations (IR) - shared across programming languages - to improve the multilingual capabilities of Code-LMs and facilitate cross-lingual transfer. To this end, we first compile SLTrans, a parallel dataset consisting of nearly 4M self-contained source code files coupled with respective intermediate representations. Next, starting from various base Code-LMs (ranging in size from 1.1B to 7.3B parameters), we carry out continued causal language modelling training on SLTrans, forcing the Code-LMs to (1) learn the IR language and (2) align the IR constructs with respective constructs of various programming languages. Our resulting models, dubbed IRCoder, display sizeable and consistent gains across a wide variety of code generation tasks and metrics, including prompt robustness, multilingual code completion, code understanding, and instruction following.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2403.03894

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

  • Measuring The Impact Of Programming Language Distribution

    Orlanski, G., Xiao, K., Garcia, X., Hui, J., Howland, J., Malmaud, J., Austin, J., Singh, R., & Catasta, M. (2023) · arXiv · doi:10.48550/arxiv.2302.01973 · Cited by 21 · cited 3×

    “Our findings are in contrast with the findings of Orlanski et al. (2023) who show a trade-off be-tween the performance on high and low-resource languages: we, instead, observe gains across the board with no evidence of interference even be-tween typologically disparate programming languages.”
    BackgroundResult

Flagged first, then by citations

Research works printed but not linked: 82

  1. Toufique Ahmed and Premkumar Devanbu. 2022. Mul- tilingual training for software engineering. In Pro- ceedings of the 44th International Conference on Software Engineering, pages 1443-1455. · DOI · Paper page
  2. Miltiadis Allamanis. 2019. The adverse effects of code duplication in machine learning models of code. In Proceedings of the 2019 ACM SIGPLAN Interna- tional Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, pages 143-153. · DOI · Paper page
  3. Alan Ansell, Edoardo Ponti, Anna Korhonen, and Ivan Vulić. 2022. Composable sparse fine-tuning for cross- lingual transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1778-1796, Dublin, Ireland. Association for Computational Lin- guistics. · DOI
  4. Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. 2019. Massively multilingual neural machine translation in the wild: Findings and chal- lenges. arXiv preprint 1907.05019. · arXiv
  5. Mikel Artetxe and Holger Schwenk. 2019. Mas- sively multilingual sentence embeddings for zero- shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597-610. · DOI
  6. Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Su- jan Kumar Gonugondla, Hantian Ding, Varun Ku- mar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ramanathan, and Ramesh Nallapati. 2023. Multi- lingual evaluation of code generation models. In The Eleventh International Conference on Learning Rep- resentations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
  7. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint 2108.07732. · arXiv
  8. Mehdi Bahrami, NC Shrikanth, Yuji Mizobuchi, Lei Liu, Masahiro Fukuyori, Wei-Peng Chen, and Kazuki Munakata. 2021. Augmentedcode: Examining the effects of natural language resources in code retrieval models. arXiv preprint 2110.08512. · arXiv
  9. Terra Blevins, Tomasz Limisiewicz, Suchin Gururan- gan, Margaret Li, Hila Gonen, Noah A Smith, and Luke Zettlemoyer. 2024. Breaking the curse of multi- linguality with cross-lingual expert language models. arXiv preprint 2401.10440. · arXiv
  10. Alexander Brauckmann, Andrés Goens, Sebastian Er- tel, and Jeronimo Castrillon. 2020. Compiler-based graph representations for deep learning models of code. In Proceedings of the 29th International Con- ference on Compiler Construction, pages 201-211. · DOI
  11. Marc Brockschmidt, Miltiadis Allamanis, Alexander L. Gaunt, and Oleksandr Polozov. 2019. Generative code modeling with graphs. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
  12. Andrei Z Broder. 1997. On the resemblance and con- tainment of documents. In Proceedings. Compres- sion and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171), pages 21-29. IEEE.
  13. Federico Cassano, John Gouwar, Daniel Nguyen, Syd- ney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. 2022. Multipl-e: A scal- able and extensible approach to benchmarking neural code generation. arXiv preprint 2208.08227. · arXiv
  14. Federico Cassano, Luisa Li, Akul Sethi, Noah Shinn, Abby Brennan-Jones, Anton Lozhkov, Carolyn An- derson, and Arjun Guha. 2023b. Can it edit? evalu- ating the ability of large language models to follow code editing instructions. arXiv preprint 2312.12450. · arXiv
  15. Yekun Chai, Shuohuan Wang, Chao Pang, Yu Sun, Hao Tian, and Hua Wu. 2023. ERNIE-code: Beyond English-centric cross-lingual pretraining for program- ming languages. In Findings of the Association for Computational Linguistics: ACL 2023, pages 10628- 10650, Toronto, Canada. Association for Computa- tional Linguistics. · DOI · Paper page
  16. Saikat Chakraborty, Toufique Ahmed, Yangruibo Ding, Premkumar T Devanbu, and Baishakhi Ray. 2022. Natgen: generative pre-training by "naturalizing" source code. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Sym- posium on the Foundations of Software Engineering, pages 18-30. · DOI · Paper page
  17. Fuxiang Chen, Fatemeh H. Fard, David Lo, and Timofey Bryksin. 2022. On the transferability of pre-trained language models for low-resource programming lan- guages. In Proceedings of the 30th IEEE/ACM In- ternational Conference on Program Comprehension, ICPC '22, page 401-412, New York, NY, USA. As- sociation for Computing Machinery. · DOI
  18. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint 2107.03374. · arXiv
  19. Nuo Chen, Qiushi Sun, Jianing Wang, Xiang Li, and Ming Gao. 2023a. Pass-tuning: Towards structure- aware parameter-efficient tuning for code represen- tation learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 577-591, Singapore. Association for Computational Linguistics. · DOI
  20. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023b. Teaching large language mod- els to self-debug. arXiv preprint 2304.05128. · arXiv
  21. Hyung Won Chung, Xavier Garcia, Adam Roberts, Yi Tay, Orhan Firat, Sharan Narang, and Noah Con- stant. 2023. Unimax: Fairer and more effective language sampling for large-scale multilingual pre- training. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
  22. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics. · DOI · Paper page
  23. Kaustubh D Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abi- naya Mahendiran, Simon Mille, Ashish Shrivastava, Samson Tan, et al. 2021. Nl-augmenter: A frame- work for task-sensitive natural language augmenta- tion. arXiv preprint 2112.02721. · arXiv
  24. Tuan Dinh, Jinman Zhao, Samson Tan, Renato Ne- grinho, Leonard Lausen, Sheng Zha, and George Karypis. 2023. Large language models of code fail at completing code with potential bugs. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Sys- tems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023.
  25. Omer Dunay, Daniel Cheng, Adam Tait, Parth Thakkar, Peter C Rigby, Andy Chiu, Imad Ahmad, Arun Gane- san, Chandra Maddila, Vijayaraghavan Murali, et al. 2024. Multi-line ai-assisted code authoring. arXiv preprint 2402.04141. · arXiv
  26. Aleksandra Eliseeva, Yaroslav Sokolov, Egor Bogo- molov, Yaroslav Golubev, Danny Dig, and Timofey Bryksin. 2023. From commit message generation to history-aware commit message completion. In 38th IEEE/ACM International Conference on Auto- mated Software Engineering, ASE 2023, Luxembourg, September 11-15, 2023, pages 723-735. IEEE. · DOI
  27. Alexander Frömmgen, Jacob Austin, Peter Choy, Nimesh Ghelani, Lera Kharatyan, Gabriela Surita, Elena Khrapko, Pascal Lamblin, Pierre-Antoine Man- zagol, Marcus Revaj, Maxim Tabachnyk, Daniel Tar- low, Kevin Villela, Dan Zheng, Satish Chandra, and Petros Maniatis. 2024. Resolving code review com- ments with machine learning. In 2024 IEEE/ACM 46th International Conference on Software Engineer- ing: Software Engineering in Practice (ICSE-SEIP).
  28. Pengzhi Gao, Liwen Zhang, Zhongjun He, Hua Wu, and Haifeng Wang. 2023. Learning multilingual sentence representations with cross-lingual consistency regu- larization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 243-262, Singapore. Associa- tion for Computational Linguistics. · DOI
  29. Linyuan Gong, Mostafa Elhoushi, and Alvin Che- ung. 2024. Ast-t5: Structure-aware pretraining for code generation and understanding. arXiv preprint 2401.03003. · arXiv
  30. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2023. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint 2305.11738. · arXiv
  31. Aiden Grossman, Ludger Paehler, Konstantinos Parasyris, Tal Ben-Nun, Jacob Hegna, William Moses, Jose M Monsalve Diaz, Mircea Trofin, and Johannes Doerfert. 2023. Compile: A large ir dataset from production sources. arXiv preprint 2309.15432. · arXiv
  32. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. 2023. Textbooks are all you need. arXiv preprint 2306.11644. · arXiv
  33. Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. UniXcoder: Unified cross- modal pre-training for code representation. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 7212-7225, Dublin, Ireland. Associa- tion for Computational Linguistics. · DOI · Paper page
  34. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. 2024. Deepseek-coder: When the large language model meets programming-the rise of code intelligence. arXiv preprint 2401.14196. · arXiv
  35. Dan Hendrycks, Steven Basart, Saurav Kadavath, Man- tas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring coding challenge com- petence with APPS. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Bench- marks 2021, December 2021, virtual.
  36. Ashish Hooda, Mihai Christodorescu, Miltos Allama- nis, Aaron Wilson, Kassem Fawaz, and Somesh Jha. 2024. Do large code models understand pro- gramming concepts? a black-box approach. arXiv preprint 2402.05980. · arXiv
  37. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
  38. Paras Jain, Ajay Jain, Tianjun Zhang, Pieter Abbeel, Joseph Gonzalez, and Ion Stoica. 2021. Contrastive code representation learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5954-5971, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI · Paper page
  39. Hui Jiang, Linfeng Song, Yubin Ge, Fandong Meng, Junfeng Yao, and Jinsong Su. 2022. An AST struc- ture enhanced decoder for code generation. IEEE ACM Trans. Audio Speech Lang. Process., 30:468- 476. · DOI
  40. Shuyang Jiang, Yuhao Wang, and Yu Wang. 2023. Self- evolve: A code evolution framework via large lan- guage models. arXiv preprint 2306.02907. · arXiv
  41. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
  42. Denis Kocetkov, Raymond Li, Loubna Ben allal, Jia LI, Chenghao Mou, Yacine Jernite, Margaret Mitchell, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro Von Werra, and Harm de Vries. 2023. The stack: 3 TB of permissively li- censed source code. Transactions on Machine Learn- ing Research.
  43. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stan- ley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexan- der Mattick. 2023. Openassistant conversations - democratizing large language model alignment. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Pro- cessing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023.
  44. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonza- lez, Hao Zhang, and Ion Stoica. 2023. Efficient mem- ory management for large language model serving with pagedattention. In Proceedings of the 29th Sym- posium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023, pages 611- 626. ACM. · DOI · Paper page
  45. Chris Lattner and Vikram S. Adve. 2004. LLVM: A compilation framework for lifelong program analysis & transformation. In 2nd IEEE / ACM International Symposium on Code Generation and Optimization (CGO 2004), 20-24 March 2004, San Jose, CA, USA, pages 75-88. IEEE Computer Society. · DOI
  46. Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero-shot language transfer with mul- tilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 4483-4499, On- line. Association for Computational Linguistics. · DOI
  47. Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu-Hong Hoi. 2022. Coderl: Mastering code generation through pretrained mod- els and deep reinforcement learning. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Sys- tems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 -December 9, 2022.
  48. Jaeseong Lee and Seung-won Hwang. 2023. Multilin- gual lottery tickets to pretrain language models. In Findings of the Association for Computational Lin- guistics: EMNLP 2023, pages 9387-9398, Singapore. Association for Computational Linguistics. · DOI
  49. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8424-8445, Dublin, Ireland. Association for Computational Linguistics. · DOI · Paper page
  50. Raymond Li, Loubna Ben allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia LI, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Joel Lamy-Poirier, Joao Monteiro, Nicolas Gontier, Ming-Ho Yee, Lo- gesh Kumar Umapathi, Jian Zhu, Ben Lipkin, Muh- tasham Oblokulov, Zhiruo Wang, Rudra Murthy, Ja- son T Stillerman, Siva Sankalp Patel, Dmitry Ab- ulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Urvashi Bhattacharyya, Wenhao Yu, Sasha Luccioni, Paulo Villegas, Fedor Zhdanov, Tony Lee, Nadav Timor, Jennifer Ding, Claire S Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Carolyn Jane Anderson, Brendan Dolan- Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro Von Werra, and Harm de Vries. 2023. Star- coder: may the source be with you! Transactions on Machine Learning Research. Reproducibility Certifi- cation.
  51. Zongjie Li, Pingchuan Ma, Huaijin Wang, Shuai Wang, Qiyi Tang, Sen Nie, and Shi Wu. 2022. Unleashing the power of compiler intermediate representation to enhance neural program embeddings. In 44th IEEE/ACM 44th International Conference on Soft- ware Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022, pages 2253-2265. ACM. · DOI
  52. Chin-Yew Lin and Franz Josef Och. 2004. ORANGE: a method for evaluating automatic evaluation metrics for machine translation. In COLING 2004: Pro- ceedings of the 20th International Conference on Computational Linguistics, pages 501-507, Geneva, Switzerland. COLING.
  53. Jiate Liu, Yiqin Zhu, Kaiwen Xiao, Qiang Fu, Xiao Han, Wei Yang, and Deheng Ye. 2023. Rltf: Reinforce- ment learning from unit test feedback. arXiv preprint 2307.04349. · arXiv
  54. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. The flan collection: Designing data and methods for effective instruction tuning. In International Con- ference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 22631-22648.
  55. PMLR.
  56. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Li- dong Zhou, Linjun Shou, Long Zhou, Michele Tu- fano, Ming Gong, Ming Zhou, Nan Duan, Neel Sun- daresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021. Codexglue: A machine learning bench- mark dataset for code understanding and generation. In Proceedings of the Neural Information Process- ing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual.
  57. Yingwei Ma, Yue Yu, Shanshan Li, Zhouyang Jia, Jun Ma, Rulin Xu, Wei Dong, and Xiangke Liao. 2023. Mulcs: Towards a unified deep representation for multilingual code search. In IEEE International Con- ference on Software Analysis, Evolution and Reengi- neering, SANER 2023, Taipa, Macao, March 21-24, 2023, pages 120-131. IEEE. · DOI
  58. Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. 2016. Convolutional neural networks over tree structures for programming language processing. In Proceed- ings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Ari- zona, USA, pages 1287-1293. AAAI Press. · DOI
  59. Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. 2023. Octopack: Instruction tun- ing code large language models. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Fol- lowing.
  60. Aravind Ashok Nair, Avijit Roy, and Karl Meinke. 2020. funcgnn: A graph neural network approach to pro- gram similarity. In ESEM '20: ACM / IEEE Interna- tional Symposium on Empirical Software Engineer- ing and Measurement, Bari, Italy, October 5-7, 2020, pages 10:1-10:11. ACM. · DOI
  61. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
  62. PMLR.
  63. Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. 2023. Openwebmath: An open dataset of high-quality mathematical web text. In The 3rd Workshop on Mathematical Reasoning and AI at NeurIPS'23.
  64. Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022. Lifting the curse of multilinguality by pre-training modular transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3479-3495, Seattle, United States. Association for Computational Lin- guistics. · DOI
  65. Weiguo Pian, Hanyu Peng, Xunzhu Tang, Tiezhu Sun, Haoye Tian, Andrew Habib, Jacques Klein, and Tegawendé F. Bissyandé. 2023. Metatptrans: A meta learning approach for multilingual code representa- tion learning. In Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pages 5239-5247. AAAI Press. · DOI
  66. Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir R. Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. 2021. Codenet: A large-scale AI for code dataset for learning a diversity of coding tasks. In Proceedings of the Neural Information Process- ing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual.
  67. Erwin Quiring, Alwin Maier, and Konrad Rieck. 2019. Misleading authorship attribution of source code us- ing adversarial learning. In 28th USENIX Security Symposium, USENIX Security 2019, Santa Clara, CA, USA, August 14-16, 2019, pages 479-496. USENIX Association.
  68. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. In KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pages 3505-3506. ACM. · DOI · Paper page
  69. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Code llama: Open foundation models for code. arXiv preprint 2308.12950. · arXiv
  70. Baptiste Rozière, Jie Zhang, François Charton, Mark Harman, Gabriel Synnaeve, and Guillaume Lample. 2022. Leveraging automated unit tests for unsuper- vised code translation. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
  71. Zeyu Sun, Qihao Zhu, Yingfei Xiong, Yican Sun, Lili Mou, and Lu Zhang. 2020. Treegen: A tree-based transformer architecture for code generation. In The Thirty-Fourth AAAI Conference on Artificial Intelli- gence, AAAI 2020, The Thirty-Second Innovative Ap- plications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8984- 8991. AAAI Press.
  72. Marc Szafraniec, Baptiste Rozière, Hugh Leather, Patrick Labatut, François Charton, and Gabriel Syn- naeve. 2023. Code translation with compiler repre- sentations. In The Eleventh International Confer- ence on Learning Representations, ICLR 2023, Ki- gali, Rwanda, May 1-5, 2023. OpenReview.net.
  73. Sindhu Tipirneni, Ming Zhu, and Chandan K. Reddy. 2024. Structcoder: Structure-aware transformer for code generation. ACM Trans. Knowl. Discov. Data, 18(3):70:1-70:20. · DOI
  74. Jennifer Tracey, Stephanie Strassel, Ann Bies, Zhiyi Song, Michael Arrigo, Kira Griffitt, Dana Delgado, Dave Graff, Seth Kulick, Justin Mott, and Neil Kuster. 2019. Corpus building for low resource languages in the DARPA LORELEI program. In Proceedings of the 2nd Workshop on Technologies for MT of Low Resource Languages, pages 48-55, Dublin, Ireland. European Association for Machine Translation.
  75. Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Sam- son Tan, Baishakhi Ray, Parminder Bhatia, Ramesh Nallapati, Murali Krishna Ramanathan, Dan Roth, and Bing Xiang. 2023. ReCode: Robustness eval- uation of code generation models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13818-13843, Toronto, Canada. Association for Computational Linguistics. · DOI · Paper page
  76. Zirui Wang, Zachary C. Lipton, and Yulia Tsvetkov. 2020. On negative interference in multilingual mod- els: Findings and a meta-learning treatment. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4438-4450, Online. Association for Computa- tional Linguistics. · DOI
  77. Fangzhou Wu, Xiaogeng Liu, and Chaowei Xiao. 2023. Deceptprompt: Exploiting llm-driven code gener- ation via adversarial natural language instructions. arXiv preprint 2312.04730. · arXiv
  78. Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120-130, Online. Association for Com- putational Linguistics. · DOI
  79. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguistics. · DOI · Paper page
  80. Wei Yuan, Quanjun Zhang, Tieke He, Chunrong Fang, Nguyen Quoc Viet Hung, Xiaodong Hao, and Hongzhi Yin. 2022. CIRCLE: continual repair across programming languages. In ISSTA '22: 31st ACM SIGSOFT International Symposium on Software Test- ing and Analysis, Virtual Event, South Korea, July 18 -22, 2022, pages 678-690. ACM. · DOI
  81. Kechi Zhang, Wenhan Wang, Huangzhao Zhang, Ge Li, and Zhi Jin. 2022. Learning to represent programs with heterogeneous graphs. In Proceedings of the 30th IEEE/ACM International Conference on Pro- gram Comprehension, ICPC 2022, Virtual Event, May 16-17, 2022, pages 378-389. ACM. · DOI
  82. Yu Zhou, Xiaoqing Zhang, Juanjuan Shen, Tingting Han, Taolue Chen, and Harald C. Gall. 2022. Ad- versarial robustness of deep code comment genera- tion. ACM Trans. Softw. Eng. Methodol., 31(4):60:1- 60:30. · DOI

Cited by

Sorted by citations

Metadata

Authors
Paul, I., Glavaš, G., & Gurevych, I.
Affiliations
  • Technische Universität Darmstadt (DE)
  • University of Würzburg (DE)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Pages
15023-15041
ISSN
0736-587X (ISSN-L)
DOI
10.48550/arxiv.2403.03894
arXiv
2403.03894
OpenAlex
W4402671203
Cited by
12
References
1
Citations by year
2026: 6 · 2025: 6
FWCI
4.52
Type
conference paper
Language
en
Open access
Licence
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • TUbilio (Technical University of Darmstadt) (submittedVersion)
  • arXiv (Cornell University)
Funders
Bundesministerium für Bildung und Forschung
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
Keywords
multilingual code generation, intermediate representation, compiler intermediate representation, cross-lingual transfer, code language models, prompt robustness, program comprehension, instruction following, code completion, data augmentation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review