CoChat
Sign Up
PreprintCited by 46Software Engineering ResearchNatural Language Processing TechniquesTeaching and Learning Programming

AVATAR: A Parallel Corpus for Java-Python Program Translation

Open access
Ahmad, Wasi Uddin, Tushar, Md Golam Rahman, Chakraborty, Saikat, Chang, Kai-Wei·2021·arXiv
DOI 10.48550/arxiv.2108.11590arXiv 2108.11590
Open this paper in CoChat — free accountOpen PDF

Abstract

Program translation refers to migrating source code from one programming language to another. It has tremendous practical value in software development, as porting software across languages is time-consuming and costly. Automating program translation is of paramount importance in software migration, and recently researchers explored unsupervised approaches due to the unavailability of parallel corpora. However, the availability of pre-trained language models for programming languages enables supervised fine-tuning with a small number of labeled examples. Therefore, we present AVATAR, a collection of 9,515 programming problems and their solutions written in two popular languages, Java and Python. AVATAR is collected from competitive programming sites, online platforms, and open-source repositories. Furthermore, AVATAR includes unit tests for 250 examples to facilitate functional correctness evaluation. We benchmark several pre-trained language models fine-tuned on AVATAR. Experiment results show that the models lack in generating functionally accurate code.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2108.11590

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Without adequate context to LLMs via prompt crafting, they may only serve as next code token predictors, without understanding the overall task.

    “However, research has shown that without providing adequate context to LLMs via prompt crafting, they may only serve as”

    From Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code (2024) · cites this paper as [22]

    Section 3.2 (page unavailable) says decoder-only models such as CodeGPT generate autoregressively, but does not link prompt context to task understanding or describe LLMs as merely next-token predictors. The Limitations section (page unavailable) says the authors did not evaluate LLMs. Searches for prompt crafting, adequate context, token prediction, and task understanding found no passage addressing the claim. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 21

  1. Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2020. A transformer-based ap- proach for source code summarization. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4998-5007, On- line. Association for Computational Linguistics. · DOI
  2. Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Unified pre-training for pro- gram understanding and generation. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 2655-2668, Online. Association for Computational Linguistics. · DOI · Paper page
  3. Marie anne Lachaux, Baptiste Roziere, Marc Szafraniec, and Guillaume Lample. 2021. DOBF: A deobfus- cation pre-training objective for programming lan- guages. In Advances in Neural Information Process- ing Systems. 10 Estimations were conducted using the MachineLearning Impact calculator presented in (Lacoste et al., 2019). We use Amazon Web Services as the provider.
  4. Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Su- jan Kumar Gonugondla, Hantian Ding, Varun Ku- mar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ra- manathan, Ramesh Nallapati, Baishakhi Ray, Par- minder Bhatia, Sudipta Sengupta, Dan Roth, and Bing Xiang. 2023. Multi-lingual evaluation of code generation models. In The Eleventh International Conference on Learning Representations.
  5. Xinyun Chen, Chang Liu, and Dawn Song. 2018. Tree- to-tree neural networks for program translation. In Advances in Neural Information Processing Systems, volume 31.
  6. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xi- aocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. Code- BERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1536-1547, Online. Association for Computational Linguistics. · DOI · Paper page
  7. Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. 2023. Incoder: A generative model for code infilling and synthesis. In The Eleventh International Conference on Learn- ing Representations.
  8. Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Jian Yin, Daxin Jiang, et al. 2021. Graphcodebert: Pre-training code representations with data flow. In International Conference on Learning Representations.
  9. Svetoslav Karaivanov, Veselin Raychev, and Martin Vechev. 2014. Phrase-based statistical translation of programming languages. In Proceedings of the · DOI
  10. Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700. · arXiv
  11. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Li- dong Zhou, Linjun Shou, Long Zhou, Michele Tu- fano, MING GONG, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shu- jie LIU. 2021. CodeXGLUE: A machine learning benchmark dataset for code understanding and gener- ation. In Thirty-fifth Conference on Neural Informa- tion Processing Systems Datasets and Benchmarks Track (Round 1).
  12. Kawser Wazed Nafi, Tonny Shekha Kar, Banani Roy, Chanchal K Roy, and Kevin A Schneider. 2019. Clcdsa: cross language code clone detection using syntactical features and api documentation. In 2019 34th IEEE/ACM International Conference on Au- tomated Software Engineering (ASE), pages 1026- 1037. IEEE. · DOI
  13. Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. 2013. Lexical statistical machine transla- tion for language migration. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, pages 651-654. · DOI
  14. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations.
  15. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. · DOI · Paper page
  16. Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. 2021. Codenet: A large-scale AI for code dataset for learning a diversity of coding tasks. In Thirty-fifth Conference on Neural Information Pro- cessing Systems Datasets and Benchmarks Track (Round 2).
  17. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  18. Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for auto- matic evaluation of code synthesis. arXiv preprint arXiv:2009.10297. · arXiv
  19. Baptiste Roziere, Jie Zhang, Francois Charton, Mark Harman, Gabriel Synnaeve, and Guillaume Lample. 2022. Leveraging automated unit tests for unsuper- vised code translation. In International Conference on Learning Representations.
  20. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30, pages 5998-6008.
  21. Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. 2021. CodeT5: Identifier-aware unified pre- trained encoder-decoder models for code understand- ing and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 8696-8708, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI · Paper page

Cited by

Sorted by citations

Authors
Ahmad, W. U., Tushar, M. G. R., Chakraborty, S., & Chang, K.-W.
Affiliations
  • University of California, Los Angeles (US)
  • Microsoft (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
Pages
2268-2281
DOI
10.48550/arxiv.2108.11590
arXiv
2108.11590
OpenAlex
W4385571480
Cited by
46
References
0
Citations by year
2026: 11 · 2025: 15 · 2024: 9 · 2023: 8 · 2022: 3
FWCI
11.05
Type
conference paper
Language
en
Open access
Licence
Locations
  • Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
Keywords
program translation, parallel corpus, software migration, code porting, supervised fine-tuning, pre-trained language models, functional correctness, unit testing, avatars

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review