CoChat
Sign Up
Cited by 1Software Engineering ResearchNatural Language Processing TechniquesTopic Modeling

The DOI matches its Crossref record.

TRACE: Evaluating Execution Efficiency of LLM-Based Code Translation

Open access
Gong, Zhihao, Sun, Zeyu, Huang, Dong, Liang, Qingyuan, Zhang, Jie M., Hao, Dan·2026·Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
DOI 10.18653/v1/2026.acl-long.140
Open this paper in CoChat — free accountOpen PDF

Abstract

While Large Language Models (LLMs) have substantially improved the functional correctness of code translation, the critical dimension of execution efficiency remains overlooked.We present TRACE, the first benchmark to explicitly assess efficiency in LLM-translated code.TRACE includes 1,000 efficiency-critical tasks across C++, Java, and Python, each augmented with stress tests that reveal efficiency disparities often overlooked by small-scale tests.Using TRACE, we conduct an extensive evaluation of 28 representative LLMs and highlight several key insights: 1) Correctness and efficiency are often misaligned: the correctness leader Claude-Sonnet-4-Think achieves only moderate time efficiency, outperformed by smaller open-source LLMs such as Qwen2.5-Coder-14B-Instruct.2) Inefficiency is both prevalent and patterned: 23.5% of correct translations suffer from notable inefficiency, mainly arising from algorithm implementation discrepancy (11.9%), language construct mismatch (66.4%), and resource management inefficiency (21.7%).3) Inference-time prompt strategies bring only modest improvements, indicating that simple prompting alone is insufficient to improve translation efficiency.Together, our results establish execution efficiency as an essential dimension of code translation and position TRACE as a principled foundation for efficiency-oriented evaluation.Our

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2026.acl-long.140

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AlphaTrans [95] is a neuro-symbolic compositional technique that decomposes repositories into fragments and translates them in reverse call order with GraalVM-based validation.

    “AlphaTrans [95] is proposed as a neuro-symbolic compositional technique that decomposes repositories into fragments and translates them in reverse call order with GraalVM-based validation.”

    From RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation (2026) · cites this paper as [95]

    The References section identifies AlphaTrans as a neuro-symbolic compositional approach to repository-level code translation and validation. Our copy does not describe repository fragmentation, translation in reverse call order, or GraalVM-based validation; searches for those details found no supporting passage. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Still being checked.

Part of this paper’s reference list could not be split into single entries.

Research works printed but not linked: 41

  1. Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2022. Summarize and generate to back-translate: Unsupervised transla- tion of programming languages. arXiv preprint arXiv:2205.11116. · arXiv
  2. Wasi Uddin Ahmad, Md Golam Rahman Tushar, Saikat Chakraborty, and Kai-Wei Chang. 2021. Avatar: A parallel corpus for java-python program translation. arXiv preprint arXiv:2108.11590. · arXiv
  3. Mingzhe Du, Anh Tuan Luu, Bin Ji, Qian Liu, and See- Kiong Ng. 2024. Mercury: A code efficiency bench- mark for code large language models. Advances in Neural Information Processing Systems, 37:16601- 16622.
  4. Mingzhe Du, Luu Anh Tuan, Yue Liu, Yuhao Qing, Dong Huang, Xinyi He, Qian Liu, Zejun Ma, and See-kiong Ng. 2025. Afterburner: Reinforcement learning facilitates self-improving code efficiency optimization. arXiv preprint arXiv:2505.23387. · arXiv
  5. Peter Emerson. 2013. The original borda count and partial voting. Social Choice and Welfare, 40(2):353- 358.
  6. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, and 1 others. 2024. Deepseek- coder: When the large language model meets programming-the rise of code intelligence. arXiv preprint arXiv:2401.14196. · arXiv
  7. Minghua He, Fangkai Yang, Pu Zhao, Wenjie Yin, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dong- mei Zhang, and Qi Zhang. 2025. Execoder: Em- powering large language models with executability representation for code translation. arXiv preprint arXiv:2501.18460. · arXiv
  8. Dong Huang, Jianbo Dai, Han Weng, Puzhen Wu, Yuhao Qing, Heming Cui, Zhijiang Guo, and Jie Zhang. 2024a. Effilearner: Enhancing efficiency of generated code via self-optimization. Advances in Neural Information Processing Systems, 37:84482- 84522.
  9. Dong Huang, Yuhao Qing, Weiyi Shang, Heming Cui, and Jie M Zhang. 2024b. Effibench: Benchmarking the efficiency of automatically generated code. arXiv preprint arXiv:2402.02037. · arXiv
  10. Dong Huang, Guangtao Zeng, Jianbo Dai, Meng Luo, Han Weng, Yuhao Qing, Heming Cui, Zhijiang Guo, and Jie M Zhang. 2024c. Swiftcoder: En- hancing code generation in large language models through efficiency-aware fine-tuning. arXiv preprint arXiv:2410.10209. · arXiv
  11. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, and 1 others. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186. · arXiv
  12. Ali Reza Ibrahimzada, Kaiyao Ke, Mrigank Pawagi, Muhammad Salman Abid, Rangeet Pan, Saurabh Sinha, and Reyhaneh Jabbarvand. 2025. Alpha- trans: A neuro-symbolic compositional approach for repository-level code translation and validation. Proceedings of the ACM on Software Engineering, 2(FSE):2454-2476.
  13. Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A survey on large lan- guage models for code generation. arXiv preprint arXiv:2406.00515. · arXiv
  14. Mingsheng Jiao, Tingrui Yu, Xuan Li, Guanjie Qiu, Xiaodong Gu, and Beijun Shen. 2023. On the evalua- tion of neural code translation: Taxonomy and bench- mark. In 2023 38th IEEE/ACM International Con- ference on Automated Software Engineering (ASE), pages 1529-1541. IEEE.
  15. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language mod- els resolve real-world github issues? arXiv preprint arXiv:2310.06770. · arXiv
  16. Svetoslav Karaivanov, Veselin Raychev, and Martin Vechev. 2014. Phrase-based statistical translation of programming languages. In Proceedings of the 2014 ACM international symposium on new ideas, new paradigms, and reflections on programming & software, pages 173-184.
  17. Marie-Anne Lachaux, Baptiste Roziere, Lowik Chanus- sot, and Guillaume Lample. 2020. Unsupervised translation of programming languages. arXiv preprint arXiv:2006.03511. · arXiv
  18. Fang Liu, Jia Li, and Li Zhang. 2023. Syntax and domain aware model for unsupervised program trans- lation. In 2023 IEEE/ACM 45th International Con- ference on Software Engineering (ICSE), pages 755- 767. IEEE.
  19. Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang. 2024. Evaluating language models for efficient code generation. arXiv preprint arXiv:2408.06450. · arXiv
  20. Marcos Macedo, Yuan Tian, Pengyu Nie, Filipe R Cogo, and Bram Adams. 2024. Intertrans: Leveraging tran- sitive intermediate translations to enhance llm-based code translation. arXiv preprint arXiv:2411.01063. · arXiv
  21. Leland McInnes, John Healy, Steve Astels, and 1 others. 2017. hdbscan: Hierarchical density based clustering. J. Open Source Softw., 2(11):205.
  22. Trong Duc Nguyen, Anh Tuan Nguyen, and Tien N Nguyen. 2016. Mapping api elements for code mi- gration with vector representations. In Proceedings of the 38th international conference on software en- gineering companion, pages 756-758.
  23. Vikram Nitin, Rahul Krishna, and Baishakhi Ray. 2024. Spectra: Enhancing the code translation ability of language models by generating multi-modal specifi- cations. arXiv preprint arXiv:2405.18574. · arXiv
  24. Changan Niu, Ting Zhang, Chuanyi Li, Bin Luo, and Vincent Ng. 2024. On evaluating the efficiency of source code generated by llms. In Proceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering, pages 103-107.
  25. Rangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar, Lambert Pouguem Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha, and Reyhaneh Jabbarvand. 2024. Lost in transla- tion: A study of bugs introduced by large language models while translating code. In Proceedings of the IEEE/ACM 46th International Conference on Soft- ware Engineering, pages 1-13.
  26. Yun Peng, Jun Wan, Yichen Li, and Xiaoxue Ren. 2025. Coffe: A code efficiency benchmark for code genera- tion. Proceedings of the ACM on Software Engineer- ing, 2(FSE):242-265.
  27. Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, and 1 others. 2021. Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks. arXiv preprint arXiv:2105.12655. · arXiv
  28. Yuhao Qing, Boyu Zhu, Mingzhe Du, Zhijiang Guo, Terry Yue Zhuo, Qianru Zhang, Jie M Zhang, Hem- ing Cui, Siu-Ming Yiu, Dong Huang, and 1 others. 2025. Effibench-x: A multi-language benchmark for measuring efficiency of llm-generated code. arXiv preprint arXiv:2505.13004. · arXiv
  29. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, and 1 others. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950. · arXiv
  30. Baptiste Roziere, Jie M. Zhang, Francois Charton, Mark Harman, Gabriel Synnaeve, and Guillaume Lample. 2021. Leveraging automated unit tests for unsuper- vised code translation. ICLR.
  31. Jingwei Shi, Xinxiang Yin, Jing Huang, Jinman Zhao, and Shengyu Tao. 2026. Codehacker: Automated test case generation for detecting vulnerabilities in competitive programming solutions. arXiv preprint arXiv:2602.20213. · arXiv
  32. Alexander Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob Gardner, Milad Hashemi, Gra- ham Neubig, Parthasarathy Ranganathan, Osbert Bastani, and Amir Yazdanbakhsh. 2023. Learning performance-improving code edits. arXiv preprint arXiv:2302.07867. · arXiv
  33. Marc Szafraniec, Baptiste Roziere, Hugh Leather, Fran- cois Charton, Patrick Labatut, and Gabriel Synnaeve. 2022. Code translation with compiler representations. arXiv preprint arXiv:2207.03578. · arXiv
  34. Qingxiao Tao, Tingrui Yu, Xiaodong Gu, and Beijun Shen. 2024. Unraveling the potential of large lan- guage models in code translation: How far are we? In 2024 31st Asia-Pacific Software Engineering Con- ference (APSEC), pages 353-362. IEEE.
  35. Tina Vartziotis, Ippolyti Dellatolas, George Dasoulas, Maximilian Schmidt, Florian Schneider, Tim Hoff- mann, Sotirios Kotsopoulos, and Michael Keckeisen. 2024. Learn to code sustainably: An empirical study on llm-based green code generation. arXiv preprint arXiv:2403.03344. · arXiv
  36. Zihan Wang, Siyao Liu, Yang Sun, Hongyan Li, and Kai Shen. 2025. Codecontests+: High-quality test case generation for competitive programming. arXiv preprint arXiv:2506.05817. · arXiv
  37. Pengyu Xue, Linhao Wu, Zhen Yang, Chengyi Wang, Xiang Li, Yuxiang Zhang, Jia Li, Ruikai Jin, Yifei Pei, Zhaoyan Shen, and 1 others. 2025. Classeval-t: Evaluating large language models in class-level code translation. Proceedings of the ACM on Software Engineering, 2(ISSTA):1421-1444.
  38. Zhen Yang, Fang Liu, Zhongxing Yu, Jacky Wai Ke- ung, Jia Li, Shuo Liu, Yifan Hong, Xiaoxue Ma, Zhi Jin, and Ge Li. 2024. Exploring and unleashing the power of large language models in automated code translation. Proceedings of the ACM on Software Engineering, 1(FSE):1585-1608.
  39. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, and 1 others. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223, 1(2). · arXiv
  40. Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, and 1 others. 2023a. Codegeex: A pre- trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5673-5684.
  41. Zibin Zheng, Kaiwen Ning, Yanlin Wang, Jingwen Zhang, Dewu Zheng, Mingxi Ye, and Jiachi Chen. 2023b. A survey of large language models for code: Evolution, benchmarking, and future trends. arXiv preprint arXiv:2311.10372. · arXiv

Cited by

Sorted by citations

Related

Metadata

Authors
Gong, Z., Sun, Z., Huang, D., Liang, Q., Zhang, J. M., & Hao, D.
Affiliations
  • Peking University (CN)
  • Chinese Academy of Sciences (CN)
  • National University of Singapore (SG)
  • King's College London (GB)
ORCID
Venue
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Publisher
Association for Computational Linguistics
Published
2026
Pages
3089-3117
ISSN
0736-587X (ISSN-L)
DOI
10.18653/v1/2026.acl-long.140
OpenAlex
W7166824387
Cited by
1
References
0
Citations by year
2026: 1
FWCI
10.67
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
Funders
  • ITEA
  • National Natural Science Foundation of China (62372005, 62402482)
  • National Key Research and Development Program of China
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
execution efficiency, benchmark, large language models, program correctness, stress testing, resource management, prompt strategies, inference latency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review