RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation
Open accessAbstract
Repository-level code translation refers to translating an entire code repository from one programming language to another while preserving the functionality of the source repository. Many benchmarks have been proposed to evaluate the performance of such code translators. However, previous benchmarks mostly provide fine-grained samples, focusing at either code snippet, function, or file-level code translation. Such benchmarks do not accurately reflect real-world demands, where entire repositories often need to be translated, involving longer code length and more complex functionalities. To address this gap, we propose a new benchmark, named RepoTransBench, which is a real-world multilingual repository-level code translation benchmark featuring 1,897 real-world repository samples across 13 language pairs with automatically executable test suites. Besides, we introduce RepoTransAgent, a general agent framework to perform repository-level code translation. We evaluate both our benchmark’s challenges and agent’s effectiveness using several methods and backbone LLMs, revealing that repository-level translation remains challenging, where the best-performing method achieves only a 32.8% success rate. Furthermore, our analysis reveals that translation difficulty varies significantly by language pair direction, with dynamic-to-static language translation being much more challenging than the reverse direction (achieving below 10% vs. static-to-dynamic at 45-63%). Finally, we conduct a detailed error analysis and highlight current LLMs’ deficiencies in repository-level code translation, which could provide a reference for further improvements. We provide the code and data athttps://github.com/DeepSoftwareAnalytics/RepoTransBench.
AI snapshot
Written from the paper’s full text.
TL;DR
RepoTransBench is a real-world multilingual repository-level code translation benchmark with 1,897 samples across 13 language pairs and automatically executable test suites, paired with RepoTransAgent, a ReAct-based agent framework. The best-performing method reaches only 32.8% success rate, and translation difficulty is strongly asymmetric: static-to-dynamic translation is much easier than dynamic-to-static.
Key findings
- RepoTransBench contains 1,897 repository samples across 13 translation pairs and 7 languages, drawn from 970 unique projects. Samples average 23,966 tokens, 2,394 lines of code, 177 functions, 35 classes, and 163 import statements, with 87.9 cross-file and 1,443.7 intra-file dependencies; test suites achieve 81.89% line coverage and 72.61% branch coverage.
- Repository-level translation remains challenging: the best configuration, RepoTransAgent with Claude or GPT-4.1, achieves 32.8% success rate. TranslationOnly mostly achieves 0.0% success, while ErrorFeedback reaches 11.3% with Claude and 15.6% with GPT-4.1; RepoTransAgent improves over ErrorFeedback by up to 21.5%.
- Translation difficulty varies sharply by direction. Static-to-dynamic translations reach 45–63% success in the paper’s summary, while dynamic-to-static translations are typically below 10%; for example, Claude achieves 61.5% for C→Python, 63.0% for C++→Python, and 45.9% for Java→Python, whereas Python→Java reaches only 0.0–7.0% and Python→C++ only 0.0–9.9%.
- Error analysis identifies five categories: E1 configuration file issues, E2 limited understanding ability issues, E3 incomplete generation issues, E4 language feature issues, and E5 encoding issues.
Methods
- Data collection began with a questionnaire of 21 professional developers yielding 86 translation-pair requests; the authors selected 13 pairs across 7 languages, ranked GitHub repositories by stars, and retained repositories with more than 50 stars. Rule-based filtering enforced language dominance, popularity threshold, and package exclusion.
- Benchmark construction used a multi-agent framework with four agents: Test Case Generator, Test Case Runner, Coverage Analyst, and Test Case Translator. It produced execution-based test suites and validated source repositories in runnable environments.
- Evaluation used 8 LLMs (Qwen3-235B-A22B, Qwen3-235B-A22B-think, DeepSeek-Chat, DeepSeek-Reasoner, Claude-Sonnet-4, Gemini-2.5-Flash-Lite, GPT-4.1, o3-mini) and three methods (TranslationOnly, ErrorFeedback, RepoTransAgent), with metrics SR, CR, APR, and AMPR; generated code was executed in a Docker sandbox.
Limitations
- Paper-stated internal threats: evaluated LLMs may not represent newer models, no fine-tuning was applied for repository-level translation, and the 13 translation pairs across 7 languages may not generalize to less common or domain-specific languages.
- Paper-stated external threats: LLM generation variability may affect reproducibility; execution-based metrics using existing test suites may not capture all edge cases or guarantee complete semantic equivalence; repository filtering may bias toward popular, well-maintained projects; and benchmark temporal validity may decay as languages and frameworks evolve.
- Evident from the text: test coverage is below 100% (81.89% line, 72.61% branch overall), so untested behavior may escape evaluation; some source languages have relatively few samples (e.g., Matlab 64, C# 97), which may limit per-pair conclusions.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
19 of 32 cited sources checked · 9 supported · 0 flagged · accuracy not yet measured
- Premisecitation§ I. INTRODUCTION · p. 1
Snippet-level benchmarks such as CoST and XLCost focus on evaluating translation of program segments between two consecutive code comments.
“snippetlevel code translation benchmarks, such as CoST [32], XLCost [33], typically focus on evaluating the translation of program segments located between two consecutive code comments.”
Quote in the paperNot a substantial claim2 of 2 sources checked- cites [32] Zhu, 2022, Multilingual code snippets training for program translation source
The Introduction (p. 1) identifies CoST as a benchmark for snippet-level translation and says its snippets are aligned by matching code comments across languages. Data Collection and Processing (p. 3) likewise describes using comments and their corresponding code to align snippets. Neither passage specifies that the evaluated segments lie between two consecutive comments; the paper also evaluates program-level translation. XLCost has a separate citation in the citing sentence and is not judged here. (LLM, conf=0.82)
Experimental — accuracy not yet measured - cites [33] Zhu, 2022, Xlcost: A benchmark dataset for cross-lingual code intelligence source
- Premisecitation§ I. INTRODUCTION · p. 1
Function-level benchmarks such as TransCoder-test, CodeXGLUE, and HumanEvalX focus on evaluating translation of a function.
“Function-level code translation benchmarks, such as TransCoder-test [34], CodeXGLUE [35] and HumanEvalX [36], focus on evaluating the translation of a function.”
Quote in the paperSubstantial claim2 of 3 sources checked - Premisecitation§ C. Evaluation Metrics · p. 9
Similarity-based metrics ignore the syntactic correctness and functional correctness of translations.
“However, similarity-based metrics ignore the syntactic correctness and functional correctness of translations [34].”
Quote in the paperSubstantial claim1 of 1 sources checked- cites [34] Lachaux, 2020, Unsupervised translation of programming languages source
- Premisecitation§ II. BACKGROUND · p. 4
Real repositories typically contain existing test suites that can serve as validation mechanisms.
“Furthermore, unlike artificially crafted datasets used in fine-grained translation [37], [38], real repositories typically contain existing test suites that can serve as valuable validation mechanisms.”
Quote in the paperSubstantial claim2 of 2 sources checked- cites [37] Puri, 2021, CodeNet: A large-scale AI for code dataset for learning a diversity of coding tasks source
The indexed passages discuss CodeNet datasets and online-judge acceptance metadata (Sections 5.1 and 9, pp. 6 and 19), not test suites in real repositories. Searches for existing test suites, repository validation, and test cases as correctness oracles found no passage supporting the attributed claim. (LLM, conf=0.74)
Experimental — accuracy not yet measured - cites [38] source
The Data Statistics section (page unavailable) says AVATAR uses publicly available AtCoder test cases to evaluate translation accuracy, supporting the use of existing tests for validation. The Introduction and Related Works mention collecting solutions from GitHub repositories, but do not say that real repositories typically contain test suites or that repository test suites supplied AVATAR’s tests. (LLM, conf=0.78)
Experimental — accuracy not yet measured
- Premisereasoning§ VIII. RELATED WORK · p. 14
Pan et al. [1] do not provide a sufficient number of repositories and corresponding automatic test suites for evaluation.
“However, they do not provide a sufficient number of repositories and corresponding automatic test suites for evaluation.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites [1] source
The Dataset Collection and Pre-Processing section identifies only two real-world projects, Apache Commons CLI and Click, which supports the concern about the number of repositories. But it says the projects have unit tests and does not assess whether the number of repositories or corresponding test suites is sufficient for evaluation. It also notes that the translated projects never reached test execution because the code was syntactically incorrect. (LLM, conf=0.83)
Experimental — accuracy not yet measured
- Premisereasoning§ VIII. RELATED WORK · p. 14
Existing fine-grained code translation benchmarks cannot evaluate the performance of current techniques on real-world repository-level code translation tasks.
“Although these benchmarks can evaluate the capabilities of existing code translation techniques to some extent, they cannot evaluate the performance of current techniques on real-world repositorylevel code translation tasks.”
Quote in the paperSubstantial claim3 of 3 sources checked- cites [32] Zhu, 2022, Multilingual code snippets training for program translation source
The Dataset Comparisons and Characteristics section (p. 3) says existing translation datasets have method-level mappings, limited language coverage, or quality problems. That supports the general concern about benchmark limitations, but the paper does not establish that these benchmarks cannot evaluate techniques on real-world repository-level translation tasks. (LLM, conf=0.84)
Experimental — accuracy not yet measured - cites [33] Zhu, 2022, Xlcost: A benchmark dataset for cross-lingual code intelligence source
The Limitations and Future Work section (p. 7) says that not all of XLCoST’s tasks have practical real-world applications, especially its snippet-level tasks. The paper also evaluates program-level translation (Code Tasks, p. 5), but it does not establish that existing benchmarks cannot evaluate real-world repository-level translation; that stronger, repository-specific claim is unsupported. (LLM, conf=0.78)
Experimental — accuracy not yet measured - cites [35] Lu, 2021, CodeXGLUE: A machine learning benchmark dataset for code understanding and generation source
Section 3.5 (p. 4) describes mining paired functions or methods from open-source projects, and Section 5.7 (p. 9) evaluates matching Java and C# functions. This supports the characterization of CodeXGLUE’s translation benchmark as fine-grained, but the paper does not state that existing benchmarks cannot evaluate real-world repository-level translation tasks. (LLM, conf=0.82)
Experimental — accuracy not yet measured
- Findingreasoning§ D. Statistics of RepoTransBench · p. 6
RepoTransBench requires proper configuration files such as pom.xml, package.json, or requirements.txt.
“another crucial distinction of RepoTransBench is the requirement for proper configuration files (such as 'pom.xml' for Java Maven projects, 'package.json' for JavaScript projects, or 'requirements.txt' for Python projects).”
Quote in the paperSubstantial claim - Findingown data§ I. INTRODUCTION · p. 2
RepoTransAgent achieves up to 32.8% success rate on RepoTransBench.
“We propose RepoTransAgent , a general agent framework for multilingual repository-level code translation based on reasoning and action paradigms, achieving up to 32.8% success rate on RepoTransBench.”
Quote in the paperSubstantial claim - Findingown data§ B. RQ2: Performance Differences Across translation pairs · p. 10
There is a fundamental asymmetry between translating from statically-typed languages to dynamically-typed languages versus the reverse direction.
“The most striking pattern in our results is the fundamental asymmetry between translating from statically-typed languages to dynamically-typed languages versus the reverse direction.”
- Also stated in: § B. RQ2: Performance Differences Across translation pairs · p. 11 · plain
“revealing a fundamental asymmetry in translation difficulty”
Quote in the paperSubstantial claim - Also stated in: § B. RQ2: Performance Differences Across translation pairs · p. 11 · plain
- Findingown data§ A. Data Collection · p. 5
The current dataset achieves an average of 81.89% line coverage and 72.61% branch coverage across all translation pairs.
“with our current dataset achieving an average of 81.89% line coverage and 72.61% branch coverage across all translation pairs as shown in Table II.”
- Also stated in: § D. Statistics of RepoTransBench · p. 7 · plain
“the overall dataset maintains 81.89% line coverage and 72.61% branch coverage”
Quote in the paperSubstantial claim- Rests on an assumption: Line and branch coverage metrics indicate high-quality test suites.
- Also stated in: § D. Statistics of RepoTransBench · p. 7 · plain
- Findingown data§ A. RQ1: Performance of LLM-based Translation Methods · p. 10
DeepSeek-Reasoner performs dramatically worse than DeepSeek-Chat (1.2% vs 22.5% SR).
“More surprisingly, DeepSeekReasoner performs dramatically worse than DeepSeek-Chat (1.2% vs 22.5%)”
- Also stated in: § A. RQ1: Performance of LLM-based Translation Methods · p. 10 · plain
“DeepSeek-Reasoner performing significantly worse than DeepSeek-Chat”
Quote in the paperSubstantial claim - Also stated in: § A. RQ1: Performance of LLM-based Translation Methods · p. 10 · plain
- Findingnone given§ I. INTRODUCTION · p. 2
To construct RepoTransBench, the authors developed a multi-agent framework that automatically generates comprehensive test suites and handles complex requirements of repository-level translation validation.
“To construct RepoTransBench, we develop a multi-agent framework that automatically generates comprehensive test suites and handles the complex requirements of repository-level translation validation.”
Quote in the paperSubstantial claim - Findingown data§ A. Data Collection · p. 4
JavaScript → Python translation shows strong demand.
“JavaScript → Python translation also shows strong demand as organizations seek to consolidate their tech stacks by moving web-based logic into Python's rich ecosystem for data processing and machine learning workflows.”
Quote in the paperNot a substantial claim - Findingown data§ I. INTRODUCTION · p. 2
Translation difficulty varies by different translation pairs and project complexity.
“We conduct an extensive evaluation across multiple dimensions, revealing that translation difficulty varies by different translation pairs and project complexity.”
Quote in the paperSubstantial claim - Findingown data§ A. Data Collection · p. 4
Based on survey findings, the benchmark includes 13 translation pairs reflecting real-world developer needs.
“Based on these survey findings, we expand our benchmark to include 13 translation pairs that reflect real-world developer needs.”
Quote in the paperSubstantial claim - Findingown data§ A. Data Collection · p. 4
The highest demand in the survey is for Python ↔ C++ bidirectional translation.
“The highest demand is observed for Python ↔ C++ bidirectional translation, with developers frequently needing to migrate between Python's rapid prototyping capabilities and C++'s performance-critical applications in systems programming and high-performance computing.”
Quote in the paperNot a substantial claim - Findingown data§ A. RQ1: Performance of LLM-based Translation Methods · p. 9
RepoTransAgent consistently outperforms baseline approaches across all evaluated backbone models.
“our RepoTransAgent framework consistently demonstrates superior performance across all evaluated models.”
- Also stated in: § A. RQ1: Performance of LLM-based Translation Methods · p. 10 · plain
“Our RepoTransAgent framework consistently outperforms baseline approaches across all evaluated backbone models.”
Quote in the paperSubstantial claim - Also stated in: § A. RQ1: Performance of LLM-based Translation Methods · p. 10 · plain
- Findingown data§ B. RQ2: Performance Differences Across translation pairs · p. 11
Different LLM backbones demonstrate specialized advantages for specific translation pairs (e.g., o3-mini excelling at C → Rust with 67.2% vs. Claude's 47.5%).
“Finding 5: Different LLM backbones demonstrate specialized advantages for specific translation pairs (e.g., o3-mini excelling at C → Rust with 67.2% vs. Claude's 47.5%), likely reflecting training data composition and architectural biases that favor particular programming languages.”
Quote in the paperSubstantial claim - Findingown data§ A. RQ1: Performance of LLM-based Translation Methods · p. 10
Reasoning-focused models do not show clear advantages over their standard counterparts.
“Notably, reasoning-focused models do not show clear advantages over their standard counterparts.”
- Also stated in: § A. RQ1: Performance of LLM-based Translation Methods · p. 10 · plain
“Reasoning-based models exhibit no clear advantages”
Quote in the paperSubstantial claim - Also stated in: § A. RQ1: Performance of LLM-based Translation Methods · p. 10 · plain
- Findingown data§ C. RQ3: Impact of Dependency Complexity · p. 12
Successful translations consistently exhibit lower median values for cross-file dependencies compared to failed translations.
“For cross-file dependencies, successful translations consistently exhibit lower median values compared to failed translations, indicating that repositories with fewer intermodule dependencies are more amenable to successful translation.”
Quote in the paperSubstantial claim - Findingown data§ A. RQ1: Performance of LLM-based Translation Methods · p. 10
Direct translation approaches (TranslationOnly) consistently fail to produce repositories that pass all test cases, with 0.0% success rate across most models (except o3-mini with 0.2%).
“An important observation is that direct translation approaches (TranslationOnly) consistently fail to produce repositories that pass all test cases, with 0.0% success rate across most models (except o3-mini with 0.2%).”
- Also stated in: § A. RQ1: Performance of LLM-based Translation Methods · p. 10 · plain
“The translation-only method rarely produces repositories that pass all test cases.”
Quote in the paperSubstantial claim - Also stated in: § A. RQ1: Performance of LLM-based Translation Methods · p. 10 · plain
- Findingown data§ B. RQ2: Performance Differences Across translation pairs · p. 11
Different LLM backbones exhibit specialization patterns for specific translation pairs.
“Our analysis reveals that different LLM backbones exhibit surprising specialization patterns for specific translation pairs”
Quote in the paperSubstantial claim - Findingown data§ A. RQ1: Performance of LLM-based Translation Methods · p. 10
There is a substantial performance gap between leading closed-source models (Claude and GPT-4.1) and open-source alternatives.
“The evaluation reveals a substantial performance gap between leading closed-source models and open-source alternatives.”
- Also stated in: § A. RQ1: Performance of LLM-based Translation Methods · p. 10 · boosted
“Claude and GPT-4.1 demonstrate superior performance with RepoTransAgent achieving 32.8% SR for both models, significantly outperforming open-source models such as Qwen3 (16.9%), DeepSeek-Chat (22.5%), and Gemini (11.3%).”
Quote in the paperSubstantial claim - Also stated in: § A. RQ1: Performance of LLM-based Translation Methods · p. 10 · boosted
- Findingown data§ D. RQ4: Error Analysis · p. 13
Non-US-ASCII characters commonly occur in repositories.
“Finding 11: Non-US-ASCII characters commonly occur in repositories, and encoding-related issues can sometimes arise when reading or writing resources.”
Quote in the paperSubstantial claim - Findingown data§ D. RQ4: Error Analysis · p. 12
Common errors in repository-level code translation can be classified into five categories: E1 Configuration File Issues, E2 Limited Understanding Ability Issues, E3 Incomplete Generation Issues, E4 Language Feature Issues, E5 Encoding Issues.
“These errors are classified into five categories: E1 (Configuration File Issues), E2 (Limited Understanding Ability Issues), E3 (Incomplete Generation Issues), E4 (Language Feature Issues), E5 (Encoding Issues).”
Quote in the paperNot a substantial claim - Findingreasoning§ D. RQ4: Error Analysis · p. 12
Improper configuration in repository-level code translation can lead to dependency-related issues such as non-existent dependencies and version mismatches.
“This can lead to dependency-related issues (e.g., non-existent dependencies, version mismatches, etc.).”
Quote in the paperNot a substantial claim - Findingown data§ C. RQ3: Impact of Dependency Complexity · p. 12
Repository complexity across all dimensions (cross-file dependencies, intra-file dependencies, code length, and structural complexity) inversely correlates with translation success.
“Finding 6: Repository complexity across all dimensions (cross-file dependencies, intra-file dependencies, code length, and structural complexity) inversely correlates with translation success, indicating that current LLMs struggle systematically with complex repository structures.”
- Also stated in: § IX. CONCLUSION · p. 14 · plain
“We also observe strong directional asymmetry in translation difficulty (static-todynamic achieving 45-63% vs. reverse direction below 10%), model-specific advantages for certain translation pairs reflecting training biases, and inverse correlation between repository complexity and translation success.”
Quote in the paperSubstantial claim - Also stated in: § IX. CONCLUSION · p. 14 · plain
- Findingown data§ D. RQ4: Error Analysis · p. 13
Encoding-related issues can sometimes arise when reading or writing resources.
“Finding 11: Non-US-ASCII characters commonly occur in repositories, and encoding-related issues can sometimes arise when reading or writing resources.”
Quote in the paperNot a substantial claim - Findingown data§ IX. CONCLUSION · p. 14
The best-performing method for repository-level translation achieves only a 32.8% success rate.
“Our evaluation reveals that repositorylevel translation remains challenging, with the best-performing method achieving only a 32.8% success rate.”
Quote in the paperSubstantial claim - Findingown data§ C. RQ3: Impact of Dependency Complexity · p. 12
Successful translations tend to have simpler internal dependency structures (intra-file dependencies).
“Similarly, intra-file dependencies show that successful translations tend to have simpler internal dependency structures.”
Quote in the paperSubstantial claim - Premisecitation§ I. INTRODUCTION · p. 1
LLM training increasingly faces limitations regarding the availability of open-source code data.
“However, the training of these models increasingly faces limitations regarding the availability of opensource code data.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [5], [6]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Premisecitation§ RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation · p. 1
Previous code translation benchmarks mostly provide fine-grained samples at the snippet, function, or file level.
“However, previous benchmarks mostly provide fine-grained samples, focusing at either code snippet, function, or file-level code translation.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [32]-[40]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationreasoning§ I. INTRODUCTION · p. 1
Fine-grained code translation benchmarks may not meet the demands of real development scenarios, which require translation of entire repositories.
“However, these fine-grained code translation benchmarks may not meet the demands of real development scenarios, which require the translation of entire repositories.”
Quote in the paperSubstantial claim - Interpretationreasoning§ RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation · p. 1
Fine-grained code translation benchmarks do not accurately reflect real-world demands, where entire repositories often need to be translated, involving longer code length and more complex functionalities.
“Such benchmarks do not accurately reflect real-world demands, where entire repositories often need to be translated, involving longer code length and more complex functionalities.”
Quote in the paperSubstantial claim - Interpretationown data§ D. Statistics of RepoTransBench · p. 7
Comprehensive test coverage enables reliable execution-based evaluation of translation quality, moving beyond similarity metrics.
“This comprehensive test coverage enables reliable executionbased evaluation of translation quality, moving beyond similarity metrics used in previous benchmarks to assess actual functional correctness.”
Quote in the paperSubstantial claim - Interpretationreasoning§ B. RQ2: Performance Differences Across translation pairs · p. 10
The dramatic performance gap between static-to-dynamic and dynamic-to-static translations reflects the fundamental challenge of inferring static type information, memory management strategies, and explicit interface definitions from Python's dynamic and implicit programming model.
“This dramatic performance gap reflects the fundamental challenge of inferring static type information, memory management strategies, and explicit interface definitions from Python's dynamic and implicit programming model.”
Quote in the paperSubstantial claim - Interpretationreasoning§ B. RQ2: Performance Differences Across translation pairs · p. 10
The translation process must essentially reverse-engineer the implicit contracts and assumptions present in dynamically-typed code.
“The translation process must essentially reverse-engineer the implicit contracts and assumptions present in dynamically-typed code.”
Quote in the paperNot a substantial claim - Interpretationreasoning§ B. RQ2: Performance Differences Across translation pairs · p. 11
The specialized advantages of LLM backbones for specific translation pairs likely reflect training data composition and architectural biases that favor particular programming languages.
“Finding 5: Different LLM backbones demonstrate specialized advantages for specific translation pairs (e.g., o3-mini excelling at C → Rust with 67.2% vs. Claude's 47.5%), likely reflecting training data composition and architectural biases that favor particular programming languages.”
- Also stated in: § IX. CONCLUSION · p. 14 · plain
“We also observe strong directional asymmetry in translation difficulty (static-todynamic achieving 45-63% vs. reverse direction below 10%), model-specific advantages for certain translation pairs reflecting training biases, and inverse correlation between repository complexity and translation success.”
Quote in the paperSubstantial claim - Also stated in: § IX. CONCLUSION · p. 14 · plain
- Interpretationreasoning§ A. RQ1: Performance of LLM-based Translation Methods · p. 10
The worse performance of DeepSeek-Reasoner is due to its excessively long reasoning chains that lead to context management difficulties, request timeouts, and other technical issues.
“which can be attributed to its excessively long reasoning chains that lead to context management difficulties, request timeouts, and other technical issues that hinder effective translation performance.”
- Also stated in: § A. RQ1: Performance of LLM-based Translation Methods · p. 10 · plain
“due to context management and request timeout issues resulting from overly long reasoning chains.”
Quote in the paperSubstantial claim - Also stated in: § A. RQ1: Performance of LLM-based Translation Methods · p. 10 · plain
- Premisenone given§ C. Evaluation Metrics · p. 9
Translations with high similarity to the reference cannot avoid having grammatical errors and functional errors.
“On the one hand, translations with high similarity to the reference cannot avoid having grammatical errors and functional errors.”
Quote in the paperNot a substantial claim - Interpretationown data§ A. RQ1: Performance of LLM-based Translation Methods · p. 9
The consistent improvement of RepoTransAgent across different LLMs validates the effectiveness of the agent-based approach for repository-level code translation.
“This consistent improvement across different LLMs validates the effectiveness of our agent-based approach for repository-level code translation.”
Quote in the paperSubstantial claim - Premisenone given§ C. Evaluation Metrics · p. 9
Equivalent programs with different implementations may have low similarity.
“On the other hand, equivalent programs with different implementations may have low similarity.”
Quote in the paperNot a substantial claim - Interpretationreasoning§ B. RQ2: Performance Differences Across translation pairs · p. 11
o3-mini's training included substantial Rust-related content or system-level programming examples that enhanced its understanding of memory safety patterns and ownership concepts critical for C-to-Rust translation.
“This suggests that o3-mini's training included substantial Rustrelated content or system-level programming examples that enhanced its understanding of memory safety patterns and ownership concepts critical for C-to-Rust translation.”
Quote in the paperNot a substantial claim - Interpretationown data§ A. RQ1: Performance of LLM-based Translation Methods · p. 10
There is a gap between syntactic correctness and complete functional correctness in translation-only methods.
“highlighting the gap between syntactic and complete functional correctness.”
Quote in the paperSubstantial claim - Interpretationreasoning§ B. RQ2: Performance Differences Across translation pairs · p. 10
The high success rates of static-to-dynamic translations stem from static languages providing explicit type information, memory management details, and structured interfaces that can be effectively simplified and adapted to Python's more flexible paradigm.
“This success stems from the fact that static languages provide explicit type information, memory management details, and structured interfaces that can be effectively simplified and adapted to Python's more flexible paradigm.”
Quote in the paperSubstantial claim - Interpretationown data§ D. RQ4: Error Analysis · p. 13
Some LLMs may tend to generate example code with some elements omitted.
“Finding 9: Some LLMs may struggle to continue generating due to limited instruction following capability, and others may tend to generate example code with some elements omitted.”
Quote in the paperNot a substantial claim - Interpretationown data§ D. RQ4: Error Analysis · p. 12
Repository-level code translation requires proper configuration of files such as the 'CMakeLists.txt' file in C++ repositories.
“Finding 7: Unlike fine-grained code translation works, repository-level code translation requires proper configuration of files such as the 'CMakeLists.txt' file in C++ repositories.”
Quote in the paperSubstantial claim - Interpretationown data§ D. RQ4: Error Analysis · p. 13
A lack of understanding of the repository context may result in generating inappropriate code, due to long code length and complex repository structure.
“Finding 8: Due to the long code length and complex repository structure, a lack of understanding of the repository context may result in generating inappropriate code.”
Quote in the paperSubstantial claim - Interpretationown data§ C. RQ3: Impact of Dependency Complexity · p. 12
Repositories with fewer intermodule dependencies are more amenable to successful translation.
“For cross-file dependencies, successful translations consistently exhibit lower median values compared to failed translations, indicating that repositories with fewer intermodule dependencies are more amenable to successful translation.”
Quote in the paperSubstantial claim - Interpretationreasoning§ D. RQ4: Error Analysis · p. 13
Encoding-related problems can be resolved by configuring the correct encoding format of the repository.
“These problems can be resolved by configuring correct encoding format of the repository.”
Quote in the paperNot a substantial claim - Study design and conductreasoning§ VII. THREATS TO VALIDITY · p. 14
The filtering criteria (star count, language composition, executability) may favor well-maintained, popular repositories while excluding experimental or domain-specific projects.
“Our filtering criteria (star count, language composition, executability) may favor well-maintained, popular repositories while excluding experimental or domain-specific projects.”
Quote in the paperSubstantial claim - Interpretationown data§ D. RQ4: Error Analysis · p. 13
Some LLMs may lack sufficient understanding of language features when translating a repository, leading to related issues.
“Finding 10: When translating a repository to another language, some LLMs may lack sufficient understanding of language features, leading to related issues.”
Quote in the paperSubstantial claim - Interpretationreasoning§ D. RQ4: Error Analysis · p. 13
Incomplete generation issues can be automatically identified by dependency analysis and syntax checking.
“These issues can be automatically identified by dependency analysis and syntax checking.”
Quote in the paperSubstantial claim - Interpretationown data§ D. RQ4: Error Analysis · p. 13
Some LLMs may struggle to continue generating due to limited instruction following capability.
“Finding 9: Some LLMs may struggle to continue generating due to limited instruction following capability, and others may tend to generate example code with some elements omitted.”
Quote in the paperNot a substantial claim - Interpretationown data§ IX. CONCLUSION · p. 14
Repository-level translation remains challenging.
“Our evaluation reveals that repositorylevel translation remains challenging, with the best-performing method achieving only a 32.8% success rate.”
Quote in the paperSubstantial claim - Study design and conductown data§ VII. THREATS TO VALIDITY · p. 13
The evaluation includes 8 state-of-the-art LLMs across different categories (open-source, closed-source, and reasoning-focused models).
“While we evaluate 8 state-ofthe-art LLMs across different categories (open-source, closedsource, and reasoning-focused models)”
Quote in the paperNot a substantial claim- Rests on an assumption: The evaluated LLMs represent the current state-of-the-art.
- Interpretationown data§ C. RQ3: Impact of Dependency Complexity · p. 12
Current LLMs struggle systematically with complex repository structures.
“This consistent trend across all complexity dimensions suggests that current LLMs struggle systematically with increased repository complexity, regardless of whether the complexity stems from structural dependencies, code volume, or functional richness.”
- Also stated in: § C. RQ3: Impact of Dependency Complexity · p. 12 · hedged
“Finding 6: Repository complexity across all dimensions (cross-file dependencies, intra-file dependencies, code length, and structural complexity) inversely correlates with translation success, indicating that current LLMs struggle systematically with complex repository structures.”
Quote in the paperSubstantial claim - Also stated in: § C. RQ3: Impact of Dependency Complexity · p. 12 · hedged
- Interpretationreasoning§ D. RQ4: Error Analysis · p. 12
Solving configuration-related problems in repository-level code translation requires a clear understanding of the related calls and the latest dependencies.
“Solving these problems requires a clear understanding of the related calls and the latest dependencies.”
Quote in the paperNot a substantial claim - Study design and conductnone given§ VII. THREATS TO VALIDITY · p. 14
The evaluation primarily relies on execution-based metrics using existing test suites from source repositories.
“Our evaluation primarily relies on execution-based metrics using existing test suites from source repositories.”
Quote in the paperSubstantial claim
- Classified as probably not a substantial claim. 44
- May combine two claims. 42
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ I. INTRODUCTION · p. 1Code translation facilitates refactoring outdated languages, transitions to faster languages, and supports programming language migration.
“It facilitates the refactoring of code written in outdated languages [10], transitions from simpler but slower languages to more complex and faster ones [11], and supports programming language migration in software development [12]-[19].”
Quote in the paperNot a substantial claim1 of 3 sources checkedProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ I. INTRODUCTION · p. 1File-level benchmarks including CodeNet, Avatar, CodeScope, and CodeTransOcean evaluate translation of a complete program file.
“File-level code translation benchmarks which include CodeNet [37], Avatar [38], CodeScope [39] and CodeTransOcean [40] refer to evaluating the translation of a complete program file.”
Quote in the paperNot a substantial claim2 of 3 sources checked- cites [37] Puri, 2021, CodeNet: A large-scale AI for code dataset for learning a diversity of coding tasks source
The ‘Further Uses of CodeNet’ section (p. 19) says CodeNet may be used for program translation, supporting its inclusion as a translation dataset. The passages do not establish that it evaluates translation of a complete program file, or describe Avatar, CodeScope, and CodeTransOcean as file-level benchmarks. (LLM, conf=0.78)
Experimental — accuracy not yet measured - cites [38] source
- cites [39], [40]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim. May combine two claims.
Premisecitation§ I. INTRODUCTION · p. 1Pan et al. manually studied two open-source repositories and found current LLMs struggle to perform code translation for entire repositories.
“Recently, Pan et al. [1] manually study two opensource repositories (Apache Commons CLI [41] and Python Click [42]), and find that current LLMs struggle to perform code translation for entire repositories.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites [1] source
Classified as probably not a substantial claim. May combine two claims.
Premisecitation§ II. BACKGROUND · p. 4Pan et al. found advanced LLMs are largely ineffective at repository-level translation, with success rates of 8.1% for GPT-4 and 0% for the rest of the models.
“Pan et al. [1] attempt to perform mutual conversion between Python and Java projects [41], [42], but find that the advanced LLMs are largely ineffective, with success rates of 8.1% for GPT-4 and 0% for the rest of the models.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites [1] source
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ VIII. RELATED WORK · p. 14AlphaTrans [95] is a neuro-symbolic compositional technique that decomposes repositories into fragments and translates them in reverse call order with GraalVM-based validation.
“AlphaTrans [95] is proposed as a neuro-symbolic compositional technique that decomposes repositories into fragments and translates them in reverse call order with GraalVM-based validation.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites [95] Gong, 2025, Tracy: Benchmarking execution efficiency of LLM-based code translation source
The References section identifies AlphaTrans as a neuro-symbolic compositional approach to repository-level code translation and validation. Our copy does not describe repository fragmentation, translation in reverse call order, or GraalVM-based validation; searches for those details found no supporting passage. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ VIII. RELATED WORK · p. 14Pan et al. [1] ignore resource and configuration files.
“Besides, the resource and configuration files are ignored in this research.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites [1] source
Dataset Collection and Pre-Processing describes translating Java and Python code broken into classes and files, and says comments were removed. It does not say whether resource or configuration files were ignored. Searches for those files and for non-code-file exclusions found no passage that settles the claim. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Classified as probably not a substantial claim.
Premisecitation§ VIII. RELATED WORK · p. 14Pan et al. [1] manually studied two open-source repositories (Apache Commons CLI and Python Click) and found that current LLMs struggle to complete the translation tasks of entire repositories.
“Recently, pan et al. [1] manually study two open-source repositories (Apache Commons CLI [41] and Python Click [42]) and find that current LLMs struggle to complete the translation tasks of entire repositories.”
Quote in the paperSubstantial claim1 of 3 sources checked- cites [1] source
- cites [41] 2025, Apache commons cli
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [42] 2025, Click
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
Findingown data§ RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation · p. 1Repository-level translation remains challenging; the best-performing method achieves only a 32.8% success rate.
“revealing that repositorylevel translation remains challenging, where the best-performing method achieves only a 32.8% success rate.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation · p. 1RepoTransBench is a real-world multilingual repository-level code translation benchmark with 1,897 repository samples across 13 translation pairs covering 7 programming languages and automatic execution-based test suites for compilability and functional correctness.
“we propose a new benchmark, named RepoTransBench, which is a real-world multilingual repository-level code translation benchmark featuring 1,897 real-world repository samples across 13 language pairs with automatically executable test suites.”
- Also stated in: § I. INTRODUCTION · p. 2 · plain
“RepoTransBench encompasses 1,897 repository samples across 13 translation pairs covering 7 programming languages, providing automatic execution-based test suites to evaluate both compilability and functional correctness of translated repositories.”
Quote in the paperSubstantial claim- Also stated in: § I. INTRODUCTION · p. 2 · plain
May combine two claims.
Findingown data§ RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation · p. 1Translation difficulty varies significantly by language pair direction; dynamic-to-static translation is much more challenging than the reverse direction, achieving below 10% versus static-to-dynamic at 45-63%.
“translation difficulty varies significantly by language pair direction, with dynamic-to-static language translation being much more challenging than the reverse direction (achieving below 10% vs. static-to-dynamic at 45-63%).”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Findingnone given§ RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation · p. 1RepoTransAgent is a general agent framework to perform repository-level code translation.
“Besides, we introduce RepoTransAgent, a general agent framework to perform repository-level code translation.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ D. Statistics of RepoTransBench · p. 7JavaScript has the highest coverage (93.56% line, 88.12% branch) and Matlab the lowest (61.33% line, 55.99% branch).
“with JavaScript achieving the highest coverage (93.56% line coverage, 88.12% branch coverage) and Matlab showing the most room for improvement (61.33% line coverage, 55.99% branch coverage).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ A. RQ1: Performance of LLM-based Translation Methods · p. 10TranslationOnly approaches achieve partial success, with compilation rates ranging from 10.9% to 33.3% and nonzero average pass rates.
“However, these approaches can still achieve partial success, with compilation rates ranging from 10.9% (DeepSeek-Reasoner) to 33.3% (o3mini), and some test cases passing as evidenced by nonzero APR values.”
- Also stated in: § A. RQ1: Performance of LLM-based Translation Methods · p. 10 · plain
“However, it can still achieve partial success with some repositories being compilable and passing individual test cases”
Quote in the paperSubstantial claim- Also stated in: § A. RQ1: Performance of LLM-based Translation Methods · p. 10 · plain
May combine two claims.
Findingown data§ B. RQ2: Performance Differences Across translation pairs · p. 11o3-mini demonstrates exceptional performance on C→Rust translation with 67.2% success rate, outperforming Claude (47.5%) and GPT-4.1 (47.5%).
“Most notably, o3-mini demonstrates exceptional performance on C → Rust translation with 67.2% success rate, significantly outperforming leading models like Claude (47.5%) and GPT-4.1 (47.5%) on the same pair.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S4 (based).
Findingnone given§ I. INTRODUCTION · p. 2RepoTransBench demonstrates substantially higher context and dependency complexity than previous benchmarks.
“RepoTransBench demonstrates substantially higher context and dependency complexity than previous benchmarks.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ D. Statistics of RepoTransBench · p. 7The dataset encompasses 970 unique projects, with Python most represented (684 samples from 171 projects) followed by Java (438 samples from 146 projects).
“Our dataset encompasses 970 unique projects, with Python being the most represented source language (684 samples from 171 projects) followed by Java (438 samples from 146 projects).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ D. Statistics of RepoTransBench · p. 6RepoTransBench has an average of 87.9 cross-file dependencies and 1,443.7 intra-file dependencies.
“RepoTransBench has an average of 87.9 cross-file dependencies and 1,443.7 intra-file dependencies.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ I. INTRODUCTION · p. 2RepoTransBench repositories contain an average of 23,966 tokens, 2,394 lines of code, 177 functions, 35 classes, and 163 import statements.
“Our benchmark demonstrates higher complexity than previous work, with repositories containing an average of 23,966 tokens, 2,394 lines of code, 177 functions, 35 classes, and 163 import statements.”
- Also stated in: § D. Statistics of RepoTransBench · p. 6 · boosted
“Each sample contains an average of 177 functions, 35 classes, and 163 import statements, demonstrating the complex interdependencies and architectural patterns inherent in real-world software projects.”
Quote in the paperSubstantial claim- Also stated in: § D. Statistics of RepoTransBench · p. 6 · boosted
May combine two claims.
Findingown data§ B. RQ2: Performance Differences Across translation pairs · p. 11Claude particularly excels at translating to Python, achieving the highest success rates for C → Python, C++ → Python, and Matlab → Python.
“Claude shows strong performance across most translation pairs but particularly excels at translating to Python (achieving the highest success rates for C → Python, C++ → Python, and Matlab → Python)”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingnone given§ A. Data Collection · p. 4Interest in Python → Go translation reflects organizational shifts toward Go's concurrency model and deployment simplicity for backend services.
“The interest in Python → Go translation similarly reflects organizational shifts toward Go's concurrency model and deployment simplicity for backend services.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based). May combine two claims.
Findingnone given§ IV. REPOTRANSAGENT · p. 7RepoTransAgent adopts a holistic understanding of entire software repositories, enabling it to handle complex interdependencies, maintain project structure, and ensure functional correctness during translation.
“RepoTransAgent adopts a holistic understanding of entire software repositories, enabling it to handle complex interdependencies, maintain project structure, and ensure functional correctness during translation.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ B. RQ2: Performance Differences Across translation pairs · p. 11Gemini demonstrates highly inconsistent performance, achieving strong compilation rates for some pairs (95.8% for JavaScript → Python compilation) but near-zero functional success rates.
“In contrast, Gemini demonstrates highly inconsistent performance, achieving strong compilation rates for some pairs (95.8% for JavaScript → Python compilation) but nearzero functional success rates, indicating potential gaps in understanding semantic equivalence across languages.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone given§ C. Multi-Agent-Based Test Suite Construction · p. 5The multi-agent framework comprises four specialized agents: Test Case Generator, Test Case Runner, Coverage Analyst, and Test Case Translator.
“The framework comprises four specialized agents that work collaboratively to ensure high-quality benchmark construction: Test Case Generator, Test Case Runner, Coverage Analyst, and Test Case Translator.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ B. RQ2: Performance Differences Across translation pairs · p. 10Python-to-static-language translations face severe challenges, with Python→Java achieving 0.0-7.0% success rates, Python→C++ 0.0-9.9%, and Python→Rust 1.2-26.3%.
“Conversely, Python-to-static-language translations face severe challenges across all evaluated models. Python → Java achieves only 0.0-7.0% success rates, Python → C++ reaches merely 0.0-9.9%, and Python → Rust performs between 1.226.3%.”
Quote in the paperSubstantial claimMay combine two claims.
Findingnone given§ I. INTRODUCTION · p. 2RepoTransAgent can analyze repository structures, understand cross-file dependencies, manage configuration files, and iteratively refine translations based on execution feedback.
“RepoTransAgent can analyze repository structures, understand cross-file dependencies, manage configuration files, and iteratively refine translations based on execution feedback.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ D. Statistics of RepoTransBench · p. 6RepoTransBench contains 1,897 translation samples across 13 translation pairs, significantly surpassing the scale of previous work.
“Our benchmark contains 1,897 translation samples across 13 translation pairs, significantly surpassing the scale of previous work.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Findingnone given§ I. INTRODUCTION · p. 2RepoTransAgent operates through five core capabilities: reading files, creating files, executing commands, searching content, and marking completion.
“The agent operates through five core capabilities: reading files, creating files, executing commands, searching content, and marking completion, enabling repository-level translation through a reasoning-action loop that adapts to diverse programming language ecosystems.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ A. RQ1: Performance of LLM-based Translation Methods · p. 9With Claude, RepoTransAgent achieves 32.8% SR compared to 0.0% for TranslationOnly and 11.3% for ErrorFeedback; with GPT-4.1, it achieves 32.8% SR versus 0.0% and 15.6% respectively.
“For instance, with Claude, RepoTransAgent achieves 32.8% SR compared to 0.0% for TranslationOnly and 11.3% for ErrorFeedback, while with GPT-4.1, it achieves 32.8% SR versus 0.0% and 15.6% respectively.”
Quote in the paperSubstantial claimMay combine two claims.
Findingreasoning§ B. RQ2: Performance Differences Across translation pairs · p. 10Translations from static languages (C, C++, Java) to Python consistently achieve high success rates across most models (e.g., Claude 61.5% for C→Python, 63.0% for C++→Python, 45.9% for Java→Python; GPT-4.1 54.9%, 55.2%, 58.2% respectively).
“Translations from static languages (C, C++, Java) to Python consistently achieve high success rates across most models: Claude achieves 61.5% for C → Python, 63.0% for C++ → Python, and 45.9% for Java → Python, while GPT-4.1 reaches 54.9%, 55.2%, and 58.2% respectively.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ D. Statistics of RepoTransBench · p. 6Cross-file dependencies range from 2.1 in Matlab to 156.9 in Java, while intra-file dependencies range from 600.9 in C# to 2,494.4 in C.
“cross-file dependencies range from 2.1 in Matlab to 156.9 in Java, while intra-file dependencies range from 600.9 in C# to 2,494.4 in C.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ D. Statistics of RepoTransBench · p. 6The average sample in RepoTransBench contains 23,966 tokens and 2,394 lines of code, representing an increase of over 95× in tokens and 108× in lines compared to xCodeEval (209 tokens and 22 lines per sample).
“The average sample in our benchmark contains 23,966 tokens and 2,394 lines of code, representing a dramatic increase of over 95× in tokens and 108× in lines compared to the largest previous benchmark (xCodeEval with 209 tokens and 22 lines per sample).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ A. RQ1: Performance of LLM-based Translation Methods · p. 9Repository-level code translation remains a challenging task for current LLMs, with the best-performing configuration (RepoTransAgent with Claude and GPT-4.1) achieving only 32.8% success rate.
“The results demonstrate that repository-level code translation remains a challenging task for current LLMs. Even the best-performing configuration (RepoTransAgent with Claude and GPT-4.1) achieves only 32.8% success rate”
- Also stated in: § A. RQ1: Performance of LLM-based Translation Methods · p. 10 · plain
“However, repository-level code translation remains highly challenging for current LLM-based methods, with the best-performing method achieving only a 32.8% success rate.”
Quote in the paperSubstantial claim- Also stated in: § A. RQ1: Performance of LLM-based Translation Methods · p. 10 · plain
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingnone given§ A. Data Collection · p. 4C → Rust translation demand indicates growing adoption of Rust for systems programming.
“Meanwhile, C → Rust translation demand indicates the growing adoption of Rust for systems programming, where developers aim to modernize legacy C codebases while gaining memory safety guarantees.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ A. Data Collection · p. 4The survey received responses from 21 professional developers, comprising 86 requests for specific translation pairs.
“We receive responses from 21 professional developers, comprising 86 requests for specific translation pairs.”
Quote in the paperSubstantial claim- Rests on an assumption: The 21 surveyed professional developers' translation-pair requests represent real-world developer needs.
May combine two claims.
Findingown data§ I. INTRODUCTION · p. 2RepoTransAgent consistently outperforms baseline approaches, demonstrating improvements of up to 21.5% over the error feedback method.
“However, our RepoTransAgent framework consistently outperforms baseline approaches, demonstrating improvements of up to 21.5% over the error feedback method.”
Quote in the paperSubstantial claimMay combine two claims.
Findingnone given§ I. INTRODUCTION · p. 2RepoTransAgent is based on the ReAct (Reasoning + Acting) paradigm and is specifically designed to solve repository-level code translation problems.
“RepoTransAgent addresses general translation challenges through an intelligent agent framework based on the ReAct (Reasoning + Acting) paradigm, specifically designed to solve repository-level code translation problems.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingreasoning§ A. Data Collection · p. 4There is sustained interest in Python → Java translation.
“The sustained interest in Python → Java translation reflects the common enterprise pattern where Python prototypes must be productionized in Java environments for scalability and integration with existing enterprise systems.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ C. RQ3: Impact of Dependency Complexity · p. 12Successful translations typically involve repositories with fewer lines of code, fewer functions, and fewer classes.
“The pattern extends to basic size metrics, where successful translations typically involve repositories with fewer lines of code, fewer functions, and fewer classes.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ IX. CONCLUSION · p. 14There is strong directional asymmetry in translation difficulty, with static-to-dynamic translation achieving 45-63% success and the reverse direction below 10%.
“We also observe strong directional asymmetry in translation difficulty (static-todynamic achieving 45-63% vs. reverse direction below 10%)”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
Premisecitation§ I. INTRODUCTION · p. 1High-quality synthetic data has become essential for further improving LLMs' performance.
“High-quality synthetic data has become essential for further improving LLMs' performance [5], [6].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [5], [6]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ I. INTRODUCTION · p. 1By leveraging code translation, vast amounts of code in various programming languages can be transformed into high-quality training data, offering a viable solution to data scarcity.
“By leveraging code translation, vast amounts of code written in various programming languages can be transformed into high-quality training data, offering a viable solution to this scarcity [7]-[9].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [7]-[9]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S4 (based).
Premisenone given§ I. INTRODUCTION · p. 1Existing repository-level translation studies are severely limited in scope, with most work examining only one or two translation pairs and a handful of repositories.
“Existing repository-level translation studies are severely limited in scope, with most work examining only one or two translation pairs and a handful of repositories.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ I. INTRODUCTION · p. 1With the popularity of LLMs, researchers are trying to translate code with LLMs, yielding promising results.
“With the popularity of large language models (LLMs), researchers are trying to translate code with LLMs, yielding promising results [29]-[31].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [29]-[31]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ I. INTRODUCTION · p. 1Automatic code translation can significantly reduce manual effort.
“Automatic code translation can significantly reduce manual effort and has thus garnered widespread attention in recent years [20][28].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [20][28]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S4 (based). May combine two claims.
Interpretationreasoning§ II. BACKGROUND · p. 4Successful repository translation necessitates appropriate configuration of ecosystem-specific files and comprehensive resource migration beyond source code.
“Additionally, successful repository translation necessitates appropriate configuration of ecosystem-specific files and comprehensive resource migration beyond source code, where resource files that perform I/O operations with code components must be carefully handled and potentially transformed to maintain functional equivalence.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
Interpretationreasoning§ B. RQ2: Performance Differences Across translation pairs · p. 11The specialization patterns likely reflect the LLMs' training data composition and architectural biases.
“likely reflecting their training data composition and architectural biases.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Premisenone given§ I. INTRODUCTION · p. 2Current translation methods often rely on language-specific heuristics tailored to particular translation pairs, making them difficult to generalize across diverse language combinations.
“Current translation methods often rely on language-specific heuristics tailored to particular translation pairs, making them difficult to generalize across diverse programming language combinations.”
Quote in the paperNot a substantial claim- Rests on an assumption: Current translation methods often rely on language-specific heuristics tailored to particular translation pairs.
Classified as probably not a substantial claim. May combine two claims.
Study design and conductnone given§ A. Data Collection · p. 4To ensure repository quality, repositories were ranked by star count and only those with more than 50 stars were retained.
“To ensure repository quality, we rank repositories by star count and retain only those with more than 50 stars.”
Quote in the paperSubstantial claim- Rests on an assumption: A repository's star count above 50 indicates sufficient code quality and practical relevance.
Classified as probably not a substantial claim. May combine two claims.
Study design and conductreasoning§ B. Rule-Based Filtering · p. 5For Python repositories, they required Python code comprises the largest share, the repository exceeds the popularity threshold, and it does not have substantial dependencies on excluded packages such as PyTorch, TensorFlow, and NLTK.
“To illustrate, for Python repositories, we require that (1) Python code comprises the largest share of the codebase, (2) the repository exceeds the popularity threshold, and (3) it does not have substantial dependencies on excluded packages (e.g., PyTorch, TensorFlow, NLTK).”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationreasoning§ B. RQ2: Performance Differences Across translation pairs · p. 11Training data distribution and architectural choices significantly influence model performance on specific language combinations.
“These patterns suggest that training data distribution and architectural choices significantly influence model performance on specific language combinations.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S4 (based).
Premisenone given§ I. INTRODUCTION · p. 2Repository-level code translation benchmark construction is labor-intensive, requiring extensive manual effort to ensure executability, generate test suites, validate correctness, and manage dependencies.
“Repository-level code translation benchmark construction is labor-intensive, requiring extensive manual effort to ensure repository executability, generate comprehensive test suites, validate functional correctness, and manage complex dependencies.”
Quote in the paperNot a substantial claim- Rests on an assumption: Repository-level code translation benchmark construction is labor-intensive.
Probably not a substantial claim: may not pass S4 (based).
Premisenone given§ II. BACKGROUND · p. 4Real-world code repositories typically include numerous functions, classes, and import statements, requiring translators to understand entire repository context and intricate interdependencies across files and modules.
“Real-world code repositories typically include numerous functions, classes, and import statements to realize complex functionalities, requiring translators to understand the entire repository context rather than isolated code fragments with intricate interdependencies between components across multiple files and modules.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
Interpretationreasoning§ II. BACKGROUND · p. 4Repository-level code translation presents three fundamentally different challenges: challenging context management, complex dependency analysis, and difficult environment setup.
“Compared with previous fine-grained code translation granularity, repository-level code translation presents three fundamentally different challenges: challenging context management, complex dependency analysis, and difficult environment setup.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationcitation§ II. BACKGROUND · p. 4Repository-level translation requires fundamentally different approaches from fine-grained translation because current LLM-based methods struggle with large-scale context management, complex dependency analysis, and comprehensive environment configuration.
“This stark performance gap reveals that repository-level translation requires fundamentally different approaches from fine-grained translation, as current LLM-based methods struggle with the challenges of large-scale context management, complex dependency analysis, and comprehensive environment configuration that are inherent to repository-level code translation.”
Quote in the paperSubstantial claimMay combine two claims.
Premisenone given§ D. Statistics of RepoTransBench · p. 6Previous benchmarks typically contain at most 2 functions per sample and rarely include classes or substantial import dependencies.
“In contrast, previous benchmarks typically contain at most 2 functions per sample and rarely include classes or substantial import dependencies.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Study design and conductreasoning§ B. Rule-Based Filtering · p. 5Rule-based filtering retained only repositories where the target language constitutes the largest proportion of the codebase.
“Rule 1: Language Dominance. We retain only repositories where the target language constitutes the largest proportion of the codebase.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.
Premisereasoning§ I. INTRODUCTION · p. 2Real-world code repository migration often requires proper handling of resources and correct configuration management.
“A real-world code repository migration often requires proper handling of resources and correct configuration management.”
Quote in the paperNot a substantial claimMay combine two claims.
Premisenone given§ II. BACKGROUND · p. 3Neural network-based approaches still struggle with cross-file dependencies and large-scale context management, limiting their effectiveness beyond file-level translation.
“However, neural network-based approaches still struggle with cross-file dependencies and large-scale context management, limiting their effectiveness beyond file-level translation.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Premisenone given§ I. INTRODUCTION · p. 2Many programming language repositories contain configuration files and resource files, such as CMakeLists.txt for C++ and pom.xml for Java.
“Many programming language repositories contain configuration files and resource files, such as CMakeLists.txt for C++ and pom.xml for Java.”
Quote in the paperNot a substantial claim- Rests on an assumption: Many programming language repositories contain configuration files and resource files.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductreasoning§ B. Rule-Based Filtering · p. 5Rule-based filtering excluded repositories that depend on packages difficult to translate into target languages.
“Rule 3: Package Exclusion. We exclude repositories that depend on packages difficult to translate into target languages.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Study design and conductreasoning§ B. Rule-Based Filtering · p. 5Rule-based filtering filtered repositories based on a minimum star threshold θ_star.
“Rule 2: Popularity Threshold. We filter repositories based on a minimum star threshold θ star to ensure code quality and practical relevance.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
Study design and conductreasoning§ VII. THREATS TO VALIDITY · p. 14The focus on fundamental language features and well-established frameworks ensures that the findings remain relevant across reasonable time horizons.
“However, our focus on fundamental language features and well-established frameworks ensures that our findings remain relevant across reasonable time horizons.”
Quote in the paperNot a substantial claim- Rests on an assumption: Programming languages, frameworks, and development practices evolve continuously.
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone given§ VII. THREATS TO VALIDITY · p. 14The automated benchmark construction framework can facilitate future updates and extensions to maintain benchmark currency.
“The automated benchmark construction framework we develop can facilitate future updates and extensions to maintain benchmark currency.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationreasoning§ D. RQ4: Error Analysis · p. 13Incomplete generation issues can be fixed by regenerating or replacing them with stronger LLMs.
“Then we can fix these issues by regenerating or replacing them with stronger LLMs.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductreasoning§ VII. THREATS TO VALIDITY · p. 14The study employs consistent experimental settings and analyzes performance patterns across multiple samples rather than isolated instances to mitigate generation variability.
“To mitigate this threat, we employ consistent experimental settings and analyze performance patterns across multiple samples rather than isolated instances.”
Quote in the paperSubstantial claim- Rests on an assumption: Large language models exhibit inherent randomness in their outputs.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone given§ VII. THREATS TO VALIDITY · p. 13No fine-tuning methods were applied to the LLMs specifically for repository-level translation.
“Additionally, we do not apply fine-tuning methods to these LLMs specifically for repository-level translation, which may impact their performance.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductnone given§ VII. THREATS TO VALIDITY · p. 14Passing all test cases provides strong evidence of functional correctness but may not capture all edge cases or guarantee complete semantic equivalence.
“While passing all test cases provides strong evidence of functional correctness, it may not capture all edge cases or guarantee complete semantic equivalence.”
Quote in the paperSubstantial claim- Rests on an assumption: Real-world test suites from production repositories provide more realistic evaluation scenarios compared to artificially constructed benchmarks.
Classified as probably not a substantial claim.
Study design and conductnone given§ VII. THREATS TO VALIDITY · p. 14The evaluation focuses on objective metrics (compilation success, test passage) that are less susceptible to generation variance compared to subjective quality assessments.
“Additionally, our evaluation focuses on objective metrics (compilation success, test passage) that are less susceptible to generation variance compared to subjective quality assessments.”
Quote in the paperSubstantial claim- Rests on an assumption: Large language models exhibit inherent randomness in their outputs.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ VIII. RELATED WORK · p. 14TRACY [96] is a benchmark proposed to evaluate the execution efficiency of function-level code translation.
“TRACY [96] is a benchmark proposed to evaluate the execution efficiency of function-level code translation.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [96]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
Interpretationreasoning§ D. RQ4: Error Analysis · p. 13Providing LLMs with more information about language features in the prompt may help improve translation performance.
“Providing LLMs with more information about the language features in the prompt may help improve the translation performance.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based). May combine two claims.
Study design and conductnone given§ VII. THREATS TO VALIDITY · p. 13The language selection is based on TIOBE rankings and developer survey data, ensuring relevance to real-world translation needs.
“However, our language selection is based on TIOBE rankings and developer survey data, ensuring relevance to real-world translation needs.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Study design and conductreasoning§ VII. THREATS TO VALIDITY · p. 14The bias toward well-maintained, popular repositories is intentional to ensure benchmark quality and practical relevance.
“This bias is intentional to ensure benchmark quality and practical relevance, as successful repository-level translation tools should prioritize handling well-structured, maintainable codebases that represent common development scenarios.”
Quote in the paperSubstantial claim- Rests on an assumption: Successful repository-level translation tools should prioritize handling well-structured, maintainable codebases that represent common development scenarios.
The paper relies on this without support.
Repository-level code translation benchmark construction is labor-intensive.
“Repository-level code translation benchmark construction is labor-intensive, requiring extensive manual effort to ensure repository executability, generate comprehensive test suites, validate functional correctness, and manage complex dependencies.”
The paper relies on this without support.
Current translation methods often rely on language-specific heuristics tailored to particular translation pairs.
“Current translation methods often rely on language-specific heuristics tailored to particular translation pairs, making them difficult to generalize across diverse programming language combinations.”
The paper relies on this without support.
Many programming language repositories contain configuration files and resource files.
“Many programming language repositories contain configuration files and resource files, such as CMakeLists.txt for C++ and pom.xml for Java.”
The paper relies on this without support.
A repository's star count above 50 indicates sufficient code quality and practical relevance.
“To ensure repository quality, we rank repositories by star count and retain only those with more than 50 stars.”
The paper relies on this without support.
The 21 surveyed professional developers' translation-pair requests represent real-world developer needs.
“We receive responses from 21 professional developers, comprising 86 requests for specific translation pairs.”
The paper relies on this without support.
Line and branch coverage metrics indicate high-quality test suites.
“The coverage analysis ensures that our benchmark maintains high-quality standards, with our current dataset achieving an average of 81.89% line coverage and 72.61% branch coverage across all translation pairs as shown in Table II.”
The paper relies on this without support.
The evaluated LLMs represent the current state-of-the-art.
“our evaluation includes leading models from major providers (OpenAI, Anthropic, Google, Alibaba, DeepSeek) that represent the current state-of-the-art”
The paper relies on this without support.
Large language models exhibit inherent randomness in their outputs.
“Large language models exhibit inherent randomness in their outputs, which could affect the reproducibility of our results.”
The paper relies on this without support.
Real-world test suites from production repositories provide more realistic evaluation scenarios compared to artificially constructed benchmarks.
“Real-world test suites from production repositories provide more realistic evaluation scenarios compared to artificially constructed benchmarks.”
The paper relies on this without support.
Successful repository-level translation tools should prioritize handling well-structured, maintainable codebases that represent common development scenarios.
“as successful repository-level translation tools should prioritize handling well-structured, maintainable codebases that represent common development scenarios.”
The paper relies on this without support.
Programming languages, frameworks, and development practices evolve continuously.
“as programming languages, frameworks, and development practices evolve continuously.”
The paper relies on this without support.
The inclusion of diverse language combinations provides insights into various translation scenarios encountered in practice.
“The inclusion of diverse language combinations (e.g., C ↔ Python, Java ↔ Go, Python ↔ Rust) provides insights into various translation scenarios encountered in practice.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Gemini is classified as both open-source and closed-source
MinorThe results discussion lists Gemini among open-source alternatives, whereas the model-selection section explicitly places it among the closed-source LLMs.
“significantly outperforming open-source models such as Qwen3 (16.9%), DeepSeek-Chat (22.5%), and Gemini (11.3%).”
“For closed-source LLMs, we examine four leading models from major AI companies. From Anthropic, we evaluate Claude-Sonnet-4, released in May 2025 with a 200K context window, representing one of the most capable reasoning models available. Google's Gemini-2.5-Flash-Lite”
Translation pairs involve nine distinct languages, not seven
MinorThe seven-language count matches the source languages, but the listed translation pairs also include Go and Rust as targets, making nine distinct languages involved in the benchmark.
“RepoTransBench encompasses 1,897 repository samples across 13 translation pairs covering 7 programming languages”
“Translation pairs: C →{ Python, Rust } , C++ → Python, C# → Java, Java →{ C#, Go, Python } , JavaScript → Python, Matlab → Python, Python →{ C++, Go, Java, Rust } .”
Sample count does not surpass every previous benchmark
MinorThe claimed 1,897 samples do not surpass the 1,942 samples listed for the earlier xCodeEval benchmark.
“Our benchmark contains 1,897 translation samples across 13 translation pairs, significantly surpassing the scale of previous work.”
“| xCodeEval [59] | 2023 | Codeforces [61] | File | Not Req | Execution | 1942 | 209 | 22 | 1 | 0 | 1 |”
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 1× · see citing works
“It is worth noting that many previous works use similarity-based metrics like BLEU [90], CodeBLEU [8], or other metrics which calculate the overlapping tokens between references and translations [10], [12], [40], [91], [92] to evaluate the quality of translations.”
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · see citing works
Moses: open source toolkit for statistical machine translation
Koehn, P., Zens, R., Dyer, C., Bojar, O., Constantin, A., Herbst, E., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., et al. (2007) · Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions - ACL '07 · doi:10.3115/1557769.1557821 · Cited by 4,735 · cited 1× · see citing works
“Aggarwal et al. [10] convert Python 2 code to Python 3 code using trained Moses [54], which is an open-source toolkit for statistical machine translation.”
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · see citing works
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824 · cited 3× · see citing works
“Early learning-based approaches often train a neural network to achieve the ability of code translation [24], [25], [27], [53].”
Self-citationLlama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495 · cited 2× · see citing works
“Existing repository-level translation studies are severely limited in scope, with most work examining only one or two translation pairs and a handful of repositories.”
Self-citationWang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495 · see citing works
Self-citationEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
UniXcoder: Unified Cross-Modal Pre-training for Code Representation
Guo, D., Lu, S., Duan, N., et al. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2203.03850 · Cited by 1,151
Self-citationUnified Pre-training for Program Understanding and GenerationPreprint
Ahmad, W. U., Chakraborty, S., Ray, B., & Chang, K.-W. (2021) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2103.06333 · Cited by 1,000 · see citing works
UniXcoder: Unified Cross-Modal Pre-training for Code Representation
Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., & Yin, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.499 · Cited by 673 · see citing works
Self-citationUnified Pre-training for Program Understanding and Generation
Ahmad, W., Chakraborty, S., Ray, B., & Chang, K.-W. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.211 · Cited by 642 · see citing works
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.04664 · Cited by 536 · see citing works
Self-citationGrattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505 · see citing works
Self-citationGraphCodeBERT: Pre-training Code Representations with Data Flow
Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Liu, S., Zhou, L., Duan, N., Svyatkovskiy, A., Fu, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.08366 · Cited by 480 · see citing works
Self-citationProgram Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388 · see citing works
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., & Xiong, C. (2022) · arXiv · doi:10.48550/arxiv.2203.13474 · Cited by 330 · see citing works
Unsupervised Statistical Machine Translation
Artetxe, M., Labaka, G., & Agirre, E. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1399 · Cited by 321 · cited 2× · see citing works
“Automatic code translation can significantly reduce manual effort and has thus garnered widespread attention in recent years [20], [21], [22], [23], [24], [25], [26], [27], [28].”
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Shen, L., Wang, Z., Wang, A., Li, Y., et al. (2023) · Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3580305.3599790 · Cited by 263 · cited 2× · see citing works
“Automatic code translation can significantly reduce manual effort and has thus garnered widespread attention in recent years [20], [21], [22], [23], [24], [25], [26], [27], [28].”
Self-citationStarCoder: may the source be with you!
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.06161 · Cited by 261 · see citing works
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · arXiv · doi:10.48550/arxiv.2002.08155 · Cited by 243 · see citing works
Self-citationCodeBLEU: a Method for Automatic Evaluation of Code Synthesis
Ren, S., Guo, D., Lu, S., Zhou, L., Liu, S., Tang, D., Sundaresan, N., Zhou, M., Blanco, A., & Ma, S. (2020) · arXiv · doi:10.48550/arxiv.2009.10297 · Cited by 214 · see citing works
Self-citationTree-to-tree neural networks for program translation
Chen, X., Liu, C., & Song, D. (2018) · arXiv · doi:10.48550/arxiv.1802.03691 · Cited by 157 · see citing works
Mining API mapping for language migration
Zhong, H., Thummalapenta, S., Xie, T., Zhang, L., & Wang, Q. (2010) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/1806799.1806831 · Cited by 146 · cited 1× · see citing works
“It facilitates the refactoring of code written in outdated languages [10], transitions from simpler but slower languages to more complex and faster ones [11], and supports programming language migration in software development [12], [13], [14], [15], [16], [17], [18], [19].”
Measuring Coding Challenge Competence With APPS
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.09938 · Cited by 142 · see citing works
Lexical statistical machine translation for language migration
Nguyen, A. T., Nguyen, T. T., & Nguyen, T. N. (2013) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2491411.2494584 · Cited by 136 · cited 3× · see citing works
“It facilitates the refactoring of code written in outdated languages [10], transitions from simpler but slower languages to more complex and faster ones [11], and supports programming language migration in software development [12], [13], [14], [15], [16], [17], [18], [19].”
Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code
Pan, R., Ibrahimzada, A. R., Krishna, R., Sankar, D., Wassi, L. P., Merler, M., Sobolev, B., Pavuluri, R., Sinha, S., & Jabbarvand, R. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639226 · Cited by 133 · see citing works
Barone, A. V. M., & Sennrich, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.02275 · Cited by 123 · see citing works
CodeTransOcean: A Comprehensive Multilingual Benchmark for Code TranslationPreprint
Yan, W., Tian, Y., Li, Y., et al. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2310.04951 · Cited by 120 · see citing works
Phrase-Based Statistical Translation of Programming Languages
Karaivanov, S., Raychev, V., & Vechev, M. (2014) · ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) · doi:10.1145/2661136.2661148 · Cited by 115 · cited 1× · see citing works
“It is worth noting that many previous works use similarity-based metrics like BLEU [90], CodeBLEU [8], or other metrics which calculate the overlapping tokens between references and translations [10], [12], [40], [91], [92] to evaluate the quality of translations.”
(2000) · International Journal of Productivity and Performance Management · doi:10.1108/ws.2000.07949fab.004 · Cited by 104 · see citing works
Exploring and Unleashing the Power of Large Language Models in Automated Code Translation
Yang, Z., Liu, F., Yu, Z., Keung, J. W., Li, J., Liu, S., Hong, Y., Ma, X., Jin, Z., & Li, G. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660778 · Cited by 104 · cited 2× · see citing works
“With the popularity of large language models (LLMs), researchers are trying to translate code with LLMs, yielding promising results [29], [30], [31].”
Statistical learning approach for mining API usage mappings for code migration
Nguyen, A. T., Nguyen, H. A., Nguyen, T. T., & Nguyen, T. N. (2014) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/2642937.2643010 · Cited by 97 · cited 1× · see citing works
“It facilitates the refactoring of code written in outdated languages [10], transitions from simpler but slower languages to more complex and faster ones [11], and supports programming language migration in software development [12], [13], [14], [15], [16], [17], [18], [19].”
CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks
Puri, R., Kung, D. S., Janssen, G., Zhang, W., Domeniconi, G., Zolotov, V., Dolby, J., Chen, J., Choudhury, M., Decker, L., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.12655 · Cited by 92
Self-citationUnsupervised translation of programming languages
Lachaux, M.-A., Roziere, B., Chanussot, L., & Lample, G. (2020) · arXiv · doi:10.48550/arxiv.2006.03511 · Cited by 83 · see citing works
StarCoder 2 and The Stack v2: The Next Generation
Lozhkov, A., Li, R., Allal, L. B., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.19173 · Cited by 74 · cited 1× · see citing works
“In recent years, large language models such as StarCoder [68], [69], SantaCoder [70] and more latest models such as the Llama series [71], [72], [73], the ChatGPT series [74], the DeepSeek series [75], and the Claude series [76] have shown remarkable performance on traditional code translation…”
Yan, W., Liu, H., Wang, Y.-K., Li, Y., Chen, Q., Wang, W., Lin, T., Zhao, W.-S., Zhu, L., Deng, S.-G., et al. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2311.08588 · Cited by 71 · see citing works
Self-citationMigrating code with statistical machine translation
Nguyen, A. T., Nguyen, T. T., & Nguyen, T. N. (2014) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1145/2591062.2591072 · Cited by 62 · cited 1× · see citing works
“However, the training of these models increasingly faces limitations regarding the availability of open-source code data.”
DeepAM: Migrate APIs with Multi-modal Sequence to Sequence Learning
Gu, X., Zhang, H., Zhang, D., & Kim, S. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/514 · Cited by 62 · cited 2× · see citing works
“It facilitates the refactoring of code written in outdated languages [10], transitions from simpler but slower languages to more complex and faster ones [11], and supports programming language migration in software development [12], [13], [14], [15], [16], [17], [18], [19].”
Self-citationSantaCoder: don't reach for the stars!
Allal, L. B., Li, R., Kocetkov, D., Mou, C., Akiki, C., Ferrandis, C. M., Muennighoff, N., Mishra, M., Gu, A., Dey, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.03988 · Cited by 61 · see citing works
Swing to SWT and back: Patterns for API migration by wrapping
Bartolomei, T. T., Czarnecki, K., & Lämmel, R. (2010) · Proceedings/Proceedings - Conference on Software Maintenance · doi:10.1109/icsm.2010.5610429 · Cited by 48 · cited 1× · see citing works
“It facilitates the refactoring of code written in outdated languages [10], transitions from simpler but slower languages to more complex and faster ones [11], and supports programming language migration in software development [12], [13], [14], [15], [16], [17], [18], [19].”
AVATAR: A Parallel Corpus for Java-Python Program Translation
Ahmad, W. U., Tushar, M. G. R., Chakraborty, S., & Chang, K.-W. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.143 · Cited by 46 · cited 5× · see citing works
“File-level code translation benchmarks which include CodeNet [37], Avatar [38], CodeScope [39] and Code-TransOcean [40] refer to evaluating the translation of a complete program file.”
InfluentialAVATAR: A Parallel Corpus for Java-Python Program Translation
Ahmad, W. U., Tushar, M. G. R., Chakraborty, S., & Chang, K.-W. (2021) · arXiv · doi:10.48550/arxiv.2108.11590 · Cited by 46 · see citing works
What is wrong with your code generated by large language models? An extensive study
Dou, S., Jia, H., Wu, S., Zheng, H., Wu, M., Tao, Y., Zhang, M., Chai, M., Fan, J., Xu, N., et al. (2026) · Science China Information Sciences · doi:10.1007/s11432-025-4632-8 · Cited by 44 · cited 2× · see citing works
“Code translation involves converting source code written in one programming language into another language while preserving the original program’s functionality and logic [1], [2], [3], [4].”
Automated Cobol to Java recycling
Mossienko, M. (2003) · Proceedings of the ... European Conference on Software Maintenance and Reengineering/Proceedings of the European Conference on Software Maintenance and Reengineering · doi:10.1109/csmr.2003.1192409 · Cited by 43 · cited 1× · see citing works
“It facilitates the refactoring of code written in outdated languages [10], transitions from simpler but slower languages to more complex and faster ones [11], and supports programming language migration in software development [12], [13], [14], [15], [16], [17], [18], [19].”
Multilingual Code Snippets Training for Program Translation
Zhu, M., Suresh, K., & Reddy, C. K. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21434 · Cited by 39 · see citing works
TRANSAGENT: An LLM-Based Multi-Agent System for Code Translation
Yuan, Z.-Q., Chen, W.-T., Wang, H.-L., Yu, K., Peng, X., & Lou, Y. (2024) · arXiv.org · doi:10.48550/arxiv.2409.19894 · Cited by 39
Maximum Expected Likelihood Estimation for Zero-resource Neural Machine Translation
Zheng, H., Cheng, Y., & Liu, Y. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/594 · Cited by 34 · cited 1× · see citing works
“Zheng et al. [55] propose an approach for zero-resource NMT using maximum expected likelihood estimation.”
Syntax and Domain Aware Model for Unsupervised Program Translation
Liu, F., Li, J., & Zhang, L. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00072 · Cited by 32 · cited 1× · see citing works
“The key contributions of this research are:”
Khan, M. A. M., Bari, M. S., Long, X., et al. (2024) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.18653/v1/2024.acl-long.367 · Cited by 32 · see citing works
CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
Yan, W., Tian, Y., Li, Y., Chen, Q., & Wang, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.337 · Cited by 31 · see citing works
Ibrahimzada, A. R., Ke, K., Pawagi, M., Abid, M. S., Pan, R., Sinha, S., & Jabbarvand, R. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729379 · Cited by 27 · cited 1× · see citing works
“AlphaTrans [94] is proposed as a neuro-symbolic compositional technique that decomposes repositories into fragments and translates them in reverse call order with GraalVM-based validation.”
Code translation with compiler representations
Szafraniec, M., Roziere, B., Leather, H., Charton, F., Labatut, P., & Synnaeve, G. (2022) · arXiv · doi:10.48550/arxiv.2207.03578 · Cited by 26 · see citing works
Using model-driven engineering to automate software language translation
Lano, K., & Siala, H. A. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00419-y · Cited by 24 · cited 1× · see citing works
“With the popularity of large language models (LLMs), researchers are trying to translate code with LLMs, yielding promising results [29], [30], [31].”
XLCoST: A Benchmark Dataset for Cross-lingual Code Intelligence
Zhu, M., Jain, A., Suresh, K., Ravindran, R., Tipirneni, S., & Reddy, C. K. (2022) · arXiv · doi:10.48550/arxiv.2206.08474 · Cited by 24 · see citing works
Data Augmentation for Code Translation with Comparable Corpora and Multiple References
Xie, Y.-Q., Naik, A., Fried, D., & Rosé, C. P. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2311.00317 · Cited by 23
A lightweight approach for migrating web frameworks
Hassan, A. E., & Holt, R. C. (2004) · Information and Software Technology · doi:10.1016/j.infsof.2004.10.002 · Cited by 21 · cited 1× · see citing works
“It facilitates the refactoring of code written in outdated languages [10], transitions from simpler but slower languages to more complex and faster ones [11], and supports programming language migration in software development [12], [13], [14], [15], [16], [17], [18], [19].”
Using machine translation for converting <i>Python</i> <i>2</i> to <i>Python</i> <i>3</i> code
Aggarwal, K., Salameh, M., & Hindle, A. (2015) · doi:10.7287/peerj.preprints.1459v1 · Cited by 20 · cited 4× · see citing works
“It facilitates the refactoring of code written in outdated languages [10], transitions from simpler but slower languages to more complex and faster ones [11], and supports programming language migration in software development [12], [13], [14], [15], [16], [17], [18], [19].”
InfluentialOn the Evaluation of Neural Code Translation: Taxonomy and Benchmark
Jiao, M., Yu, T., Li, X., Qiu, G., Gu, X., & Shen, B. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00114 · Cited by 20 · cited 2× · see citing works
“G-TransEval [67] provides a more fine-grained taxonomy, including token-level, syntax-level, library-level, and algorithm-level, which is part of a function.”
Yan, W., Liu, H., Wang, Y., et al. (2024) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.18653/v1/2024.acl-long.301 · Cited by 18 · see citing works
Self-citationDeepAM: Migrate APIs with Multi-modal Sequence to Sequence LearningPreprint
Gu, X., Zhang, H., Zhang, D., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.07734 · Cited by 16 · see citing works
Self-citationBhattarai, M., Santos, J. E., Jones, S., Biswas, A., Alexandrov, B., & O'Malley, D. (2024) · 2024 IEEE High Performance Extreme Computing Conference (HPEC) · doi:10.1109/hpec62836.2024.10938485 · Cited by 13 · cited 1× · see citing works
“Bhat-tarai et al. [78] enhance code translation in LLMs with few-shot learning via retrieval-augmented generation.”
Khan, M. A. M., Bari, M. S., Do, X. L., Wang, W., Parvez, M. R., & Joty, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.03004 · Cited by 13 · see citing works
Towards Translating Real-World Code with LLMs: A Study of Translating to RustPreprint
Enişer, H. F., Zhang, H., David, C., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.11514 · Cited by 9 · see citing works
Self-citationData Augmentation for Code Translation with Comparable Corpora and Multiple References
Xie, Y., Naik, A., Fried, D., & Rosé, C. P. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.917 · Cited by 6 · see citing works
What's Wrong with Your Code Generated by Large Language Models? An Extensive StudyPreprint
Dou, S., Jia, H., Wu, S., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.06153 · Cited by 6 · see citing works
Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code
Pan, R., Ibrahimzada, A. R., Krishna, R., Sankar, D., Wassi, L. P., Merler, M., Sobolev, B., Pavuluri, R., Sinha, S., & Jabbarvand, R. (2023) · arXiv · Cited by 6
A Survey of Neural Code Intelligence: Paradigms, Advances and BeyondPreprint
Sun, Q., Chen, Z., Xu, F., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.14734 · Cited by 5 · see citing works
GlueTest: Testing Code Translation via Language Interoperability
Abid, M. S., Pawagi, M., Adhikari, S., Cheng, X., Badr, R., Wahiduzzaman, M., Rathi, V., Qi, R., Li, C., Liu, L., et al. (2024) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58944.2024.00061 · Cited by 4 · cited 1× · see citing works
“GlueTest [85] systematically and semiautomatically validates translations for non-trivial libraries.”
Nitin, V., Krishna, R., & Ray, B. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.18574 · Cited by 4 · see citing works
Rectifier: Code Translation with Corrector via LLMsPreprint
Yin, X. L., Ni, C., Nguyen, T. N., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.07472 · Cited by 4 · see citing works
VERT: Verified Equivalent Rust Transpilation with Large Language Models as Few-Shot LearnersPreprint
Yang, A. Z. H., Takashima, Y., Paulsen, B., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.18852 · Cited by 3 · see citing works
Using machine translation for converting Python 2 to Python 3 code
Aggarwal, K., Salameh, M., & Hindle, A. (2015) · doi:10.7287/peerj.preprints.1459 · Cited by 1
Bhattarai, M., Santos, J. E., Jones, S., Biswas, A., Alexandrov, B., & O'Malley, D. (2024) · arXiv
Teaching Machines to Code: Smart Contract Translation with LLMsPreprint
Karanjai, R., Xu, L., & Shi, W. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.09740
TRACE: Evaluating Execution Efficiency of LLM-Based Code TranslationPreprint
Gong, Z., Sun, Z., Huang, D., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.11468
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Unsupervised Statistical Machine Translation
Artetxe, M., Labaka, G., & Agirre, E. (2018) · arXiv · doi:10.48550/arxiv.1809.01272
Shiraishi, M., Cao, Y., & Shinagawa, T. (2024) · arXiv · doi:10.48550/arxiv.2409.10506
Rectifier: Code Translation with Corrector via LLMs
Yin, X., Ni, C., Nguyen, T., et al. (2025) · doi:10.22541/au.174548366.63392632/v1
Flagged first, then by citations
Cited by
Sorted by citations
Koohestani, R., de Bekker, P., Koç, B., & Izadi, M. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3644183 · Cited by 6 · see citing works
LongCodeZip: Compress Long Context for Code Language Models
Shi, Y., Qian, Y., Zhang, H., Shen, B., & Gu, X. (2025) · 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00020 · Cited by 7 · see citing works
Self-citationLongCodeZip: Compress Long Context for Code Language Models
Shi, Y., Qian, Y., Zhang, H., Shen, B., & Gu, X. (2025) · arXiv · Cited by 7 · see citing works
Self-citationIbrahimzada, A. R., Paulsen, B., Kroening, D., & Jabbarvand, R. (2026) · Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3832783.3837486 · Cited by 7 · see citing works
“This report identifies the failing functions and provides actionable feedback for the Translator Agent’s repair step in the next iteration.”
Project-Level C-to-Rust Translation via Pointer Knowledge Graphs
Yuan, Z.-Q., Mao, W., Chen, Z.-F., Shang, X., Wang, C., Lou, Y., & Peng, X. (2025) · Proc. ACM Softw. Eng. · doi:10.1145/3808169 · Cited by 5 · see citing works
Code–Test Co-translation: Towards Practical and Effective Program Migration in the Wild
Li, X., Xie, X., Wu, J., Liu, T., & Wang, H. (2026) · Proceedings of the ACM on Programming Languages · doi:10.1145/3839490 · Cited by 1 · see citing works
“Even repository-level benchmarks, such as [36], provide pre-existing test cases in the target language for translation and evaluation.”
InfluentialRustRepoTrans: Repository-level Context Code Translation Benchmark Targeting Rust
Ou, G., Liu, M., Chen, Y., Wang, Y., Peng, X., & Zheng, Z. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00057 · Cited by 30 · see citing works
Self-citationAgileAgent-Orch: Governance-Aware Agentic Repository Migration within Scaled Agile AWS Pipelines
Penumarthi, S. V. G. (2026) · doi:10.1109/iciice69672.2026.11565314
“RepoTransBench supplies a benchmark, in multiple languages, for translating repositories at a large scale, and points out how tricky it is to keep the arrangement of things in files the same [8].”
RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models
Wang, Y., Wang, S., Wang, Y., Zhang, B., Guo, D., Chen, J., & Zheng, Z. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808131
Self-citationZhang, X., & Yang, B. (2026) · IEEE Access · doi:10.1109/access.2026.3722027 · Cited by 1 · see citing works
“Third, code translation and migration, including IRCoder [14], MigrationBench [15] and RepoTransBench [16], show that migration tasks require repository context and executable validation.”
Trustworthy Programming Language Translation with Large Language Models: A Focused Review
Wen, C., Wang, Q., Guo, Y., et al. (2026) · doi:10.1109/wccis70285.2026.11650848
“Repository-scale translation remains challenging because interfaces, dependencies, and build systems matter as much as local correctness [5], [12].”
SPDZCoder: Combining Expert Knowledge with LLMs for Generating Privacy-Computing Code
Dong, X., Xin, P., Huang, L., Wang, Y., Li, J., & Xu, W. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3845985
“Therefore, we examine existing code-generation benchmarks such as HumanEval [9], MBPP [5], LLMTrans [63], and RepoTransBench [61].”
Self-citationAnchorRepair: A Reference-Guided Dual-Agent Framework for Code Translation Repair
Sun, J., Zhao, Y., Li, F., & Jiang, J. (2026) · IEEE/ACM International Workshop on Automated Program Repair (APR) · doi:10.1145/3843776.3844669
“Multi-agent translation systems [43] and benchmark studies [30] further characterize the difficulty of repository-level translation.”
- Authors
- Wang, Y., Wang, Y., Wang, S., Guo, D., Chen, J., Grundy, J., Liu, X., Ma, Y., Mao, M., Zhang, H., & Zheng, Z.
- Affiliations
- Sun Yat-sen University (CN)
- Monash University (AU)
- Huawei Technologies (China) (CN)
- Chongqing University (CN)
- ORCID
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2026-02
- Volume
- 52
- Issue
- 2
- Pages
- 675-690
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W4417439109
- Semantic Scholar
- e73d76841a8bcfb4707c2a2ad3a8602e32f0a0a5
- Cited by
- 12
- References
- 35
- Citations by year
- 2026: 9 · 2025: 3
- FWCI
- 13.11
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- CCF-Huawei Populus Grove Fund (CCF-HuaweiSE202403)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, test suite execution, error analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-16.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-16.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
