CoChat
Sign Up
Cited by 34Software Engineering ResearchNatural Language Processing TechniquesTopic Modeling

The DOI matches its Crossref record; 76 of the 77 references it cites were found in a registry or an index; 34 of the 46 sources its claims cite were checked for support.

CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation

Open access
Yan, Weixiang, Tian, Yuchen, Li, Yunzhe, Chen, Qian, Wang, Wen·2023·Findings of the Association for Computational Linguistics: EMNLP 2023
DOI 10.18653/v1/2023.findings-emnlp.337
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent code translation techniques exploit neural machine translation models to translate source code from one programming language to another to satisfy production compatibility or to improve efficiency of codebase maintenance.Most existing code translation datasets only focus on a single pair of popular programming languages.To advance research on code translation and meet diverse requirements of real-world applications, we construct CodeTransOcean, a large-scale comprehensive benchmark that supports the largest variety of programming languages for code translation.CodeTransOcean consists of three novel multilingual datasets, namely, MultilingualTrans supporting translations between multiple popular programming languages, NicheTrans for translating between niche programming languages and popular ones, and LLMTrans for evaluating executability of translated code by large language models (LLMs).CodeTransOcean also includes a novel cross-framework dataset, DLTrans, for translating deep learning code across different frameworks.We develop multilingual modeling approaches for code translation and demonstrate their great potential in improving the translation quality of both low-resource and high-resource language pairs and boosting the training efficiency.We also propose a novel evaluation metric Debugging Success Rate@K for program-level code translation.Last but not least, we evaluate LLM ChatGPT on our datasets and investigate its potential for fuzzy execution predictions.We build baselines for CodeTransOcean and analyze challenges of code translation for guiding future research.The CodeTransOcean datasets and code are publicly available at https://github. com/WeixiangYAN/CodeTransOcean.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

CodeTransOcean is a large-scale multilingual and cross-framework code translation benchmark with 270,507 program-level samples across 45 programming languages and 4 deep learning frameworks, plus new datasets and evaluation metrics. Experiments show multilingual modeling improves BLEU over one-to-one baselines, ChatGPT achieves 48.57% DSR@0 on LLMTrans with self-debugging/one-shot gains, and fuzzy execution remains impractical.

Key findings

  • CodeTransOcean contains 270,507 samples (over 200K unit tests), covering 45 programming languages and 4 deep learning frameworks: MultilingualTrans has 30,419 program samples over 8 popular languages; NicheTrans has 236,468 samples from 37 niche languages to 8 popular ones; LLMTrans has 350 samples translating 7 popular languages to Python; DLTrans has 408 coarse-grained and 3,270 fine-grained samples across PyTorch, TensorFlow, MXNet, and Paddle.
  • Multilingual modeling improves average BLEU over the One-to-One baseline: All language pairs rise from 5.19 to 5.31 (Many-to-One, +0.12), 5.81 (Many-to-Many, +0.62), and 6.42 (One-to-Many, +1.23); high-resource pairs rise from 4.68 to 6.18 under One-to-Many, and low-resource pairs from 4.83 to 5.84.
  • On LLMTrans, ChatGPT baseline achieves 48.57% DSR@0; self-debugging raises DSR@1 to 51.43%, DSR@2 to 52.29%, and DSR@3 to 52.57%; the best one-shot strategy reaches 50.29% DSR@0 (+1.72 absolute), while Chain-of-Thought degrades performance (e.g., Experiment #2 falls to 42.57%); CodeT5+_220M achieves DSR@0 of 0.
  • For fuzzy execution, ChatGPT zero-shot gets 59.00% accuracy, 88.57% precision, 20.67% recall, and 33.52% F1; few-shot gets 59.00% accuracy, 93.55% precision, 19.33% recall, and 32.04% F1, with about an 88% tendency to predict code is executable and only 4% ability to predict execution results.

Methods

  • Data construction: MultilingualTrans and NicheTrans were collected from Rosetta Code under CC BY-SA 4.0; DLTrans was derived from Dive into Deep Learning under Apache-2.0. Samples are program-level, balanced across languages/frameworks, and non-overlapping with existing code translation datasets; manual quality assessment of 1K samples per dataset found >90% compilability for MultilingualTrans.
  • Modeling and evaluation: CodeT5+_220M was fine-tuned and compared across One-to-One, One-to-Many, Many-to-One, and Many-to-Many strategies using BLEU; baselines were established for DLTrans and NicheTrans. The paper proposes DSR@K and the AutoTransExecuter pipeline, evaluates ChatGPT (gpt-3.5-turbo) on LLMTrans with prompt, hyperparameter, context, self-debugging, one-shot, and Chain-of-Thought experiments, and tests fuzzy execution on an ExecuteStatus dataset of 300 executable and 300 non-executable samples.
  • Analysis used BLEU, EM, and CodeBLEU for translation quality; DSR@K for executability; and accuracy, precision, recall, and F1 for fuzzy execution prediction.

Limitations

  • The paper states that existing match-based metrics focus solely on semantics and overlook executability and functional equivalence, while execution-based metrics are expensive and carry unresolved overhead and security risks. It also states that AutoTransExecuter currently only supports Python as the target language due to runtime, dependency, compiler-configuration, and security challenges, and that fuzzy execution with ChatGPT is not yet practical.
  • Evident from the text: LLMTrans is limited to 350 samples and only seven popular languages translated to Python; DLTrans experiments use only the coarse-grained subset; the fuzzy execution study uses 600 samples; ChatGPT experiments use gpt-3.5-turbo; and the multilingual modeling experiments use CodeT5+_220M.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/2023.findings-emnlp.337

Cites

76 of 77 references verified0 citation errors · 1 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

34 of 46 cited sources checked · 18 supported · 2 flagged · accuracy not yet measured

  • Premisecitation

    XLCoST is the only existing multilingual code translation dataset.

    “Note that XLCoST (Zhu et al., 2022a) is the only existing multilingual code translation dataset.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Zhu et al., 2022a source

      The Related work section (p. 2) identifies CoST as another code translation dataset covering seven programming languages. That directly contradicts the claim that XLCoST is the only existing multilingual code translation dataset. (LLM, conf=0.94)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
  • Premisecitation

    Existing match-based evaluation metrics for code translation focus solely on semantics, overlooking executability of the code and the functional equivalence under different implementations.

    “Existing match-based evaluation metrics for code translation (Papineni et al., 2002;Ren et al., 2020;Eghbali and Pradel, 2022;Zhou et al., 2023;Tran et al., 2019) focus solely on semantics, overlooking executability of the code and the functional equivalence under different implementations.”
    • Also stated in: § A.1.2 Code Translation Metrics · plain
      “These match-based evaluation metrics can only evaluate the surface form and semantic differences of the code, while neglecting the executability of the code and the functional equivalence of implementation variations.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Papineni et al., 2002 source
    • cites Ren et al., 2020 source

      The Introduction (p. 1) describes n-gram accuracy as token overlap, not a metric focused solely on semantics; it says n-gram accuracy can favor code with logical errors. The Abstract (p. 1) likewise says BLEU neglects syntactic and semantic features. The paper does support the narrower concern that exact-match accuracy underestimates different implementations with the same semantic logic. (LLM, conf=0.88)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites Eghbali and Pradel, 2022 source

      The Introduction explains that BLEU matches token n-grams and can give misleading similarity scores, while §3.2 treats equal execution behavior as the standard for semantic equivalence. This supports the concern that matching need not establish functional equivalence, but not the claim that these metrics focus solely on semantics: the paper also describes metrics using syntax and structural dependencies. It does not establish a general claim about whether the metrics assess executability. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Premisecitation

    Multilingual modeling has not been systematically explored for code translation.

    “Multilingual modeling shows great potential in neural machine translation (Aharoni et al., 2019;Wang et al., 2020;Zhu et al., 2023), but it has not been systematically explored for code translation.”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites Aharoni et al., 2019 source
    • cites Wang et al., 2020 source
    • cites Zhu et al., 2023 source
  • Interpretationcitation

    The performance of ChatGPT is sensitive to the provided contextual example in One-shot learning.

    “The results corroborate previous findings in natural language translation (Peng et al., 2023) that the performance of ChatGPT is sensitive to the provided contextual example in One-shot learning.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Peng et al., 2023 source
  • Study design and conductcitation

    CodeT5+ is used as the backbone based on its superior performance on code understanding and generation evaluations.

    “In this work, we use pretrained CodeT5+ (Wang et al., 2023b) foot_3 as the backbone based on its superior performance on code understanding and generation evaluations reported in (Wang et al., 2023b).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Wang et al., 2023b source
  • Probably not a substantial claim. 40
  • It may state two separate claims as one. 36
  • The quote only partly states this claim. 1

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Premisecitation

    Multilingual modeling has been pivotal in broadening the applicability of neural machine translation, primarily evidenced in enhancing performance of low-resource languages and cross-language transfer learning.

    “Multilingual modeling has been pivotal in broadening the applicability of neural machine translation (Aharoni et al., 2019;Wang et al., 2020;Zhu et al., 2023;Johnson et al., 2017).This is primarily evidenced in enhancing the performance of low-resource languages and cross-language transfer learning (Mohammadshahi et al., 2022;Zoph et al., 2016;Nguyen and Chiang, 2017;Johnson et al., 2017).”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites Aharoni et al., 2019 source
    • cites Wang et al., 2020 source
    • cites Zhu et al., 2023 source

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    Prior works show that prompts can influence the performance of ChatGPT.

    “Prior works show that prompts can influence the performance of ChatGPT (Zhong et al., 2023;Peng et al., 2023;Jiao et al., 2023).”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites Zhong et al., 2023 source
    • cites Peng et al., 2023 source
    • cites Jiao et al., 2023 source

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    ChatGPT demonstrates competitive performance on language generation tasks such as summarization and machine translation.

    “The recent LLM ChatGPT demonstrates competitive performance on language generation tasks such as summarization and machine translation (Yang et al., 2023;Peng et al., 2023;Gao et al., 2023).”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites Yang et al., 2023 source

      Section 4.2.1 describes LLMs as strong at summarization by human evaluation and competent at machine translation, supporting the general performance claim. But its summarization examples concern LLMs such as OPT, not ChatGPT specifically; it also notes that translation performance can trail commercial tools. The paper therefore does not fully establish the claim about ChatGPT on both tasks. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites Peng et al., 2023 source

      The Introduction and Results report competitive machine-translation performance, particularly for high-resource languages and with task-specific prompting. The paper does not report competitive summarization performance; summarization appears only in a reference title in the passages found (page numbers unavailable). (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites Gao et al., 2023 source

      The Introduction and Results (sections 1 and 3.5; page numbers unavailable) examine ChatGPT as an evaluator of summaries, not its performance at generating them. The Related Work section discusses evaluation of translation quality, not ChatGPT's machine-translation performance. Searches for summarization generation and machine translation found no passage supporting the attributed claim. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim.

    Study design and conductcitation

    BLEU is used to measure translation accuracy for the four strategies because CodeBLEU does not cover all eight languages in MultilingualTrans.

    “Since the CodeBLEU metric (Ren et al., 2020) does not cover all eight languages in MultilingualTrans, we use BLEU to measure translation accuracy for the four strategies.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Ren et al., 2020 source

      Section 3 (p. 2) identifies this paper as the source of CodeBLEU, and Section 4 (p. 4) evaluates it on Java and Java-to-C# tasks. Neither those passages nor the passages on AST and data-flow matching (Sections 3.2–3.3, p. 3) establish whether CodeBLEU covers all eight languages in MultilingualTrans. The cited paper therefore supports the identification of the metric, but not the stated reason for choosing BLEU. (LLM, conf=0.83)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim.

    Premisecitation

    In-context learning allows the model to learn from input examples, enabling it to understand and manage each new task.

    “In-context learning (Brown et al., 2020) allows the model to learn from input examples, enabling it to understand and manage each new task.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Brown et al., 2020 source

  • It may state two separate claims as one.

    Premisecitation

    The significant overhead of executing numerous test cases and the heightened security risks during the execution process remain unresolved.

    “the significant overhead of executing numerous test cases and the heightened security risks during the execution process remain unresolved.”
    • Also stated in: § A.6 Prompt Variations · plain
      “Furthermore, due to potential security threats from the execution environment and the code, they need to be run in an isolated sandbox.”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites Kulal et al., 2019 source

      The Problem statement (p. 3) describes a computation budget in which each synthesis trial compiles and executes a program on public test cases. The Introduction (p. 2) reports up to 100 such trials, and the Experiments (p. 7) note that one method requires additional compilation calls. These passages support a computational-cost concern, but do not establish that executing numerous test cases creates significant overhead or discuss security risks during execution. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites Hao et al., 2022 source

      The Backgrounds and Related Work section (p. 2) says the benchmark uses several hand-crafted test cases to check generated code. The Experiment section (pp. 4–5) reports test-case-based metrics. Neither passage addresses execution overhead or security risks; searches for those concerns found no passage that does. (LLM, conf=0.73)

      Experimental — accuracy not yet measured
    • cites Hendrycks et al., 2021 source

      Section 3 (p. 4) says executing and evaluating arbitrary Python code is challenging, while the Introduction (p. 2) and Metrics (p. 7) describe using a large bank of test cases. None of these passages says that test-case execution has significant overhead or that security risks during execution remain unresolved; searches for execution costs and security risks found no such passage. (LLM, conf=0.72)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    CoT helps guide the model to output the final answer step by step.

    “It helps guide the model to output the final answer step by step (Wei et al., 2022;Peng et al., 2023;Kojima et al., 2022).”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites Wei et al., 2022 source
    • cites Peng et al., 2023 source
    • cites Kojima et al., 2022 source

  • It may state two separate claims as one.

    Findingown data

    Self-debugging and one-shot improve ChatGPT performance while chain-of-thought strategies degrade translation accuracy.

    “We find that self-debugging and one-shot improve the performance while chain-of-thought strategies degrade the translation accuracy.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data

    LLMTrans translates seven popular programming languages to Python, totaling 350 program samples.

    “The dataset translates seven popular programming languages to Python, totaling 350 program samples.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim.

    Findingnone given

    The paper proposes a novel evaluation metric Debugging Success Rate@K for program-level code translation.

    “We also propose a novel evaluation metric Debugging Success Rate@K for program-level code translation.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingnone given

    CodeTransOcean includes an unprecedented dataset for translating code across different deep learning frameworks and a dataset and automated pipeline for evaluating LLMs on code translation.

    “It also includes an unprecedented dataset for translating code across different deep learning frameworks and a dataset and an automated pipeline for evaluating LLMs on code translation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Findingnone given

    The work is the first to systematically investigate the potential of ChatGPT in code translation.

    “To the best of our knowledge, our work is the first to systematically investigate the potential of ChatGPT in code translation.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data

    MultilingualTrans contains 30,419 program samples covering eight popular programming languages: C, C++, C#, Java, Python, Go, PHP, and Visual Basic.

    “This dataset contains 30,419 program samples covering eight popular programming languages, namely, C, C++, C#, Java, Python, Go, PHP, and Visual Basic.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim.

    Findingnone given

    The paper develops a fully automated translation-execution-evaluation pipeline AutoTransExecuter.

    “We develop a fully automated translationexecution-evaluation pipeline AutoTransExecuter to support this study.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data

    Multilingual modeling approaches improve translation quality for both low-resource and high-resource language pairs and boost training efficiency.

    “We develop multilingual modeling approaches for code translation and demonstrate their great potential in improving the translation quality of both low-resource and high-resource language pairs and boosting the training efficiency.”
    • Also stated in: § 1 Introduction · boosted
      “Experimental results demonstrate that multilingual modeling significantly improves translation quality for both high-resource and low-resource language pairs and improves training efficiency.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. It may state two separate claims as one.

    Findingown data

    AutoTransExecuter automatically uses LLMs to conduct code translation, execution, debugging, and success-rate calculation.

    “Based on this dataset, we design and implement an automated pipeline, AutoTransExecuterfoot_0 , automatically using LLMs to conduct code translation, execution, debugging, and calculating the success rate.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim.

    Findingnone given

    DSR@K evaluates translation and debugging capabilities of LLMs.

    “We propose DSR@K to evaluate translation and debugging capabilities of LLMs.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data

    ChatGPT still has difficulty in identifying errors in the code, exhibiting about an 88% tendency to predict that the code is executable.

    “The low accuracy, recall and F1 scores show that ChatGPT still has difficulty in identifying errors in the code, exhibiting about an 88% tendency to predict that the code is executable.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds.

    Findingnone given

    ChatGPT has superior code translation capability.

    “We also reveal the superior code translation capability of ChatGPT”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data

    DLTrans includes four deep learning frameworks and spans twelve directions.

    “We construct the DLTrans dataset for Cross-DL-framework Translation, including four deep learning frameworks and spanning twelve directions.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data

    DLTrans has 408 coarse-grained and 3,270 fine-grained samples.

    “The coarse-grained and finegrained datasets have 408 and 3,270 samples, respectively.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim.

    Findingnone given

    The paper provides a comprehensive suite of resources, tools, and baselines for code translation.

    “In summary, we provide a comprehensive suite of resources, tools, and baselines for code translation.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data

    The performance of the polite inquiry prompt is comparable to but still worse than the baseline performance.

    “The performance of polite inquiry prompt is comparable to but still worse than the baseline performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds.

    Findingnone given

    Advanced strategies lead to significant performance gains in code translation.

    “advanced strategies lead to significant performance gains.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data

    The Experiment#2 strategy achieves the best performance, yielding 1.72% absolute gain in DSR@0.

    “The Experiment#2 strategy (provided contextual example has both same source and target languages as the original task) achieves the best performance, yielding 1.72% absolute gain in DSR@0”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Findingreasoning

    DSR@K is the first metric designed to accurately reflect real-world software development scenarios.

    “To the best of our knowledge, DSR@K is the first metric designed to accurately reflect real-world software development scenarios.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: In real-world software development, developers typically ensure functionality by testing and debugging upon completion, rather than writing and testing multiple versions of code.

  • The quote only partly states this claim.

    Findingown data

    Each subsequent round of self-debugging brings further gain.

    “Each subsequent round of selfdebugging brings further gain but DSR begins to plateau after the second debugging round.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingreasoning

    For high-resource and low-resource languages, the ranking of the four strategies is the same as for all language pairs, but there is notable difference in their adaptability across languages of varying resource scales.

    “For High-resource and Low-resource languages, as shown in Table 3, the ranking of the four strategies is the same as for All, but there is notable difference in their adaptability across languages of varying resource scales.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data

    Swapping source and target languages in the training set facilitates data augmentation and training a bidirectional model, achieving significant improvements in translation accuracy on DLTrans and NicheTrans.

    “The experimental results suggest that significant improvements in translation accuracy can be achieved by swapping the source and target languages in the training set to facilitate data augmentation and training a bidirectional model.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data

    CodeT5+_220M Zero-shot is unable to generate executable translation results on LLMTrans with the Many-to-Many strategy, with DSR@0 of 0.

    “We also evaluate the CodeT5+_220M model on LLMTrans with the Many-to-Many strategy and find that DSR@0 is 0, suggesting that CodeT5+_220M Zero-shot is unable to generate executable translation results.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingnone given

    The work is the first to define the cross-DL-framework translation task and construct a corresponding dataset.

    “To the best of our knowledge, our work is the first to define the cross-DLframework translation task and construct a corresponding dataset.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data

    Implementing role assignments, clarifying usage, and polite inquiry in prompts all degrade performance compared to the baseline prompt.

    “As shown in Table 6, implementing role assignments, clarifying usage, and polite inquiry in prompts all degrade the performance compared to the baseline prompt.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Findingnone given

    CodeTransOcean includes various code examples ranging from basic syntactic structures to complex algorithm implementations, as well as building neural networks from scratch and conducting training and inference.

    “CodeTransOcean includes various code examples ranging from basic syntactic structures to complex algorithm implementations, as well as building neural networks from scratch and conducting training and inference.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Interpretationreasoning

    Compared to XLCoST, MultilingualTrans is advantageous in more balanced data distribution across programming languages, practicality of language pairs, and data quality.

    “Compared to XLCoST, MultilingualTrans is advantageous in more balanced data distribution across various programming languages, practicality of language pairs, and data quality.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

    Interpretationnone given

    Cross-DL-framework Translation achieves re-implementation under multiple DL frameworks through an automated process, generates highly readable code, enables secondary development, and provides flexibility in combining advantages of multiple frameworks.

    “Compared to existing cross-framework adaptive technologies, Cross-DLframework Translation achieves re-implementation under multiple DL frameworks through an automated process, which not only generates highly readable code and enables secondary development, but also provides developers with flexibility on combining advantages of multiple frameworks.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: There are significant differences in syntax and dependency libraries between different deep learning frameworks, severely impeding reusability of projects.

  • It may state two separate claims as one.

    Study design and conductnone given

    MultilingualTrans data originates from a programming chrestomathy website, and all data was reviewed and verified by the website.

    “As to data quality, our MultilingualTrans originates from a programming chrestomathy website, with all data already reviewed and verified by the website.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds.

    Study design and conductnone given

    There is no overlap between CodeTransOcean datasets and existing code translation datasets.

    “There is no overlap between CodeTransOcean datasets and existing code translation datasets.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds.

    Premisenone given

    Existing code translation datasets neglect important real-world demands, including modernizing early software systems in niche programming languages and migrating code across different deep learning frameworks.

    “Existing code translation datasets also neglect important demands from real-world applications, including modernizing early software systems developed in niche programming languages and migrating code across different deep learning frameworks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.

    Premisenone given

    Existing cross-DL-framework adaptive technologies such as ONNX require both parties to import and export based on agreed data formats or convert only the final model through computation graphs, and have obvious limitations.

    “Existing cross-DL-framework adaptive technologies such as the ONNXfoot_2 model conversion protocol require both parties to import and export based on agreed data formats or to convert only the final model through the computation graphs.These technologies have obvious limitations.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Premisecitation

    CodeTrans is a small dataset containing only Java-C# parallel data for short code samples.

    “The widely used CodeTrans (Lu et al., 2021) is a small dataset containing only Java-C# parallel data for quite short code samples.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Lu et al., 2021

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim. It may state two separate claims as one.

    Study design and conductown data

    LLMTrans samples were compiled and tested, and execution results were recorded.

    “We compile and test these samples and record the execution results.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.

    Interpretationnone given

    CodeTransOcean covers the largest number of popular and niche programming languages so far with the largest scale.

    “CodeTransOcean covers the largest number of popular and niche programming languages so far with the largest scale.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    Studies illustrate significant disparities in energy consumption and economic costs during training and inference between various deep learning frameworks.

    “Moreover, studies illustrate significant disparities in energy consumption and economic costs during training and inference between various frameworks (Georgiou et al., 2022).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Georgiou et al., 2022 source

  • Probably not a substantial claim.

    Premisereasoning

    Converting code between different deep learning frameworks is challenging due to differences between frameworks, code complexity, structural inconsistencies, and cross-platform compatibility.

    “Converting code between different DL frameworks is challenging, mainly due to differences between frameworks, code complexity, structural inconsistencies, and cross-platform compatibility (more details are in Appendix A.3).”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Interpretationreasoning

    The baseline with the most straightforward prompt produces the best performance, possibly because it provides clear, short, and unambiguous instructions for the task to the model.

    “These results show that the baseline with the most straightforward prompt produces the best performance, possibly because it provides clear, short, and unambiguous instructions for the task to the model.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Study design and conductnone given

    The experiments only use the coarse-grained subset of DLTrans.

    “In this work, we only experiment on the coarse-grained subset.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

    Interpretationnone given

    Fuzzy execution may overcome limitations of existing metrics but requires future research.

    “Moreover, we introduce fuzzy execution that may overcome limitations of existing metrics but requires future research.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Premisereasoning

    Existing code translation evaluation metrics such as EM, BLEU, and CodeBLEU are based on surface-form matching and are not suitable for program-level translation tasks because they cannot reliably evaluate functional correctness of translated code.

    “However, all these metrics are based on surface-form matching (or with some adaptations as for CodeBLEU) and are not suitable for our program-level translation tasks since they cannot reliably evaluate functional correctness of translated code.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Interpretationnone given

    Fuzzy execution with LLMs holds the potential to overcome the deficiencies of current code translation evaluation metrics.

    “Despite this, fuzzy execution with LLMs holds the potential to overcome the deficiencies of current code translation evaluation metrics.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Interpretationreasoning

    High-resource languages take advantage more effectively from shared information across multiple source languages, whereas low-resource languages are less equipped to handle additional uncertainty and noise from shared parameters and rely on larger source language data.

    “High-resource languages can take advantage more effectively from the shared information across multiple source languages; whereas, low-resource languages are relatively less equipped to handle the additional uncertainty and noise introduced by shared parameters, and thus often have to rely on a larger volume of source language data to optimize their benefits.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Interpretationreasoning

    Many-to-Many can be considered as expanding One-to-Many by employing more non-source language data, and parameter sharing may undermine performance.

    “Many-to-Many can be considered as expanding the One-to-Many strategy by employing a greater volume of non-source language data for training.Since the encoder must be attuned to the features of various languages simultaneously under Many-to-Many, parameter sharing may potentially undermine the performance.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Interpretationown data

    A targeted multilingual modeling strategy for code translation can translate multiple language pairs with a single model and achieves better and more stable accuracy than baselines.

    “Overall, our results strongly recommend a targeted multilingual modeling strategy for code translation, as it not only can translate multiple language pairs with a single model, but also achieves better and more stable accuracy than baselines.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Interpretationnone given

    DSR@0 can be used for program-level code translation evaluation for any models.

    “Note that DSR@0 can be used for program-level code translation evaluation for any models.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds.

    Premisenone given

    ChatGPT for code translation has not been systematically explored.

    “However, ChatGPT for code translation has not been systematically explored.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim.

    Study design and conductnone given

    All experiments are performed under the same hyperparameters and environment for comparison.

    “We ensure all experiments are performed under the same hyperparameters and environment for comparison.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. It may state two separate claims as one.

    Study design and conductreasoning

    The original format is kept as much as possible to ensure the compilability of the data while ensuring its accuracy.

    “Given the variations in compilation requirements among programming languages, we keep the original format as much as possible to ensure the compilability of the data while ensuring its accuracy.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Study design and conductnone given

    AutoTransExecuter only supports translation from any source language to Python.

    “AutoTransExecuter only supports translation from any source language to Python.We discuss it in Limitations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

    Premisecitation

    Chain-of-Thought allows the model to simulate an orderly and structured way of thinking by sorting out the thinking process.

    “Chain-of-Thought (CoT) allows the model to simulate an orderly and structured way of thinking by sorting out the thinking process.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Study design and conductreasoning

    A duplicate-file-detection tool is employed to identify and remove duplicate data from the dataset to avoid potential data leakage during model training.

    “Additionally, we employ a duplicate-file-detection tool to identify and remove duplicate data from the dataset to avoid any potential data leakage problems during model training.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Interpretationreasoning

    The rich diversity of CodeTransOcean ensures that it reflects a wide variety of real-world scenarios.

    “This rich diversity ensures that CodeTransOcean reflects a wide variety of real-world scenarios.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may state two separate claims as one.

    Study design and conductnone given

    All automatically collected samples are manually verified, and samples that do not meet the requirements are identified and excluded.

    “We manually verify all automatically collected samples, identify and exclude samples that do not meet the requirements.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    Recent studies have begun to utilize LLMs as evaluation metrics in the field of NLP.

    “Recent studies have begun to utilize LLMs as evaluation metrics in the field of NLP (Chen et al., 2023c;Wang et al., 2023a;Fu et al., 2023;Kocmi and Federmann, 2023;Ji et al., 2023).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Chen et al., 2023c

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Wang et al., 2023a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Fu et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim. It may be too vague to check against evidence.

    Interpretationreasoning

    Achieving fully automated execution requires overcoming many technical and practical difficulties.

    “Therefore, achieving this goal requires overcoming many technical and practical difficulties.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Different programming languages necessitate distinct runtime environments and libraries, making it particularly challenging to automatically detect and install the required dependencies for each code.
    • Rests on an assumption: The configuration methods for compilers vary substantially among different programming languages, which further complicates automated configuration.
    • Rests on an assumption: Fully automated execution systems could be exploited by malicious code, thus necessitating further security measures.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Study design and conductnone given

    Each experiment is run with three different random seeds and the mean and standard deviation of BLEU scores are reported.

    “We run each experiment with three different random seeds and report the mean and standard deviation of BLEU scores.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.

    Premisenone given

    CodeBLEU, CrystalBLEU, and CodeBERTScore have limitations as they only support a limited range of programming languages and cannot be used in general multilingual scenarios.

    “However, CodeBLEU, CrystalBLEU, and CodeBERTScore have limitations as they only support a limited range of programming languages and cannot be used in general multilingual scenarios.”
    Quote in the paperNot a substantial claim
65 of 65

  • The paper relies on this without support.

    The performance of neural models for code translation relies heavily on large-scale high-quality parallel data.

    “The performance of neural models relies heavily on large-scale high-quality parallel data.”

  • The paper relies on this without support.

    There are barriers between different deep learning frameworks during the actual production process.

    “However, there are barriers between different deep learning frameworks during the actual production process.”

  • The paper relies on this without support.

    There are significant differences in syntax and dependency libraries between different deep learning frameworks, severely impeding reusability of projects.

    “However, there are significant differences in syntax and dependency libraries between different frameworks, severely impeding reusability of projectsfoot_1 .”

  • The paper relies on this without support.

    In real-world software development, developers typically ensure functionality by testing and debugging upon completion, rather than writing and testing multiple versions of code.

    “Moreover, in real-world software development scenarios, developers typically ensure the functionality of code by testing and debugging upon completion, rather than writing and testing multiple versions of the code to achieve the expected functionality as measured by the existing pass@k (Kulal et al., 2019) metric.”

  • The paper relies on this without support.

    The reference target code exhibits the same functionality as the source language code but their execution results could differ slightly.

    “Since the reference target code exhibits the same functionality as the source language code but their execution results could differ slightly, EM also does not correlate with DSR@0.”

  • The paper relies on this without support.

    Different programming languages necessitate distinct runtime environments and libraries, making it particularly challenging to automatically detect and install the required dependencies for each code.

    “This is mainly due to the fact that different programming languages necessitate distinct runtime environments and libraries, making it particularly challenging to automatically detect and install the required dependencies for each code.”

  • The paper relies on this without support.

    The configuration methods for compilers vary substantially among different programming languages, which further complicates automated configuration.

    “Moreover, the configuration methods for compilers vary substantially among different programming languages, which further complicates automated configuration.”

  • The paper relies on this without support.

    Fully automated execution systems could be exploited by malicious code, thus necessitating further security measures.

    “In addition, fully automated execution systems could be exploited by malicious code, thus necessitating further security measures.”

  • The paper relies on this without support.

    DSR@K and AutoTransExecuter measure the executability of the code and reflect real-world software development scenarios.

    “Our proposed DSR@K and the automated Auto-TransExecuter aim to measure the executability of the code and reflect the real-world software development scenarios.”
9 of 9

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Not all included samples are program-level

    Minor

    The paper says every CodeTransOcean sample is program-level, but describes a DLTrans subset of 3,270 function- or class-level samples.

  • One-to-Many does not beat One-to-One on every language pair

    Minor

    Table 14 includes counterexamples to the claim of superiority across all experiments: for PHP-to-C, One-to-Many scores 3.78 BLEU versus 4.18 for One-to-One.

  • Manual assessment sample exceeds the size of LLMTrans

    Minor

    Selecting 1,000 samples from each dataset is impossible for LLMTrans, which the paper says contains only 350 samples.

3 of 3

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. Xinyun Chen, Chang Liu, and Dawn Song. 2018. Tree- to-tree neural networks for program translation. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Pro- cessing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 2552-2562.
  2. Alireza Mohammadshahi, Vassilina Nikoulina, Alexan- dre Berard, Caroline Brun, James Henderson, and Laurent Besacier. 2022. Small-100: Introducing shal- low multilingual machine translation model for low- resource languages. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 8348-8359. Association for Computational Linguistics. · DOI

Cited by

Sorted by citations

Related

Metadata

Authors
Yan, W., Tian, Y., Li, Y., Chen, Q., & Wang, W.
Affiliations
  • University of California, Santa Barbara (US)
  • University of Hong Kong (HK)
  • University of Illinois Urbana-Champaign (US)
  • Alibaba Group (Cayman Islands) (KY)
ORCID
Venue
Findings of the Association for Computational Linguistics: EMNLP 2023
Publisher
Association for Computational Linguistics
Published
2023
Pages
5067-5089
DOI
10.18653/v1/2023.findings-emnlp.337
OpenAlex
W4389518960
Cited by
31
References
56
Citations by year
2026: 13 · 2025: 17 · 2024: 1
FWCI
7.98
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
program translation, multilingual programming languages, large language models, low-resource language pairs

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-16, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-16 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review