CoChat
Sign Up
Cited by 135Software Engineering ResearchNatural Language Processing TechniquesSoftware Testing and Debugging Techniques

Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code

Open access
Pan, Rangeet, Ibrahimzada, Ali Reza, Krishna, Rahul, Sankar, Divya, Wassi, Lambert Pouguem, Merler, Michele, Sobolev, Boris, Pavuluri, Raju +2 more·2024·Proceedings of the IEEE/ACM 46th International Conference on Software Engineering
DOI 10.1145/3597503.3639226
Open this paper in CoChat — free accountOpen PDF

Abstract

Code translation aims to convert source code from one programming language (PL) to another. Given the promising abilities of large language models (LLMs) in code synthesis, researchers are exploring their potential to automate code translation. The prerequisite for advancing the state of LLM-based code translation is to understand their promises and limitations over existing techniques. To that end, we present a large-scale empirical study to investigate the ability of general LLMs and code LLMs for code translation across pairs of different languages, including C, C++, Go, Java, and Python. Our study, which involves the translation of 1,700 code samples from three benchmarks and two real-world projects, reveals that LLMs are yet to be reliably used to automate code translation---with correct translations ranging from 2.1% to 47.3% for the studied LLMs. Further manual investigation of unsuccessful translations identifies 15 categories of translation bugs. We also compare LLM-based code translation with traditional non-LLM-based approaches. Our analysis shows that these two classes of techniques have their own strengths and weaknesses. Finally, insights from our study suggest that providing more context to LLMs during translation can help them produce better results. To that end, we propose a prompt-crafting approach based on the symptoms of erroneous translations; this improves the performance of LLM-based code translation by 5.5% on average. Our study is the first of its kind, in terms of scale and breadth, that provides insights into the current limitations of LLMs in code translation and opportunities for improving them. Our dataset---consisting of 1,700 code samples in five PLs with 10K+ tests, 43K+ translated code, 1,748 manually labeled bugs, and 1,365 bug-fix pairs---can help drive research in this area.

AI snapshot

Written from the paper’s full text.

TL;DR

This empirical study evaluates general and code LLMs for translating code across C, C++, Go, Java, and Python, finding that correct translations range from 2.1% to 47.3% and that LLMs are not yet reliable for automating code translation. It contributes a 15-category taxonomy of translation bugs, compares LLM and non-LLM translation, and shows iterative prompt crafting can improve success rates.

Key findings

  • Across 43,379 translations, average success was 11.94% (median 5.3%); GPT-4 was best at 47.3%, followed by StarCoder at 14.5%, while real-world project success was 8.1% for GPT-4 and 0% for the other models.
  • Most unsuccessful translations were compilation errors (77.8%), followed by functional errors (13.4%); data-related bugs accounted for 33.5% of bugs, syntactic/semantic differences 30.5%, and dependency/logic bugs 24.2%.
  • Manual analysis produced a taxonomy of 15 bug categories in five groups, built from 1,748 unsuccessful GPT-4 translations and taking about 630 person-hours.
  • Iterative prompt crafting increased success by 5.5% on average across studied LLMs, with GPT-4 improving by 12.33% at iteration 1 and 1.7% at iteration 2; non-LLM C2Rust achieved 95% success for C-to-Rust versus GPT-4’s 61%, while GPT-4 achieved 54.1% for Java-to-C# versus JavaToCSharp producing no compilable translations.

Methods

  • Empirical study using 1,700 executable code samples from CodeNet, Avatar, EvalPlus, Apache Commons CLI, and Python Click, covering C, C++, Go, Java, and Python, with 6,197 translation problems across 31 language pairs.
  • Seven LLMs were evaluated: GPT-4, Llama 2, TheBloke-Vicuna, TheBloke-Airoboros, StarCoder, CodeGeeX, and CodeGen; a translation was considered successful if it compiled, passed runtime checks, and passed existing tests.
  • Bug taxonomy was built manually by eight labelers in two phases, and prompt crafting was evaluated iteratively on CodeGen, StarCoder, GPT-4, and Llama 2.

Limitations

  • Stated: external validity is limited by the selected PLs, LLMs, and datasets; internal validity is threatened because each translation was performed once, no formal inter-rater reliability metric was used for manual labeling, and automation scripts could contain bugs; construct validity is threatened because weak test suites may let buggy translations pass, especially CodeNet with one test case per sample.
  • Stated: the taxonomy focused on GPT-4 translations, and prompt crafting excluded CodeGeeX, TheBloke-Airoboros, and TheBloke-Vicuna.
  • Evident from the text: real-world evaluation is limited to two projects and only Java/Python, and the paper notes that it is hard to generalize from limited observations on real-world project complexity.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3597503.3639226

Cites

73 of 73 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

4 of 12 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Without adequate context to LLMs via prompt crafting, they may only serve as next code token predictors, without understanding the overall task.

    “However, research has shown that without providing adequate context to LLMs via prompt crafting, they may only serve as”
    Quote in the paperSubstantial claim3 of 4 sources checked
    • cites [22] source

      Section 3.2 (page unavailable) says decoder-only models such as CodeGPT generate autoregressively, but does not link prompt context to task understanding or describe LLMs as merely next-token predictors. The Limitations section (page unavailable) says the authors did not evaluate LLMs. Searches for prompt crafting, adequate context, token prediction, and task understanding found no passage addressing the claim. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites [44] source

      The Related Work section says autoregressive models such as Codex are trained to predict the next token. The RQ3 Code Transformation section reports that relevant examples in prompts improve repair performance. The paper does not establish that, without adequate prompt context, LLMs only predict code tokens or fail to understand the overall task. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites [82] source

      The Introduction (p. 2) says surrounding context is crucial for a correct repair and that poor context representation can miss important relationships. The Approach and Patch Generation sections (pp. 4, 6–7) describe using context to predict code tokens. They do not establish that, without prompt crafting, LLMs merely predict the next token or fail to understand the overall task. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites [92]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingreasoning

    CodeGeeX, a model trained explicitly for code translation, performed poorly.

    “The biggest surprise here is CodeGeeX, a model trained explicitly for code translation.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
  • Findingown data

    Except for GPT-4 and StarCoder, all other models performed poorly.

    “Except for GPT-4 and StarCoder, all other models performed poorly.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
  • Findingown data

    There is no consistent pattern between unsuccessful translations and source or target language.

    “There is no consistent pattern between unsuccessful translations and source/target language”
    Quote in the paperNot a substantial claim
  • Findingown data

    There is a strong correlation between the average number of tests per translation sample and unsuccessful translation, with correlation coefficient r ranging from 0.64 to 0.85 for all models.

    “There is a strong correlation between the average number of tests per translation sample and unsuccessful translation (correlation coefficient 𝑟 ranging from 0.64 to 0.85 for all models).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
  • May combine two claims. 45
  • Classified as probably not a substantial claim. 37
  • The quote only partly states this claim. 3

  • May combine two claims.

    Study design and conductcitation

    We selected five programming languages, guided by programming language popularity while also covering different programming paradigms.

    “To mitigate these threats, we selected five PLs, guided by PL popularity [21] while also covering different programming paradigms.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [21] source

      The Code generation and Background and Terminology sections discuss several programming languages, but neither addresses their popularity or uses popularity to guide language selection. Searches for programming-language popularity and selection of five languages across paradigms found no supporting passage in our copy. The citation is attached specifically to the popularity rationale. (LLM, conf=0.72)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    LLMs are yet to be reliably used to automate code translation, with correct translations ranging from 2.1% to 47.3% for the studied LLMs.

    “Our study, which involves the translation of 1,700 code samples from three benchmarks and two real-world projects, reveals that LLMs are yet to be reliably used to automate code translationwith correct translations ranging from 2.1% to 47.3% for the studied LLMs.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.

  • May combine two claims.

    Findingown data

    Across all LLMs, 43,379 translations produced 11.94% successful translations on average (median 5.3%), with GPT-4 (47.3%) and StarCoder (14.5%) being the best-performing models.

    “We performed 43,379 translations across all LLMs, measuring translation success against the tests provided with the code samples.This produced 11.94% successful translations on average (median 5.3%), with GPT-4 (47.3% success rate) and StarCoder (14.5% success rate) being the best-performing models (details in §3.1).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.

  • May combine two claims.

    Findingown data

    On real-world projects, the LLMs were largely ineffective and failed to translate them, with success rates of 8.1% for GPT-4 and 0% for the rest of the models.

    “On realworld projects, the LLMs were largely ineffective, with success rates of 8.1% for GPT-4 and 0% for the rest of the models.”
    • Also stated in: § 3.1 Effectiveness of LLMs in Code Translation · plain
      “The LLMs fail to translate real-world projects.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.

  • May combine two claims.

    Findingown data

    Prompt crafting increases the success rate by 5.5% on average across the studied LLMs, with the largest improvement of 12% occurring for GPT-4.

    “Our results show prompt crafting increases the success rate by 5.5%, on average, across the studied LLMs, with the largest improvement, of 12%, occurring for GPT-4 (details in §6).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.

  • May combine two claims.

    Findingown data

    Identifying equivalent APIs in the target language or implementing the API functionality can introduce bugs.

    “(2) identifying equivalent APIs in the target language or implementing the API functionality can introduce bugs”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Go and C++ have comparatively stricter syntax, while it is easier for LLMs to generate syntactically correct Python code.

    “Further breakdown of the results per PLs shows that Go and C++ have comparatively stricter syntax, while it is easier for LLMs to generate syntactically correct Python code.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The bug taxonomy is structured into 15 categories and five groups.

    “Further manual investigation of unsuccessful translations identifies 15 categories of translation bugs.”
    • Also stated in: § 1 INTRODUCTION · plain
      “The resulting bug taxonomy is structured into 15 categories and five groups (details in §4.1).”
    • Also stated in: § 4.1.1 Methodology. · plain
      “The entire exercise took about 630 person-hours and produced a taxonomy with 15 bug categories organized into five groups (”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The manual labeling process and the resulting bug taxonomy are reliable and comprehensive.

  • May combine two claims.

    Findingown data

    For C2Rust, the 5% unsuccessful translations were due to compilation errors, the majority of them caused by unused imports.

    “As for C2Rust, the 5% unsuccessful translations were due to compilation errors, the majority of them caused by unused imports.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Most GPT-4 translation bugs (71.8%) are related to unused imports, variables, and other target language-specific violations, followed by incorrect input parsing (10.3%).

    “With a further investigation on the type of translation bugs, we found that most GPT-4 translation bugs (71.8%) are related to unused imports, variables, and other target language-specific violations, followed by incorrect input parsing (10.3%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Using GPT-4, 50% of the translation bugs were due to compilation errors, 39.3% were caused by runtime errors, and 10.7% were due to test failures.

    “Using GPT-4, 50% of the translation bugs were due to compilation errors, 39.3% were caused by runtime errors, and 10.7% were due to test failures.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    We observed cases of broken inheritance relations (resulting in missing behaviors/states) in translated code and incorrect translation of exception handling.

    “We also observed cases of broken inheritance relations (resulting in missing behaviors/states) in translated code and incorrect translation of exception handling.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The most prevalent bugs in real-world projects are model-specific constraints (29.4%), missing imports (27.4%), and the removal of logic from the source code (15.7%).

    “The most prevalent bugs in real-world projects are model-specific constraints (29.4%), missing imports (27.4%), and the removal of logic from the source code (15.7%),”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    None of the translated code using JavaToCSharp can compile and, in fact, require heavy rewriting of library APIs (e.g., System.in and java.util.Scanner).

    “We observed that none of the translated code using JavaToCSharp can compile and, in fact, require heavy rewriting of library APIs (e.g., System.inand java.util.Scanner).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    CxGo outperforms all the open-source models with an accuracy of 62.25%, but falls 10% behind GPT-4.

    “When compared to LLMs, we found that CxGo outperforms all the open-source models with an accuracy of 62.25%, but falls 10% behind GPT-4.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GPT-4 translates overloaded Java methods to Python in different ways, some of which are correct translations, whereas others are erroneous.

    “GPT-4 translates these to Python in different ways, some of which are correct translations, whereas others are erroneous.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    With CxGo, the majority of the bugs are related to input parsing (26.7%) and mismatch of behavior after replacing API call (26.7%).

    “Whereas, with CxGo, the majority of the bugs are related to input parsing (26.7%) and mismatch of behavior after replacing API call (26.7%).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GPT-4 and StarCoder achieve 54.1% and 18.45% success rate in translating Java code to C#.

    “On the contrary, GPT-4 and StarCoder achieve 54.1% and 18.45% success rate in translating Java code to C#.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The pattern of GPT-4 performing better on Commons CLI to Python than Click to Java is evident from successful translations (three for Commons CLI vs. none for Click) and degenerate codegeneration instances (nine for Click vs. none for Commons CLI), as well as the translation bugs observed.

    “This is evident from not only the occurrences of successful translations (three for Commons CLI vs. none for Click) and degenerate codegeneration instances (nine for Click vs none for Commons CLI), but also the translation bugs observed.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For EvalPlus and the real-world projects, the occurrence of incorrect input parsing is lesser (one for EvalPlus and none for the real-world projects).

    “For EvalPlus and the real-world projects, the occurrence of this bug is lesser (one for EvalPlus and none for the real-world projects).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
    • Rests on an assumption: The manual labeling process and the resulting bug taxonomy are reliable and comprehensive.

  • May combine two claims.

    Findingown data

    Among the bugs in real-world projects are removal of logic in the source, inclusion of logic not in source, missing imports, mismatch in API behaviors after translation, mismatch of behaviors after replacing API calls, and incorrect data types.

    “Key among these are: removal of logic in the source (e.g., source methods and field initialization not translated), inclusion of logic not in source (e.g., implementation added for a stubbed method), missing imports, mismatch in API behaviors after translation, mismatch of behaviors after replacing API calls and incorrect data types.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Commons CLI uses method overloading frequently.

    “For instance, Commons CLI uses method overloading frequently.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Choosing the correct data type in the target PL accounts for 34.3% of all data-related bugs and 11.5% of all bugs.

    “Finding 1b: Choosing the correct data type in the target PL is a crucial step that accounts for 34.3% of all data-related bugs and 11.5% of all bugs.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
    • Rests on an assumption: The manual labeling process and the resulting bug taxonomy are reliable and comprehensive.

  • May combine two claims.

    Findingown data

    Although the translation bug categories remain the same, occurrences of bug instances and their distribution vary between crafted benchmarks and real-world projects.

    “Finding 6: Although the translation bug categories remain the same, occurrences of bug instances and their distribution vary between crafted benchmarks and real-world projects.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    All of the C2Rust generated code are unsafe, whereas only three GPT-4-translated code are unsafe.

    “In fact, all of the C2Rust generated code are unsafe, whereas, only three GPT-4-translated code are unsafe.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Executing code generated by CxGo requires specific libraries, i.e., stdio for I/O operations instead of a more generic fmt library.

    “The biggest caveat is that executing code generated by CxGo requires specific libraries, i.e., stdio for I/O operations instead of a more generic fmt library.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Missing imports is the dominant category among incorrect code logic and missing dependencies bugs, being the most frequently occurring bug category (35.5% of the bugs in the group and 8.6% of the bugs overall).

    “Missing imports is the most frequently occurring bug category (35.5% of the bugs in the group and 8.6% of the bugs overall).”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    GPT-4 achieved the highest average successful translation rate (47.3%) across all datasets among the subject LLMs.

    “| Total/Average (All) | - | 1,700 10,858 | - | 6,197 | 8.1% | 2.8% | 14.5% | 47.3% 3.5% | 5.3% | 2.1% | |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    Non-LLM-based code translation approaches have determinism and reasoning, unlike LLM-based approaches.

    “| Determinism and reasoning | ✓ | X |”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    With iter1, there is a substantial reduction in compilation errors for GPT-4, with 58% completely fixed (no error) and 38% transformed to other translation bugs.

    “With 𝑖𝑡𝑒𝑟 1 , we observe a substantial reduction in compilation errors for GPT-4, with 58% completely fixed (no error) and 38% transformed to other translation bugs.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Non-LLM-based code translation approaches have broader context than LLM-based approaches.

    “| Broader context | ✓ | X |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    LLM-based code translation approaches leverage target-language idioms, unlike non-LLM-based approaches.

    “| Leveraging target-language idioms | X | ✓ |”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For other errors, the percentage is lower-45% runtime errors, 48% functional errors, and 43% non-terminating executions.

    “However, for other errors, the percentage is lower-45% runtime errors, 48% functional errors, and 43% non-terminating executions-suggesting these bugs are harder to mitigate.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In both cases, there are a few cases where the outcome of the translation degrades: functional error transforms to runtime/compilation error, or runtime error transforms to compilation error.

    “In both cases, we observe a few cases where the outcome of the translation degrades: i.e., functional error transforms to runtime/compilation error, or runtime error transforms to compilation error.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    Compilation errors account for 77.8% of unsuccessful translations overall.

    “Compilation Errors | 68.9 93.5 76.4 | 56.9 93.2 94.6 77.0 | 61.6 86.7 83.3 82.4 | 55.9 82.4 78.4 96.6 | 57.4 79.9 73.4 86.0 72.4 77.8 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Translating software or code to a modern PL can significantly reduce maintenance effort, improve overall reliability, and boost non-functional properties such as security and performance.

    “Translating the software/code to a modern PL can significantly reduce maintenance effort, improve overall reliability, and boost non-functional properties such as security and performance [1][2][3][4][5]74].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [1][2][3][4][5]74]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The poor performance of CodeGeeX is because information about the translated code (imports, class declaration, and method signature) was excluded from the prompt.

    “We believe this result is because, as mentioned, we excluded information about the translated code (imports, class declaration, and method signature) in the prompt.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    GPT-4 is the best-performing model and, compared to others, its translations exhibit a variety of quality bugs worth investigating.

    “Our initial investigation showed that GPT-4 is the best-performing model and compared to others, its translations exhibit a variety of quality bugs worth investigating.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The manual labeling process and the resulting bug taxonomy are reliable and comprehensive.

  • The quote only partly states this claim.

    Interpretationown data

    Often translated code does not preserve the behavior of the source program.

    “The next most common effect of unsuccessful translation is a functional error (13.4%), demonstrating that often translated code does not preserve the behavior of the source program.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    LLM-based and non-LLM-based code translation techniques provide different and unique advantages and have their own strengths and weaknesses.

    “Our analysis shows that these two classes of techniques have their own strengths and weaknesses.”
    • Also stated in: § 1 INTRODUCTION · hedged
      “The comparison shows that LLM-and non-LLM-based translation techniques provide different and unique advantages, suggesting that an ultimate solution for code translation should combine both techniques (details in §5).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The nature of the bugs is significantly different: non-LLM-based approaches tend to have more bugs related to the custom library, while LLM-based approaches have more syntax and semantics-related bugs.

    “The nature of these bugs is significantly different as the non-LLM-based approaches tend to have more bugs related to the custom library, while LLM-based approaches have more syntax and semantics-related bugs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    This limitation can cause inconsistencies in the translated code, e.g., renaming overloaded methods requires all the related call sites to be updated.

    “Moreover, this limitation can cause inconsistencies in the translated code, e.g., as illustrated for Finding 4, renaming overloaded methods requires all the related call sites to be updated.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning

    Python-to-Java translations can be more error-prone than Java-to-Python translations, in general, because going from a dynamic type system to a static one can be harder for an LLM to reason about.

    “For example, Python-to-Java translations can be more error-prone than Java-to-Python translations, in general, because going from a dynamic type system to a static one can be harder for an LLM to reason about.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given

    Real-world applications have much more complex code than crafted benchmarks, making them much harder for LLMs to translate.

    “However, real-world applications have much more complex code than crafted benchmarks, making them much harder for LLMs to translate.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Data type bugs occur due to incorrect choice of data type, differences between behaviors of equivalent data types across PLs, and differences in type systems of the source and target PLs.

    “These bugs occur due to incorrect choice of the data type, differences between behaviors of equivalent data types across PLs, and differences in type systems of the source and target PLs.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The manual labeling process and the resulting bug taxonomy are reliable and comprehensive.

  • May combine two claims.

    Interpretationreasoning

    Python float uses 64-bit precision whereas Java Float uses 32-bit precision, so the Java code cannot handle a large test data value that works fine in Python.

    “However, the equivalent Java test case (line 2 on right) fails because of a fundamental difference between Python float and Java Float: the former uses 64-bit precision whereas the latter uses 32-bit precision.The Java code thus cannot handle the large test data value, which works fine in Python.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Use of scanf without buffer size specifications can lead to vulnerabilities such as buffer overruns.

    “Specifically, the use of scanf (as in 5 ) without buffer size specifications can lead to vulnerabilities such as buffer overruns.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    C2Rust generates non-idiomatic and unsafe code, whereas GPT-4 tends to generate safer and idiomatic code.

    “Finding 8: C2Rust generates non-idiomatic and unsafe code, whereas GPT-4 tends to generate safer and idiomatic code.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductnone given

    The other datasets used contain fairly rigorous test suites.

    “However, the other datasets used contain fairly rigorous test suites.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Files in real-world projects do not exist as standalone programs, and providing relevant context about inter-file dependencies can help an LLM produce better code.

    “Unlike crafted benchmarks, files in real-world projects do not exist as standalone programs, and providing relevant context about inter-file dependencies can help an LLM produce better code.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    We included several state-of-the-art general and code LLMs.

    “For LLM selection, we included several SOTA general and code LLMs.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    After each labeling round, both labelers met to discuss any discrepancies and resolve them.

    “To mitigate this threat, after each labeling round, both labelers met to discuss any discrepancies and resolve them.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    At the end, each bug was assigned a single mutually agreed label.

    “At the end, each bug was assigned a single mutually agreed label.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Inadequate or weak test suites can cause buggy translations that pass the test suites to be considered correct.

    “This approach comes with the risk that inadequate or weak test suites can cause buggy translations that pass the test suites to be considered correct.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    Non-LLM-based approaches mostly do not leverage target language idioms, which can negatively affect code readability and maintainability.

    “In this respect, non-LLMbased approaches mostly do not leverage target language idioms, which can negatively affect code readability and maintainability.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The authors could not compare against mppSMT, Tree2Tree, and Sharpen due to lack of tool/artifact availability.

    “We also considered other approaches (mppSMT[57], Tree2Tree[29], and Sharpen[18]) but could not compare against them due to lack of tool/artifact availability ( §5).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [57]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [29] source
    • cites [18]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    We thoroughly tested the scripts and spot-checked the results for correctness.

    “To address this threat, we thoroughly tested the scripts and spot-checked the results for correctness.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    In some cases, non-LLM-based approaches lead to security issues (e.g., C to Rust).

    “In some cases, this even leads to security issues (e.g., C to Rust).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    LLMs tend to generate more natural, human-like code.

    “Conversely, LLMs tend to generate more natural, human-like code.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Non-LLM-based approaches, specifically transpilers, have a more comprehensive context of an application.

    “First, non-LLM-based approaches, specifically transpilers, have a more comprehensive context of an application.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    We used multiple tools covering different programming languages for non-LLM-based approaches.

    “Finally, for non-LLM-based approaches, we used multiple tools covering different PLs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    For each translation task, we performed the translation once.

    “As for threats to internal validity, one potential threat is that, for each translation task, we performed the translation once.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    There can be cases where no suitable API exists in the target programming language to translate an API call to.

    “There can be cases where no suitable API exists in the target PL to translate an API call to-techniques that combine code summarization and code synthesis could be investigated to fill such gaps.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    We measured translation correctness using the test cases provided in our datasets.

    “We measured translation correctness using the test cases provided in our datasets, which is a commonly used approach for assessing the quality of generated code.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    We make our artifacts publicly available to enable review and replication of our results.

    “Moreover, we make our artifacts publicly available [7] to enable review and replication of our results.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    We used multiple well-known datasets with different characteristics and also two real-world projects.

    “In terms of datasets, we used multiple well-known datasets with different characteristics and also two real-world projects.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    CodeNet contains one test case per sample and may be susceptible to this threat.

    “We note that one of our datasets, CodeNet [63], contains one test case per sample and may be susceptible to this threat.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [63] source

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The findings of this work show considerable scope for improving open-source and close-source LLMs for code translation.

    “The findings of this work shows considerable scope for improving open-source and close-source LLMs for code translation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    One author went through the entire labeling to ensure consistency among the groups.

    “On top of that, one author went through the entire labeling to ensure consistency among the groups.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    A significant challenge in translating real-world projects is handling library API calls and the differences in the API ecosystem of different programming languages.

    “Finally, a significant challenge in translating real-world projects is handling library API calls and the differences in the API ecosystem of different PLs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    We do not have a formal inter-rater reliability metric for our manual labeling of bugs.

    “Another threat to internal validity is that we do not have a formal inter-rater reliability metric for our manual labeling of bugs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Use of implicit casting (e.g., 0 as libc::c_int) in the method body can cause unexpected behavior.

    “This issue is further exacerbated by the use of implicit casting (e.g., 0 as libc::c_int as in 4 ) in the method body, which can cause unexpected behavior.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Unrestricted use of unsafe block circumvents Rust's safety checks for the code encompassed in that method, increasing the risk of memory safety violations.

    “The unsafe block in main function (see 3 in Figure 4-(b)) circumvents Rust's safety checks for the code encompassed in that method.This increases the risk of memory safety violations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    While translating a module, the variable types, method signatures, folder structure, and dependencies associated with that module are often unavailable to LLMs because of their restriction on the token size.

    “On the other hand, while translating a module, the variable types, method signatures, folder structure, and dependencies associated with that module are often unavailable to LLMs because of their restriction on the token size.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Performing a translation task multiple times may change the success rates in translation as LLMs are inherently nondeterministic.

    “Performing a translation task multiple times may change the success rates in translation as LLMs are inherently nondeterministic.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: LLMs are inherently nondeterministic.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Our results may be affected by bugs in our automation scripts.

    “Finally, our results may be affected by bugs in our automation scripts.”
    Quote in the paperNot a substantial claim
76 of 76

  • The paper relies on this without support.

    There is a dearth of research on understanding and cataloging the limitations of LLMs for code translation.

    “Although prior research has shown the potential of using LLMs for code translation, there is a dearth of research on understanding and cataloging their limitations for this task.”

  • The paper relies on this without support.

    Code translation is a complex task that requires LLMs to understand code syntax and semantics simultaneously.

    “This is an important undertaking because code translation is a complex task that requires LLMs to understand code syntax (to generate syntactically correct code) and semantics (to preserve functionality during translation) simultaneously.”

  • The paper relies on this without support.

    The selected datasets and real-world projects are comprehensive enough to understand the nature of LLM translation bugs.

    “To ensure the comprehensiveness of our findings and claims in understanding the nature of LLM translation bugs, we were interested in datasets used in prior studies as well as real-world projects.”

  • The paper relies on this without support.

    The tests provided with the code samples are adequate to validate code translation.

    “These datasets are accompanied by test cases to validate code translation.”

  • The paper relies on this without support.

    The selected real-world projects are reasonably complex and well-maintained software exclusively written in Java or Python.

    “Our goal was to translate reasonably complex and well-maintained software exclusively written in Java or Python.”

  • The paper relies on this without support.

    The manual labeling process and the resulting bug taxonomy are reliable and comprehensive.

    “The manual construction of the taxonomy involved eight human labelers, who are researchers or software engineers in the industry, and involved unsuccessful translations from all 31 translation pairs in Table 2.”

  • The paper relies on this without support.

    Regardless of randomness or number of repetitions, the nature of an unsuccessful translation remains unchanged.

    “Regardless of randomness or number of repetitions, the nature of an unsuccessful translation remains unchanged.”

  • The paper relies on this without support.

    LLMs are inherently nondeterministic.

    “Performing a translation task multiple times may change the success rates in translation as LLMs are inherently nondeterministic.”

  • The paper relies on this without support.

    Non-LLM-based techniques are generally deterministic.

    “Second, non-LLMbased techniques are generally deterministic, allowing for more predictable reasoning, whereas LLMs are inherently probabilistic, with a greater degree of creativity.”
9 of 9

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Pan et al. [1] do not provide a sufficient number of repositories and corresponding automatic test suites for evaluation.

    “However, they do not provide a sufficient number of repositories and corresponding automatic test suites for evaluation.”

    From RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation (2026) · cites this paper as [1]

    The Dataset Collection and Pre-Processing section identifies only two real-world projects, Apache Commons CLI and Click, which supports the concern about the number of repositories. But it says the projects have unit tests and does not assess whether the number of repositories or corresponding test suites is sufficient for evaluation. It also notes that the translated projects never reached test execution because the code was syntactically incorrect. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Pan et al. manually studied two open-source repositories and found current LLMs struggle to perform code translation for entire repositories.

    “Recently, Pan et al. [1] manually study two opensource repositories (Apache Commons CLI [41] and Python Click [42]), and find that current LLMs struggle to perform code translation for entire repositories.”

    From RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation (2026) · cites this paper as [1]

  • Pan et al. found advanced LLMs are largely ineffective at repository-level translation, with success rates of 8.1% for GPT-4 and 0% for the rest of the models.

    “Pan et al. [1] attempt to perform mutual conversion between Python and Java projects [41], [42], but find that the advanced LLMs are largely ineffective, with success rates of 8.1% for GPT-4 and 0% for the rest of the models.”

    From RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation (2026) · cites this paper as [1]

  • Pan et al. [1] manually studied two open-source repositories (Apache Commons CLI and Python Click) and found that current LLMs struggle to complete the translation tasks of entire repositories.

    “Recently, pan et al. [1] manually study two open-source repositories (Apache Commons CLI [41] and Python Click [42]) and find that current LLMs struggle to complete the translation tasks of entire repositories.”

    From RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation (2026) · cites this paper as [1]

  • Pan et al. [1] ignore resource and configuration files.

    “Besides, the resource and configuration files are ignored in this research.”

    From RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation (2026) · cites this paper as [1]

    Dataset Collection and Pre-Processing describes translating Java and Python code broken into classes and files, and says comments were removed. It does not say whether resource or configuration files were ignored. Searches for those files and for non-code-file exclusions found no passage that settles the claim. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Real-world translations are reported successful despite no test execution

    Major

    The paper says real-world project translations never reached test execution, yet Table 2 reports a 13.6% successful translation rate for Commons CLI. Its stated success criterion requires existing tests to pass, so those results cannot both hold under that criterion.

  • GPT-4 failure count is reported as 1,748 and 1,558

    Minor

    The methodology calls 1,748 GPT-4 translations unsuccessful, but the prevalence analysis reports 1,558 translation failures and identifies 1,748 as the number of bugs found in them. These are different units of analysis, since a failed translation can have multiple bugs.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 5

  1. Xinyun Chen, Chang Liu, and Dawn Song. 2018. Tree-to-tree neural networks for program translation. Advances in neural information processing systems 31 (2018).
  2. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. StarCoder: may the source be with you! arXiv preprint arXiv:2305.06161 (2023). · arXiv
  3. Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, et al. 2021. CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
  4. Yuanbo Wen, Qi Guo, Qiang Fu, Xiaqing Li, Jianxing Xu, Yanlin Tang, Yongwei Zhao, Xing Hu, Zidong Du, Ling Li, et al. 2022. BabelTower: Learning to Auto-parallelized Program Translation. In International Conference on Machine Learning. PMLR, 23685--23700.
  5. WG Zhang, Arne J Berre, Dumitru Roman, and Hans Aage Huru. 2009. Migrating legacy applications to the service Cloud. In Proceedings of the 14th Conference Companion on Object Oriented Programming Systems Languages and Applications. 59--68.

Cited by

Sorted by citations

Related

Authors
Pan, R., Ibrahimzada, A. R., Krishna, R., Sankar, D., Wassi, L. P., Merler, M., Sobolev, B., Pavuluri, R., Sinha, S., & Jabbarvand, R.
Affiliations
  • IBM (United States) (US)
  • University of Illinois Urbana-Champaign (US)
ORCID
Venue
Proceedings of the IEEE/ACM 46th International Conference on Software Engineering
Publisher
ACM
Published
2024-04-12
Pages
1-13
DOI
10.1145/3597503.3639226
OpenAlex
W4394769102
Cited by
133
References
33
Citations by year
2026: 43 · 2025: 72 · 2024: 18
FWCI
51.11
Type
conference paper
Language
en
Open access
Licence
Locations
IEEE/ACM International Conference on Software Engineering (ICSE) (publishedVersion, cc-by-nc)
Funders
National Science Foundation
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
program translation, large language models, bug classification, prompt engineering, program synthesis, programming language translation, LLM limitations

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-16.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-16.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review