The DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index; 4 of the 12 sources its claims cite were checked for support.
Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code
Open accessAbstract
Code translation aims to convert source code from one programming language (PL) to another. Given the promising abilities of large language models (LLMs) in code synthesis, researchers are exploring their potential to automate code translation. The prerequisite for advancing the state of LLM-based code translation is to understand their promises and limitations over existing techniques. To that end, we present a large-scale empirical study to investigate the ability of general LLMs and code LLMs for code translation across pairs of different languages, including C, C++, Go, Java, and Python. Our study, which involves the translation of 1,700 code samples from three benchmarks and two real-world projects, reveals that LLMs are yet to be reliably used to automate code translation---with correct translations ranging from 2.1% to 47.3% for the studied LLMs. Further manual investigation of unsuccessful translations identifies 15 categories of translation bugs. We also compare LLM-based code translation with traditional non-LLM-based approaches. Our analysis shows that these two classes of techniques have their own strengths and weaknesses. Finally, insights from our study suggest that providing more context to LLMs during translation can help them produce better results. To that end, we propose a prompt-crafting approach based on the symptoms of erroneous translations; this improves the performance of LLM-based code translation by 5.5% on average. Our study is the first of its kind, in terms of scale and breadth, that provides insights into the current limitations of LLMs in code translation and opportunities for improving them. Our dataset---consisting of 1,700 code samples in five PLs with 10K+ tests, 43K+ translated code, 1,748 manually labeled bugs, and 1,365 bug-fix pairs---can help drive research in this area.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This empirical study evaluates general and code LLMs for translating code across C, C++, Go, Java, and Python, finding that correct translations range from 2.1% to 47.3% and that LLMs are not yet reliable for automating code translation. It contributes a 15-category taxonomy of translation bugs, compares LLM and non-LLM translation, and shows iterative prompt crafting can improve success rates.
Key findings
- Across 43,379 translations, average success was 11.94% (median 5.3%); GPT-4 was best at 47.3%, followed by StarCoder at 14.5%, while real-world project success was 8.1% for GPT-4 and 0% for the other models.
- Most unsuccessful translations were compilation errors (77.8%), followed by functional errors (13.4%); data-related bugs accounted for 33.5% of bugs, syntactic/semantic differences 30.5%, and dependency/logic bugs 24.2%.
- Manual analysis produced a taxonomy of 15 bug categories in five groups, built from 1,748 unsuccessful GPT-4 translations and taking about 630 person-hours.
- Iterative prompt crafting increased success by 5.5% on average across studied LLMs, with GPT-4 improving by 12.33% at iteration 1 and 1.7% at iteration 2; non-LLM C2Rust achieved 95% success for C-to-Rust versus GPT-4’s 61%, while GPT-4 achieved 54.1% for Java-to-C# versus JavaToCSharp producing no compilable translations.
Methods
- Empirical study using 1,700 executable code samples from CodeNet, Avatar, EvalPlus, Apache Commons CLI, and Python Click, covering C, C++, Go, Java, and Python, with 6,197 translation problems across 31 language pairs.
- Seven LLMs were evaluated: GPT-4, Llama 2, TheBloke-Vicuna, TheBloke-Airoboros, StarCoder, CodeGeeX, and CodeGen; a translation was considered successful if it compiled, passed runtime checks, and passed existing tests.
- Bug taxonomy was built manually by eight labelers in two phases, and prompt crafting was evaluated iteratively on CodeGen, StarCoder, GPT-4, and Llama 2.
Limitations
- Stated: external validity is limited by the selected PLs, LLMs, and datasets; internal validity is threatened because each translation was performed once, no formal inter-rater reliability metric was used for manual labeling, and automation scripts could contain bugs; construct validity is threatened because weak test suites may let buggy translations pass, especially CodeNet with one test case per sample.
- Stated: the taxonomy focused on GPT-4 translations, and prompt crafting excluded CodeGeeX, TheBloke-Airoboros, and TheBloke-Vicuna.
- Evident from the text: real-world evaluation is limited to two projects and only Java/Python, and the paper notes that it is hard to generalize from limited observations on real-world project complexity.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
73 Own result · 46 Own interpretation · 12 Own method · 11 No basis given · 3 Sources not found · 1 Partly supported · 1 Couldn’t confirm · 1 No full text of the sources · 1 Not checked yet · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, research has shown that without providing adequate context to LLMs via prompt crafting, they may only serve as”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 1 INTRODUCTION
What the sources say
- Could not verifycites [22] sourceExperimental — accuracy not yet measured
Section 3.2 (page unavailable) says decoder-only models such as CodeGPT generate autoregressively, but does not link prompt context to task understanding or describe LLMs as merely next-token predictors. The Limitations section (page unavailable) says the authors did not evaluate LLMs. Searches for prompt crafting, adequate context, token prediction, and task understanding found no passage addressing the claim. (LLM, conf=0.76)
- Partly supportedcites [44] sourceExperimental — accuracy not yet measured
The Related Work section says autoregressive models such as Codex are trained to predict the next token. The RQ3 Code Transformation section reports that relevant examples in prompts improve repair performance. The paper does not establish that, without adequate prompt context, LLMs only predict code tokens or fail to understand the overall task. (LLM, conf=0.85)
- Partly supportedcites [82] sourceExperimental — accuracy not yet measured
The Introduction (p. 2) says surrounding context is crucial for a correct repair and that poor context representation can miss important relationships. The Approach and Patch Generation sections (pp. 4, 6–7) describe using context to predict code tokens. They do not establish that, without prompt crafting, LLMs merely predict the next token or fail to understand the overall task. (LLM, conf=0.83)
- Not matchedcites [92]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The biggest surprise here is CodeGeeX, a model trained explicitly for code translation.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.1 Effectiveness of LLMs in Code Translation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Except for GPT-4 and StarCoder, all other models performed poorly.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.1 Effectiveness of LLMs in Code Translation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There is no consistent pattern between unsuccessful translations and source/target language”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.1 Effectiveness of LLMs in Code Translation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There is a strong correlation between the average number of tests per translation sample and unsuccessful translation (correlation coefficient 𝑟 ranging from 0.64 to 0.85 for all models).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.1 Effectiveness of LLMs in Code Translation
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
In the paper
“That said, further manual investigation showed that for the Commons CLI, three out of 22 translated files could be compiled using py_compile [17].”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.1 Effectiveness of LLMs in Code Translation
What the sources say
- Not matchedcites [17]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(3) replacing language-specific features, such as method overloading and annotations, can be challenging, especially in real-world projects.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“but translating to Go results in more compilation errors due to its strict syntax constraints, e.g., forbidding unused variables or imports.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.1 Effectiveness of LLMs in Code Translation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Some notable findings are: (1) identifying suitable data type in the target PL that preserves the source behavior is challenging”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From these results, we observe that most unsuccessful translations result in compilation errors (77.8%), meaning both general and code LLMs have difficulty understanding code syntax.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.2 Outcome of Unsuccessful Translations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
- It may state two separate claims as one. 45
- Probably not a substantial claim. 37
- The quote only partly states this claim. 3
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“To mitigate these threats, we selected five PLs, guided by PL popularity [21] while also covering different programming paradigms.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
What the sources say
- Could not verifycites [21] sourceExperimental — accuracy not yet measured
The Code generation and Background and Terminology sections discuss several programming languages, but neither addresses their popularity or uses popularity to guide language selection. Searches for programming-language popularity and selection of five languages across paradigms found no supporting passage in our copy. The citation is attached specifically to the popularity rationale. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our study, which involves the translation of 1,700 code samples from three benchmarks and two real-world projects, reveals that LLMs are yet to be reliably used to automate code translationwith correct translations ranging from 2.1% to 47.3% for the studied LLMs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Abstract
plainn = 1,700- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We performed 43,379 translations across all LLMs, measuring translation success against the tests provided with the code samples.This produced 11.94% successful translations on average (median 5.3%), with GPT-4 (47.3% success rate) and StarCoder (14.5% success rate) being the best-performing models (details in §3.1).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On realworld projects, the LLMs were largely ineffective, with success rates of 8.1% for GPT-4 and 0% for the rest of the models.”
- Also stated in: § 3.1 Effectiveness of LLMs in Code Translation · plain
“The LLMs fail to translate real-world projects.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
- Also stated in: § 3.1 Effectiveness of LLMs in Code Translation · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results show prompt crafting increases the success rate by 5.5%, on average, across the studied LLMs, with the largest improvement, of 12%, occurring for GPT-4 (details in §6).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(2) identifying equivalent APIs in the target language or implementing the API functionality can introduce bugs”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further breakdown of the results per PLs shows that Go and C++ have comparatively stricter syntax, while it is easier for LLMs to generate syntactically correct Python code.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.2 Outcome of Unsuccessful Translations
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further manual investigation of unsuccessful translations identifies 15 categories of translation bugs.”
- Also stated in: § 1 INTRODUCTION · plain
“The resulting bug taxonomy is structured into 15 categories and five groups (details in §4.1).”
- Also stated in: § 4.1.1 Methodology. · plain
“The entire exercise took about 630 person-hours and produced a taxonomy with 15 bug categories organized into five groups (”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: The manual labeling process and the resulting bug taxonomy are reliable and comprehensive.
- Also stated in: § 1 INTRODUCTION · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As for C2Rust, the 5% unsuccessful translations were due to compilation errors, the majority of them caused by unused imports.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With a further investigation on the type of translation bugs, we found that most GPT-4 translation bugs (71.8%) are related to unused imports, variables, and other target language-specific violations, followed by incorrect input parsing (10.3%).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using GPT-4, 50% of the translation bugs were due to compilation errors, 39.3% were caused by runtime errors, and 10.7% were due to test failures.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also observed cases of broken inheritance relations (resulting in missing behaviors/states) in translated code and incorrect translation of exception handling.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.3 Translation Bugs in Real-world Projects
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The most prevalent bugs in real-world projects are model-specific constraints (29.4%), missing imports (27.4%), and the removal of logic from the source code (15.7%),”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 6:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observed that none of the translated code using JavaToCSharp can compile and, in fact, require heavy rewriting of library APIs (e.g., System.inand java.util.Scanner).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When compared to LLMs, we found that CxGo outperforms all the open-source models with an accuracy of 62.25%, but falls 10% behind GPT-4.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 translates these to Python in different ways, some of which are correct translations, whereas others are erroneous.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.3 Translation Bugs in Real-world Projects
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Whereas, with CxGo, the majority of the bugs are related to input parsing (26.7%) and mismatch of behavior after replacing API call (26.7%).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the contrary, GPT-4 and StarCoder achieve 54.1% and 18.45% success rate in translating Java code to C#.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is evident from not only the occurrences of successful translations (three for Commons CLI vs. none for Click) and degenerate codegeneration instances (nine for Click vs none for Commons CLI), but also the translation bugs observed.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 5:
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For EvalPlus and the real-world projects, the occurrence of this bug is lesser (one for EvalPlus and none for the real-world projects).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.2 Prevalence of LLM-based Translation Bugs
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
- Rests on an assumption: The manual labeling process and the resulting bug taxonomy are reliable and comprehensive.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Key among these are: removal of logic in the source (e.g., source methods and field initialization not translated), inclusion of logic not in source (e.g., implementation added for a stubbed method), missing imports, mismatch in API behaviors after translation, mismatch of behaviors after replacing API calls and incorrect data types.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.3 Translation Bugs in Real-world Projects
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For instance, Commons CLI uses method overloading frequently.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.3 Translation Bugs in Real-world Projects
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finding 1b: Choosing the correct data type in the target PL is a crucial step that accounts for 34.3% of all data-related bugs and 11.5% of all bugs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 1b:
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
- Rests on an assumption: The manual labeling process and the resulting bug taxonomy are reliable and comprehensive.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finding 6: Although the translation bug categories remain the same, occurrences of bug instances and their distribution vary between crafted benchmarks and real-world projects.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 6:
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In fact, all of the C2Rust generated code are unsafe, whereas, only three GPT-4-translated code are unsafe.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The biggest caveat is that executing code generated by CxGo requires specific libraries, i.e., stdio for I/O operations instead of a more generic fmt library.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Missing imports is the most frequently occurring bug category (35.5% of the bugs in the group and 8.6% of the bugs overall).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 2a:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Total/Average (All) | - | 1,700 10,858 | - | 6,197 | 8.1% | 2.8% | 14.5% | 47.3% 3.5% | 5.3% | 2.1% | |”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 10 CONCLUDING REMARKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“| Determinism and reasoning | ✓ | X |”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 10 CONCLUDING REMARKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With 𝑖𝑡𝑒𝑟 1 , we observe a substantial reduction in compilation errors for GPT-4, with 58% completely fixed (no error) and 38% transformed to other translation bugs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 6 MITIGATING TRANSLATION BUGS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“| Broader context | ✓ | X |”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 10 CONCLUDING REMARKS
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Leveraging target-language idioms | X | ✓ |”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 10 CONCLUDING REMARKS
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, for other errors, the percentage is lower-45% runtime errors, 48% functional errors, and 43% non-terminating executions-suggesting these bugs are harder to mitigate.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 6 MITIGATING TRANSLATION BUGS
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In both cases, we observe a few cases where the outcome of the translation degrades: i.e., functional error transforms to runtime/compilation error, or runtime error transforms to compilation error.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 6 MITIGATING TRANSLATION BUGS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Compilation Errors | 68.9 93.5 76.4 | 56.9 93.2 94.6 77.0 | 61.6 86.7 83.3 82.4 | 55.9 82.4 78.4 96.6 | 57.4 79.9 73.4 86.0 72.4 77.8 |”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 10 CONCLUDING REMARKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
In the paper
“Translating the software/code to a modern PL can significantly reduce maintenance effort, improve overall reliability, and boost non-functional properties such as security and performance [1][2][3][4][5]74].”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 1 INTRODUCTION
What the sources say
- Not matchedcites [1][2][3][4][5]74]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We believe this result is because, as mentioned, we excluded information about the translated code (imports, class declaration, and method signature) in the prompt.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.1 Effectiveness of LLMs in Code Translation
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our initial investigation showed that GPT-4 is the best-performing model and compared to others, its translations exhibit a variety of quality bugs worth investigating.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.1 Taxonomy of Translation Bugs
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The manual labeling process and the resulting bug taxonomy are reliable and comprehensive.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The next most common effect of unsuccessful translation is a functional error (13.4%), demonstrating that often translated code does not preserve the behavior of the source program.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.2 Outcome of Unsuccessful Translations
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our analysis shows that these two classes of techniques have their own strengths and weaknesses.”
- Also stated in: § 1 INTRODUCTION · hedged
“The comparison shows that LLM-and non-LLM-based translation techniques provide different and unique advantages, suggesting that an ultimate solution for code translation should combine both techniques (details in §5).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Abstract
plaincorrelationaln = 1,700- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 INTRODUCTION · hedged
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The nature of these bugs is significantly different as the non-LLM-based approaches tend to have more bugs related to the custom library, while LLM-based approaches have more syntax and semantics-related bugs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, this limitation can cause inconsistencies in the translated code, e.g., as illustrated for Finding 4, renaming overloaded methods requires all the related call sites to be updated.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 6:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, Python-to-Java translations can be more error-prone than Java-to-Python translations, in general, because going from a dynamic type system to a static one can be harder for an LLM to reason about.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 5:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, real-world applications have much more complex code than crafted benchmarks, making them much harder for LLMs to translate.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.3 Translation Bugs in Real-world Projects
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These bugs occur due to incorrect choice of the data type, differences between behaviors of equivalent data types across PLs, and differences in type systems of the source and target PLs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 1b:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The manual labeling process and the resulting bug taxonomy are reliable and comprehensive.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the equivalent Java test case (line 2 on right) fails because of a fundamental difference between Python float and Java Float: the former uses 64-bit precision whereas the latter uses 32-bit precision.The Java code thus cannot handle the large test data value, which works fine in Python.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 1b:
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specifically, the use of scanf (as in 5 ) without buffer size specifications can lead to vulnerabilities such as buffer overruns.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finding 8: C2Rust generates non-idiomatic and unsafe code, whereas GPT-4 tends to generate safer and idiomatic code.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the other datasets used contain fairly rigorous test suites.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unlike crafted benchmarks, files in real-world projects do not exist as standalone programs, and providing relevant context about inter-file dependencies can help an LLM produce better code.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For LLM selection, we included several SOTA general and code LLMs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To mitigate this threat, after each labeling round, both labelers met to discuss any discrepancies and resolve them.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“At the end, each bug was assigned a single mutually agreed label.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This approach comes with the risk that inadequate or weak test suites can cause buggy translations that pass the test suites to be considered correct.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this respect, non-LLMbased approaches mostly do not leverage target language idioms, which can negatively affect code readability and maintainability.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“We also considered other approaches (mppSMT[57], Tree2Tree[29], and Sharpen[18]) but could not compare against them due to lack of tool/artifact availability ( §5).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 10 CONCLUDING REMARKS
What the sources say
- Not matchedcites [57]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [29] source
- Not matchedcites [18]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To address this threat, we thoroughly tested the scripts and spot-checked the results for correctness.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In some cases, this even leads to security issues (e.g., C to Rust).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Conversely, LLMs tend to generate more natural, human-like code.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, non-LLM-based approaches, specifically transpilers, have a more comprehensive context of an application.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, for non-LLM-based approaches, we used multiple tools covering different PLs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“As for threats to internal validity, one potential threat is that, for each translation task, we performed the translation once.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There can be cases where no suitable API exists in the target PL to translate an API call to-techniques that combine code summarization and code synthesis could be investigated to fill such gaps.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We measured translation correctness using the test cases provided in our datasets, which is a commonly used approach for assessing the quality of generated code.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Moreover, we make our artifacts publicly available [7] to enable review and replication of our results.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
What the sources say
- Not matchedcites [7]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In terms of datasets, we used multiple well-known datasets with different characteristics and also two real-world projects.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“We note that one of our datasets, CodeNet [63], contains one test case per sample and may be susceptible to this threat.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
What the sources say
- Not checked — no full text of the sourcecites [63] source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The findings of this work shows considerable scope for improving open-source and close-source LLMs for code translation.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“On top of that, one author went through the entire labeling to ensure consistency among the groups.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, a significant challenge in translating real-world projects is handling library API calls and the differences in the API ecosystem of different PLs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Another threat to internal validity is that we do not have a formal inter-rater reliability metric for our manual labeling of bugs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This issue is further exacerbated by the use of implicit casting (e.g., 0 as libc::c_int as in 4 ) in the method body, which can cause unexpected behavior.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The unsafe block in main function (see 3 in Figure 4-(b)) circumvents Rust's safety checks for the code encompassed in that method.This increases the risk of memory safety violations.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, while translating a module, the variable types, method signatures, folder structure, and dependencies associated with that module are often unavailable to LLMs because of their restriction on the token size.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Performing a translation task multiple times may change the success rates in translation as LLMs are inherently nondeterministic.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
- Rests on an assumption: LLMs are inherently nondeterministic.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, our results may be affected by bugs in our automation scripts.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
There is a dearth of research on understanding and cataloging the limitations of LLMs for code translation.
“Although prior research has shown the potential of using LLMs for code translation, there is a dearth of research on understanding and cataloging their limitations for this task.”
The paper relies on this without support.
Code translation is a complex task that requires LLMs to understand code syntax and semantics simultaneously.
“This is an important undertaking because code translation is a complex task that requires LLMs to understand code syntax (to generate syntactically correct code) and semantics (to preserve functionality during translation) simultaneously.”
The paper relies on this without support.
The selected datasets and real-world projects are comprehensive enough to understand the nature of LLM translation bugs.
“To ensure the comprehensiveness of our findings and claims in understanding the nature of LLM translation bugs, we were interested in datasets used in prior studies as well as real-world projects.”
The paper relies on this without support.
The tests provided with the code samples are adequate to validate code translation.
“These datasets are accompanied by test cases to validate code translation.”
The paper relies on this without support.
The selected real-world projects are reasonably complex and well-maintained software exclusively written in Java or Python.
“Our goal was to translate reasonably complex and well-maintained software exclusively written in Java or Python.”
The paper relies on this without support.
The manual labeling process and the resulting bug taxonomy are reliable and comprehensive.
“The manual construction of the taxonomy involved eight human labelers, who are researchers or software engineers in the industry, and involved unsuccessful translations from all 31 translation pairs in Table 2.”
The paper relies on this without support.
Regardless of randomness or number of repetitions, the nature of an unsuccessful translation remains unchanged.
“Regardless of randomness or number of repetitions, the nature of an unsuccessful translation remains unchanged.”
The paper relies on this without support.
LLMs are inherently nondeterministic.
“Performing a translation task multiple times may change the success rates in translation as LLMs are inherently nondeterministic.”
The paper relies on this without support.
Non-LLM-based techniques are generally deterministic.
“Second, non-LLMbased techniques are generally deterministic, allowing for more predictable reasoning, whereas LLMs are inherently probabilistic, with a greater degree of creativity.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, they do not provide a sufficient number of repositories and corresponding automatic test suites for evaluation.”
From RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation (2026) · cites this paper as [1]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Recently, Pan et al. [1] manually study two opensource repositories (Apache Commons CLI [41] and Python Click [42]), and find that current LLMs struggle to perform code translation for entire repositories.”
From RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation (2026) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“Pan et al. [1] attempt to perform mutual conversion between Python and Java projects [41], [42], but find that the advanced LLMs are largely ineffective, with success rates of 8.1% for GPT-4 and 0% for the rest of the models.”
From RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation (2026) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“Recently, pan et al. [1] manually study two open-source repositories (Apache Commons CLI [41] and Python Click [42]) and find that current LLMs struggle to complete the translation tasks of entire repositories.”
From RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation (2026) · cites this paper as [1]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Besides, the resource and configuration files are ignored in this research.”
From RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation (2026) · cites this paper as [1]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Dataset Collection and Pre-Processing.
“The translation of real-world projects never reached test execution, as it produced syntactically incorrect code (more discussion in §3).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 10 CONCLUDING REMARKS
“Commons CLI [6] | Java | 22 | 310 | Python | 22 | 0.0% | 0.0% | 0.0% | 13.6% 0.0% | 0.0% | 0.0% | |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.1 Taxonomy of Translation Bugs
“To manage the manual effort for understanding and labeling bugs for building the taxonomy, we focused on 1,748 unsuccessful translations from GPT-4.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.2 Prevalence of LLM-based Translation Bugs
“Among the 6197 attempted translations over 31 language pairs (Table 2), there were 1558 translation failures.We manually checked and labeled these failures to identify 1748 bugs.”
Citations
Cites
Adaptive Mixtures of Local Experts
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991) · Neural Computation · doi:10.1162/neco.1991.3.1.79 · Cited by 5,384
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100
How to Fine-Tune BERT for Text Classification?
Sun, C., Qiu, X., Xu, Y., & Huang, X. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32381-3_16 · Cited by 1,534
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2305.10601 · Cited by 1,158
A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., & Schmidt, D. C. (2023) · arXiv · doi:10.48550/arxiv.2302.11382 · Cited by 838
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
A systematic evaluation of large language models of code
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534862 · Cited by 540
Thones, J. (2015) · IEEE Software · doi:10.1109/ms.2015.11 · Cited by 481
Liu, J., Xia, C. S., Wang, Y., & Zhang, L. (2023) · arXiv · doi:10.48550/arxiv.2305.01210 · Cited by 390
Automatic Software Repair: A Survey
Gazzola, L., Micucci, D., & Mariani, L. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2755013 · Cited by 336
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., & Xiong, C. (2022) · arXiv · doi:10.48550/arxiv.2203.13474 · Cited by 330
A comprehensive study on deep learning bug characteristics
Islam, J., Nguyen, G., Pan, R., & Rajan, H. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338955 · Cited by 315
Self-citationAn empirical study on TensorFlow program bugs
Zhang, Y., Chen, Y., Cheung, S.-C., Xiong, Y., & Zhang, L. (2018) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3213846.3213866 · Cited by 311
StarCoder: may the source be with you!
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.06161 · Cited by 261
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · arXiv · doi:10.48550/arxiv.2002.08155 · Cited by 243
Less training, more repairing please: revisiting automated program repair via zero-shot learning
Xia, C. S., & Zhang, L. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549101 · Cited by 231
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X
Zheng, Q., Xia, X., Zou, X., Dong, Y.-X., Wang, S., Xue, Y.-F., Wang, Z.-H., Shen, L., Wang, A., Li, Y., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2303.17568 · Cited by 227
CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
Ren, S., Guo, D., Lu, S., Zhou, L., Liu, S., Tang, D., Sundaresan, N., Zhou, M., Blanco, A., & Ma, S. (2020) · arXiv · doi:10.48550/arxiv.2009.10297 · Cited by 214
Traduction Non Supervisée de Langages de Programmation
Rozière, B. (2022) · theses.fr (ABES) · Cited by 167
Conversational Automated Program RepairPreprint
Xia, C. S., & Zhang, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.13246 · Cited by 152
Lexical statistical machine translation for language migration
Nguyen, A. T., Nguyen, T. T., & Nguyen, T. N. (2013) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2491411.2494584 · Cited by 136
Perfection Not Required? Human-AI Partnerships in Code Translation
Weisz, J. D., Muller, M., Houde, S., Richards, J., Ross, S. I., Martinez, F., Agarwal, M., & Talamadupula, K. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450656 · Cited by 129
Repair Is Nearly Generation: Multilingual Program Repair with LLMs
Joshi, H., Cambronero Sanchez, J., Gulwani, S., Le, V., Verbruggen, G., & Radiček, I. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i4.25642 · Cited by 126
Phrase-Based Statistical Translation of Programming Languages
Karaivanov, S., Raychev, V., & Vechev, M. (2014) · ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) · doi:10.1145/2661136.2661148 · Cited by 115
DOBF: A Deobfuscation Pre-Training Objective for Programming LanguagesPreprint
Rozière, B., Lachaux, M.-A., Szafraniec, M., & Lample, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.07492 · Cited by 113
Repairing deep neural networks
Islam, M. J., Pan, R., Nguyen, G., & Rajan, H. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380378 · Cited by 106
Self-citationKalia, A. K., Xiao, J., Krishna, R., Sinha, S., Vukovic, M., & Banerjee, D. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3473915 · Cited by 101
Self-citationHow to Fine-Tune BERT for Text Classification?Preprint
Sun, C. C., Qiu, X., Xu, Y., & Huang, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.05583 · Cited by 89
Challenges in migrating legacy software systems to the cloud - an empirical study
Fahmideh, M., Daneshgar, F., Beydoun, G., et al. (2017) · Information Systems · doi:10.1016/j.is.2017.03.008 · Cited by 87
Divide-and-Conquer Approach for Multi-phase Statistical Migration for Source Code (T)
Nguyen, A. T., Nguyen, T. T., & Nguyen, T. N. (2015) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.74 · Cited by 79
Software modernization to embrace quantum technology
Pérez‐Castillo, R., Serrano, M., & Piattini, M. G. (2020) · Advances in Engineering Software · doi:10.1016/j.advengsoft.2020.102933 · Cited by 79
Practical Program Repair in the Era of Large Pre-trained Language Models
Xia, C., Wei, Y.-X., & Zhang, L.-M. (2022) · arXiv.org · doi:10.48550/arxiv.2210.14179 · Cited by 70
Better Together? An Evaluation of AI-Supported Code Translation
Weisz, J. D., Muller, M., Ross, S. I., Martinez, F., Houde, S., Agarwal, M., Talamadupula, K., & Richards, J. T. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511157 · Cited by 69
Migrating code with statistical machine translation
Nguyen, A. T., Nguyen, T. T., & Nguyen, T. N. (2014) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1145/2591062.2591072 · Cited by 62
SantaCoder: don't reach for the stars!
Allal, L. B., Li, R., Kocetkov, D., Mou, C., Akiki, C., Ferrandis, C. M., Muennighoff, N., Mishra, M., Gu, A., Dey, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.03988 · Cited by 61
Migrating Legacy Software to Microservices Architecture
Kazanavičius, J., & Mažeika, D. (2019) · IEEE Open Conference of Electrical, Electronic and Information Sciences (eStream) · doi:10.1109/estream.2019.8732170 · Cited by 60
Breivold, H. P. (2019) · Enterprise Information Systems · doi:10.1080/17517575.2018.1556814 · Cited by 57
Unsupervised translation of programming languages
Roziere, B. (2026) · doi:10.70675/416f6582z4576z444eza710z5e930d59bf97 · Cited by 57
Nitin, V., Asthana, S., Ray, B., & Krishna, R. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556960 · Cited by 52
Self-citationMigrating Legacy Software to the Cloud with ARTIST
Bergmayr, A., Pezuela, C., Wimmer, M., et al. (2013) · Proceedings of the ... European Conference on Software Maintenance and Reengineering/Proceedings of the European Conference on Software Maintenance and Reengineering · doi:10.1109/csmr.2013.73 · Cited by 50
AVATAR: A Parallel Corpus for Java-Python Program Translation
Ahmad, W. U., Tushar, M. G. R., Chakraborty, S., & Chang, K.-W. (2021) · arXiv · doi:10.48550/arxiv.2108.11590 · Cited by 46
CrystalBLEU: Precisely and Efficiently Measuring the Similarity of Code
Eghbali, A., & Pradel, M. (2022) · Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3551349.3556903 · Cited by 42
Leveraging automated unit tests for unsupervised code translation
Roziere, B., Zhang, J. M., Charton, F., Harman, M., Synnaeve, G., & Lample, G. (2021) · arXiv · doi:10.48550/arxiv.2110.06773 · Cited by 39
Zhuo, T. Y., Li, Z., Huang, Y., Shiri, F., Wang, W., Haffari, G., & Li, Y.-F. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.77 · Cited by 31
Eghbali, A., & Pradel, M. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510454.3528648 · Cited by 29
Repairing Deep Neural Networks: Fix Patterns and Challenges
Islam, J., Pan, R., Nguyen, G., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00972 · Cited by 28
Self-citationCode translation with compiler representations
Szafraniec, M., Roziere, B., Leather, H., Charton, F., Labatut, P., & Synnaeve, G. (2022) · arXiv · doi:10.48550/arxiv.2207.03578 · Cited by 26
Migrating Monoliths to Microservices-based Customizable Multi-tenant Cloud-native Apps
Haugeland, S. G., Nguyen, P. H., Song, H., et al. (2021) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa53835.2021.00030 · Cited by 19
Repair Is Nearly Generation: Multilingual Program Repair with LLMsPreprint
Joshi, H., Cambronero, J., Gulwani, S., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.11640 · Cited by 15
Self-planning Code Generation with Large Language ModelsPreprint
Jiang, X., Dong, Y., Wang, L., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.06689 · Cited by 11
Application of back-translation
Subedi, I. M., Singh, M., Ramasamy, V., & Walia, G. (2021) · ACM Southeast Regional Conference (ACMSE) · doi:10.1145/3409334.3452068 · Cited by 10
Cloud Application Modernization and Migration Methodology
Settu, R., & Raj, P. (2013) · Computer communications and networks · doi:10.1007/978-1-4471-5107-4_12 · Cited by 8
Legacy Web Application Modernization by Generating a REST Service Layer
Echeverria, R. R., Macias, F., Pavon, V. M., Conejero, J. M., & Figueroa, F. S. (2015) · IEEE Latin America Transactions · doi:10.1109/tla.2015.7273801 · Cited by 7
Improving Automatic C-to-Rust Translation with Static Analysis
Hong, J. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion58688.2023.00074 · Cited by 7
A Survey of Learning-based Automated Program RepairPreprint
Zhang, Q., Fang, C., Ma, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.03270 · Cited by 5
End-to-End Training for Back-Translation with Categorical Reparameterization TrickPreprint
Heo, D., & Choi, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.08465 · Cited by 4
CodeStylist: A System for Performing Code Style Transfer Using Neural Networks
Ting, C.-K., Munson, K., Wade, S., Savla, A., Kate, K., & Srinivas, K. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i13.27087 · Cited by 3
Zhuo, T. Y., Li, Z., Huang, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.12868 · Cited by 3
MUFIN: Improving Neural Repair Models with Back-TranslationPreprint
Silva, A., Ferreira, J. F., Ye, H., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.02301 · Cited by 3
Transforming Monolithic Applications to Microservices with Mono2Micro
(2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678851 · Cited by 1
On Codex Prompt Engineering for OCL Generation: An Empirical Study
Abukhalaf, S., Hamdaqa, M., & Khomh, F. (2023) · arXiv · doi:10.48550/arxiv.2303.16244
InferType: A Compiler Toolkit for Implementing Efficient Constraint-Based Type Inference
Li, S., Yamazaki, T., & Chiba, S. (2024) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.ecoop.2024.23
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Workshop, B., :, Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.05100
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · arXiv · doi:10.48550/arxiv.1801.06146
TransCoder: Towards Unified Transferable Code Representation Learning Inspired by Human Skills
Sun, Q., Chen, N., Wang, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.07285
Exploring Data Augmentation for Code Generation Tasks
Chen, P., & Lampouras, G. (2023) · arXiv · doi:10.48550/arxiv.2302.03499
ADELT: Transpilation Between Deep Learning Frameworks
Gong, L., Wang, J., & Cheung, A. (2023) · arXiv · doi:10.48550/arxiv.2303.03593
Jana, P., Jha, P., Ju, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.06755
Syntax and Domain Aware Model for Unsupervised Program Translation
Liu, F., Li, J., & Zhang, L. (2023) · arXiv · doi:10.48550/arxiv.2302.03908
Research works printed but not linked: 5
- Xinyun Chen, Chang Liu, and Dawn Song. 2018. Tree-to-tree neural networks for program translation. Advances in neural information processing systems 31 (2018).
- Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. StarCoder: may the source be with you! arXiv preprint arXiv:2305.06161 (2023). · arXiv
- Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, et al. 2021. CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
- Yuanbo Wen, Qi Guo, Qiang Fu, Xiaqing Li, Jianxing Xu, Yanlin Tang, Yongwei Zhao, Xing Hu, Zidong Du, Ling Li, et al. 2022. BabelTower: Learning to Auto-parallelized Program Translation. In International Conference on Machine Learning. PMLR, 23685--23700.
- WG Zhang, Arne J Berre, Dumitru Roman, and Hans Aage Huru. 2009. Migrating legacy applications to the service Cloud. In Proceedings of the 14th Conference Companion on Object Oriented Programming Systems Languages and Applications. 59--68.
Cited by
Sorted by citations
Towards an understanding of large language models in software engineering tasks
Zheng, Z., Ning, K., Zhong, Q., Chen, J., Chen, W., Guo, L., Wang, W., & Wang, Y. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10602-0 · Cited by 212
On the Evaluation of Large Language Models in Unit Test Generation
Yang, L., Yang, C., Gao, S., Wang, W., Wang, B., Zhu, Q., Chu, X., Zhou, J., Liang, G., Wang, Q., et al. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695529 · Cited by 158
Look Before You Leap: An Exploratory Study of Uncertainty Analysis for Large Language Models
Huang, Y., Song, J., Wang, Z., Zhao, S., Chen, H., Juefei-Xu, F., & Ma, L. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3519464 · Cited by 275
Bugs in large language models generated code: an empirical study
Tambon, F., Dakhel, A. M., Nikanjam, A., Khomh, F., Desmarais, M. C., & Antoniol, G. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10614-4 · Cited by 70
Generative Artificial Intelligence for Software Engineering—A Research Agenda
Nguyen‐Duc, A., Cabrero‐Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K., et al. (2025) · Software: Practice and Experience · doi:10.1002/spe.70005 · Cited by 60
Metadata
- Authors
- Pan, R., Ibrahimzada, A. R., Krishna, R., Sankar, D., Wassi, L. P., Merler, M., Sobolev, B., Pavuluri, R., Sinha, S., & Jabbarvand, R.
- Affiliations
- IBM (United States) (US)
- University of Illinois Urbana-Champaign (US)
- ORCID
- Venue
- Proceedings of the IEEE/ACM 46th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2024-04-12
- Pages
- 1-13
- OpenAlex
- W4394769102
- Cited by
- 133
- References
- 33
- Citations by year
- 2026: 43 · 2025: 72 · 2024: 18
- FWCI
- 51.11
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- IEEE/ACM International Conference on Software Engineering (ICSE) (publishedVersion, cc-by-nc)
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- program translation, large language models, bug classification, prompt engineering, program synthesis, programming language translation, LLM limitations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-16, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-16 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
