The DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index; 4 of the 12 sources its claims cite were checked for support.
Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code
Open accessAbstract
Code translation aims to convert source code from one programming language (PL) to another. Given the promising abilities of large language models (LLMs) in code synthesis, researchers are exploring their potential to automate code translation. The prerequisite for advancing the state of LLM-based code translation is to understand their promises and limitations over existing techniques. To that end, we present a large-scale empirical study to investigate the ability of general LLMs and code LLMs for code translation across pairs of different languages, including C, C++, Go, Java, and Python. Our study, which involves the translation of 1,700 code samples from three benchmarks and two real-world projects, reveals that LLMs are yet to be reliably used to automate code translation---with correct translations ranging from 2.1% to 47.3% for the studied LLMs. Further manual investigation of unsuccessful translations identifies 15 categories of translation bugs. We also compare LLM-based code translation with traditional non-LLM-based approaches. Our analysis shows that these two classes of techniques have their own strengths and weaknesses. Finally, insights from our study suggest that providing more context to LLMs during translation can help them produce better results. To that end, we propose a prompt-crafting approach based on the symptoms of erroneous translations; this improves the performance of LLM-based code translation by 5.5% on average. Our study is the first of its kind, in terms of scale and breadth, that provides insights into the current limitations of LLMs in code translation and opportunities for improving them. Our dataset---consisting of 1,700 code samples in five PLs with 10K+ tests, 43K+ translated code, 1,748 manually labeled bugs, and 1,365 bug-fix pairs---can help drive research in this area.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This empirical study evaluates general and code LLMs for translating code across C, C++, Go, Java, and Python, finding that correct translations range from 2.1% to 47.3% and that LLMs are not yet reliable for automating code translation. It contributes a 15-category taxonomy of translation bugs, compares LLM and non-LLM translation, and shows iterative prompt crafting can improve success rates.
Key findings
- Across 43,379 translations, average success was 11.94% (median 5.3%); GPT-4 was best at 47.3%, followed by StarCoder at 14.5%, while real-world project success was 8.1% for GPT-4 and 0% for the other models.
- Most unsuccessful translations were compilation errors (77.8%), followed by functional errors (13.4%); data-related bugs accounted for 33.5% of bugs, syntactic/semantic differences 30.5%, and dependency/logic bugs 24.2%.
- Manual analysis produced a taxonomy of 15 bug categories in five groups, built from 1,748 unsuccessful GPT-4 translations and taking about 630 person-hours.
- Iterative prompt crafting increased success by 5.5% on average across studied LLMs, with GPT-4 improving by 12.33% at iteration 1 and 1.7% at iteration 2; non-LLM C2Rust achieved 95% success for C-to-Rust versus GPT-4’s 61%, while GPT-4 achieved 54.1% for Java-to-C# versus JavaToCSharp producing no compilable translations.
Methods
- Empirical study using 1,700 executable code samples from CodeNet, Avatar, EvalPlus, Apache Commons CLI, and Python Click, covering C, C++, Go, Java, and Python, with 6,197 translation problems across 31 language pairs.
- Seven LLMs were evaluated: GPT-4, Llama 2, TheBloke-Vicuna, TheBloke-Airoboros, StarCoder, CodeGeeX, and CodeGen; a translation was considered successful if it compiled, passed runtime checks, and passed existing tests.
- Bug taxonomy was built manually by eight labelers in two phases, and prompt crafting was evaluated iteratively on CodeGen, StarCoder, GPT-4, and Llama 2.
Limitations
- Stated: external validity is limited by the selected PLs, LLMs, and datasets; internal validity is threatened because each translation was performed once, no formal inter-rater reliability metric was used for manual labeling, and automation scripts could contain bugs; construct validity is threatened because weak test suites may let buggy translations pass, especially CodeNet with one test case per sample.
- Stated: the taxonomy focused on GPT-4 translations, and prompt crafting excluded CodeGeeX, TheBloke-Airoboros, and TheBloke-Vicuna.
- Evident from the text: real-world evaluation is limited to two projects and only Java/Python, and the paper notes that it is hard to generalize from limited observations on real-world project complexity.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
73 Own result · 46 Own interpretation · 12 Own method · 11 No basis given · 3 Sources not found · 1 Partly supported · 1 Couldn’t confirm · 1 No full text of the sources · 1 Not checked yet · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, research has shown that without providing adequate context to LLMs via prompt crafting, they may only serve as”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 1 INTRODUCTION
What the sources say
- Could not verifycites [22] sourceExperimental — accuracy not yet measured
Section 3.2 (page unavailable) says decoder-only models such as CodeGPT generate autoregressively, but does not link prompt context to task understanding or describe LLMs as merely next-token predictors. The Limitations section (page unavailable) says the authors did not evaluate LLMs. Searches for prompt crafting, adequate context, token prediction, and task understanding found no passage addressing the claim. (LLM, conf=0.76)
- Partly supportedcites [44] sourceExperimental — accuracy not yet measured
The Related Work section says autoregressive models such as Codex are trained to predict the next token. The RQ3 Code Transformation section reports that relevant examples in prompts improve repair performance. The paper does not establish that, without adequate prompt context, LLMs only predict code tokens or fail to understand the overall task. (LLM, conf=0.85)
- Partly supportedcites [82] sourceExperimental — accuracy not yet measured
The Introduction (p. 2) says surrounding context is crucial for a correct repair and that poor context representation can miss important relationships. The Approach and Patch Generation sections (pp. 4, 6–7) describe using context to predict code tokens. They do not establish that, without prompt crafting, LLMs merely predict the next token or fail to understand the overall task. (LLM, conf=0.83)
- Not matchedcites [92]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The biggest surprise here is CodeGeeX, a model trained explicitly for code translation.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.1 Effectiveness of LLMs in Code Translation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Except for GPT-4 and StarCoder, all other models performed poorly.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.1 Effectiveness of LLMs in Code Translation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There is no consistent pattern between unsuccessful translations and source/target language”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.1 Effectiveness of LLMs in Code Translation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There is a strong correlation between the average number of tests per translation sample and unsuccessful translation (correlation coefficient 𝑟 ranging from 0.64 to 0.85 for all models).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.1 Effectiveness of LLMs in Code Translation
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
In the paper
“That said, further manual investigation showed that for the Commons CLI, three out of 22 translated files could be compiled using py_compile [17].”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.1 Effectiveness of LLMs in Code Translation
What the sources say
- Not matchedcites [17]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(3) replacing language-specific features, such as method overloading and annotations, can be challenging, especially in real-world projects.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“but translating to Go results in more compilation errors due to its strict syntax constraints, e.g., forbidding unused variables or imports.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.1 Effectiveness of LLMs in Code Translation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Some notable findings are: (1) identifying suitable data type in the target PL that preserves the source behavior is challenging”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From these results, we observe that most unsuccessful translations result in compilation errors (77.8%), meaning both general and code LLMs have difficulty understanding code syntax.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.2 Outcome of Unsuccessful Translations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
- It may state two separate claims as one. 45
- Probably not a substantial claim. 37
- The quote only partly states this claim. 3
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“To mitigate these threats, we selected five PLs, guided by PL popularity [21] while also covering different programming paradigms.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
What the sources say
- Could not verifycites [21] sourceExperimental — accuracy not yet measured
The Code generation and Background and Terminology sections discuss several programming languages, but neither addresses their popularity or uses popularity to guide language selection. Searches for programming-language popularity and selection of five languages across paradigms found no supporting passage in our copy. The citation is attached specifically to the popularity rationale. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our study, which involves the translation of 1,700 code samples from three benchmarks and two real-world projects, reveals that LLMs are yet to be reliably used to automate code translationwith correct translations ranging from 2.1% to 47.3% for the studied LLMs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Abstract
plainn = 1,700- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We performed 43,379 translations across all LLMs, measuring translation success against the tests provided with the code samples.This produced 11.94% successful translations on average (median 5.3%), with GPT-4 (47.3% success rate) and StarCoder (14.5% success rate) being the best-performing models (details in §3.1).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On realworld projects, the LLMs were largely ineffective, with success rates of 8.1% for GPT-4 and 0% for the rest of the models.”
- Also stated in: § 3.1 Effectiveness of LLMs in Code Translation · plain
“The LLMs fail to translate real-world projects.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
- Also stated in: § 3.1 Effectiveness of LLMs in Code Translation · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results show prompt crafting increases the success rate by 5.5%, on average, across the studied LLMs, with the largest improvement, of 12%, occurring for GPT-4 (details in §6).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(2) identifying equivalent APIs in the target language or implementing the API functionality can introduce bugs”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further breakdown of the results per PLs shows that Go and C++ have comparatively stricter syntax, while it is easier for LLMs to generate syntactically correct Python code.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.2 Outcome of Unsuccessful Translations
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further manual investigation of unsuccessful translations identifies 15 categories of translation bugs.”
- Also stated in: § 1 INTRODUCTION · plain
“The resulting bug taxonomy is structured into 15 categories and five groups (details in §4.1).”
- Also stated in: § 4.1.1 Methodology. · plain
“The entire exercise took about 630 person-hours and produced a taxonomy with 15 bug categories organized into five groups (”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: The manual labeling process and the resulting bug taxonomy are reliable and comprehensive.
- Also stated in: § 1 INTRODUCTION · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As for C2Rust, the 5% unsuccessful translations were due to compilation errors, the majority of them caused by unused imports.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With a further investigation on the type of translation bugs, we found that most GPT-4 translation bugs (71.8%) are related to unused imports, variables, and other target language-specific violations, followed by incorrect input parsing (10.3%).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using GPT-4, 50% of the translation bugs were due to compilation errors, 39.3% were caused by runtime errors, and 10.7% were due to test failures.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also observed cases of broken inheritance relations (resulting in missing behaviors/states) in translated code and incorrect translation of exception handling.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.3 Translation Bugs in Real-world Projects
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The most prevalent bugs in real-world projects are model-specific constraints (29.4%), missing imports (27.4%), and the removal of logic from the source code (15.7%),”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 6:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observed that none of the translated code using JavaToCSharp can compile and, in fact, require heavy rewriting of library APIs (e.g., System.inand java.util.Scanner).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When compared to LLMs, we found that CxGo outperforms all the open-source models with an accuracy of 62.25%, but falls 10% behind GPT-4.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 translates these to Python in different ways, some of which are correct translations, whereas others are erroneous.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.3 Translation Bugs in Real-world Projects
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Whereas, with CxGo, the majority of the bugs are related to input parsing (26.7%) and mismatch of behavior after replacing API call (26.7%).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the contrary, GPT-4 and StarCoder achieve 54.1% and 18.45% success rate in translating Java code to C#.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is evident from not only the occurrences of successful translations (three for Commons CLI vs. none for Click) and degenerate codegeneration instances (nine for Click vs none for Commons CLI), but also the translation bugs observed.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 5:
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For EvalPlus and the real-world projects, the occurrence of this bug is lesser (one for EvalPlus and none for the real-world projects).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.2 Prevalence of LLM-based Translation Bugs
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
- Rests on an assumption: The manual labeling process and the resulting bug taxonomy are reliable and comprehensive.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Key among these are: removal of logic in the source (e.g., source methods and field initialization not translated), inclusion of logic not in source (e.g., implementation added for a stubbed method), missing imports, mismatch in API behaviors after translation, mismatch of behaviors after replacing API calls and incorrect data types.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.3 Translation Bugs in Real-world Projects
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For instance, Commons CLI uses method overloading frequently.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.3 Translation Bugs in Real-world Projects
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finding 1b: Choosing the correct data type in the target PL is a crucial step that accounts for 34.3% of all data-related bugs and 11.5% of all bugs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 1b:
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
- Rests on an assumption: The manual labeling process and the resulting bug taxonomy are reliable and comprehensive.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finding 6: Although the translation bug categories remain the same, occurrences of bug instances and their distribution vary between crafted benchmarks and real-world projects.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 6:
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In fact, all of the C2Rust generated code are unsafe, whereas, only three GPT-4-translated code are unsafe.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The biggest caveat is that executing code generated by CxGo requires specific libraries, i.e., stdio for I/O operations instead of a more generic fmt library.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Missing imports is the most frequently occurring bug category (35.5% of the bugs in the group and 8.6% of the bugs overall).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 2a:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Total/Average (All) | - | 1,700 10,858 | - | 6,197 | 8.1% | 2.8% | 14.5% | 47.3% 3.5% | 5.3% | 2.1% | |”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 10 CONCLUDING REMARKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“| Determinism and reasoning | ✓ | X |”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 10 CONCLUDING REMARKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With 𝑖𝑡𝑒𝑟 1 , we observe a substantial reduction in compilation errors for GPT-4, with 58% completely fixed (no error) and 38% transformed to other translation bugs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 6 MITIGATING TRANSLATION BUGS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“| Broader context | ✓ | X |”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 10 CONCLUDING REMARKS
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Leveraging target-language idioms | X | ✓ |”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 10 CONCLUDING REMARKS
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, for other errors, the percentage is lower-45% runtime errors, 48% functional errors, and 43% non-terminating executions-suggesting these bugs are harder to mitigate.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 6 MITIGATING TRANSLATION BUGS
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In both cases, we observe a few cases where the outcome of the translation degrades: i.e., functional error transforms to runtime/compilation error, or runtime error transforms to compilation error.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 6 MITIGATING TRANSLATION BUGS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Compilation Errors | 68.9 93.5 76.4 | 56.9 93.2 94.6 77.0 | 61.6 86.7 83.3 82.4 | 55.9 82.4 78.4 96.6 | 57.4 79.9 73.4 86.0 72.4 77.8 |”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 10 CONCLUDING REMARKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
In the paper
“Translating the software/code to a modern PL can significantly reduce maintenance effort, improve overall reliability, and boost non-functional properties such as security and performance [1][2][3][4][5]74].”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 1 INTRODUCTION
What the sources say
- Not matchedcites [1][2][3][4][5]74]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We believe this result is because, as mentioned, we excluded information about the translated code (imports, class declaration, and method signature) in the prompt.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.1 Effectiveness of LLMs in Code Translation
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our initial investigation showed that GPT-4 is the best-performing model and compared to others, its translations exhibit a variety of quality bugs worth investigating.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.1 Taxonomy of Translation Bugs
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The manual labeling process and the resulting bug taxonomy are reliable and comprehensive.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The next most common effect of unsuccessful translation is a functional error (13.4%), demonstrating that often translated code does not preserve the behavior of the source program.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.2 Outcome of Unsuccessful Translations
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our analysis shows that these two classes of techniques have their own strengths and weaknesses.”
- Also stated in: § 1 INTRODUCTION · hedged
“The comparison shows that LLM-and non-LLM-based translation techniques provide different and unique advantages, suggesting that an ultimate solution for code translation should combine both techniques (details in §5).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Abstract
plaincorrelationaln = 1,700- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 INTRODUCTION · hedged
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The nature of these bugs is significantly different as the non-LLM-based approaches tend to have more bugs related to the custom library, while LLM-based approaches have more syntax and semantics-related bugs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, this limitation can cause inconsistencies in the translated code, e.g., as illustrated for Finding 4, renaming overloaded methods requires all the related call sites to be updated.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 6:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, Python-to-Java translations can be more error-prone than Java-to-Python translations, in general, because going from a dynamic type system to a static one can be harder for an LLM to reason about.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 5:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, real-world applications have much more complex code than crafted benchmarks, making them much harder for LLMs to translate.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.3 Translation Bugs in Real-world Projects
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These bugs occur due to incorrect choice of the data type, differences between behaviors of equivalent data types across PLs, and differences in type systems of the source and target PLs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 1b:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The manual labeling process and the resulting bug taxonomy are reliable and comprehensive.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the equivalent Java test case (line 2 on right) fails because of a fundamental difference between Python float and Java Float: the former uses 64-bit precision whereas the latter uses 32-bit precision.The Java code thus cannot handle the large test data value, which works fine in Python.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 1b:
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specifically, the use of scanf (as in 5 ) without buffer size specifications can lead to vulnerabilities such as buffer overruns.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finding 8: C2Rust generates non-idiomatic and unsafe code, whereas GPT-4 tends to generate safer and idiomatic code.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the other datasets used contain fairly rigorous test suites.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unlike crafted benchmarks, files in real-world projects do not exist as standalone programs, and providing relevant context about inter-file dependencies can help an LLM produce better code.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For LLM selection, we included several SOTA general and code LLMs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To mitigate this threat, after each labeling round, both labelers met to discuss any discrepancies and resolve them.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“At the end, each bug was assigned a single mutually agreed label.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This approach comes with the risk that inadequate or weak test suites can cause buggy translations that pass the test suites to be considered correct.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this respect, non-LLMbased approaches mostly do not leverage target language idioms, which can negatively affect code readability and maintainability.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“We also considered other approaches (mppSMT[57], Tree2Tree[29], and Sharpen[18]) but could not compare against them due to lack of tool/artifact availability ( §5).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 10 CONCLUDING REMARKS
What the sources say
- Not matchedcites [57]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [29] source
- Not matchedcites [18]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To address this threat, we thoroughly tested the scripts and spot-checked the results for correctness.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In some cases, this even leads to security issues (e.g., C to Rust).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Conversely, LLMs tend to generate more natural, human-like code.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, non-LLM-based approaches, specifically transpilers, have a more comprehensive context of an application.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, for non-LLM-based approaches, we used multiple tools covering different PLs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“As for threats to internal validity, one potential threat is that, for each translation task, we performed the translation once.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There can be cases where no suitable API exists in the target PL to translate an API call to-techniques that combine code summarization and code synthesis could be investigated to fill such gaps.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We measured translation correctness using the test cases provided in our datasets, which is a commonly used approach for assessing the quality of generated code.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Moreover, we make our artifacts publicly available [7] to enable review and replication of our results.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
What the sources say
- Not matchedcites [7]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In terms of datasets, we used multiple well-known datasets with different characteristics and also two real-world projects.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“We note that one of our datasets, CodeNet [63], contains one test case per sample and may be susceptible to this threat.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
What the sources say
- Not checked — no full text of the sourcecites [63] source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The findings of this work shows considerable scope for improving open-source and close-source LLMs for code translation.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“On top of that, one author went through the entire labeling to ensure consistency among the groups.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, a significant challenge in translating real-world projects is handling library API calls and the differences in the API ecosystem of different PLs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Another threat to internal validity is that we do not have a formal inter-rater reliability metric for our manual labeling of bugs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This issue is further exacerbated by the use of implicit casting (e.g., 0 as libc::c_int as in 4 ) in the method body, which can cause unexpected behavior.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The unsafe block in main function (see 3 in Figure 4-(b)) circumvents Rust's safety checks for the code encompassed in that method.This increases the risk of memory safety violations.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, while translating a module, the variable types, method signatures, folder structure, and dependencies associated with that module are often unavailable to LLMs because of their restriction on the token size.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Performing a translation task multiple times may change the success rates in translation as LLMs are inherently nondeterministic.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
- Rests on an assumption: LLMs are inherently nondeterministic.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, our results may be affected by bugs in our automation scripts.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
There is a dearth of research on understanding and cataloging the limitations of LLMs for code translation.
“Although prior research has shown the potential of using LLMs for code translation, there is a dearth of research on understanding and cataloging their limitations for this task.”
The paper relies on this without support.
Code translation is a complex task that requires LLMs to understand code syntax and semantics simultaneously.
“This is an important undertaking because code translation is a complex task that requires LLMs to understand code syntax (to generate syntactically correct code) and semantics (to preserve functionality during translation) simultaneously.”
The paper relies on this without support.
The selected datasets and real-world projects are comprehensive enough to understand the nature of LLM translation bugs.
“To ensure the comprehensiveness of our findings and claims in understanding the nature of LLM translation bugs, we were interested in datasets used in prior studies as well as real-world projects.”
The paper relies on this without support.
The tests provided with the code samples are adequate to validate code translation.
“These datasets are accompanied by test cases to validate code translation.”
The paper relies on this without support.
The selected real-world projects are reasonably complex and well-maintained software exclusively written in Java or Python.
“Our goal was to translate reasonably complex and well-maintained software exclusively written in Java or Python.”
The paper relies on this without support.
The manual labeling process and the resulting bug taxonomy are reliable and comprehensive.
“The manual construction of the taxonomy involved eight human labelers, who are researchers or software engineers in the industry, and involved unsuccessful translations from all 31 translation pairs in Table 2.”
The paper relies on this without support.
Regardless of randomness or number of repetitions, the nature of an unsuccessful translation remains unchanged.
“Regardless of randomness or number of repetitions, the nature of an unsuccessful translation remains unchanged.”
The paper relies on this without support.
LLMs are inherently nondeterministic.
“Performing a translation task multiple times may change the success rates in translation as LLMs are inherently nondeterministic.”
The paper relies on this without support.
Non-LLM-based techniques are generally deterministic.
“Second, non-LLMbased techniques are generally deterministic, allowing for more predictable reasoning, whereas LLMs are inherently probabilistic, with a greater degree of creativity.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, they do not provide a sufficient number of repositories and corresponding automatic test suites for evaluation.”
From RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation (2026) · cites this paper as [1]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Recently, Pan et al. [1] manually study two opensource repositories (Apache Commons CLI [41] and Python Click [42]), and find that current LLMs struggle to perform code translation for entire repositories.”
From RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation (2026) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“Pan et al. [1] attempt to perform mutual conversion between Python and Java projects [41], [42], but find that the advanced LLMs are largely ineffective, with success rates of 8.1% for GPT-4 and 0% for the rest of the models.”
From RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation (2026) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“Recently, pan et al. [1] manually study two open-source repositories (Apache Commons CLI [41] and Python Click [42]) and find that current LLMs struggle to complete the translation tasks of entire repositories.”
From RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation (2026) · cites this paper as [1]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Besides, the resource and configuration files are ignored in this research.”
From RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation (2026) · cites this paper as [1]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Dataset Collection and Pre-Processing.
“The translation of real-world projects never reached test execution, as it produced syntactically incorrect code (more discussion in §3).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 10 CONCLUDING REMARKS
“Commons CLI [6] | Java | 22 | 310 | Python | 22 | 0.0% | 0.0% | 0.0% | 13.6% 0.0% | 0.0% | 0.0% | |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.1 Taxonomy of Translation Bugs
“To manage the manual effort for understanding and labeling bugs for building the taxonomy, we focused on 1,748 unsuccessful translations from GPT-4.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.2 Prevalence of LLM-based Translation Bugs
“Among the 6197 attempted translations over 31 language pairs (Table 2), there were 1558 translation failures.We manually checked and labeled these failures to identify 1748 bugs.”
Citations
Cites
Adaptive Mixtures of Local Experts
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991) · Neural Computation · doi:10.1162/neco.1991.3.1.79 · Cited by 5,384
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100
How to Fine-Tune BERT for Text Classification?
Sun, C., Qiu, X., Xu, Y., & Huang, X. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32381-3_16 · Cited by 1,534
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2305.10601 · Cited by 1,158
A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., & Schmidt, D. C. (2023) · arXiv · doi:10.48550/arxiv.2302.11382 · Cited by 838
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Research works printed but not linked: 5
- Xinyun Chen, Chang Liu, and Dawn Song. 2018. Tree-to-tree neural networks for program translation. Advances in neural information processing systems 31 (2018).
- Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. StarCoder: may the source be with you! arXiv preprint arXiv:2305.06161 (2023). · arXiv
- Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, et al. 2021. CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
- Yuanbo Wen, Qi Guo, Qiang Fu, Xiaqing Li, Jianxing Xu, Yanlin Tang, Yongwei Zhao, Xing Hu, Zidong Du, Ling Li, et al. 2022. BabelTower: Learning to Auto-parallelized Program Translation. In International Conference on Machine Learning. PMLR, 23685--23700.
- WG Zhang, Arne J Berre, Dumitru Roman, and Hans Aage Huru. 2009. Migrating legacy applications to the service Cloud. In Proceedings of the 14th Conference Companion on Object Oriented Programming Systems Languages and Applications. 59--68.
Cited by
Sorted by citations
Towards an understanding of large language models in software engineering tasks
Zheng, Z., Ning, K., Zhong, Q., Chen, J., Chen, W., Guo, L., Wang, W., & Wang, Y. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10602-0 · Cited by 212
On the Evaluation of Large Language Models in Unit Test Generation
Yang, L., Yang, C., Gao, S., Wang, W., Wang, B., Zhu, Q., Chu, X., Zhou, J., Liang, G., Wang, Q., et al. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695529 · Cited by 158
Look Before You Leap: An Exploratory Study of Uncertainty Analysis for Large Language Models
Huang, Y., Song, J., Wang, Z., Zhao, S., Chen, H., Juefei-Xu, F., & Ma, L. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3519464 · Cited by 275
Bugs in large language models generated code: an empirical study
Tambon, F., Dakhel, A. M., Nikanjam, A., Khomh, F., Desmarais, M. C., & Antoniol, G. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10614-4 · Cited by 70
Generative Artificial Intelligence for Software Engineering—A Research Agenda
Nguyen‐Duc, A., Cabrero‐Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K., et al. (2025) · Software: Practice and Experience · doi:10.1002/spe.70005 · Cited by 60
Metadata
- Authors
- Pan, R., Ibrahimzada, A. R., Krishna, R., Sankar, D., Wassi, L. P., Merler, M., Sobolev, B., Pavuluri, R., Sinha, S., & Jabbarvand, R.
- Affiliations
- IBM (United States) (US)
- University of Illinois Urbana-Champaign (US)
- ORCID
- Venue
- Proceedings of the IEEE/ACM 46th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2024-04-12
- Pages
- 1-13
- OpenAlex
- W4394769102
- Cited by
- 133
- References
- 33
- Citations by year
- 2026: 43 · 2025: 72 · 2024: 18
- FWCI
- 51.11
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- IEEE/ACM International Conference on Software Engineering (ICSE) (publishedVersion, cc-by-nc)
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- program translation, large language models, bug classification, prompt engineering, program synthesis, programming language translation, LLM limitations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-16, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-16 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
