The DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index; 4 of the 12 sources its claims cite were checked for support.
Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code
Open accessAbstract
Code translation aims to convert source code from one programming language (PL) to another. Given the promising abilities of large language models (LLMs) in code synthesis, researchers are exploring their potential to automate code translation. The prerequisite for advancing the state of LLM-based code translation is to understand their promises and limitations over existing techniques. To that end, we present a large-scale empirical study to investigate the ability of general LLMs and code LLMs for code translation across pairs of different languages, including C, C++, Go, Java, and Python. Our study, which involves the translation of 1,700 code samples from three benchmarks and two real-world projects, reveals that LLMs are yet to be reliably used to automate code translation---with correct translations ranging from 2.1% to 47.3% for the studied LLMs. Further manual investigation of unsuccessful translations identifies 15 categories of translation bugs. We also compare LLM-based code translation with traditional non-LLM-based approaches. Our analysis shows that these two classes of techniques have their own strengths and weaknesses. Finally, insights from our study suggest that providing more context to LLMs during translation can help them produce better results. To that end, we propose a prompt-crafting approach based on the symptoms of erroneous translations; this improves the performance of LLM-based code translation by 5.5% on average. Our study is the first of its kind, in terms of scale and breadth, that provides insights into the current limitations of LLMs in code translation and opportunities for improving them. Our dataset---consisting of 1,700 code samples in five PLs with 10K+ tests, 43K+ translated code, 1,748 manually labeled bugs, and 1,365 bug-fix pairs---can help drive research in this area.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This empirical study evaluates general and code LLMs for translating code across C, C++, Go, Java, and Python, finding that correct translations range from 2.1% to 47.3% and that LLMs are not yet reliable for automating code translation. It contributes a 15-category taxonomy of translation bugs, compares LLM and non-LLM translation, and shows iterative prompt crafting can improve success rates.
Key findings
- Across 43,379 translations, average success was 11.94% (median 5.3%); GPT-4 was best at 47.3%, followed by StarCoder at 14.5%, while real-world project success was 8.1% for GPT-4 and 0% for the other models.
- Most unsuccessful translations were compilation errors (77.8%), followed by functional errors (13.4%); data-related bugs accounted for 33.5% of bugs, syntactic/semantic differences 30.5%, and dependency/logic bugs 24.2%.
- Manual analysis produced a taxonomy of 15 bug categories in five groups, built from 1,748 unsuccessful GPT-4 translations and taking about 630 person-hours.
- Iterative prompt crafting increased success by 5.5% on average across studied LLMs, with GPT-4 improving by 12.33% at iteration 1 and 1.7% at iteration 2; non-LLM C2Rust achieved 95% success for C-to-Rust versus GPT-4’s 61%, while GPT-4 achieved 54.1% for Java-to-C# versus JavaToCSharp producing no compilable translations.
Methods
- Empirical study using 1,700 executable code samples from CodeNet, Avatar, EvalPlus, Apache Commons CLI, and Python Click, covering C, C++, Go, Java, and Python, with 6,197 translation problems across 31 language pairs.
- Seven LLMs were evaluated: GPT-4, Llama 2, TheBloke-Vicuna, TheBloke-Airoboros, StarCoder, CodeGeeX, and CodeGen; a translation was considered successful if it compiled, passed runtime checks, and passed existing tests.
- Bug taxonomy was built manually by eight labelers in two phases, and prompt crafting was evaluated iteratively on CodeGen, StarCoder, GPT-4, and Llama 2.
Limitations
- Stated: external validity is limited by the selected PLs, LLMs, and datasets; internal validity is threatened because each translation was performed once, no formal inter-rater reliability metric was used for manual labeling, and automation scripts could contain bugs; construct validity is threatened because weak test suites may let buggy translations pass, especially CodeNet with one test case per sample.
- Stated: the taxonomy focused on GPT-4 translations, and prompt crafting excluded CodeGeeX, TheBloke-Airoboros, and TheBloke-Vicuna.
- Evident from the text: real-world evaluation is limited to two projects and only Java/Python, and the paper notes that it is hard to generalize from limited observations on real-world project complexity.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
73 Own result · 46 Own interpretation · 12 Own method · 11 No basis given · 3 Sources not found · 1 Partly supported · 1 Couldn’t confirm · 1 No full text of the sources · 1 Not checked yet · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, research has shown that without providing adequate context to LLMs via prompt crafting, they may only serve as”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 1 INTRODUCTION
What the sources say
- Could not verifycites [22] sourceExperimental — accuracy not yet measured
Section 3.2 (page unavailable) says decoder-only models such as CodeGPT generate autoregressively, but does not link prompt context to task understanding or describe LLMs as merely next-token predictors. The Limitations section (page unavailable) says the authors did not evaluate LLMs. Searches for prompt crafting, adequate context, token prediction, and task understanding found no passage addressing the claim. (LLM, conf=0.76)
- Partly supportedcites [44] sourceExperimental — accuracy not yet measured
The Related Work section says autoregressive models such as Codex are trained to predict the next token. The RQ3 Code Transformation section reports that relevant examples in prompts improve repair performance. The paper does not establish that, without adequate prompt context, LLMs only predict code tokens or fail to understand the overall task. (LLM, conf=0.85)
- Partly supportedcites [82] sourceExperimental — accuracy not yet measured
The Introduction (p. 2) says surrounding context is crucial for a correct repair and that poor context representation can miss important relationships. The Approach and Patch Generation sections (pp. 4, 6–7) describe using context to predict code tokens. They do not establish that, without prompt crafting, LLMs merely predict the next token or fail to understand the overall task. (LLM, conf=0.83)
- Not matchedcites [92]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The biggest surprise here is CodeGeeX, a model trained explicitly for code translation.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.1 Effectiveness of LLMs in Code Translation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Except for GPT-4 and StarCoder, all other models performed poorly.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.1 Effectiveness of LLMs in Code Translation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There is no consistent pattern between unsuccessful translations and source/target language”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.1 Effectiveness of LLMs in Code Translation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There is a strong correlation between the average number of tests per translation sample and unsuccessful translation (correlation coefficient 𝑟 ranging from 0.64 to 0.85 for all models).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.1 Effectiveness of LLMs in Code Translation
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
In the paper
“That said, further manual investigation showed that for the Commons CLI, three out of 22 translated files could be compiled using py_compile [17].”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.1 Effectiveness of LLMs in Code Translation
What the sources say
- Not matchedcites [17]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(3) replacing language-specific features, such as method overloading and annotations, can be challenging, especially in real-world projects.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“but translating to Go results in more compilation errors due to its strict syntax constraints, e.g., forbidding unused variables or imports.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.1 Effectiveness of LLMs in Code Translation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Some notable findings are: (1) identifying suitable data type in the target PL that preserves the source behavior is challenging”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From these results, we observe that most unsuccessful translations result in compilation errors (77.8%), meaning both general and code LLMs have difficulty understanding code syntax.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.2 Outcome of Unsuccessful Translations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
- It may state two separate claims as one. 45
- Probably not a substantial claim. 37
- The quote only partly states this claim. 3
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“To mitigate these threats, we selected five PLs, guided by PL popularity [21] while also covering different programming paradigms.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
What the sources say
- Could not verifycites [21] sourceExperimental — accuracy not yet measured
The Code generation and Background and Terminology sections discuss several programming languages, but neither addresses their popularity or uses popularity to guide language selection. Searches for programming-language popularity and selection of five languages across paradigms found no supporting passage in our copy. The citation is attached specifically to the popularity rationale. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our study, which involves the translation of 1,700 code samples from three benchmarks and two real-world projects, reveals that LLMs are yet to be reliably used to automate code translationwith correct translations ranging from 2.1% to 47.3% for the studied LLMs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Abstract
plainn = 1,700- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We performed 43,379 translations across all LLMs, measuring translation success against the tests provided with the code samples.This produced 11.94% successful translations on average (median 5.3%), with GPT-4 (47.3% success rate) and StarCoder (14.5% success rate) being the best-performing models (details in §3.1).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On realworld projects, the LLMs were largely ineffective, with success rates of 8.1% for GPT-4 and 0% for the rest of the models.”
- Also stated in: § 3.1 Effectiveness of LLMs in Code Translation · plain
“The LLMs fail to translate real-world projects.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
- Also stated in: § 3.1 Effectiveness of LLMs in Code Translation · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results show prompt crafting increases the success rate by 5.5%, on average, across the studied LLMs, with the largest improvement, of 12%, occurring for GPT-4 (details in §6).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(2) identifying equivalent APIs in the target language or implementing the API functionality can introduce bugs”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further breakdown of the results per PLs shows that Go and C++ have comparatively stricter syntax, while it is easier for LLMs to generate syntactically correct Python code.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.2 Outcome of Unsuccessful Translations
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further manual investigation of unsuccessful translations identifies 15 categories of translation bugs.”
- Also stated in: § 1 INTRODUCTION · plain
“The resulting bug taxonomy is structured into 15 categories and five groups (details in §4.1).”
- Also stated in: § 4.1.1 Methodology. · plain
“The entire exercise took about 630 person-hours and produced a taxonomy with 15 bug categories organized into five groups (”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: The manual labeling process and the resulting bug taxonomy are reliable and comprehensive.
- Also stated in: § 1 INTRODUCTION · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As for C2Rust, the 5% unsuccessful translations were due to compilation errors, the majority of them caused by unused imports.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With a further investigation on the type of translation bugs, we found that most GPT-4 translation bugs (71.8%) are related to unused imports, variables, and other target language-specific violations, followed by incorrect input parsing (10.3%).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using GPT-4, 50% of the translation bugs were due to compilation errors, 39.3% were caused by runtime errors, and 10.7% were due to test failures.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also observed cases of broken inheritance relations (resulting in missing behaviors/states) in translated code and incorrect translation of exception handling.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.3 Translation Bugs in Real-world Projects
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The most prevalent bugs in real-world projects are model-specific constraints (29.4%), missing imports (27.4%), and the removal of logic from the source code (15.7%),”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 6:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observed that none of the translated code using JavaToCSharp can compile and, in fact, require heavy rewriting of library APIs (e.g., System.inand java.util.Scanner).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When compared to LLMs, we found that CxGo outperforms all the open-source models with an accuracy of 62.25%, but falls 10% behind GPT-4.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 translates these to Python in different ways, some of which are correct translations, whereas others are erroneous.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.3 Translation Bugs in Real-world Projects
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Whereas, with CxGo, the majority of the bugs are related to input parsing (26.7%) and mismatch of behavior after replacing API call (26.7%).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the contrary, GPT-4 and StarCoder achieve 54.1% and 18.45% success rate in translating Java code to C#.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is evident from not only the occurrences of successful translations (three for Commons CLI vs. none for Click) and degenerate codegeneration instances (nine for Click vs none for Commons CLI), but also the translation bugs observed.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 5:
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For EvalPlus and the real-world projects, the occurrence of this bug is lesser (one for EvalPlus and none for the real-world projects).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.2 Prevalence of LLM-based Translation Bugs
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
- Rests on an assumption: The manual labeling process and the resulting bug taxonomy are reliable and comprehensive.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Key among these are: removal of logic in the source (e.g., source methods and field initialization not translated), inclusion of logic not in source (e.g., implementation added for a stubbed method), missing imports, mismatch in API behaviors after translation, mismatch of behaviors after replacing API calls and incorrect data types.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.3 Translation Bugs in Real-world Projects
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For instance, Commons CLI uses method overloading frequently.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.3 Translation Bugs in Real-world Projects
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finding 1b: Choosing the correct data type in the target PL is a crucial step that accounts for 34.3% of all data-related bugs and 11.5% of all bugs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 1b:
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The tests provided with the code samples are adequate to validate code translation.
- Rests on an assumption: The manual labeling process and the resulting bug taxonomy are reliable and comprehensive.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finding 6: Although the translation bug categories remain the same, occurrences of bug instances and their distribution vary between crafted benchmarks and real-world projects.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 6:
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In fact, all of the C2Rust generated code are unsafe, whereas, only three GPT-4-translated code are unsafe.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The biggest caveat is that executing code generated by CxGo requires specific libraries, i.e., stdio for I/O operations instead of a more generic fmt library.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Missing imports is the most frequently occurring bug category (35.5% of the bugs in the group and 8.6% of the bugs overall).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 2a:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Total/Average (All) | - | 1,700 10,858 | - | 6,197 | 8.1% | 2.8% | 14.5% | 47.3% 3.5% | 5.3% | 2.1% | |”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 10 CONCLUDING REMARKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“| Determinism and reasoning | ✓ | X |”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 10 CONCLUDING REMARKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With 𝑖𝑡𝑒𝑟 1 , we observe a substantial reduction in compilation errors for GPT-4, with 58% completely fixed (no error) and 38% transformed to other translation bugs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 6 MITIGATING TRANSLATION BUGS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“| Broader context | ✓ | X |”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 10 CONCLUDING REMARKS
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Leveraging target-language idioms | X | ✓ |”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 10 CONCLUDING REMARKS
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, for other errors, the percentage is lower-45% runtime errors, 48% functional errors, and 43% non-terminating executions-suggesting these bugs are harder to mitigate.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 6 MITIGATING TRANSLATION BUGS
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In both cases, we observe a few cases where the outcome of the translation degrades: i.e., functional error transforms to runtime/compilation error, or runtime error transforms to compilation error.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 6 MITIGATING TRANSLATION BUGS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Compilation Errors | 68.9 93.5 76.4 | 56.9 93.2 94.6 77.0 | 61.6 86.7 83.3 82.4 | 55.9 82.4 78.4 96.6 | 57.4 79.9 73.4 86.0 72.4 77.8 |”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 10 CONCLUDING REMARKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
In the paper
“Translating the software/code to a modern PL can significantly reduce maintenance effort, improve overall reliability, and boost non-functional properties such as security and performance [1][2][3][4][5]74].”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 1 INTRODUCTION
What the sources say
- Not matchedcites [1][2][3][4][5]74]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We believe this result is because, as mentioned, we excluded information about the translated code (imports, class declaration, and method signature) in the prompt.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.1 Effectiveness of LLMs in Code Translation
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our initial investigation showed that GPT-4 is the best-performing model and compared to others, its translations exhibit a variety of quality bugs worth investigating.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.1 Taxonomy of Translation Bugs
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The manual labeling process and the resulting bug taxonomy are reliable and comprehensive.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The next most common effect of unsuccessful translation is a functional error (13.4%), demonstrating that often translated code does not preserve the behavior of the source program.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 3.2 Outcome of Unsuccessful Translations
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our analysis shows that these two classes of techniques have their own strengths and weaknesses.”
- Also stated in: § 1 INTRODUCTION · hedged
“The comparison shows that LLM-and non-LLM-based translation techniques provide different and unique advantages, suggesting that an ultimate solution for code translation should combine both techniques (details in §5).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Abstract
plaincorrelationaln = 1,700- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 INTRODUCTION · hedged
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The nature of these bugs is significantly different as the non-LLM-based approaches tend to have more bugs related to the custom library, while LLM-based approaches have more syntax and semantics-related bugs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, this limitation can cause inconsistencies in the translated code, e.g., as illustrated for Finding 4, renaming overloaded methods requires all the related call sites to be updated.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 6:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, Python-to-Java translations can be more error-prone than Java-to-Python translations, in general, because going from a dynamic type system to a static one can be harder for an LLM to reason about.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 5:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, real-world applications have much more complex code than crafted benchmarks, making them much harder for LLMs to translate.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.3 Translation Bugs in Real-world Projects
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These bugs occur due to incorrect choice of the data type, differences between behaviors of equivalent data types across PLs, and differences in type systems of the source and target PLs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 1b:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The manual labeling process and the resulting bug taxonomy are reliable and comprehensive.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the equivalent Java test case (line 2 on right) fails because of a fundamental difference between Python float and Java Float: the former uses 64-bit precision whereas the latter uses 32-bit precision.The Java code thus cannot handle the large test data value, which works fine in Python.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Finding 1b:
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specifically, the use of scanf (as in 5 ) without buffer size specifications can lead to vulnerabilities such as buffer overruns.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finding 8: C2Rust generates non-idiomatic and unsafe code, whereas GPT-4 tends to generate safer and idiomatic code.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the other datasets used contain fairly rigorous test suites.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unlike crafted benchmarks, files in real-world projects do not exist as standalone programs, and providing relevant context about inter-file dependencies can help an LLM produce better code.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For LLM selection, we included several SOTA general and code LLMs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To mitigate this threat, after each labeling round, both labelers met to discuss any discrepancies and resolve them.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“At the end, each bug was assigned a single mutually agreed label.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This approach comes with the risk that inadequate or weak test suites can cause buggy translations that pass the test suites to be considered correct.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this respect, non-LLMbased approaches mostly do not leverage target language idioms, which can negatively affect code readability and maintainability.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“We also considered other approaches (mppSMT[57], Tree2Tree[29], and Sharpen[18]) but could not compare against them due to lack of tool/artifact availability ( §5).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 10 CONCLUDING REMARKS
What the sources say
- Not matchedcites [57]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [29] source
- Not matchedcites [18]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To address this threat, we thoroughly tested the scripts and spot-checked the results for correctness.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In some cases, this even leads to security issues (e.g., C to Rust).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Conversely, LLMs tend to generate more natural, human-like code.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, non-LLM-based approaches, specifically transpilers, have a more comprehensive context of an application.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, for non-LLM-based approaches, we used multiple tools covering different PLs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“As for threats to internal validity, one potential threat is that, for each translation task, we performed the translation once.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There can be cases where no suitable API exists in the target PL to translate an API call to-techniques that combine code summarization and code synthesis could be investigated to fill such gaps.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We measured translation correctness using the test cases provided in our datasets, which is a commonly used approach for assessing the quality of generated code.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Moreover, we make our artifacts publicly available [7] to enable review and replication of our results.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
What the sources say
- Not matchedcites [7]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In terms of datasets, we used multiple well-known datasets with different characteristics and also two real-world projects.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“We note that one of our datasets, CodeNet [63], contains one test case per sample and may be susceptible to this threat.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
What the sources say
- Not checked — no full text of the sourcecites [63] source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The findings of this work shows considerable scope for improving open-source and close-source LLMs for code translation.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“On top of that, one author went through the entire labeling to ensure consistency among the groups.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, a significant challenge in translating real-world projects is handling library API calls and the differences in the API ecosystem of different PLs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Another threat to internal validity is that we do not have a formal inter-rater reliability metric for our manual labeling of bugs.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This issue is further exacerbated by the use of implicit casting (e.g., 0 as libc::c_int as in 4 ) in the method body, which can cause unexpected behavior.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The unsafe block in main function (see 3 in Figure 4-(b)) circumvents Rust's safety checks for the code encompassed in that method.This increases the risk of memory safety violations.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 5 LLM-VS. NON-LLM-BASED TRANSLATION
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, while translating a module, the variable types, method signatures, folder structure, and dependencies associated with that module are often unavailable to LLMs because of their restriction on the token size.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 7 DISCUSSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Performing a translation task multiple times may change the success rates in translation as LLMs are inherently nondeterministic.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
- Rests on an assumption: LLMs are inherently nondeterministic.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, our results may be affected by bugs in our automation scripts.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 8 THREATS TO VALIDITY
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
There is a dearth of research on understanding and cataloging the limitations of LLMs for code translation.
“Although prior research has shown the potential of using LLMs for code translation, there is a dearth of research on understanding and cataloging their limitations for this task.”
The paper relies on this without support.
Code translation is a complex task that requires LLMs to understand code syntax and semantics simultaneously.
“This is an important undertaking because code translation is a complex task that requires LLMs to understand code syntax (to generate syntactically correct code) and semantics (to preserve functionality during translation) simultaneously.”
The paper relies on this without support.
The selected datasets and real-world projects are comprehensive enough to understand the nature of LLM translation bugs.
“To ensure the comprehensiveness of our findings and claims in understanding the nature of LLM translation bugs, we were interested in datasets used in prior studies as well as real-world projects.”
The paper relies on this without support.
The tests provided with the code samples are adequate to validate code translation.
“These datasets are accompanied by test cases to validate code translation.”
The paper relies on this without support.
The selected real-world projects are reasonably complex and well-maintained software exclusively written in Java or Python.
“Our goal was to translate reasonably complex and well-maintained software exclusively written in Java or Python.”
The paper relies on this without support.
The manual labeling process and the resulting bug taxonomy are reliable and comprehensive.
“The manual construction of the taxonomy involved eight human labelers, who are researchers or software engineers in the industry, and involved unsuccessful translations from all 31 translation pairs in Table 2.”
The paper relies on this without support.
Regardless of randomness or number of repetitions, the nature of an unsuccessful translation remains unchanged.
“Regardless of randomness or number of repetitions, the nature of an unsuccessful translation remains unchanged.”
The paper relies on this without support.
LLMs are inherently nondeterministic.
“Performing a translation task multiple times may change the success rates in translation as LLMs are inherently nondeterministic.”
The paper relies on this without support.
Non-LLM-based techniques are generally deterministic.
“Second, non-LLMbased techniques are generally deterministic, allowing for more predictable reasoning, whereas LLMs are inherently probabilistic, with a greater degree of creativity.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, they do not provide a sufficient number of repositories and corresponding automatic test suites for evaluation.”
From RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation (2026) · cites this paper as [1]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Recently, Pan et al. [1] manually study two opensource repositories (Apache Commons CLI [41] and Python Click [42]), and find that current LLMs struggle to perform code translation for entire repositories.”
From RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation (2026) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“Pan et al. [1] attempt to perform mutual conversion between Python and Java projects [41], [42], but find that the advanced LLMs are largely ineffective, with success rates of 8.1% for GPT-4 and 0% for the rest of the models.”
From RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation (2026) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“Recently, pan et al. [1] manually study two open-source repositories (Apache Commons CLI [41] and Python Click [42]) and find that current LLMs struggle to complete the translation tasks of entire repositories.”
From RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation (2026) · cites this paper as [1]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Besides, the resource and configuration files are ignored in this research.”
From RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation (2026) · cites this paper as [1]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > Dataset Collection and Pre-Processing.
“The translation of real-world projects never reached test execution, as it produced syntactically incorrect code (more discussion in §3).”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 10 CONCLUDING REMARKS
“Commons CLI [6] | Java | 22 | 310 | Python | 22 | 0.0% | 0.0% | 0.0% | 13.6% 0.0% | 0.0% | 0.0% | |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.1 Taxonomy of Translation Bugs
“To manage the manual effort for understanding and labeling bugs for building the taxonomy, we focused on 1,748 unsuccessful translations from GPT-4.”
§ Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code > 4.2 Prevalence of LLM-based Translation Bugs
“Among the 6197 attempted translations over 31 language pairs (Table 2), there were 1558 translation failures.We manually checked and labeled these failures to identify 1748 bugs.”
Citations
Cites
Adaptive Mixtures of Local Experts
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991) · Neural Computation · doi:10.1162/neco.1991.3.1.79 · Cited by 5,384
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100
How to Fine-Tune BERT for Text Classification?
Sun, C., Qiu, X., Xu, Y., & Huang, X. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32381-3_16 · Cited by 1,534
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2305.10601 · Cited by 1,158
A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., & Schmidt, D. C. (2023) · arXiv · doi:10.48550/arxiv.2302.11382 · Cited by 838
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Research works printed but not linked: 5
- Xinyun Chen, Chang Liu, and Dawn Song. 2018. Tree-to-tree neural networks for program translation. Advances in neural information processing systems 31 (2018).
- Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. StarCoder: may the source be with you! arXiv preprint arXiv:2305.06161 (2023). · arXiv
- Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, et al. 2021. CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
- Yuanbo Wen, Qi Guo, Qiang Fu, Xiaqing Li, Jianxing Xu, Yanlin Tang, Yongwei Zhao, Xing Hu, Zidong Du, Ling Li, et al. 2022. BabelTower: Learning to Auto-parallelized Program Translation. In International Conference on Machine Learning. PMLR, 23685--23700.
- WG Zhang, Arne J Berre, Dumitru Roman, and Hans Aage Huru. 2009. Migrating legacy applications to the service Cloud. In Proceedings of the 14th Conference Companion on Object Oriented Programming Systems Languages and Applications. 59--68.
Cited by
Sorted by citations
Towards an understanding of large language models in software engineering tasks
Zheng, Z., Ning, K., Zhong, Q., Chen, J., Chen, W., Guo, L., Wang, W., & Wang, Y. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10602-0 · Cited by 212
On the Evaluation of Large Language Models in Unit Test Generation
Yang, L., Yang, C., Gao, S., Wang, W., Wang, B., Zhu, Q., Chu, X., Zhou, J., Liang, G., Wang, Q., et al. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695529 · Cited by 158
Look Before You Leap: An Exploratory Study of Uncertainty Analysis for Large Language Models
Huang, Y., Song, J., Wang, Z., Zhao, S., Chen, H., Juefei-Xu, F., & Ma, L. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3519464 · Cited by 275
Bugs in large language models generated code: an empirical study
Tambon, F., Dakhel, A. M., Nikanjam, A., Khomh, F., Desmarais, M. C., & Antoniol, G. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10614-4 · Cited by 70
Generative Artificial Intelligence for Software Engineering—A Research Agenda
Nguyen‐Duc, A., Cabrero‐Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K., et al. (2025) · Software: Practice and Experience · doi:10.1002/spe.70005 · Cited by 60
Prompt Engineering or Fine-Tuning: An Empirical Assessment of LLMs for Code
Shin, J., Tang, C. J., Mohati, T., Nayebi, M., Wang, S., & Hemmati, H. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00082 · Cited by 165
Formal requirements engineering and large language models: A two-way roadmap
Ferrari, A., & Spoletini, P. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107697 · Cited by 71
Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMs
Cassano, F., Gouwar, J., Lucchetti, F., Schlesinger, C., Freeman, A., Anderson, C. J., Feldman, M. Q., Greenberg, M., Jangda, A., & Guha, A. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689735 · Cited by 88
Agents in software engineering: survey, landscape, and vision
Wang, Y., Zhong, W., Huang, Y., Shi, E., Yang, M., Chen, J., Li, H., Ma, Y., Wang, Q., & Zheng, Z. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00544-2 · Cited by 84
Farghal, M., & Haider, A. S. (2024) · Cogent Social Sciences · doi:10.1080/23311886.2024.2410998 · Cited by 33
Maltracker: A Fine-Grained NPM Malware Tracker Copiloted by LLM-Enhanced Dataset
Yu, Z., Wen, M., Guo, X., & Jin, H. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680397 · Cited by 42
Type-migrating C-to-Rust translation using a large language model
Hong, J., & Ryu, S. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10573-2 · Cited by 31
Ibrahimzada, A. R., Ke, K., Pawagi, M., Abid, M. S., Pan, R., Sinha, S., & Jabbarvand, R. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729379 · Cited by 27
Self-citationTowards Understanding the Characteristics of Code Generation Errors Made by Large Language Models
Wang, Z., Zhou, Z., Song, D., Huang, Y., Chen, S., Ma, L., & Zhang, T. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00180 · Cited by 87
Scalable, Validated Code Translation of Entire Projects using Large Language Models
Zhang, H., David, C., Wang, M., Paulsen, B., & Kroening, D. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3729315 · Cited by 23
An LLM-Based Agent-Oriented Approach for Automated Code Design Issue Localization
Batole, F., O'Brien, D. P., Nguyen, T. N., Dyer, R., & Rajan, H. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00100 · Cited by 20
Do Advanced Language Models Eliminate the Need for Prompt Engineering in Software Engineering?
Wang, G., Sun, Z., Ye, S., Gong, Z., Chen, Y., Zhao, Y., Liang, Q., & Hao, D. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3771933 · Cited by 26
Cracking SQL Barriers: An LLM-based Dialect Translation System
Zhou, W., Gao, Y., Zhou, X., & Li, G. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3725278 · Cited by 21
C2SaferRust: Transforming C Projects Into Safer Rust With NeuroSymbolic Techniques
Nitin, V., Krishna, R., do Valle, L. L., et al. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3641486 · Cited by 15
Self-citationINTERTRANS: Leveraging Transitive Intermediate Translations to Enhance LLM-Based Code Translation
Macedo, M., Tian, Y., Nie, P., Cogo, F. R., & Adams, B. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00236 · Cited by 14
Bhattarai, M., Santos, J. E., Jones, S., Biswas, A., Alexandrov, B., & O'Malley, D. (2024) · 2024 IEEE High Performance Extreme Computing Conference (HPEC) · doi:10.1109/hpec62836.2024.10938485 · Cited by 13
A 2030 Roadmap for Software Engineering
Pezzè, M., Abrahão, S., Penzenstadler, B., Poshyvanyk, D., Roychoudhury, A., & Yue, T. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3731559 · Cited by 13
RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation
Wang, Y., Wang, Y., Wang, S., Guo, D., Chen, J., Grundy, J., Liu, X., Ma, Y., Mao, M., Zhang, H., et al. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3645056 · Cited by 12
Translating code with Large Language Models and human-in-the-loop feedback
Siano, G. D. D., Fasolino, A. R., Sperlí, G., & Vignali, A. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107785 · Cited by 16
To Tag, or Not to Tag: Translating C's Unions to Rust's Tagged Unions
Hong, J., & Ryu, S. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3694985 · Cited by 28
Unraveling the Potential of Large Language Models in Code Translation: How Far are We?
Tao, Q., Yu, T., Gu, X., & Shen, B. (2024) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec65559.2024.00046 · Cited by 13
Type-Constrained Code Generation with Language Models
Mündler-Sasahara, N., He, J.-X., Wang, H., Sen, K., Song, D., & Vechev, M. T. (2025) · Proc. ACM Program. Lang. · doi:10.1145/3729274 · Cited by 77
Large language models in source code static analysis
Ignatyev, V. N., Shimchik, N. V., Panov, D. D., & Mitrofanov, A. A. (2024) · Ivannikov Memorial Workshop (IVMEM) · doi:10.1109/ivmem63006.2024.10659715 · Cited by 15
ClassEval-T: Evaluating Large Language Models in Class-Level Code Translation
Xue, P., Wu, L., Yang, Z., Wang, C., Li, X., Zhang, Y., Li, J., Jin, R., Pei, Y., Shen, Z., et al. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728940 · Cited by 15
One Size Does Not Fit All: Investigating Efficacy of Perplexity in Detecting LLM-Generated Code
Xu, J., Zhang, H., Yang, Y., Yang, L., Cheng, Z., Lyu, J., Liu, B., Zhou, X., Bacchelli, A., Chiam, Y. K., et al. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3748506 · Cited by 21
ZS4C: Zero-Shot Synthesis of Compilable Code for Incomplete Code Snippets Using LLMs
Kabir, A., Wang, S., Tian, Y., Chen, T.-H. P., Asaduzzaman, M., & Zhang, W. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3702979 · Cited by 13
Prompt Alchemy: Automatic Prompt Refinement for Enhancing Code Generation
Ye, S.-X., Sun, Z., Wang, G.-Q., Guo, L.-W., Liang, Q.-L., Li, Z., & Liu, Y. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3589634 · Cited by 19
RustMap: Towards Project-Scale C-to-Rust Migration via Program Analysis and LLM
Cai, X., Liu, J., Huang, X., et al. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-00828-2_16 · Cited by 7
Exploring the Potential of Llama Models in Automated Code Refinement: A Replication Study
Caumartin, G., Qin, Q., Chatragadda, S., Panjrolia, J., Li, H., & Costa, D. E. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00070 · Cited by 9
Program Slicing in the Era of Large Language Models
Shahandashti, K. K., Mohajer, M. M., Belle, A. B., Wang, S., & Hemmati, H. (2025) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac65507.2025.00155 · Cited by 9
Understanding Code Changes Practically with Small-Scale Language Models
Li, C., Xu, Z., Di, P., Wang, D., Li, Z., & Zheng, Q. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3694999 · Cited by 6
Challenging Bug Prediction and Repair Models with Synthetic Bugs
Ibrahimzada, A. R., Chen, Y., Rong, R., & Jabbarvand, R. (2025) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam67354.2025.00021 · Cited by 17
Self-citationVERT: Polyglot Verified Equivalent Rust Transpilation with Large Language Models
Yang, A. Z. H., Takashima, Y., Paulsen, B., et al. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00123 · Cited by 5
Zhang, R., Zhang, S., & Xie, L. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00570-0 · Cited by 5
Model-Driven Quantum Code Generation Using Large Language Models and Retrieval-Augmented Generation
Siavash, N., & Moin, A. (2025) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1109/models67397.2025.00031 · Cited by 4
IaC Generation with LLMs: An Error Taxonomy and A Study on Configuration Knowledge Injection
Nekrasov, R., Fossati, S., Kumara, I., Tamburri, D. A., & van den Heuvel, W. J. A. M. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3817608 · Cited by 6
EvoC2Rust: A Skeleton-guided Framework for Project-Level C-to-Rust Translation
Wang, C., Yu, T., Shen, B., Wang, J., Chen, D., Zhang, W., Shi, Y., Xie, C., & Gu, X. (2026) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3786583.3786856 · Cited by 4
Enterprise-Scale COBOL-to-Java Translation: LLMs Augmented with Program Analysis
Chakaravarthy, V. T., Choudhury, A. R., Garg, D., Kanvar, V., Pandian, S. S., Raghuvanshi, A., Sabharwal, Y., & Singhee, A. (2026) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3786583.3786915 · Cited by 4
Optimizing Type Migration for LLM-Based C-to-Rust Translation: A Data Flow Graph Approach
Xu, Q., & Huang, J. (2025) · doi:10.1145/3735544.3735582 · Cited by 4
Program Skeletons for Automated Program Translation
Wang, B., Li, T., Li, R., Mathur, U., & Saxena, P. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3729287 · Cited by 4
GlueTest: Testing Code Translation via Language Interoperability
Abid, M. S., Pawagi, M., Adhikari, S., Cheng, X., Badr, R., Wahiduzzaman, M., Rathi, V., Qi, R., Li, C., Liu, L., et al. (2024) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58944.2024.00061 · Cited by 4
Context Conquers Parameters: Outperforming Proprietary Llm in Commit Message Generation
Imani, A., Ahmed, I., & Moshirpour, M. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00048 · Cited by 14
A Tool for In-depth Analysis of Code Execution Reasoning of Large Language Models
Liu, C., & Jabbarvand, R. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3728605 · Cited by 14
Self-citationUsing AI to Automate the Modernization of Legacy Software Applications
Nitin, V. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695610 · Cited by 3
SwiftEval: Developing a Language-Specific Benchmark for LLM-generated Code Evaluation
Petrukha, I., Kurliak, Y., & Stulova, N. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00015 · Cited by 3
Automated description generation for software patches
Vu, T. T., Bui, T.-D., Do, T.-D., Nguyen, T.-T., Vo, H. D., & Nguyen, S. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107543 · Cited by 3
Dandamudi, R., & Rodríguez-Pérez, G. (2024) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1145/3691621.3694939 · Cited by 3
Li, H., Bezemer, C., & Hassan, A. E. (2025) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00033 · Cited by 12
Translating to a Low-Resource Language with Compiler Feedback: A Case Study on Cangjie
Wang, J., Su, C., Ou, Y., Li, Y., Tan, J., Chen, L., & Zhou, Y. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3594908 · Cited by 2
RustAssure: Differential Symbolic Testing for LLM-Transpiled C-to-Rust Code
Bai, Y., & Palit, T. (2025) · International Conference on Automated Software Engineering · doi:10.1109/ase63991.2025.00051 · Cited by 14
DLBench: A Comprehensive Benchmark for SQL Translation with Large Language Models
Lin, L., Chen, H., Zhu, Q., Chen, L., Tang, L., & Wu, R. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00076 · Cited by 3
Araújo, H. C., Lisboa, M. O., Coura, P., Freitas, I. D. C., Villela, M. L. B., & Ferreira, R. (2026) · Revista Brasileira de Informática na Educação · doi:10.5753/rbie.2026.6311 · Cited by 2
Environment-in-the-Loop: Rethinking Code Migration with LLM-based Agents
Li, X., Fei, Z., Ma, Y., Zhang, J., Sarro, F., & Ye, H. (2026) · doi:10.1145/3786180.3788315 · Cited by 8
Efficient Translation of Long Code Blocks Using Large Language Models
Chakaravarthy, V. T., Choudhury, A. R., Kanvar, V., Katan, R., Pandian, S. S., Raghuvanshi, A., & Sabharwal, Y. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner67736.2026.00048 · Cited by 2
RepoMod-Bench: A Benchmark for Code Repository Modernization via Implementation-Agnostic Testing
Li, X., Ben-Israel, N. I., Raz, Y., Ahmed, B., Serebro, D., & Raux, A. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817463 · Cited by 2
Native cross-platform app development using the SequalsK transpiler
Schultes, D., Schneider, L., Heymann, T., & Wild, F. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107626 · Cited by 2
Mass Generation of Programming Learning Problems from Public Code Repositories
Александрович, С. О., & Shashkov, D. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9030057 · Cited by 2
Porting Software Libraries to OpenHarmony: Transitioning from TypeScript or JavaScript to ArkTS
Zhou, B., Shi, J., Wang, Y., Li, L., Li, T. O., Yu, H., & Zhu, Z. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728941 · Cited by 2
Piccolo, S., & Stevens, H. P. (2025) · GigaScience · doi:10.1093/gigascience/giaf149 · Cited by 1
ApiRAT: Integrating Multi-source API Knowledge for Enhanced Code Translation with LLMs
Wang, C., Qiu, G., Gu, X., & Shen, B. (2025) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac65507.2025.00176 · Cited by 1
The fools are certain; the wise are doubtful: Exploring LLM confidence in code completion
Kotti, Z., Dritsa, K., Spinellis, D. D., & Λουρίδας, Π. (2026) · Automated Software Engineering · doi:10.21203/rs.3.rs-7302683/v1 · Cited by 3
RAG and Agentic Assistant: A Combined Approach
HajHmida, M. B., & Lee, E. A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-07132-3_4 · Cited by 1
Optimizing the Optimizer: An Example Showing the Power of LLM Code Generation
Sartori, C. C., & Blum, C. (2025) · Annals of Computer Science and Information Systems · doi:10.15439/2025f1481 · Cited by 1
Prompting Matters: Assessing the Effect of Prompting Techniques on LLM-Generated Class Code
Yuen, A., Pangas, J., Polash, M. M. H., & Abdellatif, A. (2025) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme64153.2025.00083 · Cited by 1
COB2PY - A Non-AI, Rule-Based COBOL to Python Translator
Challa, K. R., Sonith, M. V., Chiranjeevi, B., et al. (2025) · IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme64153.2025.00113 · Cited by 1
Can Emulating Semantic Translation Help LLMs with Code Translation? A Study Based on Pseudocode
Chen, S., Xu, C., Chen, J., Cao, J., Wu, J., & Cheung, S.-C. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3790101 · Cited by 1
Feischl, C., & Kern, R. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3777383 · Cited by 2
Output format biases in the evaluation of large language models for code translation
Macedo, M., Tian, Y., Cogo, F. R., & Adams, B. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10768-1 · Cited by 2
NL in the Middle: Code Translation with LLMs and Intermediate Representations
Tai, C.-E. A., Nie, P., Golab, L., & Wong, A. K. C. (2025) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1109/cascon66301.2025.00037 · Cited by 2
HFuzzer: Testing Large Language Models for Package Hallucinations via Phrase-based Fuzzing
Zhao, Y., Wu, M., Hu, X., & Xia, X. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00225 · Cited by 1
Security Risks of Transpiling C Programs to Rust
Wu, S., Yang, H., & Hua, B. (2025) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom66490.2025.00217 · Cited by 1
Ibrahimzada, A. R., Paulsen, B., Kroening, D., & Jabbarvand, R. (2026) · Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3832783.3837486 · Cited by 7
Self-citationMinsC2Rust: LLM-driven project-level code migration from C to safe Rust
Kang, Z., Zhu, Q., 缪文瑞, Wang, B., Zhang, X., & Huang, H. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10905-4 · Cited by 1
Project-Level C-to-Rust Translation via Pointer Knowledge Graphs
Yuan, Z.-Q., Mao, W., Chen, Z.-F., Shang, X., Wang, C., Lou, Y., & Peng, X. (2025) · Proc. ACM Softw. Eng. · doi:10.1145/3808169 · Cited by 5
Code–Test Co-translation: Towards Practical and Effective Program Migration in the Wild
Li, X., Xie, X., Wu, J., Liu, T., & Wang, H. (2026) · Proceedings of the ACM on Programming Languages · doi:10.1145/3839490
Bhutani, V., Toosi, F. G., & Buckley, J. (2025) · Applied Computer Systems · doi:10.2478/acss-2025-0013 · Cited by 1
RustRepoTrans: Repository-level Context Code Translation Benchmark Targeting Rust
Ou, G., Liu, M., Chen, Y., Wang, Y., Peng, X., & Zheng, Z. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00057 · Cited by 32
Data Race Detection through Vibe Translation
Hückelheim, J., Sathia, V., & Qian, S. B. (2025) · doi:10.1145/3731599.3767363 · Cited by 1
Translating and Optimising Computational Microscopy Algorithms with Large Language Models
Guzzi, F., Kourousias, G., Pugliese, R., Gianoncelli, A., & Billè, F. (2024) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro60963.2024.10569791
OptCodeTrans: Boost LLMs on Low-Resource Programming Language Translation
Lin, J., Shen, Y., Li, C., Niu, C., & Luo, B. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00014
Rethinking BFT: Leveraging Diverse Software Components with LLMs
Imperadeiro, J., Alonso, A. N., & Pereira, J. (2025) · IEEE/IFIP International Conference on Dependable Systems and Networks - Supplemental Volume (DSN-S) · doi:10.1109/dsn-s65789.2025.00067
Leveraging Large Language Models for MATLAB to Python Translation
Pietrini, R. (2025) · Smart innovation, systems and technologies · doi:10.1007/978-3-031-87154-2_49
Zhang, J., Zhang, J., Shen, X., et al. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3769305
Challenges in C++ to Rust Translation with Large Language Models: A Preliminary Empirical Study
Yan, Y., Feng, Y., He, Q., et al. (2025) · doi:10.1145/3759425.3763380
Evaluating and Improving GPT-Based Expansion of Abbreviations
Jiang, Y.-J., Li, C.-X., Zhao, Z.-X., Fan, F., Zhang, L., & Liu, H. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3623625
Bugs in AI-Generated Code - Understanding Bug Patterns and Possible Fix Strategies
Gao, R. (2025) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme64153.2025.00098
Project Phoenix – Resurrecting ASAPS, an Obsolete IDN Authoring System Using Generative AI
Koenitz, H. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-12408-1_25
Kustagi, V., Kulkarni, R., Patil, M., et al. (2025) · doi:10.1109/csitss67709.2025.11294543
Holopotyliuk, Y., Romashchenko, V., & Brutscheck, M. (2025) · Irish Signals and Systems Conference · doi:10.1109/issc67739.2025.11291512
Tayde, S. G., Mishra, A., & Agarwal, A. (2025) · International Conference on Sustainable Communication Networks and Application (ICSCNA) · doi:10.1109/icscn67106.2025.11308271
A Fully Automated Agent for End-to-End Code Translation and Validation
Erer, E., Bozanta, A., Aytac, T., & Basar, A. (2025) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem64174.2025.00058
WIP: Literature Review on LLMs to Improve Code
Nadendla, S., Karanam, S., & Gehringer, E. F. (2025) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie63693.2025.11328320
From Kotlin to Swift and Back: Toward Fully Automated Cross-Language Code Transpilation
Lad, S., Hanna, C., & Petke, J. (2025) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew67777.2025.00030
Machine Learning Solutions to Automated Governance and Code Analysis to Compliance
Paul, A. A., Harshini, T. V., & Maheswari, U. (2025) · International Conference on Applied Artificial Intelligence and Computing (ICAAIC) · doi:10.1109/icaaic64647.2025.11331106
A Fully Automated Agent for End-to-End Code Translation and Validation
Erer, E., Bozanta, A., Aytac, T., & Basar, A. (2025) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1109/cascon66301.2025.00128
Metadata
- Authors
- Pan, R., Ibrahimzada, A. R., Krishna, R., Sankar, D., Wassi, L. P., Merler, M., Sobolev, B., Pavuluri, R., Sinha, S., & Jabbarvand, R.
- Affiliations
- IBM (United States) (US)
- University of Illinois Urbana-Champaign (US)
- ORCID
- Venue
- Proceedings of the IEEE/ACM 46th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2024-04-12
- Pages
- 1-13
- OpenAlex
- W4394769102
- Cited by
- 133
- References
- 33
- Citations by year
- 2026: 43 · 2025: 72 · 2024: 18
- FWCI
- 51.11
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- IEEE/ACM International Conference on Software Engineering (ICSE) (publishedVersion, cc-by-nc)
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- program translation, large language models, bug classification, prompt engineering, program synthesis, programming language translation, LLM limitations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-16, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-16 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
