CoChat

An Empirical Study of Problem-Aligned Multilingual Code Generation and Directed Code Translation by Large Language Models

Open access
Wang, Quanhe, Wen, Cheng, Liu, Dugang, Han, Xingjian, Yu, Bin, Chen, Ping, Qin, Shengchao, Tian, Cong · 2023 · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)
Cited by 282

The DOI matches its OpenAlex record; 2 of the 2 references it cites were found in a registry or an index.

Abstract

Background. Large language models (LLMs) can generate executable code, but their reliability across programming languages remains difficult to characterize. Aims. We investigate multilingual code capability under problem alignment, focusing on functional reliability, cross-language consistency, translation behavior, problem and sampling effects, and execution quality. Method. We present MciBench, covering 1,489 programming problems and eight languages, of which 1,466 have execution-validated coverage in all eight languages. We evaluate representative LLMs through multilingual generation and analyze source-code-conditioned translation on 100 randomly selected all-language-covered problems. Results. Performance varies substantially across models and target languages, while source-code-conditioned translation also exhibits marked variation across source-target settings. Problem difficulty and candidate budget materially affect correctness, strong aggregate performance does not necessarily imply cross-language consistency, and accepted programs differ in runtime and memory efficiency. Conclusions. Aggregate pass@k alone is insufficient for characterizing multilingual code capability, motivating evaluation that considers language consistency, problem sensitivity, sampling behavior, translation settings, and post-correctness execution quality.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

OpenAlex · 10.4230/lipics.esem.2026.25

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 2 references

Cited by

See all 282

Metadata

Authors
Wang, Q., Wen, C., Liu, D., Han, X., Yu, B., Chen, P., Qin, S., & Tian, C.
Venue
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)
Publisher
Schloss Dagstuhl – Leibniz Center for Informatics
Published
2023-01-01
OpenAlex
W4287024925
Cited by
282
References
2
FWCI
23.10
Type
journal article
Language
en
Open access
green

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: An Empirical Study of Problem-Aligned Multilingual Code Generation and Directed Code Translation by Large Language Models. https://app.cochat.ai/papers/10.4230/lipics.esem.2026.25
The paper
Wang, Q., Wen, C., Liu, D., Han, X., Yu, B., Chen, P., Qin, S., & Tian, C. (2023). An Empirical Study of Problem-Aligned Multilingual Code Generation and Directed Code Translation by Large Language Models. DROPS (Schloss Dagstuhl – Leibniz Center for Informatics). https://doi.org/10.4230/lipics.esem.2026.25

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.