An Empirical Study of Problem-Aligned Multilingual Code Generation and Directed Code Translation by Large Language Models
Open accessThe DOI matches its OpenAlex record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
Background. Large language models (LLMs) can generate executable code, but their reliability across programming languages remains difficult to characterize. Aims. We investigate multilingual code capability under problem alignment, focusing on functional reliability, cross-language consistency, translation behavior, problem and sampling effects, and execution quality. Method. We present MciBench, covering 1,489 programming problems and eight languages, of which 1,466 have execution-validated coverage in all eight languages. We evaluate representative LLMs through multilingual generation and analyze source-code-conditioned translation on 100 randomly selected all-language-covered problems. Results. Performance varies substantially across models and target languages, while source-code-conditioned translation also exhibits marked variation across source-target settings. Problem difficulty and candidate budget materially affect correctness, strong aggregate performance does not necessarily imply cross-language consistency, and accepted programs differ in runtime and memory efficiency. Conclusions. Aggregate pass@k alone is insufficient for characterizing multilingual code capability, motivating evaluation that considers language consistency, problem sensitivity, sampling behavior, translation settings, and post-correctness execution quality.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., et al. (2022) · arXiv · doi:10.48550/arxiv.2208.08227 · Cited by 17
Cited by
Code as Policies: Language Model Programs for Embodied Control
Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., & Zeng, A. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10160591 · Cited by 767
Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions
Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833571 · Cited by 476
Recent advancements and challenges of NLP-based sentiment analysis: A state-of-the-art review
Jim, J. R., Talukder, M. A. R., Malakar, P., Kabir, M. M., Nur, K., & Mridha, M. F. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100059 · Cited by 297
CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models
Lemieux, C., Inala, J. P., Lahiri, S. K., & Sen, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00085 · Cited by 306
Automated Repair of Programs from Large Language Models
Fan, Z., Gao, X., Mirchev, M., Roychoudhury, A., & Tan, S. H. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00128 · Cited by 256
Metadata
- Authors
- Wang, Q., Wen, C., Liu, D., Han, X., Yu, B., Chen, P., Qin, S., & Tian, C.
- Venue
- DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)
- Publisher
- Schloss Dagstuhl – Leibniz Center for Informatics
- Published
- 2023-01-01
- OpenAlex
- W4287024925
- Cited by
- 282
- References
- 2
- FWCI
- 23.10
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: An Empirical Study of Problem-Aligned Multilingual Code Generation and Directed Code Translation by Large Language Models. https://app.cochat.ai/papers/10.4230/lipics.esem.2026.25
- The paper
- Wang, Q., Wen, C., Liu, D., Han, X., Yu, B., Chen, P., Qin, S., & Tian, C. (2023). An Empirical Study of Problem-Aligned Multilingual Code Generation and Directed Code Translation by Large Language Models. DROPS (Schloss Dagstuhl – Leibniz Center for Informatics). https://doi.org/10.4230/lipics.esem.2026.25
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.