The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index; 21 of the 42 sources its claims cite were checked for support.
MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation
Open accessAbstract
Large language models have demonstrated the ability to generate both natural language and programming language text. Although contemporary code generation models are trained on corpora with several programming languages, they are tested using benchmarks that are typically monolingual. The most widely used code generation benchmarks only target Python, so there is little quantitative evidence of how code generation models perform on other programming languages. We propose MultiPL-E, a system for translating unit test-driven code generation benchmarks to new languages. We create the first massively multilingual code generation benchmark by using MultiPL-E to translate two popular Python code generation benchmarks to 18 additional programming languages. We use MultiPL-E to extend the HumanEval benchmark [1] and MBPP benchmark [2] to 18 languages that encompass a range of programming paradigms and popularity. Using these new parallel benchmarks, we evaluate the multi-language performance of three state-of-the-art code generation models: Codex [1], CodeGen [3]and InCoder [4]. We find that Codex matches or even exceeds its performance on Python for several other languages. The range of programming languages represented in MultiPL-E allow us to explore the impact of language frequency and language features on model performance. Finally, the MultiPL-E approach of compiling code generation benchmarks to new programming languages is both scalable and extensible, making it straightforward to evaluate new models, benchmarks, and languages.
Overview
CoChat Summary
TL;DR
MultiPL-E translates unit-test-driven Python code generation benchmarks into 18 additional programming languages, producing parallel HumanEval and MBPP benchmarks across 19 languages. Evaluating Codex, CodeGen, and InCoder shows Codex often matches or exceeds Python on several languages, especially JavaScript, while perplexity and type annotations do not reliably predict correctness.
Key findings
- Codex reaches 45.9% pass@1 on Python HumanEval, compared with 28.8% reported earlier; CodeGen reaches 19.2% versus 18.3% reported earlier, and InCoder exactly replicates its prior Python HumanEval result. Codex’s JavaScript pass@1 is +2.3% over Python (p = 0.43), and it exceeds 40% pass@1 on C++, Java, TypeScript, PHP, Ruby, Rust, Scala, and Lua.
- All three models perform best on high-frequency languages, but Codex performs well on some LOW and NICHE languages: Lua is the 9th-best language in its evaluation despite only 0.2% GitHub activity and no TIOBE Top-20 rank. There is no strong correlation between Codex pass@1 and perplexity; JavaScript and TypeScript have the highest perplexity while Codex performs best on them.
- Type annotations show no overall effect on Codex pass@1 on MultiPL-HumanEval (p = 0.33) or MultiPL-MBPP (p = 0.23); removing Python annotations has no significant effect (p = 0.23), but replacing TypeScript types with `Any` hurts performance (-2.5%, p < 0.001).
- Prompt design matters: Full Translation vs. Test-Only Translation is significant (p = 0.03), No Doctests vs. Test-Only Translation is significant (p < 0.001), and omitting Perl argument naming lowers Codex pass@1 by 8% (p < 0.001). MBPP is less challenging than HumanEval for all but 6 of 57 model/language pairs.
Methods
- MultiPL-E uses 18 source-to-source compilers, each about 200 LOC, to translate function signatures, simple unit tests, doctests, Python-specific terminology, and type annotations from Python into target languages; a containerized sandbox compiles, runs, validates, and classifies generated programs.
- The authors translated HumanEval (164 problems) and MBPP into 19 languages. For MBPP, they removed assertions from prompts and mechanically added inferred function signatures; they manually validated 23 problems across every language.
- They evaluated Codex (175B `code-davinci-002`), CodeGen (16.1B), and InCoder (6.7B) using pass@k with 200 completions per prompt, temperature 0.2 for pass@1 and 0.8 for pass@10/pass@100, plus mixed-effects models for statistical comparisons.
Limitations
- The paper states that translated benchmarks may not represent real-world programming tasks, that prompt design and sampling choices can affect pass rates and may not be optimal for non-Python languages, and that all prompts use Python doctest `>>>` notation.
- The paper states that the approach excludes C and SQL, excludes 3 HumanEval problems with Python helper functions, modifies 2 to use unit tests instead of randomized testing, and fails to compile up to 5 problems for certain typed languages.
- The text makes evident that Codex’s training set is not public, so train/test overlap cannot be independently verified; dictionary-problem results are based on only 3 problems; and language selection depended on author expertise.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
11 Valid · 8 Not valid · 18 Not checked · 97 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“There is prior work on multi-language evaluation [7], but it uses perplexity as a proxy for performance, instead of benchmarks that check correctness.”
§ MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation > 1 INTRODUCTION
What the sources say
- Disputedcites [7] sourceExperimental — accuracy not yet measured
Section 5.1 (Language Use; page unavailable) compares GitHub Copilot across programming languages using completion acceptance rates. The Introduction and Section 8 (page unavailable) identify acceptance rate—not perplexity—as the paper’s proxy for performance and discuss its limitations. Section 2 discusses correctness-based evaluations as background, but the paper does not say it uses perplexity instead of them. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“[1] describe a 12B parameter version of Codex finetuned on 159 GB of deduplicated, filtered Python code from Github.”
§ MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation > 4.1 Models
What the sources say
- Supportedcites [1] Chen, 2021, Evaluating large language models trained on code source
Section 3 (p. 4) says GPT models with up to 12B parameters were fine-tuned on code to produce Codex. Section 3.1 (p. 4) says the GitHub dataset contained unique Python files and totaled 159 GB after filtering; the paper also identifies a 12B-parameter Codex (p. 2). (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“We do not observe a strong correlation between Codex pass@1 scores and their perplexity scores (Figure 8).”
§ MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation > 5.2.4 Perplexity and Code Correctness Do Not Correlate
What the sources say
- Could not verifycites [7] sourceExperimental — accuracy not yet measured
Section 5.1 discusses differences in GitHub Copilot acceptance rates across programming languages, while Section 4 correlates usage metrics with perceived productivity. Neither addresses a correlation between Codex pass@1 scores and perplexity scores. Searches for those terms and Figure 8 found no passage supporting the claim. (LLM, conf=0.76)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“Empirical results show these settings are optimal for Python [1], but it is possible that different settings would be better for other programming languages.”
§ MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation > 7 THREATS TO VALIDITY
What the sources say
- Supportedcites [1] Chen, 2021, Evaluating large language models trained on code source
Section 3.3 (pp. 4–5) reports empirically optimal sampling temperatures for Codex on the Python HumanEval tasks, with the best temperature depending on the number of samples. The finding is specific to that evaluation, not a claim of universal optimality for all Python code generation. (LLM, conf=0.88)
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“We evaluate a more recent Codex model (codedavinci-002) than the original paper and observe a large improvement on Python: a pass@1 rate of 45.9%, compared to 28.8% reported earlier [1].”
§ MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation > 5.2.1 Python Results and Replication
What the sources say
- Supportedcites [1] Chen, 2021, Evaluating large language models trained on code source
The citation supports the earlier 28.8% benchmark, not the citing authors’ own 45.9% result for code-davinci-002. Table 1 (p. 7) reports 28.81% pass@1 for Codex-12B on HumanEval, and the Figure 1 caption (p. 2) rounds this to 28.8% for the paper’s Python-function task. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
64 more claims · 1 with their cited sources checked
Create a free account to see all 134 claims and every source verdict.
- Probably not a substantial claim. 36
- It may state two separate claims as one. 38
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, their rules have limited support for type handling (e.g., they cannot handle a list with hybrid types)”
From CRUXEVAL-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution (2025) · cites this paper as Cassano et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Some works focus on the multilingual scenarios (Wang et al., 2023; Athiwaratkun et al., 2023; Peng et al., 2024; Zheng et al., 2023b) by extending the Python-only HumanEval/MBPP benchmark (e.g. MultiPL-E (Cassano et al., 2023)), which is challenged by the number of the languages and data leaking problem (Jain et al., 2024).”
From McEval: Massively Multilingual Code Evaluation (2024) · cites this paper as Cassano et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Prior work has extended HumanEval to new programming languages via automatic translation mechanisms (Athiwaratkun et al., 2022;Cassano et al., 2023;Orlanski et al., 2023).”
From OctoPack: Instruction Tuning Code Large Language Models (2023) · cites this paper as Cassano et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“This improves performance significantly for less common languages as measured by the MultiPL-E (Cassano et al., 2023) benchmark.”
From The Llama 3 Herd of Models (2024) · cites this paper as Cassano et al., 2023
In the citing paper
“Rule-based translation (Cassano et al., 2023;Ling et al., 2022) usually suffers from generalizability issues, making them limited in handling prescribed code structures.”
From CRUXEVAL-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution (2025) · cites this paper as Cassano et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation > 5.2.2 Codex Performs Best on JavaScript
“However, that Codex matches or exceeds its Python performance on 18 new languages suggests a negligible impact of any train/test overlap.”
§ MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation > 5.2 MultiPL-HumanEval results
“We find reliable differences between Codex pass@1 rates on MultiPL-HumanEval for Python and all but 4 languages: C++, JavaScript, Scala, and TypeScript.For these languages, Codex performance is similar to Python.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation > 6.3 Type Annotations
“In Figure 7 and Figure 9, the dashed lines in each category separate languages with type annotations (left) from languages without (right).”
§ MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation > 9 CONCLUSION
“Languages to the left of dashed line are dynamically typed, and languages to the right are statically typed.”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“We evaluate three state-of-the-art code generation models, each of which use a Transformer architecture [29] and are trained with a language modeling objective on a mixture of natural language and code.”
MethodFitting Linear Mixed-Effects Models Using lme4
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015) · Journal of Statistical Software · doi:10.18637/jss.v067.i01 · Cited by 86,835
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 1×
“For instance, Feng et al. [16] evaluate their CodeBERT model on six programming languages using BLEU [35].”
MethodCodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824 · cited 6×
“Two influential early models were CodeBERT [16] and PyMT [17].”
BackgroundMethodInfluentialPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 1×
“More recent models include PolyCoder [6], CodeParrot [39], AlphaCode [40], and PaLM-Coder [41].”
MethodGebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 9×
“Over the past few years, code generation models have demonstrated their utility on a wide variety of software engineering tasks, including test generation, documentation generation, and even synthesizing working programs from natural language descriptions [1], [3], [4], [5], [6].”
BackgroundMethodInfluentialThe Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304
Research works printed but not linked: 3
- Allal, SantaCoder: Dont reach for the stars!
- 10.18653/v1/D18-1425 · DOI · Paper page
- Tunstall, 2022, Natural Language Processing With Transformers
Cited by
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695988 · Cited by 1,029
Code Llama: Open Foundation Models for Code
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12950 · Cited by 531
“For that, we use the MultiPL-E benchmark (Cassano et al., 2023).”
BackgroundMethodInfluentialReflexion: Language Agents with Verbal Reinforcement Learning
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11366 · Cited by 1,087
Self-citationSWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang, J., Jimenez, C., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1601 · Cited by 264
Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues
Liu, Y., Le-Cong, T., Widyasari, R., Tantithamthavorn, C., Li, L., Le, X.-B. D., & Lo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3643674 · Cited by 149
“Specifically, popular benchmarks like HumanEval [9] encompassing 164 Python tasks, and MBPP [3] containing 974 programming tasks, have been widely used by prior research [7–9 , 32 ].”
Method
Metadata
- Authors
- Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., Guha, A., Greenberg, M., & Jangda, A.
- Affiliations
- Northeastern University (US)
- Hanover High School, Hanover, NH, USA
- Wellesley College (US)
- Oberlin College (US)
- Stevens Institute of Technology (US)
- Microsoft (United States) (US)
- ORCID
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2023-07
- Volume
- 49
- Issue
- 7
- Pages
- 3675-3691
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W4366204357
- Semantic Scholar
- f9acfdf58ccc27ac7d4b815ef8a2b9e03c5b215a
- Cited by
- 394
- References
- 54
- Citations by year
- 2026: 149 · 2025: 142 · 2024: 88 · 2023: 15
- FWCI
- 72.25
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- IEEE Transactions on Software Engineering (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (CCF-2052696)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- code generation benchmarks, multilingual programming languages, HumanEval, MBPP, codices, CodeGen, linguistic features, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation. https://app.cochat.ai/papers/10.1109/tse.2023.3267446
- The paper
- Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., Guha, A., Greenberg, M., & Jangda, A. (2023). MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation. IEEE Transactions on Software Engineering. https://doi.org/10.1109/tse.2023.3267446
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.