Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index; 21 of the 42 sources its claims cite were checked for support.

MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation

Open access

Abstract

Large language models have demonstrated the ability to generate both natural language and programming language text. Although contemporary code generation models are trained on corpora with several programming languages, they are tested using benchmarks that are typically monolingual. The most widely used code generation benchmarks only target Python, so there is little quantitative evidence of how code generation models perform on other programming languages. We propose MultiPL-E, a system for translating unit test-driven code generation benchmarks to new languages. We create the first massively multilingual code generation benchmark by using MultiPL-E to translate two popular Python code generation benchmarks to 18 additional programming languages. We use MultiPL-E to extend the HumanEval benchmark [1] and MBPP benchmark [2] to 18 languages that encompass a range of programming paradigms and popularity. Using these new parallel benchmarks, we evaluate the multi-language performance of three state-of-the-art code generation models: Codex [1], CodeGen [3]and InCoder [4]. We find that Codex matches or even exceeds its performance on Python for several other languages. The range of programming languages represented in MultiPL-E allow us to explore the impact of language frequency and language features on model performance. Finally, the MultiPL-E approach of compiling code generation benchmarks to new programming languages is both scalable and extensible, making it straightforward to evaluate new models, benchmarks, and languages.

Overview

CoChat Summary

TL;DR

MultiPL-E translates unit-test-driven Python code generation benchmarks into 18 additional programming languages, producing parallel HumanEval and MBPP benchmarks across 19 languages. Evaluating Codex, CodeGen, and InCoder shows Codex often matches or exceeds Python on several languages, especially JavaScript, while perplexity and type annotations do not reliably predict correctness.

Key findings

  • Codex reaches 45.9% pass@1 on Python HumanEval, compared with 28.8% reported earlier; CodeGen reaches 19.2% versus 18.3% reported earlier, and InCoder exactly replicates its prior Python HumanEval result. Codex’s JavaScript pass@1 is +2.3% over Python (p = 0.43), and it exceeds 40% pass@1 on C++, Java, TypeScript, PHP, Ruby, Rust, Scala, and Lua.
  • All three models perform best on high-frequency languages, but Codex performs well on some LOW and NICHE languages: Lua is the 9th-best language in its evaluation despite only 0.2% GitHub activity and no TIOBE Top-20 rank. There is no strong correlation between Codex pass@1 and perplexity; JavaScript and TypeScript have the highest perplexity while Codex performs best on them.
  • Type annotations show no overall effect on Codex pass@1 on MultiPL-HumanEval (p = 0.33) or MultiPL-MBPP (p = 0.23); removing Python annotations has no significant effect (p = 0.23), but replacing TypeScript types with `Any` hurts performance (-2.5%, p < 0.001).
  • Prompt design matters: Full Translation vs. Test-Only Translation is significant (p = 0.03), No Doctests vs. Test-Only Translation is significant (p < 0.001), and omitting Perl argument naming lowers Codex pass@1 by 8% (p < 0.001). MBPP is less challenging than HumanEval for all but 6 of 57 model/language pairs.

Methods

  • MultiPL-E uses 18 source-to-source compilers, each about 200 LOC, to translate function signatures, simple unit tests, doctests, Python-specific terminology, and type annotations from Python into target languages; a containerized sandbox compiles, runs, validates, and classifies generated programs.
  • The authors translated HumanEval (164 problems) and MBPP into 19 languages. For MBPP, they removed assertions from prompts and mechanically added inferred function signatures; they manually validated 23 problems across every language.
  • They evaluated Codex (175B `code-davinci-002`), CodeGen (16.1B), and InCoder (6.7B) using pass@k with 200 completions per prompt, temperature 0.2 for pass@1 and 0.8 for pass@10/pass@100, plus mixed-effects models for statistical comparisons.

Limitations

  • The paper states that translated benchmarks may not represent real-world programming tasks, that prompt design and sampling choices can affect pass rates and may not be optimal for non-Python languages, and that all prompts use Python doctest `>>>` notation.
  • The paper states that the approach excludes C and SQL, excludes 3 HumanEval problems with Python helper functions, modifies 2 to use unit tests instead of randomized testing, and fails to compile up to 5 problems for certain typed languages.
  • The text makes evident that Codex’s training set is not public, so train/test overlap cannot be independently verified; dictionary-problem results are based on only 3 problems; and language selection depended on author expertise.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/tse.2023.3267446

Cites

54 of 54 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 6 Pass@1 rates for all languages in MultiPL-HumanEval and MultiPL-MBPP. From left to right: InCoder, CodeGen, Codex.
Figure 6 Pass@1 rates for all languages in MultiPL-HumanEval and MultiPL-MBPP. From left to right: InCoder, CodeGen, Codex.CC-BY · From the PDF
Figure 7 Model performance on MultiPL-HumanEval by language frequency and type-checking. Languages to the left of dashed line are dynamically typed, and languages to the right are statically typed. Julia is unusual: type annotations are the norm and methods are type-checked at runtime.
Figure 7 Model performance on MultiPL-HumanEval by language frequency and type-checking. Languages to the left of dashed line are dynamically typed, and languages to the right are statically typed. Julia is unusual: type annotations are the norm and methods are type-checked at runtime.CC-BY · From the PDF
Image 3
CC-BY · From the PDF
Figure 9 Model performance on MultiPL-MBPP by language frequency and type-checking. Languages to the left of dashed line are dynamically typed; languages to the right are statically typed.
Figure 9 Model performance on MultiPL-MBPP by language frequency and type-checking. Languages to the left of dashed line are dynamically typed; languages to the right are statically typed.CC-BY · From the PDF
Figure 10 Ablation study of translation components, showing Codex pass@1 with original prompts; translated doctests; translated text and doctests; and doctests removed.
Figure 10 Ablation study of translation components, showing Codex pass@1 with original prompts; translated doctests; translated text and doctests; and doctests removed.CC-BY · From the PDF
Figure 11 Impact of programming language features on Codex pass@1 performance by language
Figure 11 Impact of programming language features on Codex pass@1 performance by languageCC-BY · From the PDF

Verification

Claims the paper makes

11 Valid · 8 Not valid · 18 Not checked · 97 Own results and interpretations

64 more claims · 1 with their cited sources checked

Create a free account to see all 134 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 36
  • It may state two separate claims as one. 38
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 54 references

Research works printed but not linked: 3

  1. Allal, SantaCoder: Dont reach for the stars!
  2. 10.18653/v1/D18-1425 · DOI · Paper page
  3. Tunstall, 2022, Natural Language Processing With Transformers

Cited by

See all 394

Metadata

Authors
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., Guha, A., Greenberg, M., & Jangda, A.
Affiliations
  • Northeastern University (US)
  • Hanover High School, Hanover, NH, USA
  • Wellesley College (US)
  • Oberlin College (US)
  • Stevens Institute of Technology (US)
  • Microsoft (United States) (US)
Venue
IEEE Transactions on Software Engineering
Published
2023-07
Volume
49
Issue
7
Pages
3675-3691
ISSN
  • 0098-5589 (ISSN-L)
  • 1939-3520
  • 2326-3881
OpenAlex
W4366204357
Cited by
394
References
54
Citations by year
2026: 149 · 2025: 142 · 2024: 88 · 2023: 15
FWCI
72.25
Type
journal article
Language
en
Open access
Locations
  • IEEE Transactions on Software Engineering (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
National Science Foundation (CCF-2052696)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
code generation benchmarks, multilingual programming languages, HumanEval, MBPP, codices, CodeGen, linguistic features, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation. https://app.cochat.ai/papers/10.1109/tse.2023.3267446
The paper
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., Guha, A., Greenberg, M., & Jangda, A. (2023). MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation. IEEE Transactions on Software Engineering. https://doi.org/10.1109/tse.2023.3267446

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.