MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation
Open accessThe DOI matches its Crossref record; 53 of the 54 references it cites were found in a registry or an index; 21 of the 42 sources its claims cite were checked for support.
Abstract
Large language models have demonstrated the ability to generate both natural language and programming language text. Although contemporary code generation models are trained on corpora with several programming languages, they are tested using benchmarks that are typically monolingual. The most widely used code generation benchmarks only target Python, so there is little quantitative evidence of how code generation models perform on other programming languages. We propose MultiPL-E, a system for translating unit test-driven code generation benchmarks to new languages. We create the first massively multilingual code generation benchmark by using MultiPL-E to translate two popular Python code generation benchmarks to 18 additional programming languages. We use MultiPL-E to extend the HumanEval benchmark [1] and MBPP benchmark [2] to 18 languages that encompass a range of programming paradigms and popularity. Using these new parallel benchmarks, we evaluate the multi-language performance of three state-of-the-art code generation models: Codex [1], CodeGen [3]and InCoder [4]. We find that Codex matches or even exceeds its performance on Python for several other languages. The range of programming languages represented in MultiPL-E allow us to explore the impact of language frequency and language features on model performance. Finally, the MultiPL-E approach of compiling code generation benchmarks to new programming languages is both scalable and extensible, making it straightforward to evaluate new models, benchmarks, and languages.
Overview
CoChat Summary
TL;DR
MultiPL-E translates unit-test-driven Python code generation benchmarks into 18 additional programming languages, producing parallel HumanEval and MBPP benchmarks across 19 languages. Evaluating Codex, CodeGen, and InCoder shows Codex often matches or exceeds Python on several languages, especially JavaScript, while perplexity and type annotations do not reliably predict correctness.
Key findings
- Codex reaches 45.9% pass@1 on Python HumanEval, compared with 28.8% reported earlier; CodeGen reaches 19.2% versus 18.3% reported earlier, and InCoder exactly replicates its prior Python HumanEval result. Codex’s JavaScript pass@1 is +2.3% over Python (p = 0.43), and it exceeds 40% pass@1 on C++, Java, TypeScript, PHP, Ruby, Rust, Scala, and Lua.
- All three models perform best on high-frequency languages, but Codex performs well on some LOW and NICHE languages: Lua is the 9th-best language in its evaluation despite only 0.2% GitHub activity and no TIOBE Top-20 rank. There is no strong correlation between Codex pass@1 and perplexity; JavaScript and TypeScript have the highest perplexity while Codex performs best on them.
- Type annotations show no overall effect on Codex pass@1 on MultiPL-HumanEval (p = 0.33) or MultiPL-MBPP (p = 0.23); removing Python annotations has no significant effect (p = 0.23), but replacing TypeScript types with `Any` hurts performance (-2.5%, p < 0.001).
- Prompt design matters: Full Translation vs. Test-Only Translation is significant (p = 0.03), No Doctests vs. Test-Only Translation is significant (p < 0.001), and omitting Perl argument naming lowers Codex pass@1 by 8% (p < 0.001). MBPP is less challenging than HumanEval for all but 6 of 57 model/language pairs.
Methods
- MultiPL-E uses 18 source-to-source compilers, each about 200 LOC, to translate function signatures, simple unit tests, doctests, Python-specific terminology, and type annotations from Python into target languages; a containerized sandbox compiles, runs, validates, and classifies generated programs.
- The authors translated HumanEval (164 problems) and MBPP into 19 languages. For MBPP, they removed assertions from prompts and mechanically added inferred function signatures; they manually validated 23 problems across every language.
- They evaluated Codex (175B `code-davinci-002`), CodeGen (16.1B), and InCoder (6.7B) using pass@k with 200 completions per prompt, temperature 0.2 for pass@1 and 0.8 for pass@10/pass@100, plus mixed-effects models for statistical comparisons.
Limitations
- The paper states that translated benchmarks may not represent real-world programming tasks, that prompt design and sampling choices can affect pass rates and may not be optimal for non-Python languages, and that all prompts use Python doctest `>>>` notation.
- The paper states that the approach excludes C and SQL, excludes 3 HumanEval problems with Python helper functions, modifies 2 to use unit tests instead of randomized testing, and fails to compile up to 5 problems for certain typed languages.
- The text makes evident that Codex’s training set is not public, so train/test overlap cannot be independently verified; dictionary-problem results are based on only 3 problems; and language selection depended on author expertise.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
11 Valid · 8 Not valid · 18 Not checked · 97 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“There is prior work on multi-language evaluation [7], but it uses perplexity as a proxy for performance, instead of benchmarks that check correctness.”
§ MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation > 1 INTRODUCTION
What the sources say
- Disputedcites [7] A. Ziegler, E. Kalliamvakou, X. A. Li, A. Rice, D. Rifkin, S. Simister, G. Sittampalam, and E. Aftandilian, "Pro- ductivity assessment of neural code completion," in Proceedings of the 6th ACM SIGPLAN International Sym- posium on Machine Programming, 2022, pp. 21-29. sourceExperimental — accuracy not yet measured
Section 5.1 (Language Use; page unavailable) compares GitHub Copilot across programming languages using completion acceptance rates. The Introduction and Section 8 (page unavailable) identify acceptance rate—not perplexity—as the paper’s proxy for performance and discuss its limitations. Section 2 discusses correctness-based evaluations as background, but the paper does not say it uses perplexity instead of them. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“[1] describe a 12B parameter version of Codex finetuned on 159 GB of deduplicated, filtered Python code from Github.”
§ MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation > 4.1 Models
What the sources say
- Supportedcites [1] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., "Evaluating large language models trained on code," arXiv preprint arXiv:2107.03374, 2021. source
Section 3 (p. 4) says GPT models with up to 12B parameters were fine-tuned on code to produce Codex. Section 3.1 (p. 4) says the GitHub dataset contained unique Python files and totaled 159 GB after filtering; the paper also identifies a 12B-parameter Codex (p. 2). (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“We do not observe a strong correlation between Codex pass@1 scores and their perplexity scores (Figure 8).”
§ MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation > 5.2.4 Perplexity and Code Correctness Do Not Correlate
What the sources say
- Could not verifycites [7] A. Ziegler, E. Kalliamvakou, X. A. Li, A. Rice, D. Rifkin, S. Simister, G. Sittampalam, and E. Aftandilian, "Pro- ductivity assessment of neural code completion," in Proceedings of the 6th ACM SIGPLAN International Sym- posium on Machine Programming, 2022, pp. 21-29. sourceExperimental — accuracy not yet measured
Section 5.1 discusses differences in GitHub Copilot acceptance rates across programming languages, while Section 4 correlates usage metrics with perceived productivity. Neither addresses a correlation between Codex pass@1 scores and perplexity scores. Searches for those terms and Figure 8 found no passage supporting the claim. (LLM, conf=0.76)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“Empirical results show these settings are optimal for Python [1], but it is possible that different settings would be better for other programming languages.”
§ MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation > 7 THREATS TO VALIDITY
What the sources say
- Supportedcites [1] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., "Evaluating large language models trained on code," arXiv preprint arXiv:2107.03374, 2021. source
Section 3.3 (pp. 4–5) reports empirically optimal sampling temperatures for Codex on the Python HumanEval tasks, with the best temperature depending on the number of samples. The finding is specific to that evaluation, not a claim of universal optimality for all Python code generation. (LLM, conf=0.88)
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“We evaluate a more recent Codex model (codedavinci-002) than the original paper and observe a large improvement on Python: a pass@1 rate of 45.9%, compared to 28.8% reported earlier [1].”
§ MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation > 5.2.1 Python Results and Replication
What the sources say
- Supportedcites [1] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., "Evaluating large language models trained on code," arXiv preprint arXiv:2107.03374, 2021. source
The citation supports the earlier 28.8% benchmark, not the citing authors’ own 45.9% result for code-davinci-002. Table 1 (p. 7) reports 28.81% pass@1 for Codex-12B on HumanEval, and the Figure 1 caption (p. 2) rounds this to 28.8% for the paper’s Python-function task. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
64 more claims · 1 with their cited sources checked
Create a free account to see all 134 claims and every source verdict.
- Probably not a substantial claim. 36
- It may state two separate claims as one. 38
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, their rules have limited support for type handling (e.g., they cannot handle a list with hybrid types)”
From CRUXEVAL-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution (2025) · cites this paper as Cassano et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Some works focus on the multilingual scenarios (Wang et al., 2023; Athiwaratkun et al., 2023; Peng et al., 2024; Zheng et al., 2023b) by extending the Python-only HumanEval/MBPP benchmark (e.g. MultiPL-E (Cassano et al., 2023)), which is challenged by the number of the languages and data leaking problem (Jain et al., 2024).”
From McEval: Massively Multilingual Code Evaluation (2024) · cites this paper as Cassano et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Prior work has extended HumanEval to new programming languages via automatic translation mechanisms (Athiwaratkun et al., 2022;Cassano et al., 2023;Orlanski et al., 2023).”
From OctoPack: Instruction Tuning Code Large Language Models (2023) · cites this paper as Cassano et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“This improves performance significantly for less common languages as measured by the MultiPL-E (Cassano et al., 2023) benchmark.”
From The Llama 3 Herd of Models (2024) · cites this paper as Cassano et al., 2023
In the citing paper
“Rule-based translation (Cassano et al., 2023;Ling et al., 2022) usually suffers from generalizability issues, making them limited in handling prescribed code structures.”
From CRUXEVAL-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution (2025) · cites this paper as Cassano et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation > 5.2.2 Codex Performs Best on JavaScript
“However, that Codex matches or exceeds its Python performance on 18 new languages suggests a negligible impact of any train/test overlap.”
§ MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation > 5.2 MultiPL-HumanEval results
“We find reliable differences between Codex pass@1 rates on MultiPL-HumanEval for Python and all but 4 languages: C++, JavaScript, Scala, and TypeScript.For these languages, Codex performance is similar to Python.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation > 6.3 Type Annotations
“In Figure 7 and Figure 9, the dashed lines in each category separate languages with type annotations (left) from languages without (right).”
§ MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation > 9 CONCLUSION
“Languages to the left of dashed line are dynamically typed, and languages to the right are statically typed.”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“We evaluate three state-of-the-art code generation models, each of which use a Transformer architecture [29] and are trained with a language modeling objective on a mixture of natural language and code.”
MethodFitting Linear Mixed-Effects Models Using lme4
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015) · Journal of Statistical Software · doi:10.18637/jss.v067.i01 · Cited by 86,835
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 1×
“For instance, Feng et al. [16] evaluate their CodeBERT model on six programming languages using BLEU [35].”
MethodCodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824 · cited 6×
“Two influential early models were CodeBERT [16] and PyMT [17].”
BackgroundMethodInfluentialPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 1×
“More recent models include PolyCoder [6], CodeParrot [39], AlphaCode [40], and PaLM-Coder [41].”
MethodGebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 9×
“Over the past few years, code generation models have demonstrated their utility on a wide variety of software engineering tasks, including test generation, documentation generation, and even synthesizing working programs from natural language descriptions [1], [3], [4], [5], [6].”
BackgroundMethodInfluentialThe Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · cited 3×
“AlphaCode and PaLM-Coder are not available for academic researchers to investigate.”
MethodCompetition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043
Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., et al. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1425 · Cited by 1,034 · cited 1×
“We do not include two previously studied languages, C [6] and SQL [27], [28] because they do not meet this criterion.”
BackgroundA systematic evaluation of large language models of codePreprint
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. (2022) · MAPS@PLDI · doi:10.48550/arxiv.2202.13169 · Cited by 930
Evaluating large language models trained on code
Chen, M. (2024) · TIB Data Manager · doi:10.57702/wbv3e61b · Cited by 737
Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
Zhong, V., Xiong, C., & Socher, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.00103 · Cited by 733 · cited 2×
“Zhong et al. [28] and Yu et al. [38] propose benchmarks for SQL.”
BackgroundAlur, R., Bodik, R., Juniwal, G., Martin, M. M. K., Raghothaman, M., Seshia, S. A., Singh, R., Solar-Lezama, A., Torlak, E., & Udupa, A. (2013) · 2013 Formal Methods in Computer-Aided Design · doi:10.1109/fmcad.2013.6679385 · Cited by 656
A systematic evaluation of large language models of code
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534862 · Cited by 540 · cited 9×
“Over the past few years, code generation models have demonstrated their utility on a wide variety of software engineering tasks, including test generation, documentation generation, and even synthesizing working programs from natural language descriptions [1], [3], [4], [5], [6].”
BackgroundMethodInfluentialCodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.04664 · Cited by 536 · cited 2×
“Lu et al. [19] propose a suite of evaluation datasets for ten tasks, including code translation, docstring generation, and code summarization.”
BackgroundThe Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492 · cited 1×
“We evaluate the multilingual CodeGen model, which was trained first on The Pile [31], a 825 GB dataset of mostly natural language text with around 8% Github-scraped code.”
MethodGPT-NeoX-20B: An Open-Source Autoregressive Language Model
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. (2022) · Proceedings of BigScience Episode #5 -- Workshop on Challenges & Perspectives in Creating Large Language Models · doi:10.18653/v1/2022.bigscience-1.9 · Cited by 404
Program Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388
Unit Test Case Generation with Transformers and Focal ContextPreprint
Tufano, M., Drain, D., Svyatkovskiy, A., Deng, S. K., & Sundaresan, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.05617 · Cited by 346
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., & Xiong, C. (2022) · arXiv · doi:10.48550/arxiv.2203.13474 · Cited by 330
Gulwani, S., Polozov, O., & Singh, R. (2017) · Foundations and Trends in Programming Languages · doi:10.1561/2500000010 · Cited by 319
Wang, Q., Wen, C., Liu, D., Han, X., Yu, B., Chen, P., Qin, S., & Tian, C. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.esem.2026.25 · Cited by 282
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · arXiv · doi:10.48550/arxiv.2002.08155 · Cited by 243
Learning to mine aligned code and natural language pairs from stack overflow
Yin, P., Deng, B., Chen, E., Vasilescu, B., & Neubig, G. (2018) · Proceedings of the 15th International Conference on Mining Software Repositories · doi:10.1145/3196398.3196408 · Cited by 218 · cited 5×
“There are a number of existing single-language NL2Code benchmarks [8], [9], [10].”
BackgroundInfluentialCodeBLEU: a Method for Automatic Evaluation of Code Synthesis
Ren, S., Guo, D., Lu, S., Zhou, L., Liu, S., Tang, D., Sundaresan, N., Zhou, M., Blanco, A., & Ma, S. (2020) · arXiv · doi:10.48550/arxiv.2009.10297 · Cited by 214
Productivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534864 · Cited by 200 · cited 1×
“New products such as GitHub Copilot 1 , Amazon CodeWhisperer 2 , and Tabnine 3 built on code generation models are growing in popularity with developers [7].”
BackgroundProductivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Simister, S., Sittampalam, G., Li, A., Rice, A., Rifkin, D., & Aftandilian, E. (2022) · arXiv · doi:10.48550/arxiv.2205.06537 · Cited by 200
Rajeev, A., Rastislav, B., Eric, D., Dana, F., Pranav, G., Garvit, J., Hadas, K.-G., Madhusudan, P., Martin Milo, M. K., Mukund, R., et al. (2015) · NATO Science for Peace and Security Series - D: Information and Communication Security · doi:10.3233/978-1-61499-495-4-1 · Cited by 200
Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., et al. (2018) · arXiv · doi:10.48550/arxiv.1809.08887 · Cited by 193
Drori, I., Zhang, S., Shuttleworth, R., Tang, L., Lu, A., Ke, E., Liu, K., Chen, L., Tran, S., Cheng, N., et al. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2123433119 · Cited by 187 · cited 1×
“Code generation models have been applied to a variety of tasks, including test generation [18], docstring generation [19], code search [16], [20], type inference [21], [22], [23], and more [24].”
BackgroundInCoder: A Generative Model for Code Infilling and Synthesis
Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, W.-T., Zettlemoyer, L., & Lewis, M. (2022) · arXiv · doi:10.48550/arxiv.2204.05999 · Cited by 187 · cited 9×
“Since 2020, several code generation models have been trained or fine-tuned on programming language text [1], [3], [4], [6], [16], [17].”
BackgroundMethodInfluentialHellendoorn, V. J., Bird, C., Barr, E. T., & Allamanis, M. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3236051 · Cited by 181
Mapping Language to Code in Programmatic Context
Iyer, S., Konstas, I., Cheung, A., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1192 · Cited by 180 · cited 1×
“Iyer et al. [37] present a Java benchmark taken from online code repositories.”
Measuring Coding Challenge Competence With APPS
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.09938 · Cited by 142 · cited 3×
“There are a number of existing single-language NL2Code benchmarks [8], [9], [10].”
BackgroundMethodChaudhuri, S., Ellis, K., Polozov, O., Singh, R., Solar-Lezama, A., & Yue, Y. (2021) · Foundations and Trends® in Programming Languages · doi:10.1561/2500000049 · Cited by 133 · cited 1×
“Code generation has long been a task of interest: there is extensive work on program synthesis [11], [12], [13] using both symbolic and neuro-symbolic approaches.”
BackgroundPyMT5: multi-mode translation of natural language and Python code with transformers
Clement, C., Drain, D., Timcheck, J., Svyatkovskiy, A., & Sundaresan, N. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.728 · Cited by 120 · cited 3×
“[17] C. Clement, D. Drain, J. Timcheck, A. Svyatkovskiy, and N. Sundaresan, “PyMT5: Multi-mode translation of natural language and Python code with transformers,” in Proc.”
BackgroundMethodSPoC: Search-based Pseudocode to Code
Kulal, S., Pasupat, P., Chandra, K., Lee, M., Padon, O., Aiken, A., & Liang, P. (2019) · arXiv · doi:10.48550/arxiv.1906.04908 · Cited by 103 · cited 4×
“Moreover, of these datasets, only Kulal et al.’s includes unit tests to enable evaluation of code correctness [8].”
BackgroundInfluentialTypeWriter: neural type prediction with search-based validation
Pradel, M., Gousios, G., Liu, J., & Chandra, S. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3409715 · Cited by 102 · cited 2×
“Code generation models have been applied to a variety of tasks, including test generation [18], docstring generation [19], code search [16], [20], type inference [21], [22], [23], and more [24].”
BackgroundMethodCodeT: Code Generation with Generated Tests
Chen, B., Zhang, F., Nguyen, A., Zan, D., Lin, Z., Lou, J.-G., & Chen, W. (2022) · arXiv · doi:10.48550/arxiv.2207.10397 · Cited by 92 · cited 3×
“2) Python Results and Replication: Our Python MBPP pass@1 rates for Codex are slightly higher than what is re-ported in [34] (60.3% compared to 58.1%).”
MethodInfluentialLambdaNet: Probabilistic Type Inference using Graph Neural NetworksPreprint
Wei, J., Goyal, M., Durrett, G., & Dillig, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.02161 · Cited by 86
MCoNaLa: A Benchmark for Code Generation from Multiple Natural LanguagesPreprint
Wang, Z.-R., Cuenca, G., Zhou, S., Xu, F. F., & Neubig, G. (2022) · Findings · doi:10.48550/arxiv.2203.08388 · Cited by 70
GPT-NeoX-20B: An Open-Source Autoregressive Language Model
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.06745 · Cited by 67 · cited 2×
“neural language models (LLMs) trained for text generation have demonstrated the ability to perform program completion [5], [14], [15].”
BackgroundMCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages
Wang, Z., Cuenca, G., Zhou, S., Xu, F. F., & Neubig, G. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.20 · Cited by 65
SantaCoder: don't reach for the stars!
Allal, L. B., Li, R., Kocetkov, D., Mou, C., Akiki, C., Ferrandis, C. M., Muennighoff, N., Mishra, M., Gu, A., Dey, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.03988 · Cited by 61 · cited 2×
“However, MultiPL-E-generated solutions are used as multi-language infilling benchmarks for the SantaCoder [33] code generation models, which do support in-filling.”
MethodSelf-citationMultilingual training for software engineering
Ahmed, T., & Devanbu, P. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510049 · Cited by 57 · cited 2×
“Code generation models have been applied to a variety of tasks, including test generation [18], docstring generation [19], code search [16], [20], type inference [21], [22], [23], and more [24].”
BackgroundMethodCM3: A Causal Masked Multimodal Model of the Internet
Aghajanyan, A., Huang, B., Ross, C., Karpukhin, V., Xu, H., Goyal, N., Okhonko, D., Joshi, M., Ghosh, G., Lewis, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.07520 · Cited by 42 · cited 1×
“InCoder [4] is a 6.7B parameter language model trained using a causal masking objective [30].”
MethodMulti-lingual Evaluation of Code Generation Models
Athiwaratkun, B., Gouda, S. K., Wang, Z., Li, X., Tian, Y., Tan, M., Ahmad, W. U., Wang, S., Sun, Q., Shang, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.14868 · Cited by 38
Gulwani, S., Polozov, O., & Singh, R. K. (2017) · doi:10.1561/9781680832938 · Cited by 36
Mapping Language to Code in Programmatic Context
Iyer, S., Konstas, I., Cheung, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1808.09588
Chaudhuri, S., Ellis, K., Polozov, O., et al. (2021) · doi:10.1561/9781680839357
Research works printed but not linked: 3
- S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma, "Codebleu: a method for automatic evaluation of code synthesis," 2020. [Online]. Available: https: //arxiv.org/abs/2009.10297
- Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017. [Online]. Avail- able: https://proceedings.neurips.cc/paper/2017/ file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
- L. Tunstall, L. von Werra, and T. Wolf, Natural Language Processing with Transformers. O'Reilly Media, 2022. [Online]. Available: https://books.google.com/books? id=nTxbEAAAQBAJ
Cited by
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695988 · Cited by 1,029
Code Llama: Open Foundation Models for Code
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12950 · Cited by 531
“For that, we use the MultiPL-E benchmark (Cassano et al., 2023).”
BackgroundMethodInfluentialReflexion: Language Agents with Verbal Reinforcement Learning
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11366 · Cited by 1,087
Self-citationSWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang, J., Jimenez, C., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1601 · Cited by 264
StarCoder: may the source be with you!
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.06161 · Cited by 261
Self-citation
Metadata
- Authors
- Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., Guha, A., Greenberg, M., & Jangda, A.
- Affiliations
- Northeastern University (US)
- Hanover High School, Hanover, NH, USA
- Wellesley College (US)
- Oberlin College (US)
- Stevens Institute of Technology (US)
- Microsoft (United States) (US)
- ORCID
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2023-07
- Volume
- 49
- Issue
- 7
- Pages
- 3675-3691
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W4366204357
- Semantic Scholar
- f9acfdf58ccc27ac7d4b815ef8a2b9e03c5b215a
- Cited by
- 394
- References
- 54
- Citations by year
- 2026: 149 · 2025: 142 · 2024: 88 · 2023: 15
- FWCI
- 72.25
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- IEEE Transactions on Software Engineering (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (CCF-2052696)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- code generation benchmarks, multilingual programming languages, HumanEval, MBPP, codices, CodeGen, linguistic features, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.