CoChat
Sign Up
PreprintCited by 17Software Engineering ResearchTopic ModelingTeaching and Learning Programming

MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation

Open access
Cassano, Federico, Gouwar, John, Nguyen, Daniel, Nguyen, Sydney, Phipps-Costin, Luna, Pinckney, Donald, Yee, Ming-Ho, Zi, Yangtian +5 more·2022·arXiv
DOI 10.48550/arxiv.2208.08227arXiv 2208.08227
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models have demonstrated the ability to generate both natural language and programming language text. Such models open up the possibility of multi-language code generation: could code generation models generalize knowledge from one language to another? Although contemporary code generation models can generate semantically correct Python code, little is known about their abilities with other languages. We propose MultiPL-E, a system for translating unit test-driven code generation benchmarks to new languages. We create the first massively multilingual code generation benchmark by using MultiPL-E to translate two popular Python code generation benchmarks to 18 additional programming languages. We use MultiPL-E to extend the HumanEval benchmark and MBPP benchmark to 18 languages that encompass a range of programming paradigms and popularity. Using these new parallel benchmarks, we evaluate the multi-language performance of three state-of-the-art code generation models: Codex, CodeGen, and InCoder. We find that Codex matches or even exceeds its performance on Python for several other languages. The range of programming languages represented in MultiPL-E allow us to explore the impact of language frequency and language features on model performance. Finally, the MultiPL-E approach of compiling code generation benchmarks to new programming languages is both scalable and extensible, making it straightforward to evaluate new models, benchmarks, and languages.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2208.08227

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In many languages, MultiPL-E uses types that are not considered best practice, such as in Scala, where it relies on the Java types ArrayList instead of the native List.

    “Finally, in many languages, MultiPL-E uses types that are not considered best practice, such as in Scala, where it relies on the Java types ArrayList instead of the native List.”

    From Measuring The Impact Of Programming Language Distribution (2023) · cites this paper as Cassano et al., 2022

    The Details of Language Translations (p. 19) says MultiPL-E translates Python lists to Scala List, contrary to the claim that it uses Java ArrayList instead. That passage does not assess whether its type choices are best practice across languages. (LLM, conf=0.88)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • MultiPL-E supports 18 languages compared to BabelCode's 16.

    “The MultiPL-E framework from Cassano et al. (2022) supports 18 languages compared to BabelCode's 16.”

    From Measuring The Impact Of Programming Language Distribution (2023) · cites this paper as Cassano et al., 2022

    The introduction (p. 1) says MultiPL-E supports 18 languages, while §3.2 (p. 4) counts 19 when Python is included alongside the 18 target languages. The paper does not substantiate the comparison with BabelCode or the claim that BabelCode supports 16 languages (searched: BabelCode). (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • MultiPL-E is open-sourced, but only supports generation tasks and contains significant errors in multiple languages.

    “Multi-PLE is open-sourced, but only supports generation tasks and contains significant errors in multiple languages.”

    From Measuring The Impact Of Programming Language Distribution (2023) · cites this paper as Cassano et al., 2022

    The contributions summary (p. 2) describes MultiPL-E as a code-generation benchmark and says its system and dataset are publicly available on GitHub. Section 3.4 (p. 5) identifies a handful of translation limitations, but does not establish that MultiPL-E contains significant errors in multiple languages. The error analysis described in the conclusion (p. 13) concerns model-generated code, not errors in the benchmark. (LLM, conf=0.89)

    Experimental — accuracy not yet measured

  • Translating prompts to a target language is not novel by itself, as both Athiwaratkun et al. (2023) and Cassano et al. (2022) proposed methods to accomplish this.

    “Translating prompts to a target language is not novel by itself, as both Athiwaratkun et al. (2023) and Cassano et al. (2022) proposed methods to accomplish this.”

    From Measuring The Impact Of Programming Language Distribution (2023) · cites this paper as Cassano et al., 2022

  • CORE-Bench is one of the first benchmarks to include tasks in R.

    “While many existing benchmarks include Python tasks (Cassano et al., 2022), ours is one of the first to include tasks in R.”

    From CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark (2024) · cites this paper as Cassano et al., 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Cited by 17

Research works printed but not linked: 39

  1. [1][1] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al. , 'Evaluating large language models trained on code,' arXiv preprint arXiv:2107.03374 , 2021. - · arXiv
  2. [2][2] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, 'Program synthesis with large language models,' arXiv preprint arXiv:2108.07732 , 2021. [Online]. Available: https://arxiv.org/abs/2108.07732 - · arXiv
  3. [3][3] E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong, 'Codegen: An open large language model for code with multiturn program synthesis,' 2022. [Online]. Available: https://arxiv.org/abs/2203.13474 - · arXiv
  4. [4][4] D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, W.-t. Yih, L. Zettlemoyer, and M. Lewis, 'Incoder: A generative model for code infilling and synthesis,' arXiv preprint arXiv:2204.05999 , 2022. [Online]. Available: https://arxiv.org/abs/2204. 05999 - · arXiv
  5. [5][5] S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang, M. Pieler, U. S. Prashanth, S. Purohit, L. Reynolds, J. Tow, B. Wang, and S. Weinbach, 'Gpt-neox-20b: An open-source autoregressive language model,' arXiv preprint arXiv:2204.06745 , 2022. [Online]. Available: https://arxiv.org/abs/2204.06745 - · arXiv
  6. [6][6] E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong, 'A conversational paradigm for program synthesis,' arXiv preprint arXiv:2203.13474 , 2022. [Online]. Available: https: //arxiv.org/abs/2203.13474 - · arXiv
  7. [7][7] F. F. Xu, U. Alon, G. Neubig, and V. J. Hellendoorn, 'A systematic evaluation of large language models of code,' in Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming , 2022, pp. 1-10. -
  8. [8][8] A. Ziegler, E. Kalliamvakou, X. A. Li, A. Rice, D. Rifkin, S. Simister, G. Sittampalam, and E. Aftandilian, 'Productivity assessment of neural code completion,' in Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming , 2022, pp. 21-29. -
  9. [9][9] S. Kulal, P. Pasupat, K. Chandra, M. Lee, O. Padon, A. Aiken, and P. S. Liang, 'Spoc: Search-based pseudocode to code,' in Advances in Neural Information Processing Systems , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'AlchéBuc, E. Fox, and R. Garnett, Eds., vol. 32. Curran Associates, Inc., 2019. [Online]. Available: https://proceedings.neurips.cc/paper/2019/ file/7298332f04ac004a0ca44cc69ecf6f6b-Paper.pdf -
  10. [10][10] D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt, 'Measuring coding challenge competence with apps,' arXiv preprint arXiv:2105.09938 , 2021. [Online]. Available: https: //arxiv.org/abs/2105.09938 - · arXiv
  11. [11][11] P. Yin, B. Deng, E. Chen, B. Vasilescu, and G. Neubig, 'Learning to mine aligned code and natural language pairs from stack overflow,' in Proceedings of the 15th International Conference on Mining Software Repositories , ser. MSR '18. New York, NY, USA: Association for Computing Machinery, 2018, p. 476-486. [Online]. Available: https://doi.org/10.1145/3196398.3196408 - · DOI · Paper page
  12. [13][13] S. Chaudhuri, K. Ellis, O. Polozov, R. Singh, A. SolarLezama, and Y. Yue, 'Neurosymbolic Programming,' Foundations and Trends in Programming Languages , vol. 7, no. 3, pp. 158-243, 2021. -
  13. [14][14] S. Gulwani, O. Polozov, R. Singh et al. , 'Program synthesis,' Foundations and Trends® in Programming Languages , vol. 4, no. 1-2, pp. 1-119, 2017. -
  14. [15][15] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al. , 'Language models are few-shot learners,' Advances in neural information processing systems , vol. 33, pp. 1877-1901, 2020. -
  15. [17][17] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, 'Codebert: A pre-trained model for programming and natural languages,' arXiv preprint arXiv:2002.08155 , 2020. [Online]. Available: https://arxiv.org/abs/2002.08155 - · arXiv
  16. [18][18] C. Clement, D. Drain, J. Timcheck, A. Svyatkovskiy, and N. Sundaresan, 'PyMT5: multi-mode translation of natural language and python code with transformers,' in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) . Online: Association for Computational Linguistics, Nov. 2020, pp. 9052-9065. [Online]. Available: https: //aclanthology.org/2020.emnlp-main.728 -
  17. [19][19] M. Tufano, D. Drain, A. Svyatkovskiy, S. K. Deng, and N. Sundaresan, 'Unit test case generation with transformers and focal context,' arXiv preprint arXiv:2009.10297 , 2020. [Online]. Available: https: //arxiv.org/abs/2009.05617 - · arXiv
  18. [20][20] S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. Gong, M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. Liu, 'Codexglue: A machine learning benchmark dataset for code understanding and generation,' arXiv preprint arXiv:2102.04664 , 2021. [Online]. Available: https://arxiv.org/abs/2102.04664 - · arXiv
  19. [21][21] T. Ahmed and P. Devanbu, 'Multilingual training for software engineering,' in Proceedings of the 44th International Conference on Software Engineering . ACM, 2022. -
  20. [22][22] J. Wei, M. Goyal, G. Durrett, and I. Dillig, 'LambdaNet: Probabilistic Type Inference using Graph Neural Networks,' in International Conference on Learning Representations (ICLR) , 2020. -
  21. [25][25] I. Drori, S. Zhang, R. Shuttleworth, L. Tang, A. Lu, E. Ke, K. Liu, L. Chen, S. Tran, N. Cheng, R. Wang, N. Singh, T. L. Patti, J. Lynch, A. Shporer, N. Verma, E. Wu, and G. Strang, 'A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level,' Proceedings of the National Academy of Sciences , vol. 119, no. 32, p. e2123433119, Aug. 2022. -
  22. [26][26] A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi, 'The curious case of neural text degeneration,' in ICLR , 2020. -
  23. [27][27] S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma, 'Codebleu: a method for automatic evaluation of code synthesis,' 2020. [Online]. Available: https: //arxiv.org/abs/2009.10297 -
  24. [28][28] T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, Z. Zhang, and D. Radev, 'Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task,' in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing . Brussels, Belgium: Association for Computational Linguistics, Oct.-Nov. 2018, pp. 3911-3921. [Online]. Available: https://aclanthology.org/D18-1425 -
  25. [29][29] V. Zhong, C. Xiong, and R. Socher, 'Seq2sql: Generating structured queries from natural language using reinforcement learning,' 2017. [Online]. Available: https://arxiv.org/abs/1709.00103 - · arXiv
  26. [30][30] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, 'Attention is all you need,' in Advances in Neural Information Processing Systems , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017. [Online]. Available: https://proceedings.neurips.cc/paper/2017/ file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf -
  27. [31][31] A. Aghajanyan, B. Huang, C. Ross, V. Karpukhin, H. Xu, N. Goyal, D. Okhonko, M. Joshi, G. Ghosh, M. Lewis, and L. Zettlemoyer, 'Cm3: A causal masked multimodal model of the internet,' arXiv preprint arXiv:2201.07520 , 2022. [Online]. Available: https://arxiv.org/abs/2201.07520 - · arXiv
  28. [32][32] L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy, 'The pile: An 800gb dataset of diverse text for language modeling,' 2021. [Online]. Available: https://arxiv.org/abs/2101.00027 - · arXiv
  29. [34][34] D. Bates, M. Mächler, B. Bolker, and S. Walker, 'Fitting linear mixed-effects models using lme4,' Journal of Statistical Software , vol. 67, no. 1, pp. 1-48, 2015. -
  30. [35][35] B. Chen, F. Zhang, A. Nguyen, D. Zan, Z. Lin, J.-G. Lou, and W. Chen, 'CodeT: Code generation with generated tests,' 2022. [Online]. Available: https://arxiv.org/abs/2207.10397 - · arXiv
  31. [36][36] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, 'Bleu: A method for automatic evaluation of machine translation,' in Proceedings of the 40th Annual Meeting on Association for Computational Linguistics , ser. ACL '02. USA: Association for Computational Linguistics, 2002, p. 311-318. [Online]. Available: https://doi.org/10.3115/1073083.1073135 - · DOI · Paper page
  32. [37][37] Z. Wang, G. Cuenca, S. Zhou, F. F. Xu, and G. Neubig, 'Mconala: A benchmark for code generation from multiple natural languages,' 2022. [Online]. Available: https://arxiv.org/abs/2203.08388 - · arXiv
  33. [38][38] S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer, 'Mapping language to code in programmatic context,' arXiv preprint arXiv:1808.09588 , 2018. - · arXiv
  34. [39][39] T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, Z. Zhang, and D. Radev, 'Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task,' 2018. [Online]. Available: https://arxiv.org/abs/1809.08887 - · arXiv
  35. [40][40] L. Tunstall, L. von Werra, and T. Wolf, Natural Language Processing with Transformers . O'Reilly Media, 2022. [Online]. Available: https://books.google.com/books? id=nTxbEAAAQBAJ -
  36. [41][41] Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. D. Lago, T. Hubert, P. Choy, C. d. M. d'Autume, I. Babuschkin, X. Chen, P.-S. Huang, J. Welbl, S. Gowal, A. Cherepanov, J. Molloy, D. J. Mankowitz, E. S. Robson, P. Kohli, N. de Freitas, K. Kavukcuoglu, and O. Vinyals, 'Competition-level code generation with alphacode,' 2022. [Online]. Available: https://arxiv.org/abs/2203.07814 - · arXiv
  37. [42][42] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, - G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, - S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, - Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, - M. Isard, G. Gur-Ari, P. Yin, T. Duke, A. Levskaya, S. Ghemawat, S. Dev, H. Michalewski, X. Garcia, - V. Misra, K. Robinson, L. Fedus, D. Zhou, D. Ippolito, D. Luan, H. Lim, B. Zoph, A. Spiridonov, R. Sepassi, - D. Dohan, S. Agrawal, M. Omernick, A. M. Dai, T. S. Pillai, M. Pellat, A. Lewkowycz, E. Moreira, - R. Child, O. Polozov, K. Lee, Z. Zhou, X. Wang, - B. Saeta, M. Diaz, O. Firat, M. Catasta, J. Wei, - K. Meier-Hellstern, D. Eck, J. Dean, S. Petrov, and N. Fiedel, 'Palm: Scaling language modeling with pathways,' arXiv preprint arXiv:2204.02311 , 2022. [Online]. Available: https://arxiv.org/abs/2204.02311 - · arXiv
  38. [43][43] B. Athiwaratkun, S. K. Gouda, Z. Wang, X. Li, Y. Tian, M. Tan, W. U. Ahmad, S. Wang, Q. Sun, M. Shang, S. K. Gonugondla, H. Ding, V. Kumar, N. Fulton, A. Farahani, S. Jain, R. Giaquinto, H. Qian, M. K. Ramanathan, R. Nallapati, B. Ray, P. Bhatia, S. Sengupta, D. Roth, and B. Xiang, 'Multi-lingual evaluation of code generation models,' 2022. [Online]. Available: https://arxiv.org/abs/2210.14868 - · arXiv
  39. [44][44] T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. D. Iii, and K. Crawford, 'Datasheets for datasets,' Communications of the ACM , vol. 64, no. 12, pp. 86-92, 2021.

Authors
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., Guha, A., Greenberg, M., & Jangda, A.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2208.08227
arXiv
2208.08227
OpenAlex
W4300989059
Cited by
17
References
0
Citations by year
2026: 1 · 2025: 7 · 2024: 4 · 2023: 5
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Science Foundation (2052696)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
Keywords
multilingual code generation, code generation benchmarks, HumanEval, MBPP, codices, CodeGen, linguistic features, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review