HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization
Open accessAbstract
Large language models (LLMs) have made significant progress in generating codes from textual prompts. However, existing benchmarks have mainly concentrated on translating English prompts to multilingual codes or have been constrained to very limited natural languages (NLs). These benchmarks have overlooked the vast landscape of massively multilingual NL to multilingual code, leaving a critical gap in the evaluation of multilingual LLMs. In response, we introduce HumanEval-XL, a massively multilingual code generation benchmark specifically crafted to address this deficiency. HumanEval-XL establishes connections between 23 NLs and 12 programming languages (PLs), and comprises of a collection of 22,080 prompts with an average of 8.33 test cases. By ensuring parallel data across multiple NLs and PLs, HumanEval-XL offers a comprehensive evaluation platform for multilingual LLMs, allowing the assessment of the understanding of different NLs. Our work serves as a pioneering step towards filling the void in evaluating NL generalization in the area of multilingual code generation. We make our evaluation code and data publicly available at \url{https://github.com/FloatAI/humaneval-xl}.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
3 of 9 cited sources checked · 2 supported · 0 flagged · accuracy not yet measured
- Findingnone given
The benchmark's construction uses an iterative process inspired by back-translation to create a robust benchmark spanning 12 programming languages and 23 natural languages.
“Drawing inspiration from back-translation, we have devised an iterative process that enables the creation of a robust benchmark spanning 12 PLs and 23 NLs.”
Quote in the paperSubstantial claim - Findingown data
The experimental results show clear disparities in performance across different natural languages.
“From Figure 2, we can observe clear disparities in performance across different NLs.”
Quote in the paperSubstantial claim - Findingreasoning
CodeT5+ (2B) struggles to solve problems across most scenarios.
“CodeT5+ (2B), despite its specialization in code generation, struggles to solve problems across most scenarios.”
Quote in the paperSubstantial claim - Findingown data
Class 3 languages consistently demonstrate lower performance across different models.
“The difference in pass@1 among categorized groups is not large yet consistent, with class 3 consistently demonstrating lower performance across different models.”
Quote in the paperNot a substantial claim - Findingown data
LLMs within the same family but with different parameter scales exhibit high correlation across different levels of language resources.
“Interestingly, we observe from Table 3 that LLMs within the same family, but possessing different parameter scales, exhibit a high correlation across different levels of language resources.”
Quote in the paperSubstantial claim
- Findingown data
Python stands out as the most effectively tackled programming language by evaluated LLMs.
“Python stands out as the most effectively tackled PL by evaluated LLMs.”
Quote in the paperNot a substantial claim - Findingnone given
HumanEval-XL is a massively multilingual code generation benchmark for assessing cross-lingual natural language generalization for LLMs.
“We propose HumanEval-XL, a massively multilingual code generation benchmark for assessing cross-lingual NL generalization for LLMs.”
Quote in the paperSubstantial claim - Findingreasoning
GPT-3.5 lags behind CodeGen2-16B in all programming languages except Python, despite having considerably more parameters.
“Interestingly, despite the number of parameters of GPT-3.5 is considerably higher than CodeGen2-16B, a specifically tailored code generation mod-elLLM, GPT-3.5 lags behind CodeGen2-16B in all PLs except Python.”
Quote in the paperSubstantial claim - Findingreasoning
CodeGen2-3.7B and CodeGen2-16B exhibit markedly improved performance compared to CodeT5+ (2B).
“In contrast, both CodeGen2-3.7B and CodeGen2-16B exhibit markedly improved performance compared to CodeT5+ (2B).”
Quote in the paperSubstantial claim - Findingown data
All models seem to struggle with Scala and Go.
“Conversely, all models seem to struggle with Scala and Go, as depicted in the figure.”
Quote in the paperNot a substantial claim - Findingown data
The 23 natural languages in HumanEval-XL are distributed across 11 language families.
“Notably, these languages are distributed across 11 language families, further demonstrating the dataset's comprehensive coverage and multilingual nature.”
Quote in the paperNot a substantial claim - Findingown data
Given natural language prompts expressing the same meaning in different languages, current LLMs struggle to capture the equivalent semantic meaning.
“These findings underscore a significant challenge: Given NL prompts expressing the same meaning in dif-ferent languages, current LLMs struggle to capture the equivalent semantic meaning.”
- Also stated in: § 1. Introduction · plain
“Intriguingly, the evaluation on 23 NLs highlights a significant challenge for current LLMs, as they struggle to capture the equivalent semantic meaning expressed in different languages in the task of code generation.”
Quote in the paperSubstantial claim - Also stated in: § 1. Introduction · plain
- Findingown data
LLMs from different families have no such correlation.
“However, LLMs from different families have no such correlation.”
Quote in the paperSubstantial claim - Findingown data
Languages from the Afro-Asiatic, Indo-European (Greek), Indo-European (Iranian), and Turkic language families generally yield lower results compared to other language families.
“Table 4 highlights that languages from the Afro-Asiatic, Indo-European (Greek), Indo-European (Iranian), and Turkic language families generally yield lower results compared to other language families.”
Quote in the paperSubstantial claim - Findingown data
The 12 programming languages in HumanEval-XL are Python, Java, Go, Kotlin, PHP, Ruby, Scala, JavaScript, C#, Perl, Swift, and TypeScript.
“The 12 PLs are the same as in Multilingual HumanEval, including Python, Java, Go, Kotlin, PHP, Ruby, Scala, JavaScript, C#, Perl, Swift and TypeScript.”
Quote in the paperNot a substantial claim - Interpretationreasoning
The benchmark fills a void in evaluating natural language generalization in multilingual code generation.
“Our work serves as a pioneering step towards filling the void in evaluating NL generalization in the area of multilingual code generation.”
Quote in the paperSubstantial claim - Interpretationreasoning
The performance divergence between CodeT5+ and other models is likely attributed to model structure: CodeT5+ uses encoder-decoder while others use decoder-only.
“This divergence in performance is likely attributed to the difference in model structure.CodeT5+ follows an encoder-decoder architecture, while all other models employ a decoder-only structure.”
Quote in the paperSubstantial claim - Study design and conductreasoning
Cases within distinct natural languages in HumanEval-XL are made parallel to facilitate unbiased comparisons.
“Moreover, to facilitate unbiased comparisons, the cases within distinct NLs are made parallel.”
Quote in the paperSubstantial claim - Interpretationreasoning
Without further instruction-tuning, the encoder-decoder structure of CodeT5+ hinders its ability to complete code generation tasks proficiently across various programming languages and natural languages.
“Our findings suggest that without further instruction-tuning, the encoder-decoder structure of CodeT5+ hinders its ability to complete code generation tasks proficiently across various PLs and NLs.”
Quote in the paperSubstantial claim - Premisereasoning
The absence of multilingual code generation benchmarks prevents assessment of LLMs' cross-lingual natural language generalization.
“A significant gap in current code generation benchmarks is the absence of multilingual code generation, preventing an assessment of the capability of LLMs on cross-lingual natural language generalization.”
Quote in the paperSubstantial claim - Interpretationreasoning
Training on coding data plays a crucial role in code generation.
“This underscores the crucial role of training on coding data.”
Quote in the paperSubstantial claim
- May combine two claims. 15
- Classified as probably not a substantial claim. 8
May combine two claims.
PremisecitationODEX introduced a dataset with 4 natural languages and Python, averaging 1.8 test cases.
“ODEX (Wang et al., 2022) introduced a dataset comprising 4 NLs and Python, albeit with a limited average of 1.8 test cases.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Wang et al., 2022 source
May combine two claims.
PremisecitationExisting code generation benchmarks focus mainly on English prompts to multilingual code or cover only a small number of natural languages.
“However, existing benchmarks have mainly concentrated on translating English prompts to multilingual codes or have been constrained to very limited natural languages (NLs).”
Quote in the paperSubstantial claim1 of 3 sources checked- cites Wang et al., 2022 source
The Related Work section says ODEX uniquely draws on StackOverflow forums in various languages, and the Abstract says its prompts span English, Spanish, Japanese, and Russian. This supports the point that earlier benchmarks had limited natural-language coverage, but the passages do not establish that existing benchmarks mainly translate English prompts into multiple programming languages. (LLM, conf=0.72)
Experimental — accuracy not yet measured - cites Chai et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Scao et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
InterpretationcitationScaling laws persist while maintaining a consistent landscape of capabilities across a wide range of linguistic structures.
“This finding underscores the persistence of scaling laws (Hoffmann et al., 2022) while maintaining a consistent landscape of capabilities across a wide range of linguistic structures.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Hoffmann et al., 2022 source
May combine two claims.
Findingown dataHumanEval-XL connects 23 natural languages and 12 programming languages.
“HumanEval-XL establishes connections between 23 NLs and 12 programming languages (PLs)”
- Also stated in: § 1. Introduction · plain
“incorporating parallel coding problems across 23 NLs and 12 PLs.”
Quote in the paperSubstantial claim- Also stated in: § 1. Introduction · plain
May combine two claims.
Findingown dataHumanEval-XL contains 22,080 prompts with an average of 8.33 test cases.
“comprises of a collection of 22,080 prompts with an average of 8.33 test cases.”
- Also stated in: § 3.2. Dataset Construction · plain
“In total, this benchmark includes 22,080 coding problems.”
Quote in the paperNot a substantial claim- Also stated in: § 3.2. Dataset Construction · plain
Probably not a substantial claim: may not pass S4 (based).
Findingnone givenThe trends observed in the experimental results remain consistent across different model scales for each model.
“Due to page limits, we put all experimental results in appendix and only report different models with largest parameters in the main context, as the observed trends remain consistent across different scales for each model.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataThe difference in pass@1 among categorized groups is not large yet consistent.
“The difference in pass@1 among categorized groups is not large yet consistent, with class 3 consistently demonstrating lower performance across different models.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataAfter filtering, HumanEval-XL consists of 80 parallel coding problems spanning 12 programming languages and 23 natural languages.
“After filtering and discarding poorquality examples, we result in the HumanEval-XL benchmark, consisting of 80 parallel coding problems spanning 12 PLs and 23 NLs.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe Pearson correlation between CodeGen2-3.7B and CodeGen2-16B is 0.8, and between GPT-3.5 and GPT-4 is 0.87.
“For instance, the Pearson correlation between CodeGen2-3.7B and 16B is 0.8, and between GPT-3.5 and GPT-4, it stands at 0.87.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataGPT-4 consistently demonstrates superior performance across different programming languages and natural languages, except C#, outperforming all other models significantly.
“Notably, GPT-4 consistently demonstrates superior performance across different PLs and NLs (except C#), outperforming all other models significantly.”
Quote in the paperSubstantial claimMay combine two claims.
PremisereasoningExisting benchmarks lack evaluation of massively multilingual natural-language-to-multilingual-code, leaving a critical gap.
“These benchmarks have overlooked the vast landscape of massively multilingual NL to multilingual code, leaving a critical gap in the evaluation of multilingual LLMs.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningThe benchmark's parallel data across multiple natural languages and programming languages enables comprehensive evaluation of multilingual LLMs and assessment of understanding of different natural languages.
“By ensuring parallel data across multiple NLs and PLs, HumanEval-XL offers a comprehensive evaluation platform for multilingual LLMs, allowing the assessment of the understanding of different NLs.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationERNIE-Code was pre-trained across 116 natural languages and 6 programming languages.
“ERNIE-Code (Chai et al., 2023) pioneered multilingual LLMs by pre-training across 116 NLs and 6 PLs.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Chai et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim.
InterpretationreasoningThe study underscores the persistent challenge in achieving effective cross-lingual natural language generalization.
“Our study underscores the persistent challenge in achieving effective cross-lingual NL generalization, highlighting a key area for future research.”
Quote in the paperSubstantial claimMay combine two claims.
PremisecitationBLOOM was pre-trained on data in 46 natural languages and 13 programming languages but did not produce competitive results in Python code generation.
“BLOOM (Scao et al., 2022) was pretrained on data in 46 NLs and 13 PLs.However, it did not produce competitive results in Python code generation.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Scao et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductcitationOnly the first generated function was kept as stopping criteria following Athiwaratkun et al. (2023).
“Following Athiwaratkun et al. ( 2023), we keep only the first generated function as our stopping criteria.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Athiwaratkun et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim. May combine two claims.
Study design and conductcitationTranslations with BERTScore similarity above 0.95 were retained; those below were re-translated up to three times and discarded if still below.
“If the similarity score exceeds 0.95, indicating a high-quality translation, it is retained for further processing.In cases where the score falls below this threshold, above process is iteratively repeated up to a predefined limit (set to 3 in this work).If the score is still below the threshold, we will discard this example.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Zhang et al., 2019 source
Classified as probably not a substantial claim. May combine two claims.
Study design and conductreasoningThe study evaluated models with top-p sampling of 0.95 and temperature 0.2, ensuring comparability across models.
“The specific parameters for the sampling process were set to a top-p value of 0.95 and a temperature of 0.2.This uniform approach ensures comparability of results across different models.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown dataDue to constrained computing resources, the study reports pass@1 for all experiments.
“Due to constrained computing resources, we report pass@1 for all experiments.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductnone givenRemaining translated texts underwent heuristic quality checks, with manual checking of randomly sampled examples and corrections.
“Subsequently, the remaining translated texts undergo heuristic quality checks.We have manually checked randomly sampled examples and make corrections if the translation is problematic.”
Quote in the paperSubstantial claim
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
GPT-3.5 does not trail CodeGen2-16B on every non-Python language
MajorThe Ruby results contradict the claimed exception pattern. For English Ruby prompts, GPT-3.5 scores 5.00 while CodeGen2-16B scores 0.00.
“GPT-3.5 lags behind CodeGen2-16B in all PLs except Python.”
“| English | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 5.00 | 28.75 |”
Class 5 CodeT5+ average does not match its language scores
MajorThe six Class 5 Python scores in Table 5 average 1.67, not the 0.63 reported in Table 3. This affects the paper's comparison of performance across resource classes.
Reported 0.63 · recomputed 1.666667
“| CodeT5+ (2B) | 0.63±1.53 | 0.94±0.88 | 0.83±0.63 |”
Class 3 is not lower for every model in Table 3
MinorThe stated pattern has a counterexample in Table 3: CodeT5+ scores 0.83 for Class 3 and 0.63 for Class 5. CodeGen2 (16B) also scores higher on Class 3 than Class 4.
“The difference in pass@1 among categorized groups is not large yet consistent, with class 3 consistently demonstrating lower performance across different models.”
“| CodeT5+ (2B) | 0.63±1.53 | 0.94±0.88 | 0.83±0.63 |”
Cites
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · cited 1× · see citing works
“(3) Quality Check with BERTScore: To assess the quality of the translation, we employ BERTScore (Zhang et al., 2019) to evaluate the similarity between the back-translated English prompt and the original English text.”
MethodEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932 · cited 1× · see citing works
“This finding underscores the persistence of scaling laws (Hoffmann et al., 2022) while maintaining a consistent landscape of capabilities across a wide range of linguistic structures.”
BackgroundThe State and Fate of Linguistic Diversity and Inclusion in the NLP World
Joshi, P., Santy, S., Budhiraja, A., Bali, K., & Choudhury, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.560 · Cited by 766 · cited 2× · see citing works
“To further investigate the disparity in performance across different NLs, we have initially categorized the 23 NLs into three distinct groups based on resource availability, following the taxonomy outlined in Joshi et al. (2020) (ranging from 0 = least re-sourced to 5 = best resourced).”
MethodCodeT5+: Open Code Large Language Models for Code Understanding and Generation
Wang, Y., Le, H., Gotmare, A. D., Bui, N. D. Q., Li, J., & Hoi, S. C. H. (2023) · arXiv · doi:10.48550/arxiv.2305.07922 · Cited by 459 · cited 1×
“Some studies have delved into the intricacies of LLMs in multiple PLs (Fried et al., 2023; Wang et al., 2023; Nijkamp et al., 2023a; Li et al., 2023), while others have focused on the synergy between multilingual NLs and PLs. Notably, ERNIE-Code (Chai et al., 2023) pioneered multilingual LLMs by…”
BackgroundProgram Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388 · cited 3× · see citing works
“…Prior efforts have predominantly concentrated on English-centric Python generation, evident in benchmarks such as HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), APPS (Hendrycks et al., 2021), DSP (Chandel et al., 2022), MTPB (Nijkamp et al., 2023b), and DS-1000 (Lai et al., 2023).”
BackgroundCodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., & Xiong, C. (2022) · arXiv · doi:10.48550/arxiv.2203.13474 · Cited by 330 · cited 3× · see citing works
“…Prior efforts have predominantly concentrated on English-centric Python generation, evident in benchmarks such as HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), APPS (Hendrycks et al., 2021), DSP (Chandel et al., 2022), MTPB (Nijkamp et al., 2023b), and DS-1000 (Lai et al., 2023).”
BackgroundCodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Shen, L., Wang, Z., Wang, A., Li, Y., et al. (2023) · Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3580305.3599790 · Cited by 263 · cited 1× · see citing works
“Some endeavors have extended the HumanEval benchmark to multiple PLs (Athiwaratkun et al., 2023; Zheng et al., 2023).”
BackgroundStarCoder: may the source be with you!
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.06161 · Cited by 261 · cited 1× · see citing works
“Some studies have delved into the intricacies of LLMs in multiple PLs (Fried et al., 2023; Wang et al., 2023; Nijkamp et al., 2023a; Li et al., 2023), while others have focused on the synergy between multilingual NLs and PLs. Notably, ERNIE-Code (Chai et al., 2023) pioneered multilingual LLMs by…”
BackgroundCodeGen2: Lessons for Training LLMs on Programming and Natural LanguagesPreprint
Nijkamp, E., Hayashi, H., Xiong, C.-M., Savarese, S., & Zhou, Y.-B. (2023) · arXiv.org · doi:10.48550/arxiv.2305.02309 · Cited by 253 · see citing works
Measuring Coding Challenge Competence With APPS
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.09938 · Cited by 142 · cited 3× · see citing works
“…Prior efforts have predominantly concentrated on English-centric Python generation, evident in benchmarks such as HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), APPS (Hendrycks et al., 2021), DSP (Chandel et al., 2022), MTPB (Nijkamp et al., 2023b), and DS-1000 (Lai et al., 2023).”
BackgroundFew-shot Learning with Multilingual Generative Language Models
Lin, X. V., Mihaylov, T., Artetxe, M., Wang, T., Chen, S., Simig, D., Ott, M., Goyal, N., Bhosale, S., Du, J., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.616 · Cited by 134 · cited 1× · see citing works
“Selected data in our benchmark are sourced from materials available under permissive licenses, allowing unrestricted use and redistribution of data for research purposes.”
MethodSelf-citationMulti-lingual Evaluation of Code Generation Models
Athiwaratkun, B., Gouda, S. K., Wang, Z., Li, X., Tian, Y., Tan, M., Ahmad, W. U., Wang, S., Sun, Q., Shang, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.14868 · Cited by 38 · cited 8× · see citing works
“Some endeavors have extended the HumanEval benchmark to multiple PLs (Athiwaratkun et al., 2023; Zheng et al., 2023).”
MethodInfluentialSelf-citationDS-1000: A Natural and Reliable Benchmark for Data Science Code GenerationPreprint
Lai, Y., Li, C., Wang, Y., Zhang, T., Zhong, R., Zettlemoyer, L., Yih, S., Fried, D., Wang, S. I., & Yu, C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.11501 · Cited by 31 · cited 3× · see citing works
“…Prior efforts have predominantly concentrated on English-centric Python generation, evident in benchmarks such as HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), APPS (Hendrycks et al., 2021), DSP (Chandel et al., 2022), MTPB (Nijkamp et al., 2023b), and DS-1000 (Lai et al., 2023).”
BackgroundExecution-Based Evaluation for Open-Domain Code Generation
Wang, Z., Zhou, S., Fried, D., & Neubig, G. (2022) · arXiv · doi:10.48550/arxiv.2212.10481 · Cited by 20 · cited 2× · see citing works
“Conversely, ODEX (Wang et al., 2022) introduced a dataset comprising 4 NLs and Python, albeit with a limited average of 1.8 test cases.”
BackgroundMethodInfluentialERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languages
Chai, Y., Wang, S., Pang, C., Sun, Y., Tian, H., & Wu, H. (2022) · arXiv · doi:10.48550/arxiv.2212.06742 · Cited by 18 · cited 2×
“Some works have also made efforts to incorporate different natural languages (NLs), such as Spanish and Russian, although their coverage of different natural languages remains rather limited (Wang et al., 2022; Chai et al., 2023; Scao et al., 2022).”
BackgroundSelf-citationBLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Workshop, B., :, Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.05100 · cited 3×
“Some works have also made efforts to incorporate different natural languages (NLs), such as Spanish and Russian, although their coverage of different natural languages remains rather limited (Wang et al., 2022; Chai et al., 2023; Scao et al., 2022).”
BackgroundMethodOpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Flagged first, then by citations
3 of the 21 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- PMLR.
Cited by
Sorted by citations
Code LLMs: A Taxonomy-based Survey
Raihan, N., Newman, C. D., & Zampieri, M. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10826108 · Cited by 11 · see citing works
MacBehaviour: An R package for behavioural experimentation on large language models
Duan, X., Li, S., & Cai, Z. G. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02524-y · Cited by 5 · see citing works
“…like translation and world knowledge (Hendrycks et al., 2021; Zellers et al., 2019), math (Lewkowycz et al., 2022; Zhou et al., 2023) and coding (Peng et al., 2024); while these tests offer a good understanding of model performance, they are limited in tapping into the black box of these…”
BackgroundEvaluating the Test Adequacy of Benchmarks for LLMs on Code Generation
Liu, X., Sun, X., Bo, L., Hu, Y., Liu, X., & Ye, Z. (2025) · Journal of Software Evolution and Process · doi:10.1002/smr.70034 · Cited by 4 · see citing works
Assessing Small Language Models for Code Generation: An Empirical Study with BenchmarksPreprint
Hasan, M. M., Waseem, M., Kemell, K., Rasku, J., Ala-Rantala, J., & Abrahamsson, P. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5472927 · Cited by 3 · see citing works
McEval: Massively Multilingual Code Evaluation
Chai, L., Liu, S., Yang, J., Yin, Y., Jin, K., Liu, J., Sun, T., Zhang, G., Ren, C., Guo, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.07436 · Cited by 1 · see citing works
“Some works focus on the multilingual scenarios (Wang et al., 2023; Athiwaratkun et al., 2023; Peng et al., 2024; Zheng et al., 2023b) by extending the Python-only HumanEval/MBPP benchmark (e.g. MultiPL-E (Cassano et al., 2023)), which is challenged by the number of the languages and data leaking…”
BackgroundA benchmark for vericoding: formally verified program synthesis
Bursuc, S., Ehrenborg, T., Lin, S., Astefanoaei, L., Chiosa, I. E., Kukovec, J., Singh, A., Butterley, O., Bizid, A., Dougherty, Q., et al. (2025) · arXiv · doi:10.48550/arxiv.2509.22908 · Cited by 1 · see citing works
Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages
Ivanova, M., Zadorozhny, P., Levichev, R., Petrov, I., Pavel, A., Lopatin, I., Kutalev, A., & Babaev, D. (2026) · arXiv · doi:10.48550/arxiv.2606.20517
MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Merging
Wang, J., Tian, C., Chen, K., Liu, Z., Mao, J., Zhao, W. X., Zhang, Z., & Zhou, J. (2026) · arXiv · doi:10.48550/arxiv.2601.17858
“…math (Tang et al., 2024); (e) code (Hu-manEval (Chen et al., 2021), LiveCodeBench (Jain et al., 2025), MBPP (Tao et al., 2024), HumanEval plus (Liu et al., 2023), MBPP plus (Liu et al., 2023), HumanEval cn (Peng et al., 2024), HumanEval fim (Bavarian et al., 2022) and CruxEval (Gu et al., 2024a)).”
- Authors
- Peng, Q., Chai, Y., & Li, X.
- Affiliations
- University of Copenhagen (DK)
- Baidu (China) (CN)
- ORCID
- Li, Xuhong
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2402.16694
- OpenAlex
- W4392223004
- Semantic Scholar
- d7125d55dc1a28b08d2fdf6ccff1cfb0af4d7b67
- Cited by
- 4
- References
- 21
- Citations by year
- 2025: 2 · 2024: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, other-oa)
- Research at the University of Copenhagen (University of Copenhagen) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- multilingual code generation, code generation benchmarks, multilingual large language models, natural language to code translation, programming languages, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-05.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-05.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
