ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languages
Open accessAbstract
Software engineers working with the same programming language (PL) may speak different natural languages (NLs) and vice versa, erecting huge barriers to communication and working efficiency. Recent studies have demonstrated the effectiveness of generative pre-training in computer programs, yet they are always English-centric. In this work, we step towards bridging the gap between multilingual NLs and multilingual PLs for large language models (LLMs). We release ERNIE-Code, a unified pre-trained language model for 116 NLs and 6 PLs. We employ two methods for universal cross-lingual pre-training: span-corruption language modeling that learns patterns from monolingual NL or PL; and pivot-based translation language modeling that relies on parallel data of many NLs and PLs. Extensive results show that ERNIE-Code outperforms previous multilingual LLMs for PL or NL across a wide range of end tasks of code intelligence, including multilingual code-to-text, text-to-code, code-to-code, and text-to-text generation. We further show its advantage of zero-shot prompting on multilingual code summarization and text-to-text translation. We release our code and pre-trained checkpoints.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Still being checked.
Cited by
Sorted by citations
Towards an understanding of large language models in software engineering tasks
Zheng, Z., Ning, K., Zhong, Q., Chen, J., Chen, W., Guo, L., Wang, W., & Wang, Y. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10602-0 · Cited by 149 · see citing works
Large Language Models Meet NL2Code: A Survey
Zan, D.-G., Chen, B., Zhang, F., Lu, D., Wu, B., Guan, B., Wang, Y.-J., & Lou, J.-G. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2023.acl-long.411 · Cited by 293 · see citing works
A survey of multilingual large language models
Qin, L., Chen, Q., Zhou, Y., Zhi, C., Li, Y., Liao, L., Li, M., Che, W., & Yu, P. S. (2025) · Patterns · doi:10.1016/j.patter.2024.101118 · Cited by 197 · see citing works
Exploring the Capabilities of LLMs for Code-Change-Related Tasks
Fan, L., Liu, J., Liu, Z., Lo, D., Xia, X., & Li, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709358 · Cited by 41 · see citing works
Deep Learning for Code Intelligence: Survey, Benchmark and Toolkit
Wan, Y., Bi, Z., He, Y., Zhang, J., Zhang, H., Sui, Y., Xu, G., Jin, H., & Yu, P. S. (2024) · ACM Computing Surveys · doi:10.1145/3664597 · Cited by 34 · see citing works
Bai, X., Huang, S., Wei, C., & Wang, R. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126357 · Cited by 31 · see citing works
Transformers in source code generation: A comprehensive survey
Ghaemi, H., Alizadehsani, Z., Shahraki, A., & Corchado, J. M. (2024) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2024.103193 · Cited by 23 · see citing works
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · doi:10.31219/osf.io/ac4sp · Cited by 51 · see citing works
bjCnet: A contrastive learning-based framework for software defect prediction
Han, J., Huang, C., & Liu, J. (2024) · Computers & Security · doi:10.1016/j.cose.2024.104024 · Cited by 12 · see citing works
Private-library-oriented code generation with large language models
Zan, D., Chen, B., Gong, Y., Cao, J., Zhang, F., Wu, B., Guan, B., Yin, Y., & Wang, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113934 · Cited by 29 · see citing works
Large Language Models for Software Engineering: A Systematic Mapping Study
Görmez, M. K., Yılmaz, M., & Clarke, P. M. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-71139-8_5 · Cited by 8 · see citing works
Peng, Q., Chai, Y., & Li, X. (2024) · arXiv · doi:10.48550/arxiv.2402.16694 · Cited by 4 · see citing works
“Some works have also made efforts to incorporate different natural languages (NLs), such as Spanish and Russian, although their coverage of different natural languages remains rather limited (Wang et al., 2022; Chai et al., 2023; Scao et al., 2022).”
BackgroundSelf-citationEvaluating the Test Adequacy of Benchmarks for LLMs on Code Generation
Liu, X., Sun, X., Bo, L., Hu, Y., Liu, X., & Ye, Z. (2025) · Journal of Software Evolution and Process · doi:10.1002/smr.70034 · Cited by 4 · see citing works
LFMDiff: generation of Chinese traditional landscape paintings based on diffusion model
Li, Z., Wang, Y., Li, C., Zhang, J., Xu, S., & Gao, Y. (2025) · npj Heritage Science · doi:10.1038/s40494-025-02136-5 · Cited by 3 · see citing works
Li, M., Mishra, A., & Mujumdar, U. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11227991 · Cited by 1 · see citing works
Batwing Compiler: An Artificial Intelligence based Compiler
Harshithan, K. S. (2023) · International Conference on Advances in Computational Intelligence and Communication (ICACIC) · doi:10.1109/icacic59454.2023.10434977
A Survey on Code Pre-Trained Models for Neural Program RepairPreprint
Zhan, S., Wang, X., Wei, D., & Cao, X. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4831589
Ma, L., Hu, X., & Liu, S. (2024) · International Conference on Digital Society and Intelligent Systems (DSInS) · doi:10.1109/dsins64146.2024.10992182
Cruz, J. E., Mamani, V. S., Calsina, H., Yana-Mamani, V., Pineda, F., Goyzueta, C. A. R., Encinas, S. H., Baca, L., & Mamani, W. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8100341
- Authors
- Chai, Y., Wang, S., Pang, C., Sun, Y., Tian, H., & Wu, H.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Pages
- 10628-10650
- arXiv
- 2212.06742
- OpenAlex
- W4385571167
- Cited by
- 18
- References
- 0
- Citations by year
- 2026: 1 · 2025: 6 · 2024: 8 · 2023: 3
- FWCI
- 4.63
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- programming languages, multilingual large language models, code intelligence, text-to-text translation, code summarization, zero-shot prompting
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
