Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index; 13 of the 15 sources its claims cite were checked for support.

CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X

Open access
Zheng, Qinkai, Xia, Xiao, Zou, Xu, Dong, Yuxiao, Wang, Shan, Xue, Yufei, Shen, Lei, Wang, Zihan, Wang, Andi, Li, Yang, Su, Teng, Yang, Zhilin, Tang, Jie · 2023 · Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Abstract

Large pre-trained code generation models, such as OpenAI Codex, can generate syntax-and function-correct code, making the coding of programmers more productive. In this paper, we introduce CodeGeeX, a multilingual model with 13 billion parameters for code generation. CodeGeeX is pre-trained on 850 billion tokens of 23 programming languages as of June 2022. Our extensive experiments suggest that CodeGeeX outperforms multilingual code models of similar scale for both the tasks of code generation and translation on HumanEval-X. Building upon HumanEval (Python only), we develop the HumanEval-X benchmark for evaluating multilingual models by hand-writing the solutions in C++, Java, JavaScript, and Go. In addition, we build CodeGeeX-based extensions on Visual Studio Code, JetBrains, and Cloud Studio, generating 8 billion tokens for tens of thousands of active users per week. Our user study demonstrates that CodeGeeX can help to increase coding efficiency for 83.4% of its users. Finally, CodeGeeX is publicly accessible since Sep. 2022, we open-sourced its code, model weights, API, extensions, and HumanEval-X at https://github.com/THUDM/CodeGeeX.

Overview

CoChat Summary

TL;DR

CodeGeeX is a 13B-parameter multilingual code generation model pre-trained on 850B tokens across 23 programming languages, released with the HumanEval-X benchmark for evaluating code generation and translation in Python, C++, Java, JavaScript, and Go. It reports competitive or best average performance among same-scale multilingual baselines on HumanEval-X and positive user-reported coding-efficiency gains.

Key findings

  • CodeGeeX achieves the best average performance on HumanEval-X code generation among compared models, with average pass@1 18.40%, pass@10 33.29%, and pass@100 54.76%, slightly better than CodeGen-Multi-16B in all three metrics (0.37%–1.67% improvements).
  • On Python HumanEval, CodeGeeX reaches pass@1 22.89%, pass@10 39.57%, and pass@100 60.92%; the paper notes it is second-highest among multilingual code generation models on HumanEval, surpassed by PaLMCoder-540B.
  • For cross-lingual code translation, fine-tuned CodeGeeX-13B-FT performs best on pass@100 in 11 of 20 language pairs, while CodeGen-Multi-16B is best on 7.
  • In a user survey of 168 users, over 83.4% report that CodeGeeX improves or slightly increases coding efficiency; the deployed extensions generate 8 billion tokens per week for tens of thousands of daily active users, averaging 200+ API calls per active user per weekday.

Methods

  • CodeGeeX is a 39-layer decoder-only transformer with 13B parameters, hidden size 5120, vocabulary size 52,224, maximum sequence length 2048, and a top query layer; it was trained on 850B tokens over 213,000 steps using 1,536 Ascend 910 AI processors, Mindspore v1.7.0, 8-way model parallelism, 192-way data parallelism, and global batch size 3,072.
  • HumanEval-X was built by manually rewriting the 164 Python HumanEval problems into C++, Java, JavaScript, and Go, yielding 820 problem-solution pairs; it supports code generation and translation across 20 source-target language pairs, evaluated with pass@k using n=200 generations and k ∈ {1, 10, 100}.
  • Baselines include GPT-J-6B, GPT-NeoX-20B, InCoder-6.7B, and CodeGen-Multi-6B/16B; additional evaluations cover MBPP, CodeXGLUE code summarization, and XLCoST code translation.

Limitations

  • The paper states that it is computationally unaffordable to test different architectural designs for large pre-trained models, and that multilingual models may require larger model capacity to master different languages; it also observes asymmetry in translation performance, suggesting models lack high-level understanding between languages.
  • The paper notes language-specific weaknesses: Go is harder to translate into due to strict syntax constraints such as forbidding unused variables/imports, and Ruby is absent from CodeGeeX’s pre-training corpus.
  • Evident from the text: the main multilingual evaluation is limited to HumanEval-X and selected benchmarks, and the user-efficiency result comes from a self-reported survey of 168 users rather than a controlled productivity study.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3580305.3599790

Cites

56 of 56 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Summary of CodeGeeX. (a): In supported IDEs, users can interact with CodeGeeX by providing prompts. Different models are used to support three tasks: code generation, code translation and code explanation. (b) and (c): In HumanEval and our newly proposed HumanEval-X, CodeGeeX shows promising multilingual ability and consistently outperforms other multilingual code generation models.
Figure 1 Summary of CodeGeeX. (a): In supported IDEs, users can interact with CodeGeeX by providing prompts. Different models are used to support three tasks: code generation, code translation and code explanation. (b) and (c): In HumanEval and our newly proposed HumanEval-X, CodeGeeX shows promising multilingual ability and consistently outperforms other multilingual code generation models.CC-BY · From the PDF
Figure 2 CodeGeeX's model architecture. Themodelhas13Bparameters, consisting of 39-layer left-to-right transformer decoders and a top query layer. It takes text/code tokens as input and outputs the probability of the next token autoregressively.
Figure 2 CodeGeeX's model architecture. Themodelhas13Bparameters, consisting of 39-layer left-to-right transformer decoders and a top query layer. It takes text/code tokens as input and outputs the probability of the next token autoregressively.CC-BY · From the PDF
Figure 3 Language distribution and tags of CodeGeeX's data.
Figure 3 Language distribution and tags of CodeGeeX's data.CC-BY · From the PDF
Figure 4 CodeGeeX vs. its quantized version on code generation of 5 programming languages in HumanEval-X.
Figure 4 CodeGeeX vs. its quantized version on code generation of 5 programming languages in HumanEval-X.CC-BY · From the PDF
Image 5
CC-BY · From the PDF
Image 6
CC-BY · From the PDF
Figure 7 Test result statistics across models. For each model and language, we study 200 samples generated under 𝑡 = 0 . 8 , 𝑝 = 0 . 95. CodeGeeX has less Runtime or Syntax/Semantic Error .
Figure 7 Test result statistics across models. For each model and language, we study 200 samples generated under 𝑡 = 0 . 8 , 𝑝 = 0 . 95. CodeGeeX has less Runtime or Syntax/Semantic Error .CC-BY · From the PDF
Figure 8 Performance of translating A-to-B is negatively correlated with B-to-A. Such asymmetry indicates that multilingual models lack a high-level understanding between languages.
Figure 8 Performance of translating A-to-B is negatively correlated with B-to-A. Such asymmetry indicates that multilingual models lack a high-level understanding between languages.CC-BY · From the PDF
Figure 9 Profession vs. satisfaction. Left: Profession distribution. Right: Averaged rating score of CodeGeeX extensions.
Figure 9 Profession vs. satisfaction. Left: Profession distribution. Right: Averaged rating score of CodeGeeX extensions.CC-BY · From the PDF
Figure 10 Survey on "Has CodeGeeX improved your coding efficiency?". Over 83.4% of users have positive answers.
Figure 10 Survey on "Has CodeGeeX improved your coding efficiency?". Over 83.4% of users have positive answers.CC-BY · From the PDF

Verification

Claims the paper makes

6 Valid · 3 Not valid · 0 Not checked · 65 Own results and interpretations

24 more claims

Create a free account to see all 74 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 20
  • It may state two separate claims as one. 29
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Show all 15 references

Research works printed but not linked: 4

  1. Sid Black , Leo Gao , Phil Wang , Connor Leahy , and Stella Biderman . 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. If you use this software, please cite it using these metadata , Vol. 58 ( 2021 ). Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. If you use this software, please cite it using these metadata, Vol. 58 (2021).
  2. Alec Radford Jeffrey Wu Rewon Child David Luan Dario Amodei Ilya Sutskever etal 2019. Language models are unsupervised multitask learners. OpenAI blog Vol. 1 8 (2019) 9. Alec Radford Jeffrey Wu Rewon Child David Luan Dario Amodei Ilya Sutskever et al. 2019. Language models are unsupervised multitask learners. OpenAI blog Vol. 1 8 (2019) 9.
  3. Lewis Tunstall , Leandro von Werra , and Thomas Wolf . 2022. Natural language processing with transformers. "O'Reilly Media , Inc .". Lewis Tunstall, Leandro von Werra, and Thomas Wolf. 2022. Natural language processing with transformers. "O'Reilly Media, Inc.".
  4. Richard J Waldinger and Richard CT Lee . 1969 . PROW: A step toward automatic program writing . In Proceedings of the 1st international joint conference on Artificial intelligence. 241--252 . Richard J Waldinger and Richard CT Lee. 1969. PROW: A step toward automatic program writing. In Proceedings of the 1st international joint conference on Artificial intelligence. 241--252.

Cited by

See all 563

Metadata

Authors
Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Shen, L., Wang, Z., Wang, A., Li, Y., Su, T., Yang, Z., & Tang, J.
Affiliations
  • Tsinghua University (CN)
  • Zhipu AI (China) (CN)
  • Huawei Technologies (China) (CN)
Venue
Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
Publisher
ACM
Published
2023-08-04
Pages
5673-5684
OpenAlex
W4385562549
Cited by
563
References
15
Citations by year
2026: 148 · 2025: 205 · 2024: 157 · 2023: 53
FWCI
108.31
Type
conference paper
Language
en
Open access
Locations
  • ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
Natural Science Foundation of China (No. 62276148, No. 61836013, No. 61825602)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
Keywords
multilingual code generation, pre-trained code models, program translation, programming language support, Visual Studio Code extension, coding efficiency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X. https://app.cochat.ai/papers/10.1145/3580305.3599790
The paper
Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Shen, L., Wang, Z., Wang, A., Li, Y., Su, T., Yang, Z., & Tang, J. (2023). CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. https://doi.org/10.1145/3580305.3599790

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.