CoChat
Sign Up
Cited by 459Software Engineering ResearchTopic ModelingNatural Language Processing Techniques

CodeT5+: Open Code Large Language Models for Code Understanding and Generation

Open access
Wang, Yue, Le, Hung, Gotmare, Akhilesh, Bui, Nghi, Li, Junnan, Hoi, Steven·2023·Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
DOI 10.18653/v1/2023.emnlp-main.68
Open this paper in CoChat — free account

Abstract

Large language models (LLMs) pretrained on vast source code have achieved prominent progress in code intelligence. However, existing code LLMs have two main limitations. First, they often adopt a specific architecture (encoder-only or decoder-only) or rely on a unified encoder-decoder network for different downstream tasks, lacking the flexibility to operate in the optimal architecture for a specific task. Secondly, they often employ a limited set of pretraining objectives which might not be relevant to some tasks and hence result in substantial performance degrade. To address these limitations, we propose “CodeT5+”, a family of encoder-decoder LLMs for code in which component modules can be flexibly combined to suit a wide range of code tasks. Such flexibility is enabled by our proposed mixture of pretraining objectives, which cover span denoising, contrastive learning, text-code matching, and causal LM pretraining tasks, on both unimodal and bimodal multilingual code corpora. Furthermore, we propose to initialize CodeT5+ with frozen off-the-shelf LLMs without training from scratch to efficiently scale up our models, and explore instruction-tuning to align with natural language instructions. We extensively evaluate CodeT5+ on over 20 code-related benchmarks in different settings, including zero-shot, finetuning, and instruction-tuning. We observe state-of-the-art (SoTA) performance on various code-related tasks, and our instruction-tuned CodeT5+ 16B achieves new SoTA results of 35.0% pass@1 and 54.5% pass@10 on the HumanEval code generation task against other open code LLMs, even surpassing the OpenAI code-cushman-001 model.

Integrity signals

Registry

Crossref · 10.18653/v1/2023.emnlp-main.68

Cites

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

The text we found for this paper looks like another document, so its claims and self-check weren’t run. The registry and citation checks still apply.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The text we found for this paper looks like another document, so the self-check wasn’t run.

Cites

No references could be resolved yet.

Research works printed but not linked: 4

  1. CodeT5+: Open Code Large Language Models for Code Understanding and Generation https://aclanthology.org/2023.emnlp-main.68/ (Wang et al., EMNLP 2023)
  2. CodeT5+: Open Code Large Language Models for Code Understanding and Generation (Wang et al., EMNLP 2023)
  3. Yue Wang, Hung Le, Akhilesh Gotmare, Nghi Bui, Junnan Li, and Steven Hoi. 2023. CodeT5+: Open Code Large Language Models for Code Understanding and Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1069–1088, Singapore. Association for Computational Linguistics.
  4. ACL materials are Copyright © 1963–2026 ACL; other materials are copyrighted by their respective copyright holders. Materials prior to 2016 here are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License. Permission is granted to make copies for the purposes of teaching and research. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 International License.

Cited by

Sorted by citations

Related

Authors
Wang, Y., Le, H., Gotmare, A., Bui, N., Li, J., & Hoi, S.
ORCID
Venue
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
Publisher
Association for Computational Linguistics
Published
2023
Pages
1069-1088
DOI
10.18653/v1/2023.emnlp-main.68
OpenAlex
W4389519352
Cited by
459
References
0
Citations by year
2026: 122 · 2025: 194 · 2024: 108 · 2023: 35
FWCI
118.09
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
CodeT5, code large language models, encoder-decoder architecture, contrastive learning, instruction tuning, code generation, program comprehension, HumanEval

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review