CoChat
Sign Up
PreprintCited by 459Software Engineering ResearchTopic ModelingNatural Language Processing Techniques

CodeT5+: Open Code Large Language Models for Code Understanding and Generation

Open access
Wang, Yue, Le, Hung, Gotmare, Akhilesh Deepak, Bui, Nghi D. Q., Li, Junnan, Hoi, Steven C. H.·2023·arXiv
DOI 10.48550/arxiv.2305.07922arXiv 2305.07922
Open this paper in CoChat — free account

Abstract

Large language models (LLMs) pretrained on vast source code have achieved prominent progress in code intelligence. However, existing code LLMs have two main limitations in terms of architecture and pretraining tasks. First, they often adopt a specific architecture (encoder-only or decoder-only) or rely on a unified encoder-decoder network for different downstream tasks. The former paradigm is limited by inflexibility in applications while in the latter, the model is treated as a single system for all tasks, leading to suboptimal performance on a subset of tasks. Secondly, they often employ a limited set of pretraining objectives which might not be relevant to some downstream tasks and hence result in substantial performance degrade. To address these limitations, we propose ``CodeT5+'', a family of encoder-decoder LLMs for code in which component modules can be flexibly combined to suit a wide range of downstream code tasks. Such flexibility is enabled by our proposed mixture of pretraining objectives to mitigate the pretrain-finetune discrepancy. These objectives cover span denoising, contrastive learning, text-code matching, and causal LM pretraining tasks, on both unimodal and bimodal multilingual code corpora. Furthermore, we propose to initialize CodeT5+ with frozen off-the-shelf LLMs without training from scratch to efficiently scale up our models, and explore instruction-tuning to align with natural language instructions. We extensively evaluate CodeT5+ on over 20 code-related benchmarks in different settings, including zero-shot, finetuning, and instruction-tuning. We observe state-of-the-art (SoTA) model performance on various code-related tasks, such as code generation and completion, math programming, and text-to-code retrieval tasks. Particularly, our instruction-tuned CodeT5+ 16B achieves new SoTA results on HumanEval code generation task against other open code LLMs.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2305.07922

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • CodeT5+ is used as the backbone based on its superior performance on code understanding and generation evaluations.

    “In this work, we use pretrained CodeT5+ (Wang et al., 2023b) foot_3 as the backbone based on its superior performance on code understanding and generation evaluations reported in (Wang et al., 2023b).”

    From CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation (2023) · cites this paper as Wang et al., 2023b

  • Bugs introduced by LLM-powered code completion are often more subtle and difficult to revert.

    “Other studies further find that the bugs introduced by LLM-powered code completion are often more subtle [4, 34], and difficult to revert [42].”

    From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as [42]

    The citation [42] is attached specifically to “difficult to revert”; the “more subtle” claim has different citations. D.5 Code Completion (p. 25) describes completion benchmarks, and 5 Experiments (p. 9) discusses evaluation limitations. Neither addresses whether bugs introduced by LLM-powered completion are difficult to revert; searches for that phrase and for reverting bugs found no supporting passage. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Cited by

Sorted by citations

Related

Authors
Wang, Y., Le, H., Gotmare, A. D., Bui, N. D. Q., Li, J., & Hoi, S. C. H.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
1069-1088
DOI
10.48550/arxiv.2305.07922
arXiv
2305.07922
OpenAlex
W4389519352
Cited by
459
References
0
Citations by year
2026: 122 · 2025: 194 · 2024: 108 · 2023: 35
FWCI
118.09
Type
conference paper
Language
en
Open access
Licence
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
CodeT5, code large language models, encoder-decoder architecture, contrastive learning, instruction tuning, code generation, program comprehension, HumanEval

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review