The DOI matches its DataCite record; the references it cites are not checked yet.
CodeT5+: Open Code Large Language Models for Code Understanding and Generation
Open accessAbstract
Large language models (LLMs) pretrained on vast source code have achieved prominent progress in code intelligence. However, existing code LLMs have two main limitations in terms of architecture and pretraining tasks. First, they often adopt a specific architecture (encoder-only or decoder-only) or rely on a unified encoder-decoder network for different downstream tasks. The former paradigm is limited by inflexibility in applications while in the latter, the model is treated as a single system for all tasks, leading to suboptimal performance on a subset of tasks. Secondly, they often employ a limited set of pretraining objectives which might not be relevant to some downstream tasks and hence result in substantial performance degrade. To address these limitations, we propose ``CodeT5+'', a family of encoder-decoder LLMs for code in which component modules can be flexibly combined to suit a wide range of downstream code tasks. Such flexibility is enabled by our proposed mixture of pretraining objectives to mitigate the pretrain-finetune discrepancy. These objectives cover span denoising, contrastive learning, text-code matching, and causal LM pretraining tasks, on both unimodal and bimodal multilingual code corpora. Furthermore, we propose to initialize CodeT5+ with frozen off-the-shelf LLMs without training from scratch to efficiently scale up our models, and explore instruction-tuning to align with natural language instructions. We extensively evaluate CodeT5+ on over 20 code-related benchmarks in different settings, including zero-shot, finetuning, and instruction-tuning. We observe state-of-the-art (SoTA) model performance on various code-related tasks, such as code generation and completion, math programming, and text-to-code retrieval tasks. Particularly, our instruction-tuned CodeT5+ 16B achieves new SoTA results on HumanEval code generation task against other open code LLMs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“In this work, we use pretrained CodeT5+ (Wang et al., 2023b) foot_3 as the backbone based on its superior performance on code understanding and generation evaluations reported in (Wang et al., 2023b).”
From CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation (2023) · cites this paper as Wang et al., 2023b
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Other studies further find that the bugs introduced by LLM-powered code completion are often more subtle [4, 34], and difficult to revert [42].”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as [42]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Cited by
Sorted by citations
Items 101–116 of 116
The Convergence of AI/ML and Cybersecurity: Advancing Digital Forensic Techniques
Iyengar, S. S., Nabavirazavi, S., Hariprasad, Y., Prasad, H. B., & Mohan, C. K. (2025) · Signals and communication technology · doi:10.1007/978-3-031-89327-8_4 · Cited by 13
Lu, H., Peng, H., Nan, G., Cui, J., Wang, C., Jin, W., Wang, S., Pan, S., & Tao, X. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3583552 · Cited by 12
Geographic Knowledge Base Question Answering over OpenStreetMap
Yang, J., Jang, H., & Yu, K. (2023) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi13010010 · Cited by 11
A Preference-Driven Methodology for Efficient Code Generation
Li, Y., Zhou, Z., Peng, Z., Dong, J., You, H., Yan, R., Wen, S., Tian, Y., & Huang, T. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3639527 · Cited by 11
Evaluating SZZ Implementations: An Empirical Study on the Linux Kernel
Lyu, Y., Kang, H. J., Widyasari, R., Lawall, J., & Lo, D. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3406718 · Cited by 10
Katzy, J., Popescu, R. M., Arie Van Deursen, A., & Izadi, M. (2024) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1145/3650105.3652298 · Cited by 12
Learning to Generate Structured Code Summaries From Hybrid Code Context
Zhou, Z., Li, M., Yu, H., Fan, G., Yang, P., & Huang, Z. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3439562 · Cited by 10
Large language models based vulnerability detection: How does it enhance performance?
Xuan, C. D., Quang, D. B., & Quang, V. D. (2025) · International Journal of Information Security · doi:10.1007/s10207-025-00983-8 · Cited by 13
A Transformer-Based Approach for Smart Invocation of Automatic Code Completion
de Moor, A., van Deursen, A., & Izadi, M. (2024) · Proceedings of the 1st ACM International Conference on AI-Powered Software · doi:10.1145/3664646.3664760 · Cited by 7
Peng, Q., Chai, Y., & Li, X. (2024) · arXiv · doi:10.48550/arxiv.2402.16694 · Cited by 88
“Some studies have delved into the intricacies of LLMs in multiple PLs (Fried et al., 2023; Wang et al., 2023; Nijkamp et al., 2023a; Li et al., 2023), while others have focused on the synergy between multilingual NLs and PLs. Notably, ERNIE-Code (Chai et al., 2023) pioneered multilingual LLMs by…”
BackgroundLongCodeZip: Compress Long Context for Code Language Models
Shi, Y., Qian, Y., Zhang, H., Shen, B., & Gu, X. (2025) · 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00020 · Cited by 7
On the Impacts of Contexts on Repository-Level Code Generation
Hai, N. L., Nguyen, D. M., & Bui, N. D. Q. (2024) · arXiv · doi:10.48550/arxiv.2406.11927 · Cited by 3
Self-citationLongCodeZip: Compress Long Context for Code Language Models
Shi, Y., Qian, Y., Zhang, H., Shen, B., & Gu, X. (2025) · arXiv · Cited by 7
“CodeT5+[53] employs span denoising in an encoder-decoder framework.”
Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)
Ahmed, T., Pai, K. S., Devanbu, P., & Barr, E. T. (2023) · arXiv · Cited by 112
SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios
Le, T., Thai, M. V. T., Manh, D. N., Nhat, H. P., & Bui, N. D. Q. (2025) · arXiv · doi:10.48550/arxiv.2512.18470 · Cited by 79
Self-citationA Transformer-Based Approach for Smart Invocation of Automatic Code Completion
de Moor, A., van Deursen, A., & Izadi, M. (2024) · arXiv
Metadata
- Authors
- Wang, Y., Le, H., Gotmare, A. D., Bui, N. D. Q., Li, J., & Hoi, S. C. H.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 1069-1088
- arXiv
- 2305.07922
- OpenAlex
- W4389519352
- Cited by
- 459
- References
- 0
- Citations by year
- 2026: 122 · 2025: 194 · 2024: 108 · 2023: 35
- FWCI
- 118.09
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- CodeT5, code large language models, encoder-decoder architecture, contrastive learning, instruction tuning, code generation, program comprehension, HumanEval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
