PreprintCited by 276Software Engineering ResearchTopic ModelingNatural Language Processing Techniques

The DOI matches its DataCite record; 39 of the 39 references it cites were found in a registry or an index.

CodeBERT: A Pre-Trained Model for Programming and Natural Languages

Open access

Feng, Zhangyin, Guo, Daya, Tang, Duyu, Duan, Nan, Feng, Xiaocheng, Gong, Ming, Shou, Linjun, Qin, Bing, Liu, Ting, Jiang, Daxin, Zhou, Ming·2020·arXiv

DOI 10.48550/arxiv.2002.08155arXiv 2002.08155

Open this paper in CoChat — free account Open PDF

Abstract

We present CodeBERT, a bimodal pre-trained model for programming language (PL) and nat-ural language (NL). CodeBERT learns general-purpose representations that support downstream NL-PL applications such as natural language codesearch, code documentation generation, etc. We develop CodeBERT with Transformer-based neural architecture, and train it with a hybrid objective function that incorporates the pre-training task of replaced token detection, which is to detect plausible alternatives sampled from generators. This enables us to utilize both bimodal data of NL-PL pairs and unimodal data, where the former provides input tokens for model training while the latter helps to learn better generators. We evaluate CodeBERT on two NL-PL applications by fine-tuning model parameters. Results show that CodeBERT achieves state-of-the-art performance on both natural language code search and code documentation generation tasks. Furthermore, to investigate what type of knowledge is learned in CodeBERT, we construct a dataset for NL-PL probing, and evaluate in a zero-shot setting where parameters of pre-trained models are fixed. Results show that CodeBERT performs better than previous pre-trained models on NL-PL probing.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2002.08155

Cites

39 of 39 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Attention Is All You Need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929

Flagged first, then by citations

Show all 27 references

Research works printed but not linked: 2

Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. { ELECTRA } : Pretraining text encoders as discriminators rather than generators. In International Conference on Learning Representations .
Dan Jurafsky. 2000. Speech & language processing . Pearson Education India.

Sorted by citations

Items 201–276 of 276

A Comparison of Code Embeddings and BeyondPreprint
Han, S., Wang, D., Li, W., & Lu, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.07173 · Cited by 5
Using Document Similarity Methods to create Parallel Datasets for Code TranslationPreprint
Agarwal, M., Talamadupula, K., Martinez, F., Houde, S., Muller, M. J., Richards, J. T., Ross, S., & Weisz, J. D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.05423 · Cited by 11
Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax HierarchyPreprint
Clement, C. B., Lu, S., Liu, X., Tufano, M., Drain, D., Duan, N., Sundaresan, N., & Svyatkovskiy, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.08780 · Cited by 11
Self-citation
Invaastcluster: On Applying Invariant-Based Program Clustering to Introductory Programming AssignmentsPreprint
Orvalho, P., Janota, M., & Manquinho, V. M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4595235 · Cited by 2
A Multi-Agent Framework for Accessible Web Code Generation
Aljedaani, W., Shaik, M. A., & Aljohani, A. (2026) · Proceedings of the 23rd International Web for All Conference · doi:10.1145/3800424.3800453 · Cited by 1
JUST-IN-TIME VULNERABILITY DETECTION AND LOCALIZATION
Vo, H. D. (2024) · Journal of Computer Science and Cybernetics · doi:10.15625/1813-9663/19102 · Cited by 1
Improving Source Code Pre-Training via Type-Specific Masking
Zou, W., Li, Q., Li, C., Ge, J., Chen, X., Huang, L., & Luo, B. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3699599 · Cited by 2
Assessing Cybersecurity Threats: The Application of NLP in Advanced Threat Intelligence Systems
Islam, M. A., Islam, R., Chowdhury, S. A., Nur, A. H., Sufian, M. A., & Hasan, M. M. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-70924-1_1 · Cited by 2
E-PRedictor: an approach for early prediction of pull request acceptance
Chen, K., Bao, L., Hu, X., Xia, X., & Yang, X. (2025) · Science China Information Sciences · doi:10.1007/s11432-022-3953-4 · Cited by 2
Knowledge-guided large language models are trustworthy API recommenders
Wei, H., Su, X., Zheng, W., Tao, W., Yu, H., & Kuang, Y. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00518-4 · Cited by 1
Does it smell? A homogeneous stacking approach for code smell prediction
Jammal, R. E., & Azar, D. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107801 · Cited by 1
Analysis of software code preprocessing methods to improve the effectiveness of using large language models in vulnerability detection tasks
Charugin, V. V., Charugin, V. V., Stavtsev, A. V., & Chesalin, A. N. (2025) · Computational nanotechnology · doi:10.33693/2313-223x-2025-12-3-67-79 · Cited by 1
Pioneering agentic retrieval-augmented generation in software quality: a novel framework for code smell detection via dynamic retrieval
Aljohani, B., & Aljuhani, A. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.3642 · Cited by 1
Predicting Intermittent Job Failure Categories for Diagnosis Using Few-Shot Fine-Tuned Language Models
Aïdasso, H., Bordeleau, F., & Tizghadam, A. (2026) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3803437.3805224 · Cited by 3
Avaliação de Modelos Neurais para Sumarização de Código-fonte
Pontes, L. B. L., Oliveira, H., & Boldt, F. D. A. (2022) · Anais do Seminário Integrado de Software e Hardware (SEMISH) · doi:10.5753/semish.2022.223154 · Cited by 1
Automatic exploit assessment based on deep learning methods
Busko, N., Владимировна, Ф. Е., & Kotenko, I. V. (2024) · Ontology of Designing · doi:10.18287/2223-9537-2024-14-3-408-420 · Cited by 2
Linguacodus: a synergistic framework for transformative code generation in machine learning pipelines
Trofimova, E., Sataev, E., & Ustyuzhanin, A. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.2328
Deep Just-In-Time Inconsistency Detection Between Comments and Source Code
Panthaplackel, S., Li, J. J., Gligoric, M., & Mooney, R. J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i1.16119 · Cited by 2
Exploiting Token and Path-based Representations of Code for Identifying Security-Relevant CommitsPreprint
Ram, A., Xin, J., Nagappan, M., Yu, Y., Lozoya, R. C., Sabetta, A., & Lin, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.07620 · Cited by 3
NaturalCC: A Toolkit to Naturalize the Source Code CorpusPreprint
Wan, Y., He, Y., Zhang, J., Sui, Y., Jin, H., Xu, G., Xiong, C., & Yu, P. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.03225 · Cited by 1
Cross-Lingual Adaptation for Type Inference.Preprint
Li, Z., Xie, X., Li, H., Xu, Z., Li, Y., & Liu, Y. (2021) · arXiv (Cornell University) · Cited by 1
Distilling Transformers for Neural Cross-Domain SearchPreprint
Clement, C. B., Wu, C. H., Drain, D., & Sundaresan, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.03322 · Cited by 1
Neural Unit Test Suggestions.Preprint
Dinella, E., Lahiri, S. K., Mytkowicz, T., & Ryan, G. (2021) · arXiv (Cornell University) · Cited by 1
AugmentedCode: Examining the Effects of Natural Language Resources in Code Retrieval ModelsPreprint
Bahrami, M., Shrikanth, N. C., Mizobuchi, Y., Liu, L., Fukuyori, M., Chen, W., & Munakata, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08512 · Cited by 4
WELL: Applying Bug Detectors to Bug Localization via Weakly Supervised LearningPreprint
Li, Z., Zhang, H., Li, J., Jin, Z. G., & Li, G. (2023) · doi:10.22541/au.168810846.65018089/v1 · Cited by 1
Pre-Training Representations of Binary Code Using Contrastive LearningPreprint
Zhang, Y., Huang, C., Zhang, Y., Cao, K., Anderson, S., Shao, H., Leach, K., & Huang, Y. G. (2023) · doi:10.36227/techrxiv.24061689.v1 · Cited by 1
Structural Syntax Network for Code ClassificationPreprint
Shah, M., Patel, R., & Terry, A. (2023) · Preprints.org · doi:10.20944/preprints202312.0805.v1 · Cited by 1
A Comprehensive Study of LLM and Evolution, Varieties, and Their Role in Software Engineering and CybersecurityPreprint
Rasel, H., Didar, A. B. S., Dinar, A. A. M., Fahad, F. I., Khan, M. A., & Biplob, M. B. (2025) · Preprints.org · doi:10.20944/preprints202507.1600.v1
On the Difficulty of Selecting Few-Shot Examples for Effective LLM-based Vulnerability Detection
Hannan, M. A., Ni, R., Zhang, C., Jia, L., Mangal, R., & Pasareanu, C. S. (2026) · Proceedings 2026 Workshop on LLM Assisted Security and Trust Exploration · doi:10.14722/last-x.2026.23025
Enhancing Code Translation in Language Models with Few-Shot Learning via Retrieval-Augmented Generation
Bhattarai, M., Santos, J. E., Jones, S., Biswas, A., Alexandrov, B., & O'Malley, D. (2024) · arXiv
Self-Supervised Learning for Code Retrieval and Summarization through Semantic-Preserving Program TransformationsPreprint
Bui, N. D. Q., Yu, Y., & Jiang, L. (2020) · arXiv (Cornell University)
UoB at AI-SOCO 2020: Approaches to Source Code Classification and the Surprising Power of n-grams.
Crosby, A. E., & Madabushi, H. T. (2020) · FIRE (Working Notes)
Applying Pre-trained Model and Fine-tune to Conduct Humor Analysis on Spanish Tweets.
Kui, Y. (2021) · IberLEF@SEPLN
Searching for Replacement ClassesPreprint
Samak, M., Cambronero, J., & Rinard, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.05638
CodeQA: A Question Answering Dataset for Source Code ComprehensionPreprint
Liu, C., & Wan, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.08365 · Cited by 14
Leveraging deep learning techniques to secure software development
Siow, J. K. (2022) · doi:10.32657/10356/160286
Learning program semantics via exploring program structures with deep learning
Liu, S. (2022) · doi:10.32657/10356/164454
Machine-learning Approaches for Developing an Autograder for High School-level CS-for-all Initiatives
Tisha, S. M. (2022) · doi:10.31390/gradschool_dissertations.6089
Code-Aware Cross-Program Transfer Hyperparameter Optimization
Wang, Z., He, X., Chen, K., Chen, L., & Su, J. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26226
An Empirical Study of Adversarial Training in Code Comment Generation
Shen, Y., Ju, X., Chen, X., & Yang, G. (2023) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2023-108
A comparison of imbalanced data handling methods for pre-trained model in multi-label classification of stack overflow
Umparat, A. (2022) · doi:10.58837/chula.the.2022.338
STraceBERT: Source Code Retrieval using Semantic Application Traces
Spieß, C. K. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3617852
Guard2Vul: Vulnerability Detection viaGradient-based Adversarial TrainingEnhanced Graph LearningPreprint
Shen, H., Ju, X., Li, W., Chen, X., & Yang, G. (2024) · Research Square · doi:10.21203/rs.3.rs-3840091/v1
CHICOT: A Developer-Assistance Toolkit for Code Search with High-Level Contextual Information
Morishita, T., Koreeda, Y., Yamaguchi, A., Morio, G., Imaichi, O., & Sogawa, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i21.30575
Representation learning for algorithms implementation search using pseudo code
Sium, Y. A. (2022) · doi:10.31274/td-20240329-633
EMPIRICAL STUDY OF DOMAIN-ADAPTED PRETRAINING AND CROSS-LINGUAL TRANSFER FOR PROGRAMMING LANGUAGE UNDERSTANDING USING THE DOLMA STACK CODE SUBSET
Guerrero, L. E. M., Ceballos, Y. F., & Rojas, L. D. T. (2024) · Journal of Southwest Jiaotong University · doi:10.35741/issn.0258-2724.59.2.26
Designing artifact representation and automated pipeline for machine learning based Software Engineering
Phan, H. (2024) · doi:10.31274/td-20240617-75
A Comprehensive Evaluation of Large Language Models for Method Name PredictionPreprint
Qian, H., Xu, T., Ding, Z., Liu, W., Zhu, S., Yin, J., Feng, X., Fang, C., Zhao, J., & Sun, W. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4886577
Self-citation
Ask Me Any Type: Type Inference Plugin for Partial Code on the Web and in the Integrated Development Environment
Yu, C., HUANG, G., Yishun, W. U., Zijie, Z. H. A. O., Zhenhao, H. E., & Lu, J. (2024) · Wuhan University Journal of Natural Sciences · doi:10.1051/wujns/2024294349
Understanding Developer-Analyzer Interactions in Code Reviews
Schäf, M., Çirişci, B., Luo, L., Mansur, M. N., Tripp, O., Sanchez, D., Zhou, Q., & Zafar, M. B. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695257
Towards generalizable neural program understanding and synthesis
Li, Z. (2024) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/181413
Rapid “recycling” of logical algorithm representations in fronto-parietal reasoning systems following computer programming instructionsPreprint
Liu, Y., & Bedny, M. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.01.08.631982
AI for Code Synthesis: Can LLMs Generate Secure Code?Preprint
Bar, K. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5157837
Structure‐Aware Representation Learning for Effective Performance Prediction
Ramadan, T., Pinnow, N., Phelps, C., Thiagarajan, J. J., & Islam, T. Z. (2025) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.70046
Large Language Models for Automatic Knowledge Graph Construction From Text Documents: a Materials Science Case Study
Kaverinskiy, V. (2025) · doi:10.31274/isudp.2025.197.010
Transformer-Based Code Generation: Automating Software Development with AI
(2025) · IARJSET · doi:10.17148/iarjset.2025.12334
AI-Powered Code Review Enhancing Software Quality with Intelligent Agents
Konda, R. (2023) · International Journal of Leading Research Publication. · doi:10.70528/ijlrp.v4.i3.1541
Prompt-Optimized Fusion of Static Metrics and Code Embeddings for Software Defect PredictionPreprint
Rong, K. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5258851
Evaluating Prompt-Learning-Based API Review Classification Through Pre-Trained Models
Li, X., & Kim, A. (2025) · International Journal on Cybernetics & Informatics · doi:10.5121/ijci.2025.140301
Reductive Analysis with Compiler-Guided Large Language Models for Input-Centric Code Optimizations
Wang, X., Hui, X., Liao, C., & Shen, X. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3729282
DEEP LEARNING SOLUTIONS FOR SOURCE CODE VULNERABILITY DETECTION
HoLeViet, N., Phan, L., NgoVan, H., & TrinhQuang, T. (2025) · International Journal of Network Security & Its Applications · doi:10.5121/ijnsa.2025.17303
An Efficient Transformer-Based Model for Automated Code Generation: Leveraging Large Language Models for Software Engineering
Rose, L. (2020) · International Journal of Emerging Research in Engineering and Technology · doi:10.63282/3050-922x.ijeret-v1i3p101
An enhanced approach for automatic annotation of error codes based on Seq2edit.
Wang, J., Lin, T., Zhao, R., & Zhao, H. (2025) · Europe PMC (PubMed Central) · doi:10.7717/peerj-cs.3024
LLM-based program analysis for source codes, abstract syntax trees and webassembly instructions
Deng, L., Zhong, Q., Qiu, Y., Chen, J., Lei, H., Yang, S., Zhou, L., & Cheng, H. (2025) · Cluster Computing · doi:10.1007/s10586-025-05557-w · Cited by 1
AI4SWEng manifest: Empowering the future of software engineering with advanced trustworthy artificial intelligence
Atalay, A. S., Öztin, Ö., Kanak, A., Arif, İ., Badii, A., Dasari, N., Ergün, S., Ergün, S., Herkiloğlu, O., Mermoud, G., et al. (2025) · Open Research Europe · doi:10.12688/openreseurope.21192.1
A Study on Mixup-Inspired Augmentation Methods for Software Vulnerability Detection
Daneshvar, S. S., Tan, D., Wang, S., & Leung, C. K.-S. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3756681.3757017
Source Code Vulnerability Detection and Interpretability with Language Models
Silveira, L., Lélis, C. A. S., Marcondes, C. A. C., & Verri, F. A. N. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-11539-3_15
Website Segmentation Beyond Structure: A Benchmark on Functional and Digital Maturity Classes
Saxer, J., Gerber, J., Weiler, A., & Grossniklaus, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21300-6_14
An Overview of CodeLMs’ Security Challenges
Aaoud, H. E., Ahkouk, K., Addouni, F. E., & Moubtahij, H. E. (2026) · Advances in Science, Technology & Innovation/Advances in science, technology & innovation · doi:10.1007/978-3-031-97825-8_75
Bidirectional Program Dependency–Guided Attention for Improved Method‐Level Software Defect Prediction
Lin, J.-R., Huang, C., & Fang, C. (2026) · Journal of Software Evolution and Process · doi:10.1002/smr.70115
CoBAT: Automated Adversarial Training for Robust Software Vulnerability Detection via Bayesian Optimization
Wu, T., & Li, D. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3381-6_13
A Unified Framework for Function-Level Vulnerability Detection and Explanation in Smart Contracts
Nguyen, N. M., & Nguyen, L.-M. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3840386
Augmenting Multi-technique Static Analysis with Large Language Models: A Neuro-symbolic Approach to Smart Contract Vulnerability Detection
Wang, J., Song, F., Zhang, M., Du, B., & Pei, R. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832222
An Empirical Study of the Non-determinism of ChatGPT in Code Generation
Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2023) · arXiv · doi:10.48550/arxiv.2308.02828
GraphCodeBERT: Pre-training Code Representations with Data Flow
Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Liu, S., Zhou, L., Duan, N., Svyatkovskiy, A., Fu, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.08366 · Cited by 480
Self-citation
Unsupervised translation of programming languages
Lachaux, M.-A., Roziere, B., Chanussot, L., & Lample, G. (2020) · arXiv · doi:10.48550/arxiv.2006.03511 · Cited by 83

Authors: Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., & Zhou, M.
Affiliations: Harbin Institute of Technology (CN)
Sun Yat-sen University (CN)
Microsoft (United States) (US)
ORCID: Feng, Zhangyin
Guo, Daya
Tang, Duyu
Duan, Nan
Feng, Xiaocheng
Gong, Ming
Shou, Linjun
Qin, Bing
Liu, Ting
Jiang, Daxin
Zhou, Ming
Venue: arXiv
Publisher: arXiv
Published: 2020
ISSN: 2331-8422 (ISSN-L)
DOI: 10.48550/arxiv.2002.08155
arXiv: 2002.08155
OpenAlex: W3008088841
Cited by: 243
References: 27
Citations by year: 2026: 9 · 2025: 41 · 2024: 51 · 2023: 28 · 2022: 23 · 2021: 72 · 2020: 18
Type: posted content
Language: en
Open access: green
arxiv.org
Licence: arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations: arXiv (Cornell University) (submittedVersion)
arXiv (Cornell University)
Funders: National Natural Science Foundation of China (2018YFB1005103, 61772156, 61632011)
Topics: Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords: CodeBERT, natural language programming, transformer models, code generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

CodeBERT: A Pre-Trained Model for Programming and Natural Languages

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks