The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
UniXcoder: Unified Cross-Modal Pre-training for Code Representation
Open accessAbstract
Pre-trained models for programming languages have recently demonstrated great success on code intelligence.To support both code-related understanding and generation tasks, recent works attempt to pre-train unified encoder-decoder models.However, such encoder-decoder framework is sub-optimal for auto-regressive tasks, especially code completion that requires a decoder-only manner for efficient inference.In this paper, we present UniXcoder, a unified cross-modal pre-trained model for programming language.The model utilizes mask attention matrices with prefix adapters to control the behavior of the model and leverages cross-modal contents like AST and code comment to enhance code representation.To encode AST that is represented as a tree in parallel, we propose a one-to-one mapping method to transform AST in a sequence structure that retains all structural information from the tree.Furthermore, we propose to utilize multi-modal contents to learn representation of code fragment with contrastive learning, and then align representations among programming languages using a cross-modal generation task.We evaluate UniXcoder on five code-related tasks over nine datasets.To further evaluate the performance of code fragment representation, we also construct a dataset for a new task, called zero-shot code-to-code search.Results show that our model achieves state-of-the-art performance on most tasks and analysis reveals that comment and AST can both enhance UniXcoder.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Gao, T., Yao, X., & Chen, D. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.552 · Cited by 2,851
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824
Self-citationWang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495
Self-citationWang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495
Self-citationUnified Language Model Pre-training for Natural Language Understanding and Generation
Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., & Hon, H.-W. (2019) · arXiv · doi:10.48550/arxiv.1905.03197 · Cited by 1,225
Self-citationNeural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2015) · arXiv · doi:10.48550/arxiv.1508.07909 · Cited by 950
Convolutional Neural Networks over Tree Structures for Programming Language Processing
Mou, L., Li, G., Zhang, L., Wang, T., & Jin, Z. G. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10139 · Cited by 762
Unified Pre-training for Program Understanding and Generation
Ahmad, W., Chakraborty, S., Ray, B., & Chang, K.-W. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.211 · Cited by 642
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.04664 · Cited by 536
Self-citationCodeSearchNet Challenge: Evaluating the State of Semantic Code Search
Husain, H., Wu, H.-H., Gazit, T., Allamanis, M., & Brockschmidt, M. (2019) · arXiv · doi:10.48550/arxiv.1909.09436 · Cited by 483
IntelliCode compose: code generation using transformer
Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417058 · Cited by 474
Mining source code repositories at massive scale using language modeling
Allamanis, M., & Sutton, C. (2013) · IEEE Working Conference on Mining Software Repositories · doi:10.1109/msr.2013.6624029 · Cited by 398
Towards a Big Data Curated Benchmark of Inter-project Code Clones
Svajlenko, J., Islam, J. F., Keivanloo, I., Roy, C. K., & Mia, M. M. (2014) · 2014 IEEE International Conference on Software Maintenance and Evolution · doi:10.1109/icsme.2014.77 · Cited by 339
Probabilistic model for code with decision trees
Raychev, V., Bielik, P., & Vechev, M. T. (2016) · Conference on Object-Oriented Programming Systems, Languages, and Applications · doi:10.1145/2983990.2984041 · Cited by 294
(2004) · doi:10.3115/1220355.1220427 · Cited by 294
Learning and Evaluating Contextual Embedding of Source CodePreprint
Kanade, A., Maniatis, P., Balakrishnan, G., & Shi, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.00059 · Cited by 258
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · arXiv · doi:10.48550/arxiv.2002.08155 · Cited by 243
Self-citationMulti-task Learning based Pre-trained Language Model for Code Completion
Liu, F., Li, G., Zhao, Y., & Jin, Z. (2020) · Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3324884.3416591 · Cited by 188
Mapping Language to Code in Programmatic Context
Iyer, S., Konstas, I., Cheung, A., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1192 · Cited by 180
SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code RepresentationPreprint
Wang, X. E., Wang, Y., Mi, F., Zhou, P., Wan, Y., Xiao, L., Li, L., Wu, H., Liu, J., & Jiang, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.04556 · Cited by 170
Self-citationCoSQA: 20,000+ Web Queries for Code Search and Question Answering
Huang, J., Tang, D., Shou, L., Gong, M., Xu, K., Jiang, D., Zhou, M., & Duan, N. (2021) · doi:10.18653/v1/2021.acl-long.442 · Cited by 110
Self-citationExploring Software Naturalness through Neural Language ModelsPreprint
Buratti, L., Pujar, S., Bornea, M. A., McCarley, S., Zheng, Y., Rossiello, G., Morari, A., Laredo, J., Thost, V., Zhuang, Y., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.12641 · Cited by 95
CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks
Puri, R., Kung, D. S., Janssen, G., Zhang, W., Domeniconi, G., Zolotov, V., Dolby, J., Chen, J., Choudhury, M., Decker, L., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.12655 · Cited by 92
Probabilistic model for code with decision trees
Raychev, V., Bielik, P., & Vechev, M. (2016) · ACM SIGPLAN Notices · doi:10.1145/3022671.2984041 · Cited by 82
TreeBERT: A Tree-Based Pre-Trained Model for Programming LanguagePreprint
Jiang, X., Zheng, Z., Lyu, C., Li, L., & Lyu, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.12485 · Cited by 43
Pre-trained Contextual Embedding of Source CodePreprint
Kanade, A., Maniatis, P., Balakrishnan, G., & Shi, K. (2019) · arXiv (Cornell University) · Cited by 40
(2021) · arXiv (Cornell University) · Cited by 37
SimCSE: Simple Contrastive Learning of Sentence EmbeddingsPreprint
Gao, T., Yao, X., & Chen, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08821 · Cited by 36
CoSQA: 20,000+ Web Queries for Code Search and Question AnsweringPreprint
Huang, J., Tang, D., Shou, L., Gong, M., Xu, K., Jiang, D., Zhou, M., & Duan, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.13239 · Cited by 2
Self-citationIntelliCode Compose: Code Generation Using Transformer
Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N. (2020) · arXiv · doi:10.48550/arxiv.2005.08025
Research works printed but not linked: 2
- Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, LIU Shujie, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. 2020. Graphcode- bert: Pre-training code representations with data flow. In International Conference on Learning Rep- resentations.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Lan- guage models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Cited by
Sorted by citations
Bandi, A., Adapa, P. V. S. R., & Kuchi, Y. E. V. P. K. (2023) · Future Internet · doi:10.3390/fi15080260 · Cited by 638
A Survey on Large Language Models for Code Generation
Jiang, J., Wang, F., Shen, J., Kim, S., & Kim, S. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3747588 · Cited by 251
Automating code review activities by large-scale pre-training
Li, Z., Lu, S., Guo, D., Duan, N., Jannu, S., Jenks, G., Majumder, D., Green, J., Svyatkovskiy, A., Fu, S., et al. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549081 · Cited by 301
Self-citationRetrieval-Based Prompt Selection for Code-Related Few-Shot Learning
Nashid, N., Sintaha, M., & Mesbah, A. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00205 · Cited by 178
Self-Planning Code Generation with Large Language Models
Jiang, X., Dong, Y., Wang, L., Fang, Z., Shang, Q., Li, G., Jin, Z., & Jiao, W. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3672456 · Cited by 158
Metadata
- Authors
- Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., & Yin, J.
- Affiliations
- Sun Yat-sen University (CN)
- Microsoft Research Asia (China) (CN)
- Venue
- Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2022
- Pages
- 7212-7225
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4221166942
- Cited by
- 673
- References
- 28
- Citations by year
- 2026: 153 · 2025: 239 · 2024: 148 · 2023: 122 · 2022: 11
- FWCI
- 116.24
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Funders
- National Natural Science Foundation of China (U1811262, U1811261, U2001211, U1811264, U1911203)
- Basic and Applied Basic Research Foundation of Guangdong Province (2019B15 15130001)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- UniXcoder, code representation, programming language models, code comments, contrastive learning, cross-modal generation, code completion, code intelligence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
