IntelliCode compose: code generation using transformer
Open accessAbstract
In software development through integrated development environments (IDEs), code completion is one of the most widely used features. Nevertheless, majority of integrated development environments only support completion of methods and APIs, or arguments.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Several methods have been proposed to incorporate repository context into the training of language models, including n-grams, LSTMs, and Transformers.
“Several methods have been proposed to incorporate repository context into the training of language models, including n-grams (Tu et al., 2014), LSTMs (Hellendoorn and Devanbu, 2017), and Transformers (Svyatkovskiy et al., 2020;Liu et al., 2022;Ding et al., 2022).”
From RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation (2023) · cites this paper as Svyatkovskiy et al., 2020
Section 4.2 describes a Transformer code-completion model trained to predict tokens from preceding code-snippet context, and Section 3 says its training data came from GitHub projects. The cited paper therefore supports the Transformer example, but it does not show that the model incorporates broader repository-level context into training; the context it specifies is a sequence of code tokens. (LLM, conf=0.84)
Experimental — accuracy not yet measuredThe IntelliCode Compose system reports a Click Through Rate (acceptance rate) of 10% in online trials.
“The IntelliCode Compose system uses the term CTR (Click Through Rate) for this and reports a value of 10% in online trials [11].”
From Productivity Assessment of Neural Code Completion (2022) · cites this paper as [11]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1× · see citing works
“act syntax tree (AST) or concrete syntax tree (CST), and dataflow graph. A transformer block will typically consist of a multi-head self-attention, followed by a two-layer multi-layer perceptron (MLP) [17], optionally containing residual connections and layer normalization [23]. Recent neural architecture searches of t…”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · cited 2× · see citing works
“Several transformer models such as GPT-2, BERT, XLNet, and RoBERTa [19–22] have demonstrated the ability to learn effectively from unlabeled data to perform a wide variety of downstream tasks given supervised discriminative fine-tuning on each specific task.”
BackgroundRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 2× · see citing works
“We refer the interested reader to the numerous publications about transformer models [16, 20–22], and focus on code completion for the remainder of this section.”
BackgroundDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1× · see citing works
“lue and are most likely lower than the actual user sentiment. Over 150K requests, we have seen a surfacing rate of 9.2% and an acceptance rate of 10%. 11 Knowledge Distillation Knowledge distillation [38] is the model compression technique in which a smaller model – the student network – is trained to reproduce the re…”
InfluentialExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1× · see citing works
“(3) Language specific control codes, is an approach introduced in [29, 30] that uses target prefixes to facilitate constrained language modeling or multitask learning.”
Background(2004) · Cited by 9,718 · see citing works
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · cited 3× · see citing works
“Transformers [18–21] are a family of neural networks designed to process ordered sequential data.”
BackgroundDistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653 · cited 1× · see citing works
“n [38] is the model compression technique in which a smaller model – the student network – is trained to reproduce the results of a larger model – the teacher network. It has been shown in literature [39, 40], that it is possible to reach comparable performance on various tasks using distilled neural networks, resulti…”
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347 · see citing works
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893 · cited 1× · see citing works
“ks (RNNs) or their gated variants, including LSTM and GRU, transformers do not require tokens in a sequence to be processed in a specific order, thus allowing more options for training parallelization [25]. Composed of feed-forward layers, convolutions, and self-attention, transformers are easy to quantize and serve in…”
BackgroundHorovod: fast and easy distributed deep learning in TensorFlowPreprint
Sergeev, A., & Balso, M. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.05799 · Cited by 867 · cited 1× · see citing works
“The offline training module of the IntelliCode Compose system is implemented as a Python library integrating PyTorch and Horovod with Adasum algorithm for gradient summation [28].”
MethodCTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751 · cited 1× · see citing works
“Language specific control codes , or prefixes, is an approach introduced in [29, 30] that facilitates constrained language modeling.”
BackgroundLin, C.-Y., & Och, F. J. (2004) · Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics - ACL '04 · doi:10.3115/1218955.1219032 · Cited by 737 · see citing works
Automatically learning semantic features for defect prediction
Wang, S., Liu, T., & Tan, L. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2884781.2884804 · Cited by 720 · cited 1× · see citing works
“Some of the early applications of machine learning of source code include code search [1, 2], bug detection and localization [3], program synthesis [4], code summarization [5] and code completion [6–10].”
BackgroundSWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference
Zellers, R., Bisk, Y., Schwartz, R., & Choi, Y. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1009 · Cited by 712 · cited 1× · see citing works
“Finally, we add a programming language classification task during model pretraining, an approach inspired by natural language inference (NLI) RocStories and SWAG tasks [31, 32].”
MethodA Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories
Mostafazadeh, N., Chambers, N., He, X., Parikh, D., Batra, D., Vanderwende, L., Kohli, P., & Allen, J. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1098 · Cited by 598 · cited 1× · see citing works
“ith monolingual counterparts. 4. Finally, we add a programming language classification task during model pretraining, an approach inspired by natural language inference (NLI) RocStories and SWAG tasks [31, 32]. As such, the model is trained using two optimization objectives: language modeling and multiple choice classi…”
MethodOn the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Dévanbu, P. (2012) · doi:10.5555/2337223.2337322 · Cited by 574 · see citing works
Code completion with statistical language models
Raychev, V., Vechev, M., & Yahav, E. (2014) · Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2594291.2594321 · Cited by 555 · cited 1× · see citing works
“Numerous intelligent code completion systems for both statically and dynamically typed languages have been proposed [6, 11, 41, 42].”
BackgroundGu, X., Zhang, H., & Kim, S. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180167 · Cited by 539 · cited 1× · see citing works
“Some of the early applications of machine learning of source code include code search [1, 2], bug detection and localization [3], program synthesis [4], code summarization [5] and code completion [6–10].”
Backgroundcode2seq: Generating Sequences from Structured Representations of CodePreprint
Alon, U., Brody, S., Levy, O., & Yahav, E. (2018) · arXiv (Cornell University) · doi:10.4230/oasics.slate.2021.14 · Cited by 467 · see citing works
On the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Devanbu, P. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227135 · Cited by 425 · cited 1× · see citing works
“ation to detect programming language. Given N lang one-dimensional monolingual arrays of subtokens, the training dataset for the double-headed GPT-C is extracted according to an algorithm detailed in [7]. As seen, language specific control codes provide necessary supervision to constrain language generation for each sp…”
MethodLearning from examples to improve code completion systems
Bruch, M., Monperrus, M., & Mezini, M. (2009) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/1595696.1595728 · Cited by 383 · cited 2× · see citing works
“A large number of intelligent code completion systems for both statically and dynamically typed languages have been proposed [6, 11, 41, 42].”
BackgroundMethodTying Word Vectors and Word Classifiers: A Loss Framework for Language ModelingPreprint
Inan, H., Khosravi, K., & Socher, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01462 · Cited by 314 · cited 1× · see citing works
“We are reusing the input token embedding matrix as the output classification matrix [26], which allows to remove the large fully connected layer reducing the number of parameters by 25%.”
MethodDistilling Task-Specific Knowledge from BERT into Simple Neural NetworksPreprint
Tang, R., Lu, Y., Liu, L., Mou, L., Olga Vechtomova, A., & Lin, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.12136 · Cited by 310 · see citing works
So, D. R., Liang, C., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1901.11117 · Cited by 277 · cited 1× · see citing works
“zation [23]. Recent neural architecture searches of transformer models have shown that using depth-wise separable convolutions along with self-attention may speed up training without loss of accuracy [24]. A typical transformer architecture for a sequence-to-sequence task will have an encoder (a stack of transformer b…”
BackgroundA statistical semantic language model for source code
Nguyen, T. T., Nguyen, A., Nguyen, H., & Nguyen, T. (2013) · ESEC/FSE 2013 · doi:10.1145/2491411.2491458 · Cited by 231 · cited 1× · see citing works
“Some of the early applications of machine learning of source code include code search [1, 2], bug detection and localization [3], program synthesis [4], code summarization [5] and code completion [6–10].”
BackgroundCode completion with statistical language models
Raychev, V., Vechev, M., & Yahav, E. (2014) · ACM SIGPLAN Notices · doi:10.1145/2666356.2594321 · Cited by 194 · see citing works
Gmail Smart Compose: Real-Time Assisted Writing
Chen, M. X., Lee, B. N., Bansal, G., Cao, Y., Zhang, S., Lu, J., Tsay, J., Wang, Y., Dai, A. M., Chen, Z., et al. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330723 · Cited by 169 · cited 1× · see citing works
“IntelliCode Compose serves as a universal programming language modeling tool, effectively generating syntactically correct code in multiple programming languages, capable of completing an entire line of code in a couple of key strokes, with a user experience inspired by Gmail Smart Compose [15].”
MethodA Survey of Machine Learning for Big Code and Naturalness
Allamanis, M., Barr, E. T., Devanbu, P., & Sutton, C. (2019) · ACM Computing Surveys · doi:10.1145/3212695 · Cited by 150 · see citing works
Pythia: AI-assisted Code Completion System
Svyatkovskiy, A., Zhao, Y., Fu, S., & Sundaresan, N. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330699 · Cited by 125 · cited 2× · see citing works
“There are numerous code completion systems capable of effectively recommending method and API calls [6, 9–11], or finding the correct argument [12–14].”
MethodSelf-citationAutomatic parameter recommendation for practical API usage
Zhang, C., Yang, J., Zhang, Y., Fan, J., Zhang, X., Zhao, J., & Ou, P. (2012) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2012.6227136 · Cited by 105 · cited 1× · see citing works
“There are numerous code completion systems capable of effectively recommending method and API calls [6, 9–11], or finding the correct argument [12–14].”
MethodAutomated API-usage update for Android apps
Fazzini, M., Qi, X., & Orso, A. (2019) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3293882.3330571 · Cited by 85 · cited 1× · see citing works
“There are numerous code completion systems capable of effectively recommending method and API calls [6, 9–11], or finding the correct argument [12–14].”
MethodIntelligent Code Completion with Bayesian Networks
Proksch, S., Lerch, J., & Mezini, M. (2015) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/2744200 · Cited by 81 · cited 1× · see citing works
“A large number of intelligent code completion systems for both statically and dynamically typed languages have been proposed [6, 11, 41, 42].”
BackgroundCSCC: Simple, Efficient, Context Sensitive Code Completion
Asaduzzaman, M., Roy, C. K., Schneider, K. A., & Hou, D. (2014) · 2014 IEEE International Conference on Software Maintenance and Evolution · doi:10.1109/icsme.2014.29 · Cited by 55 · cited 2× · see citing works
“Numerous intelligent code completion systems for both statically and dynamically typed languages have been proposed [6, 11, 41, 42].”
BackgroundMethodLiu, H., Liu, Q., Staicu, C.-A., Pradel, M., & Luo, Y. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2884781.2884841 · Cited by 53 · cited 1× · see citing works
“There are numerous code completion systems capable of effectively recommending method and API calls [6, 9–11], or finding the correct argument [12–14].”
MethodProgram Synthesis and Semantic Parsing with Learned Code IdiomsPreprint
Shin, E. C. R., Allamanis, M., Brockschmidt, M., & Polozov, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.10816 · Cited by 52 · cited 1× · see citing works
“Some of the early applications of machine learning of source code include code search [1, 2], bug detection and localization [3], program synthesis [4], code summarization [5] and code completion [6–10].”
BackgroundDeep API LearningPreprint
Gu, X., Zhang, H., Zhang, D., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08535 · Cited by 3 · cited 1× · see citing works
“Some of the early applications of machine learning of source code include code search [1, 2], bug detection and localization [3], program synthesis [4], code summarization [5] and code completion [6–10].”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · cited 2×
“Several transformer models such as GPT-2, BERT, XLNet, and RoBERTa [19–22] have demonstrated the ability to learn effectively from unlabeled data to perform a wide variety of downstream tasks given supervised discriminative fine-tuning on each specific task.”
Background
Flagged first, then by citations
Cited by
Sorted by citations
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495 · see citing works
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495 · see citing works
“CuBERT employs BERT’s powerful masked language modeling objective to derive generic code-specific representation, and CodeBERT further adds a replaced token detection (Clark et al., 2020) task to learn NL-PL cross-modal representation.”
BackgroundMethodInfluentialCompetition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · see citing works
“Lanza, 2008) and large amounts of existing code data (Aye et al., 2021; Svyatkovskiy et al., 2020).”
BackgroundUniXcoder: Unified Cross-Modal Pre-training for Code Representation
Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., & Yin, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.499 · Cited by 673 · see citing works
Program Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388 · see citing works
“Code completion has been a particular focus of interest [Raychev et al., 2016, Karampatsis et al., 2020, Svyatkovskiy et al., 2020, Kim et al., 2020].”
BackgroundAn Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation
Schäfer, M., Nadi, S., Eghbali, A., & Tip, F. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3334955 · Cited by 431 · see citing works
“Neural techniques are rapidly being adopted for solving various Software Engineering problems, with promising results in several domains including code completion [34], [35], [36], [37], [38], program repair [39], [40], [41], and bug-finding [42], [43].”
MethodStudying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks
Mastropaolo, A., Scalabrino, S., Cooper, N., Palacio, D. N., Poshyvanyk, D., Oliveto, R., & Bavota, G. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00041 · Cited by 271 · see citing works
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Shen, L., Wang, Z., Wang, A., Li, Y., et al. (2023) · Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3580305.3599790 · Cited by 263 · see citing works
“From LISPbased pioneering deductive synthesis approaches [31, 37] tomodern program synthesis systems [23, 30], to end-to-end code generation via deep neural networks [19, 32, 33], tremendous efforts have been made to enable machines to automatically write correct programs as part of the quest to artificial general int…”
BackgroundAn Empirical Study of the Non-Determinism of ChatGPT in Code Generation
Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3697010 · Cited by 258 · see citing works
“As mentioned in Section 2, we especially focus on the code generated with Python3 language, since it is one of the most widely studied programming languages in code generation [4, 12, 17, 36, 59, 61, 64].”
BackgroundA Survey on Large Language Models for Code Generation
Jiang, J., Wang, F., Shen, J., Kim, S., & Kim, S. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3747588 · Cited by 251 · see citing works
Productivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Simister, S., Sittampalam, G., Li, A., Rice, A., Rifkin, D., & Aftandilian, E. (2022) · arXiv · doi:10.48550/arxiv.2205.06537 · Cited by 200 · see citing works
“The IntelliCode Compose system uses the term CTR (Click Through Rate) for this and reports a value of 10% in online trials [12].”
MethodAutomating code review activities by large-scale pre-training
Li, Z., Lu, S., Guo, D., Duan, N., Jannu, S., Jenks, G., Majumder, D., Green, J., Svyatkovskiy, A., Fu, S., et al. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549081 · Cited by 226 · see citing works
“CodeGPT [27], GPTC [36] and Codex [7] focus on generative tasks like code completion and code generation because they are pre-trained with decoderonly transformer architecture.”
BackgroundSelf-citationProductivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534864 · Cited by 200 · see citing works
“The IntelliCode Compose system uses the term CTR (Click Through Rate) for this and reports a value of 10% in online trials [12].”
MethodCode Prediction by Feeding Trees to Transformers
Kim, S., Zhao, J., Tian, Y., & Chandra, S. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00026 · Cited by 209 · see citing works
“Recently a team from Microsoft also published their ongoing efforts [62] on applying Transformers for code autocomplete.”
Natural Attack for Pre-trained Models of Code
Yang, Z., Shi, J., He, J., & Lo, D. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510146 · Cited by 166 · see citing works
“[43] introduce GPT-C, a variant of GPT-2 [37] trained on a large corpus containing multiple programming lan-”
MethodWu, H., Zhang, Z., Wang, S., Yan, L., Lin, B., Qin, Y., Zhang, H., & Mao, X. (2021) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre52982.2021.00047 · Cited by 163 · see citing works
“Svyatkovskiy et al. [35] propose GPT-C, which is a variant of the GPT-2 trained from scratch on source code data to support generative tasks like code completion.”
BackgroundMethodRepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation
Zhang, F., Chen, B., Zhang, Y., Keung, J., Liu, J., Zan, D., Mao, Y., Lou, J.-G., & Chen, W. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.151 · Cited by 173 · see citing works
No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence
Wang, C., Yang, Y., Gao, C., Peng, Y. F., Zhang, H., & Lyu, M. R.-T. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549113 · Cited by 153 · see citing works
On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot
Mastropaolo, A., Pascarella, L., Guglielmi, E., Ciniselli, M., Scalabrino, S., Oliveto, R., & Bavota, G. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00181 · Cited by 149 · see citing works
“…targeted the relatively simpler problem of code completion, that has been tackled exploiting historical information [50], coding patterns mined from software repositories [21], [42], [56], [10], [41], [45], [19] and, more recently, Deep Learning (DL) models [62], [27], [29], [8], [53], [15].”
Towards Automating Code Review Activities
Tufano, R., Pascarella, L., Tufano, M., Poshyvanyk, D., & Bavota, G. (2021) · International Conference on Software Engineering · doi:10.1109/icse43902.2021.00027 · Cited by 230 · see citing works
“[43] used Transformer models to support code completion.”
MethodExploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study
Guo, Q., Cao, J., Xie, X., Liu, S., Li, X., Chen, B., & Peng, X. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3623306 · Cited by 140 · see citing works
“Decoder-only models allow the model to access only the tokens preceding the current token during the training process [22, 39].”
BackgroundNatural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review
Wong, M.-F., Guo, S., Hang, C. N., Ho, S., & Tan, C. W. (2023) · Entropy · doi:10.3390/e25060888 · Cited by 137 · see citing works
“Intellicode [140] 2020 Code Generation, Code Completion GPT-2 Python, C#, JavaScript, and TypeScrip 7”
Is GitHub’s Copilot as bad as humans at introducing vulnerabilities in code?
Asare, O., Nagappan, M., & Asokan, N. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10380-1 · Cited by 128 · see citing works
“Similarly, IntelliCode uses a generative transformer model (GPT-C) which is a variant of GPT-2 (Svyatkovskiy et al. 2020).”
BackgroundMethodInfluentialNo Need to Lift a Finger Anymore? Assessing the Quality of Code Generation by ChatGPT
Liu, Z., Tang, Y., Luo, X., Zhou, Y., & Zhang, L. F. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3392499 · Cited by 128 · see citing works
“Lots of studies on code generation leverage AI-based approaches [1], [2], [3], [4], [5], especially for using large language models (LLMs) [6], [7], [8], [9], [10], [11] such as the recent ChatGPT [12].”
BackgroundDinella, E., Ryan, G., Mytkowicz, T., & Lahiri, S. K. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510141 · Cited by 119 · see citing works
“While not originally designed for code, transformers have found many applications in software engineering [4, 15, 26, 27].”
BackgroundMeasuring GitHub Copilot's Impact on Productivity
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2024) · Communications of the ACM · doi:10.1145/3633453 · Cited by 124 · see citing works
“20 An alternative measure is that of daily completions accepted per user (DCPU) for which a value of around 20 has been report-ed.”
BackgroundInvestigating Code Generation Performance of ChatGPT with Crowdsourcing Social Data
Feng, Y., Vanam, S., Cherukupally, M., Zheng, W., Qiu, M., & Chen, H. (2023) · 2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC) · doi:10.1109/compsac57700.2023.00117 · Cited by 117 · see citing works
“As programming generation and assistant tools, such as CodeBERT [8] and IntelliCode Compose [9], become more widely used, there has been an increased focus on investigating the usability and interactions between users and code generation tools [3], [4], [10], [11].”
Large Language Models Meet NL2Code: A Survey
Zan, D.-G., Chen, B., Zhang, F., Lu, D., Wu, B., Guan, B., Wang, Y.-J., & Lou, J.-G. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2023.acl-long.411 · Cited by 293 · see citing works
“Early works, such as GPT-C (Svyatkovskiy et al., 2020), PyMT5 (Clement et al.”
BackgroundAn Empirical Study on Fine-Tuning Large Language Models of Code for Automated Program Repair
Huang, K., Meng, X., Zhang, J., Liu, Y., Wang, W., Li, S., & Zhang, Y. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00181 · Cited by 113 · see citing works
“Typical models are CodeGPT [34], GPT-C [35], Codex [36] etc. 3) Encoder-Decoder LLMCs retain the infrastructure of Trans-former with both encoder and decoder, such as CodeT5 [37] and PLBART [38].”
MethodNiu, C., Li, C., Ng, V., Ge, J., Huang, L., & Luo, B. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510096 · Cited by 125 · see citing works
“Second, these models have largely assumed as inputs the source code [8, 27, 28, 37, 51] and the associated natural language description [16, 39].”
MethodDeep learning in electron microscopy
Ede, J. M. (2020) · Machine Learning Science and Technology · doi:10.1088/2632-2153/abd614 · Cited by 105 · see citing works
Perfection Not Required? Human-AI Partnerships in Code Translation
Weisz, J. D., Muller, M., Houde, S., Richards, J., Ross, S. I., Martinez, F., Agarwal, M., & Talamadupula, K. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450656 · Cited by 129 · see citing works
“In recent years, several efforts have focused on the use of AI and machine learning techniques for various tasks related to software engineering, including code completion [15, 41, 75, 84], code classification [49, 68], API recommendation [16, 33], variable and method naming [3, 5], type inference [39, 93], bug detect…”
BackgroundSiddiq, M. L., & Santos, J. C. S. (2022) · Proceedings of the 1st International Workshop on Mining Software Repositories Applications for Privacy and Security · doi:10.1145/3549035.3561184 · Cited by 101 · see citing works
“Later, they are fine-tuned for specialized source code related tasks such as automated code summarization [10], completion [14, 15, 29], generation [27, 28] and documentation creation [4].”
BackgroundChatGPT vs SBST: A Comparative Assessment of Unit Test Suite Generation
Tang, Y., Liu, Z., Zhou, Z., & Luo, X. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3382365 · Cited by 109 · see citing works
Bui, N. D. Q., Yu, Y., & Jiang, L. (2021) · Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3404835.3462840 · Cited by 103 · see citing works
“Deep Learning Models of Code : There has been a huge interest in applying deep learning techniques for software engineering tasks such as program functionality classification [10, 11, 55, 75], bug localization [18, 29, 37, 58], code summarization [2, 24, 66], code clone detection [13, 75], program refactoring [32], pr…”
An Empirical Study on the Usage of Transformer Models for Code Completion
Ciniselli, M., Cooper, N., Pascarella, L., Mastropaolo, A., Aghajani, E., Poshyvanyk, D., Penta, M. D., & Bavota, G. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3128234 · Cited by 101 · see citing works
“Last, but not least, Deep Learning (DL) models have been applied to code completion [8], [22], [47], [49], [73], [84], setting new standards in terms of prediction performance.”
Assessing Generalizability of CodeBERT
Zhou, X., Han, D., & Lo, D. (2021) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme52107.2021.00044 · Cited by 93 · see citing works
Machine/Deep Learning for Software Engineering: A Systematic Literature Review
Wang, S., Huang, L., Gao, A., Ge, J., Zhang, T., Feng, H., Satyarth, I., Li, M., Zhang, H., & Ng, V. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3173346 · Cited by 111 · see citing works
CodeFill: multi-token code completion by jointly learning from structure and naming sequences
Izadi, M., Gismondi, R., & Gousios, G. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510172 · Cited by 86 · see citing works
“We pit CodeFill against four baseline models and two the-stateof-the-art models, namely GPT-C [49] and TravTrans+ [25].”
BackgroundMethodInfluentialAn Empirical Study on the Usage of BERT Models for Code Completion
Ciniselli, M., Cooper, N., Pascarella, L., Poshyvanyk, D., Penta, M. D., & Bavota, G. (2021) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr52588.2021.00024 · Cited by 93 · see citing works
Tang, W., Tang, M., Ban, M., Zhao, Z., & Feng, M. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111623 · Cited by 89 · see citing works
Generative AI in mobile networks: a survey
Karapantelakis, A., Alizadeh, P., Alabbasi, A., Dey, K., & Nikou, A. (2023) · Annals of Telecommunications · doi:10.1007/s12243-023-00980-9 · Cited by 85 · see citing works
“One part of transformers works with code generation and code completion [107, 108].”
DB-GPT: Large Language Model Meets Database
Zhou, X., Sun, Z., & Li, G. (2024) · Data Science and Engineering · doi:10.1007/s41019-023-00235-6 · Cited by 83 · see citing works
“Recently LLMs have demonstrated superiority in various tasks like text generation [2], machine translation [3], program synthesis [4].”
Code Completion by Modeling Flattened Abstract Syntax Trees as Graphs
Wang, Y., & Li, H. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17650 · Cited by 81 · see citing works
An Empirical Comparison of Pre-Trained Models of Source Code
Niu, C., Li, C., Ng, V., Chen, D., Ge, J., & Luo, B. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00180 · Cited by 90 · see citing works
“Second, TF-based CoTexT, which uses MLM as the only pre-traning task, outperforms TE-based UniXcoder, which uses three more complex pre-training tasks, ULM, MCL and CMG. Finally, TF-based DeepDebug achieves better results than TE-based C-BERT when using only code as input and MLM as its only pre-training task.”
BackgroundInfluentialWhat do pre-trained code models know about code?
Karmakar, A., & Robbes, R. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678927 · Cited by 80 · see citing works
“Tabnine, IntelliCode [27], TransCoder [21], and more recently Github Copilot [11]—it becomes imperative for us to be aware of their capabilities and drawbacks.”
BackgroundSynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code RepresentationPreprint
Wang, X. E., Wang, Y., Mi, F., Zhou, P., Wan, Y., Xiao, L., Li, L., Wu, H., Liu, J., & Jiang, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.04556 · Cited by 118 · see citing works
Predicting the objective and priority of issue reports in software repositories
Izadi, M., Akbari, K., & Heydarnoori, A. (2022) · Empirical Software Engineering · doi:10.1007/s10664-021-10085-3 · Cited by 77 · see citing works
“Text normalization has been used before in preprocessing data for Machine Learning models [51,28].”
MethodUsing Transfer Learning for Code-Related Tasks
Mastropaolo, A., Cooper, N., Palacio, D. N., Scalabrino, S., Poshyvanyk, D., Oliveto, R., & Bavota, G. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3183297 · Cited by 74 · see citing works
“code summarization [24], [32], [39], [42], code reviews [77], [78], pseudo-code generation [55], code deobfuscation [31], [79], injection of code mutants [76], generation of assert statements [82], clone detection [74], [83], traceability [49] and code completion [5], [11], [17], [17], [34], [35], [70], [84].”
BackgroundRefactoring Programs Using Large Language Models with Few-Shot Examples
Shirafuji, A., Oda, Y., Suzuki, J., Morishita, M., & Watanobe, Y. (2023) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec60848.2023.00025 · Cited by 77 · see citing works
“In recent years, large language models (LLMs) trained on texts in programming languages as well as natural languages [2]–[7] have shown the potential to support programming in both programming education [8]–[11] and software development [12]–[14].”
BackgroundReentrancy Vulnerability Detection and Localization: A Deep Learning Based Two-phase Approach
Zhang, Z., Lei, Y., Yan, M. L., Yu, Y., Chen, J., Wang, S., & Mao, X. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3560428 · Cited by 69 · see citing works
“There are widely used models such as GPT [52] and BERT [17] for NLP and Scelmo [38], CodeBert [24], C-Bert [8] and GPT-C [57] for programming languages. users value value users sum users i usersBet sender It is trained on the CodeSearchNet dataset[33], which includes 2.3 millions functions of six programming languages…”
BackgroundMethodInfluentialContraBERT: Enhancing Code Pre-trained Models via Contrastive Learning
Liu, S., Wu, B., Xie, X., Meng, G., & Liu, Y. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00207 · Cited by 77 · see citing works
“To address this limitation, some researchers propose to utilize unsupervised techniques with the massive amount of data to pre-train a general model [28], [29], [30], [3], [31], [32], [33], [34], [35] and then fine-tune it for different downstream tasks.”
MethodEfficient Federated Learning for Modern NLP
Cai, D., Wu, Y., Wang, S., Lin, F. X., & Xu, M. (2023) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/3570361.3592505 · Cited by 69 · see citing works
“Examples include sentiment analysis, QA, and auto completion [42, 68, 76, 77, 92].”
BackgroundAn Empirical Study of Code Smells in Transformer-based Code Generation Techniques
Siddiq, M. L., Majumder, S. H., Mim, M. R., Jajodia, S., & Santos, J. C. S. (2022) · 2022 IEEE 22nd International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam55253.2022.00014 · Cited by 70 · see citing works
“These Large Language Learning Models (LLMs) are trained with a large dataset of source code snippets and fine-tuned for a wide range of software engineering applications, such as automated code completion [3]–[5], summarization [6], documentation [7], and generation [8], [9].”
BackgroundThongtanunam, P., Pornprasit, C., & Tantithamthavorn, C. K. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510067 · Cited by 59 · see citing works
“…[35], query reformulation [19]); (2) Text → Code (e.g., code search [24, 42]); (3) Code → Text (e.g., code summarization [25], commit message generation [29, 34]); and (4) Code → Code (e.g., automated program repair [20, 28, 33], programming language translation [48], code completion [54]).”
BackgroundLearning Deep Semantics for Test Completion
Nie, P., Banerjee, R., Li, J. J., Mooney, R. J., & Gligoric, M. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00178 · Cited by 65 · see citing works
“Despite being closely related to code completion [25]–[29], test completion is distinct in that test code has several unique”
BackgroundMethodResultInfluentialLanguage Models for Code Completion: A Practical Evaluation
Izadi, M., Katzy, J., van Dam, T., Otten, M., Popescu, R. M., & Arie Van Deursen, A. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639138 · Cited by 69 · see citing works
A Survey on Machine Learning Techniques for Source Code AnalysisPreprint
Sharma, T., Kechagia, M., Georgiou, S., Tiwari, R. K., Vats, I., Moazen, H., & Sarro, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.09610 · Cited by 56 · see citing works
“[306] presented IntelliCode that uses a Transformer model that predicts sequences of code tokens of arbitrary types, and generates entire lines of syntactically correct code.”
BackgroundMethodThe Current Challenges of Software Engineering in the Era of Large Language Models
Gao, C., Hu, X., Gao, S., Xia, X., & Jin, Z. G. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3712005 · Cited by 65 · see citing works
“Their approach reduces catastrophic forgetting in LLM and achieves superior performance in cross-lingual and low-resource scenarios.”
A survey of strategy-driven evasion methods for PE malware: Transformation, concealment, and attack
Geng, J., Wang, J., Fang, Z., Zhou, Y., Wu, D. J., & Ge, W. (2023) · Computers & Security · doi:10.1016/j.cose.2023.103595 · Cited by 54 · see citing works
Generative AI for Requirements Engineering: A Systematic Literature Review
Cheng, H., Husen, J. H., Lu, Y., Racharak, T., Yoshioka, N., Ubayashi, N., & Washizaki, H. (2025) · Software Practice and Experience · doi:10.1002/spe.70029 · Cited by 95 · see citing works
“The integration of artificial intelligence (AI) into SE has transformed traditional practices, particularly tasks such as code generation, defect prediction, and software testing, which have substantiallyimprovedefficiencyandqualityacrossdevelopment processes ([5–7]).”
Chain-of-Thought in Neural Code Generation: From and for Lightweight Language Models
Yang, G., Zhou, Y., Chen, X., Zhang, X., Zhuo, T. Y., & Chen, T. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3440503 · Cited by 56 · see citing works
“N EURAL code generation, which can automatically generate programs from natural language requirements based on deep learning, has become a promising approach to meet the challenges of the ever-increasing complexity of software and alleviate the burden on programmers [1], [2].”
A survey on large language models for software engineering
Zhang, Q., Fang, C., Xie, Y., Zhang, Y., Yu, S., Sun, W., Yang, Y., & Chen, Z. (2026) · Science China Information Sciences · doi:10.1007/s11432-025-4670-0 · Cited by 124 · see citing works
Hadi, M. A., & Fard, F. H. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10314-x · Cited by 48 · see citing works
“2022) or for tasks related to programming languages such as comment generation (Svyatkovskiy et al. 2020), transfer learning to low resource languages (Chen et al.”
BackgroundOn Multi-Modal Learning of Editing Source Code
Chakraborty, S., & Ray, B. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678559 · Cited by 50 · see citing works
“Transformers’ ability of reasoning about long-range dependency is proved useful for several source code processing task including code completions [22], code generation [23], code summarization [21].”
Compressing Pre-trained Models of Code into 3 MB
Shi, J., Yang, Z., Xu, B., Kang, H. J., & Lo, D. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556964 · Cited by 47 · see citing works
“, CodeT5 [43], GPT-C [39], and CodeGPT [28].”
BackgroundInfluentialCompilable Neural Code Generation with Compiler Feedback
Wang, X., Wang, Y., Wan, Y., Mi, F., Li, Y., Zhou, P., Liu, J., Wu, H., Jiang, X., & Liu, Q. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.2 · Cited by 56 · see citing works
One Adapter for All Programming Languages? Adapter Tuning for Code Search and Summarization
Wang, D., Chen, B., Li, S., Luo, W., Peng, S., Dong, W., & Liao, X. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00013 · Cited by 47 · see citing works
Vulnerabilities in AI Code Generators: Exploring Targeted Data Poisoning Attacks
Cotroneo, D., Improta, C., Liguori, P., & Natella, R. (2024) · Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension · doi:10.1145/3643916.3644416 · Cited by 45 · see citing works
“To assess code correctness, we adopt the Edit Distance (ED) , a metric widely used in the field to compare the similarity of the code generated by models with respect to a ground-truth implementation used as a reference for the evaluation [10, 46, 47].”
MethodGenerating bug-fixes using pretrained transformers
Drain, D., Wu, C. H., Svyatkovskiy, A., & Sundaresan, N. (2021) · MAPS@PLDI · doi:10.1145/3460945.3464951 · Cited by 49 · see citing works
“For instance, GPT-C attains perplexities in the range of 1.3 – 2 for several programming languages, whereas a comparably sized GPT-2 model attains a perplexity of 26 on Wikitext-3 [16, 17].”
BackgroundInfluentialSelf-citationPrompt Tuning in Code Intelligence: An Experimental Evaluation
Wang, C., Yang, Y., Gao, C., Peng, Y., Zhang, H., & Lyu, M. R.-T. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3313881 · Cited by 44 · see citing works
“Svyatkovskiy et al. [61] and Kim et al. [62] train GPT-2 [63] on large scale programming languages for solving code completion task.”
BackgroundExtending the Frontier of ChatGPT: Code Generation and Debugging
Sakib, F. A., Khan, S. H., & Karim, A. (2024) · International Conference on Electrical, Computer and Energy Technologies (ICECET) · doi:10.1109/icecet61485.2024.10698405 · Cited by 63 · see citing works
“Artificial intelligence has achieved remarkable prowess across a plethora of dimensions, encompassing code generation [23]– [27], program explanation [28]–[31], error correction [32]– [35], and more.”
BackgroundOn the transferability of pre-trained language models for low-resource programming languages
Chen, F., Fard, F. H., Lo, D., & Bryksin, T. (2022) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3524610.3527917 · Cited by 45 · see citing works
“IntelliCode Compose is a GPT based model that is trained on Python, C#, JavaScript, and TypeScript for code completion [41].”
Towards Efficient Fine-Tuning of Pre-trained Code Models: An Experimental Study and Beyond
Shi, E., Wang, Y., Zhang, H., Du, L., Han, S., Zhang, D., & Sun, H. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598036 · Cited by 50 · see citing works
“Similar to previous work [34], the performance is measured by EM and Levenshtein edit similarity (Edit sim) [50].”
MethodA survey on machine learning techniques applied to source code
Sharma, T., Kechagia, M., Georgiou, S., Tiwari, R., Vats, I., Moazen, H., & Sarro, F. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111934 · Cited by 130 · see citing works
Code Search is All You Need? Improving Code Suggestions with Code Search
Chen, J., Hu, X., Li, Z., Gao, C., Xia, X., & Lo, D. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639085 · Cited by 43 · see citing works
Shi, J., Yang, Z., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708525 · Cited by 43 · see citing works
“Numerous decoder-only LLMs tailored for code generation have been developed, including CodeGPT [84], GPT-C [117], Codex [15], PolyCoder [143], CodeGen [87, 88], Code Llama [99], StarCoder [73, 82], and Magicoder [137].”
CrystalBLEU: Precisely and Efficiently Measuring the Similarity of Code
Eghbali, A., & Pradel, M. (2022) · Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3551349.3556903 · Cited by 42 · see citing works
“Code prediction techniques include code completion [4, 22, 23, 26, 46], code translation between programming languages [32, 33, 43], automated program repair [11, 12, 16, 24, 49], predicting code from natural language descriptions of the desired functionality [15, 21, 27, 44, 47], injecting bugs and vulnerabilities in…”
BackgroundAljedaani, W., Habib, A., Aljohani, A., Eler, M., & Feng, Y. (2024) · Proceedings of the 21st International Web for All Conference · doi:10.1145/3677846.3677854 · Cited by 42 · see citing works
Ivie: Lightweight Anchored Explanations of Just-Generated Code
Yan, L., Hwang, A., Wu, Z., & Head, A. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642239 · Cited by 41 · see citing works
“These tools, often called “programming assistants,” include GitHub Copilot [22], OpenAI GPT-4 [40], Amazon CodeWhisperer [2], IntelliCode Compose [56], and CodeT5+ [64].”
MethodTeaching Code LLMs to Use Autocompletion Tools in Repository-Level Code Generation
Wang, C., Zhang, J., Feng, Y., Li, T., Sun, W., Liu, Y., & Peng, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3714462 · Cited by 40 · see citing works
“— Edit Similarity (EditSim) [41].”
Making the most of small Software Engineering datasets with modern machine learning
Prenner, J. A., & Robbes, R. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3135465 · Cited by 40 · see citing works
“[60] introduce IntelliCode compose, a system for intelligent code completion based on the GPT-2 Transformer language model.”
MethodPyART: Python API Recommendation in Real-Time
He, X., Xu, L., Zhang, X., Hao, R., Feng, Y., & Xu, B. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00145 · Cited by 40 · see citing works
“API recommendation approaches based on Machine Learning tend to have better performance for Python than most traditional methods [21]–[24].”
MethodDeep Learning Meets Software Engineering: A Survey on Pre-Trained Models of Source Code
Niu, C., Li, C., Luo, B., & Ng, V. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/775 · Cited by 53 · see citing works
Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacities
Ma, W., Liu, S., Zhao, M., Xie, X., Wang, W., Hu, Q., Zhang, J., & Liu, Y. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3664606 · Cited by 43 · see citing works
“Many code models [2, 4, 13, 17, 29, 31, 41, 42, 63, 71] have been proposed to greatly advance the development of code intelligence.”
BackgroundDomain Adaptive Code Completion via Language Models and Decoupled Domain Databases
Tang, Z. J., Ge, J., Liu, S., Zhu, T., Xu, T., Huang, L., & Luo, B. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00076 · Cited by 40 · see citing works
“For example, CodeFill [29] improves GPT-2 [25] by predicting the code type when completing the next token, while Gramm-former [37] generates code completions with “holes” inserted in places where the model is uncertain.”
BackgroundTransformers in health: a systematic review on architectures for longitudinal data analysis
Siebra, C., Kurpicz-Briki, M., & Wac, K. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-023-10677-z · Cited by 35 · see citing works
“Similarly, programming language code generation using transformers (Svyatkovskiy et al. 2020) employs the tokens of the programming language as vocabulary.”
How People Prompt Generative AI to Create Interactive VR Scenes
Manesh, S. A., Zhang, T., Onishi, Y., Hara, K., Bateman, S., Li, J., & Tang, A. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3661547 · Cited by 36 · see citing works
“The code generation ability of current LLMs further powers programming agents that synthesize computer programs supporting complex logic and user interactions from language input (e.g., [53, 66]).”
BackgroundUnveiling ChatGPT’s usage in open source projects: A mining-based study
Tufano, R., Mastropaolo, A., Pepe, F., Dabic, O., Di Penta, M., & Bavota, G. (2024) · Proceedings of the 21st International Conference on Mining Software Repositories · doi:10.1145/3643991.3644918 · Cited by 38 · see citing works
“…the aforementioned problems, but also to perform tasks for which no recommender was previously thought in the past, including generating entire code blocks [56, 103], automatically reviewing source code [76, 104, 107, 108], or generating scenarios for automatically reproducing issues [50, 66, 116].”
BackgroundAgents in software engineering: survey, landscape, and vision
Wang, Y., Zhong, W., Huang, Y., Shi, E., Yang, M., Chen, J., Li, H., Ma, Y., Wang, Q., & Zheng, Z. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00544-2 · Cited by 50 · see citing works
Program merge conflict resolution via neural transformers
Svyatkovskiy, A., Fakhoury, S., Ghorbani, N., Mytkowicz, T., Dinella, E., Bird, C., Jang, J., Sundaresan, N., & Lahiri, S. K. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549163 · Cited by 38 · see citing works
“Neural language models (LMs) have shown great performance in natural language generation [36, 38], and have been successfully applied to the domain of source code [17, 24, 48].”
BackgroundSelf-citationFine-Tuning Large Language Models to Improve Accuracy and Comprehensibility of Automated Code Review
Yu, Y., Rong, G., Shen, H., Zhang, H., Shao, D., Wang, M., Wei, Z., Xu, Y., & Wang, J. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695993 · Cited by 38 · see citing works
“The activities of ACR primarily encompass reviewer recommendation [46, 61], code issue prediction [27, 52], and review comment recommendation [44] and generation [27].”
Pu, K., Lazaro, D., Arawjo, I., Xia, H., Xiao, Z., Grossman, T., & Chen, Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713357 · Cited by 32 · see citing works
“Tools like Github Copilot [4] and Visual Studio’s IntelliCode [83] offer the auto-completion feature, which fills the code line as the user is typing, to alleviate prompt engineering efforts and proactively provide in-situ support.”
Large Language Model-Aware In-Context Learning for Code Generation
Li, J., Li, G., Tao, C., Li, J., Zhang, H., Liu, F., & Jin, Z. (2023) · arXiv · doi:10.48550/arxiv.2310.09748 · Cited by 32 · see citing works
Pre-training by Predicting Program Dependencies for Vulnerability Analysis Tasks
Liu, Z., Tang, Z., Zhang, J., Xia, X., & Yang, X. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639142 · Cited by 31 · see citing works
“Existing pre-trained code models can generally be divided into encoder-only [18, 20, 26, 31, 33, 62], decoder-only [40, 45, 57] and encoder-decoder models [7, 13, 25, 32, 42, 48, 49, 63, 65].”
BackgroundMethodMulti-target Backdoor Attacks for Code Pre-trained Models
Li, Y., Liu, S., Chen, K., Xie, X., Zhang, T., & Liu, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.399 · Cited by 37 · see citing works
How could Neural Networks understand Programs?Preprint
Peng, D., Zheng, S., Li, Y., Ke, G., He, D., & Liu, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.04297 · Cited by 34 · see citing works
“Svyatkovskiy et al. (2020) propose to train a variant of GPT-2 (Radford et al., 2019) from scratch on source code to improve the performance of code completion.”
BackgroundEfficient transformer with code token learner for code clone detection
Zhang, A., Fang, L., Ge, C., Li, P., & Liu, Z. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111557 · Cited by 30 · see citing works
CodeRetriever: A Large Scale Contrastive Pre-Training Method for Code Search
Li, X., Gong, Y., Shen, Y., Qiu, X., Zhang, H., Yao, B., Qi, W., Jiang, D., Chen, W., & Duan, N. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.187 · Cited by 31 · see citing works
- Authors
- Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N.
- Affiliations
- Microsoft (United States) (US)
- Microsoft, USA
- ORCID
- Sundaresan, Neel
- Venue
- Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2020-11-08
- Pages
- 1433-1443
- OpenAlex
- W3025993830
- Cited by
- 474
- References
- 38
- Citations by year
- 2026: 42 · 2025: 97 · 2024: 117 · 2023: 103 · 2022: 52 · 2021: 56 · 2020: 7
- FWCI
- 41.33
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- code generation, code completion, transformer models, integrated development environment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
