The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
IntelliCode compose: code generation using transformer
Open accessAbstract
In software development through integrated development environments (IDEs), code completion is one of the most widely used features. Nevertheless, majority of integrated development environments only support completion of methods and APIs, or arguments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Several methods have been proposed to incorporate repository context into the training of language models, including n-grams (Tu et al., 2014), LSTMs (Hellendoorn and Devanbu, 2017), and Transformers (Svyatkovskiy et al., 2020;Liu et al., 2022;Ding et al., 2022).”
From RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation (2023) · cites this paper as Svyatkovskiy et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The IntelliCode Compose system uses the term CTR (Click Through Rate) for this and reports a value of 10% in online trials [11].”
From Productivity Assessment of Neural Code Completion (2022) · cites this paper as [11]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“act syntax tree (AST) or concrete syntax tree (CST), and dataflow graph. A transformer block will typically consist of a multi-head self-attention, followed by a two-layer multi-layer perceptron (MLP) [17], optionally containing residual connections and layer normalization [23]. Recent neural architecture searches of t…”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 2×
“Several transformer models such as GPT-2, BERT, XLNet, and RoBERTa [19–22] have demonstrated the ability to learn effectively from unlabeled data to perform a wide variety of downstream tasks given supervised discriminative fine-tuning on each specific task.”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“Several transformer models such as GPT-2, BERT, XLNet, and RoBERTa [19–22] have demonstrated the ability to learn effectively from unlabeled data to perform a wide variety of downstream tasks given supervised discriminative fine-tuning on each specific task.”
BackgroundRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 2×
“We refer the interested reader to the numerous publications about transformer models [16, 20–22], and focus on code completion for the remainder of this section.”
BackgroundDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“lue and are most likely lower than the actual user sentiment. Over 150K requests, we have seen a surfacing rate of 9.2% and an acceptance rate of 10%. 11 Knowledge Distillation Knowledge distillation [38] is the model compression technique in which a smaller model – the student network – is trained to reproduce the re…”
InfluentialExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“(3) Language specific control codes, is an approach introduced in [29, 30] that uses target prefixes to facilitate constrained language modeling or multitask learning.”
Background(2004) · Cited by 9,718
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · cited 3×
“Transformers [18–21] are a family of neural networks designed to process ordered sequential data.”
BackgroundDistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653 · cited 1×
“n [38] is the model compression technique in which a smaller model – the student network – is trained to reproduce the results of a larger model – the teacher network. It has been shown in literature [39, 40], that it is possible to reach comparable performance on various tasks using distilled neural networks, resulti…”
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347
Research works printed but not linked: 7
- Alexey Svyatkovskiy , Sebastian Lee , Anna Hadjitofi , Maik Riechert , Juliana Franco , and Miltiadis Allamanis . Fast and memory-eficient neural code completion , 2020 . Alexey Svyatkovskiy, Sebastian Lee, Anna Hadjitofi, Maik Riechert, Juliana Franco, and Miltiadis Allamanis. Fast and memory-eficient neural code completion, 2020.
- Alec Radford , Jef Wu , Rewon Child , David Luan , Dario Amodei , and Ilya Sutskever . Language models are unsupervised multitask learners . 2019 . Alec Radford, Jef Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
- Microsoft Corporation . Ai-assisted development. https://marketplace. visualstudio.com/items?itemName=VisualStudioExptTeam.VSIntelliCode. Visited Jan 2020 . Microsoft Corporation. Ai-assisted development. https://marketplace. visualstudio.com/items?itemName=VisualStudioExptTeam.VSIntelliCode. Visited Jan 2020.
- Alec Radford . Improving language understanding by generative pre-training . 2018 . Alec Radford. Improving language understanding by generative pre-training. 2018.
- Jimmy Lei Ba , Jamie Ryan Kiros, and Geofrey E. Hinton. Layer normalization , 2016 . Jimmy Lei Ba, Jamie Ryan Kiros, and Geofrey E. Hinton. Layer normalization, 2016.
- Eclipse Foundation . Code Recommenders . www.eclipse.org/recommenders. Visited June 2017 . Eclipse Foundation. Code Recommenders. www.eclipse.org/recommenders. Visited June 2017.
- Eclipse Recommenders . Eclipse SnipMatch . http://www.eclipse.org/ recommenders/manual/#snipmatch , 2014 . Visited Jun 2017. Eclipse Recommenders. Eclipse SnipMatch. http://www.eclipse.org/ recommenders/manual/#snipmatch, 2014. Visited Jun 2017.
Cited by
Sorted by citations
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495
“CuBERT employs BERT’s powerful masked language modeling objective to derive generic code-specific representation, and CodeBERT further adds a replaced token detection (Clark et al., 2020) task to learn NL-PL cross-modal representation.”
BackgroundMethodInfluentialCompetition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043
“Lanza, 2008) and large amounts of existing code data (Aye et al., 2021; Svyatkovskiy et al., 2020).”
BackgroundUniXcoder: Unified Cross-Modal Pre-training for Code Representation
Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., & Yin, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.499 · Cited by 673
Program Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388
“Code completion has been a particular focus of interest [Raychev et al., 2016, Karampatsis et al., 2020, Svyatkovskiy et al., 2020, Kim et al., 2020].”
BackgroundAn Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation
Schäfer, M., Nadi, S., Eghbali, A., & Tip, F. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3334955 · Cited by 431
“Neural techniques are rapidly being adopted for solving various Software Engineering problems, with promising results in several domains including code completion [34], [35], [36], [37], [38], program repair [39], [40], [41], and bug-finding [42], [43].”
MethodStudying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks
Mastropaolo, A., Scalabrino, S., Cooper, N., Palacio, D. N., Poshyvanyk, D., Oliveto, R., & Bavota, G. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00041 · Cited by 302
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Shen, L., Wang, Z., Wang, A., Li, Y., et al. (2023) · Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3580305.3599790 · Cited by 563
“From LISPbased pioneering deductive synthesis approaches [31, 37] tomodern program synthesis systems [23, 30], to end-to-end code generation via deep neural networks [19, 32, 33], tremendous efforts have been made to enable machines to automatically write correct programs as part of the quest to artificial general int…”
BackgroundAn Empirical Study of the Non-Determinism of ChatGPT in Code Generation
Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3697010 · Cited by 258
“As mentioned in Section 2, we especially focus on the code generated with Python3 language, since it is one of the most widely studied programming languages in code generation [4, 12, 17, 36, 59, 61, 64].”
BackgroundA Survey on Large Language Models for Code Generation
Jiang, J., Wang, F., Shen, J., Kim, S., & Kim, S. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3747588 · Cited by 251
Productivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Simister, S., Sittampalam, G., Li, A., Rice, A., Rifkin, D., & Aftandilian, E. (2022) · arXiv · doi:10.48550/arxiv.2205.06537 · Cited by 200
“The IntelliCode Compose system uses the term CTR (Click Through Rate) for this and reports a value of 10% in online trials [12].”
MethodAutomating code review activities by large-scale pre-training
Li, Z., Lu, S., Guo, D., Duan, N., Jannu, S., Jenks, G., Majumder, D., Green, J., Svyatkovskiy, A., Fu, S., et al. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549081 · Cited by 301
“CodeGPT [27], GPTC [36] and Codex [7] focus on generative tasks like code completion and code generation because they are pre-trained with decoderonly transformer architecture.”
BackgroundSelf-citationProductivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534864 · Cited by 200
“The IntelliCode Compose system uses the term CTR (Click Through Rate) for this and reports a value of 10% in online trials [12].”
MethodCode Prediction by Feeding Trees to Transformers
Kim, S., Zhao, J., Tian, Y., & Chandra, S. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00026 · Cited by 209
“Recently a team from Microsoft also published their ongoing efforts [62] on applying Transformers for code autocomplete.”
Natural Attack for Pre-trained Models of Code
Yang, Z., Shi, J., He, J., & Lo, D. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510146 · Cited by 166
“[43] introduce GPT-C, a variant of GPT-2 [37] trained on a large corpus containing multiple programming lan-”
MethodWu, H., Zhang, Z., Wang, S., Yan, L., Lin, B., Qin, Y., Zhang, H., & Mao, X. (2021) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre52982.2021.00047 · Cited by 173
“Svyatkovskiy et al. [35] propose GPT-C, which is a variant of the GPT-2 trained from scratch on source code data to support generative tasks like code completion.”
BackgroundMethodRepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation
Zhang, F., Chen, B., Zhang, Y., Keung, J., Liu, J., Zan, D., Mao, Y., Lou, J.-G., & Chen, W. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.151 · Cited by 173
No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence
Wang, C., Yang, Y., Gao, C., Peng, Y. F., Zhang, H., & Lyu, M. R.-T. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549113 · Cited by 161
On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot
Mastropaolo, A., Pascarella, L., Guglielmi, E., Ciniselli, M., Scalabrino, S., Oliveto, R., & Bavota, G. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00181 · Cited by 149
“…targeted the relatively simpler problem of code completion, that has been tackled exploiting historical information [50], coding patterns mined from software repositories [21], [42], [56], [10], [41], [45], [19] and, more recently, Deep Learning (DL) models [62], [27], [29], [8], [53], [15].”
Towards Automating Code Review Activities
Tufano, R., Pascarella, L., Tufano, M., Poshyvanyk, D., & Bavota, G. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00027 · Cited by 185
“[43] used Transformer models to support code completion.”
MethodExploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study
Guo, Q., Cao, J., Xie, X., Liu, S., Li, X., Chen, B., & Peng, X. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3623306 · Cited by 186
“Decoder-only models allow the model to access only the tokens preceding the current token during the training process [22, 39].”
BackgroundNatural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review
Wong, M.-F., Guo, S., Hang, C. N., Ho, S., & Tan, C. W. (2023) · Entropy · doi:10.3390/e25060888 · Cited by 169
“Intellicode [140] 2020 Code Generation, Code Completion GPT-2 Python, C#, JavaScript, and TypeScrip 7”
Is GitHub’s Copilot as bad as humans at introducing vulnerabilities in code?
Asare, O., Nagappan, M., & Asokan, N. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10380-1 · Cited by 128
“Similarly, IntelliCode uses a generative transformer model (GPT-C) which is a variant of GPT-2 (Svyatkovskiy et al. 2020).”
BackgroundMethodInfluentialNo Need to Lift a Finger Anymore? Assessing the Quality of Code Generation by ChatGPT
Liu, Z., Tang, Y., Luo, X., Zhou, Y., & Zhang, L. F. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3392499 · Cited by 128
“Lots of studies on code generation leverage AI-based approaches [1], [2], [3], [4], [5], especially for using large language models (LLMs) [6], [7], [8], [9], [10], [11] such as the recent ChatGPT [12].”
BackgroundDinella, E., Ryan, G., Mytkowicz, T., & Lahiri, S. K. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510141 · Cited by 119
“While not originally designed for code, transformers have found many applications in software engineering [4, 15, 26, 27].”
BackgroundMeasuring GitHub Copilot's Impact on Productivity
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2024) · Communications of the ACM · doi:10.1145/3633453 · Cited by 124
“20 An alternative measure is that of daily completions accepted per user (DCPU) for which a value of around 20 has been report-ed.”
BackgroundInvestigating Code Generation Performance of ChatGPT with Crowdsourcing Social Data
Feng, Y., Vanam, S., Cherukupally, M., Zheng, W., Qiu, M., & Chen, H. (2023) · 2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC) · doi:10.1109/compsac57700.2023.00117 · Cited by 117
“As programming generation and assistant tools, such as CodeBERT [8] and IntelliCode Compose [9], become more widely used, there has been an increased focus on investigating the usability and interactions between users and code generation tools [3], [4], [10], [11].”
Large Language Models Meet NL2Code: A Survey
Zan, D., Chen, B., Zhang, F., Lu, D., Wu, B., Guan, B., Yongji, W., & Lou, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.411 · Cited by 226
“Early works, such as GPT-C (Svyatkovskiy et al., 2020), PyMT5 (Clement et al.”
BackgroundAn Empirical Study on Fine-Tuning Large Language Models of Code for Automated Program Repair
Huang, K., Meng, X., Zhang, J., Liu, Y., Wang, W., Li, S., & Zhang, Y. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00181 · Cited by 113
“Typical models are CodeGPT [34], GPT-C [35], Codex [36] etc. 3) Encoder-Decoder LLMCs retain the infrastructure of Trans-former with both encoder and decoder, such as CodeT5 [37] and PLBART [38].”
MethodNiu, C., Li, C., Ng, V., Ge, J., Huang, L., & Luo, B. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510096 · Cited by 140
“Second, these models have largely assumed as inputs the source code [8, 27, 28, 37, 51] and the associated natural language description [16, 39].”
MethodDeep learning in electron microscopy
Ede, J. M. (2020) · Machine Learning Science and Technology · doi:10.1088/2632-2153/abd614 · Cited by 118
Perfection Not Required? Human-AI Partnerships in Code Translation
Weisz, J. D., Muller, M., Houde, S., Richards, J., Ross, S. I., Martinez, F., Agarwal, M., & Talamadupula, K. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450656 · Cited by 129
“In recent years, several efforts have focused on the use of AI and machine learning techniques for various tasks related to software engineering, including code completion [15, 41, 75, 84], code classification [49, 68], API recommendation [16, 33], variable and method naming [3, 5], type inference [39, 93], bug detect…”
BackgroundSiddiq, M. L., & Santos, J. C. S. (2022) · Proceedings of the 1st International Workshop on Mining Software Repositories Applications for Privacy and Security · doi:10.1145/3549035.3561184 · Cited by 101
“Later, they are fine-tuned for specialized source code related tasks such as automated code summarization [10], completion [14, 15, 29], generation [27, 28] and documentation creation [4].”
BackgroundChatGPT vs SBST: A Comparative Assessment of Unit Test Suite Generation
Tang, Y., Liu, Z., Zhou, Z., & Luo, X. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3382365 · Cited by 136
Bui, N. D. Q., Yu, Y., & Jiang, L. (2021) · Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3404835.3462840 · Cited by 103
“Deep Learning Models of Code : There has been a huge interest in applying deep learning techniques for software engineering tasks such as program functionality classification [10, 11, 55, 75], bug localization [18, 29, 37, 58], code summarization [2, 24, 66], code clone detection [13, 75], program refactoring [32], pr…”
An Empirical Study on the Usage of Transformer Models for Code Completion
Ciniselli, M., Cooper, N., Pascarella, L., Mastropaolo, A., Aghajani, E., Poshyvanyk, D., Penta, M. D., & Bavota, G. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3128234 · Cited by 105
“Last, but not least, Deep Learning (DL) models have been applied to code completion [8], [22], [47], [49], [73], [84], setting new standards in terms of prediction performance.”
Assessing Generalizability of CodeBERT
Zhou, X., Han, D., & Lo, D. (2021) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme52107.2021.00044 · Cited by 97
Machine/Deep Learning for Software Engineering: A Systematic Literature Review
Wang, S., Huang, L., Gao, A., Ge, J., Zhang, T., Feng, H., Satyarth, I., Li, M., Zhang, H., & Ng, V. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3173346 · Cited by 128
CodeFill: multi-token code completion by jointly learning from structure and naming sequences
Izadi, M., Gismondi, R., & Gousios, G. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510172 · Cited by 86
“We pit CodeFill against four baseline models and two the-stateof-the-art models, namely GPT-C [49] and TravTrans+ [25].”
BackgroundMethodInfluentialAn Empirical Study on the Usage of BERT Models for Code Completion
Ciniselli, M., Cooper, N., Pascarella, L., Poshyvanyk, D., Penta, M. D., & Bavota, G. (2021) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr52588.2021.00024 · Cited by 101
Tang, W., Tang, M., Ban, M., Zhao, Z., & Feng, M. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111623 · Cited by 95
Generative AI in mobile networks: a survey
Karapantelakis, A., Alizadeh, P., Alabbasi, A., Dey, K., & Nikou, A. (2023) · Annals of Telecommunications · doi:10.1007/s12243-023-00980-9 · Cited by 91
“One part of transformers works with code generation and code completion [107, 108].”
DB-GPT: Large Language Model Meets Database
Zhou, X., Sun, Z., & Li, G. (2024) · Data Science and Engineering · doi:10.1007/s41019-023-00235-6 · Cited by 83
“Recently LLMs have demonstrated superiority in various tasks like text generation [2], machine translation [3], program synthesis [4].”
Code Completion by Modeling Flattened Abstract Syntax Trees as Graphs
Wang, Y., & Li, H. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17650 · Cited by 86
An Empirical Comparison of Pre-Trained Models of Source Code
Niu, C., Li, C., Ng, V., Chen, D., Ge, J., & Luo, B. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00180 · Cited by 115
“Second, TF-based CoTexT, which uses MLM as the only pre-traning task, outperforms TE-based UniXcoder, which uses three more complex pre-training tasks, ULM, MCL and CMG. Finally, TF-based DeepDebug achieves better results than TE-based C-BERT when using only code as input and MLM as its only pre-training task.”
BackgroundInfluentialWhat do pre-trained code models know about code?
Karmakar, A., & Robbes, R. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678927 · Cited by 87
“Tabnine, IntelliCode [27], TransCoder [21], and more recently Github Copilot [11]—it becomes imperative for us to be aware of their capabilities and drawbacks.”
BackgroundSynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code RepresentationPreprint
Wang, X. E., Wang, Y., Mi, F., Zhou, P., Wan, Y., Xiao, L., Li, L., Wu, H., Liu, J., & Jiang, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.04556 · Cited by 170
Predicting the objective and priority of issue reports in software repositories
Izadi, M., Akbari, K., & Heydarnoori, A. (2022) · Empirical Software Engineering · doi:10.1007/s10664-021-10085-3 · Cited by 84
“Text normalization has been used before in preprocessing data for Machine Learning models [51,28].”
MethodUsing Transfer Learning for Code-Related Tasks
Mastropaolo, A., Cooper, N., Palacio, D. N., Scalabrino, S., Poshyvanyk, D., Oliveto, R., & Bavota, G. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3183297 · Cited by 85
“code summarization [24], [32], [39], [42], code reviews [77], [78], pseudo-code generation [55], code deobfuscation [31], [79], injection of code mutants [76], generation of assert statements [82], clone detection [74], [83], traceability [49] and code completion [5], [11], [17], [17], [34], [35], [70], [84].”
BackgroundRefactoring Programs Using Large Language Models with Few-Shot Examples
Shirafuji, A., Oda, Y., Suzuki, J., Morishita, M., & Watanobe, Y. (2023) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec60848.2023.00025 · Cited by 97
“In recent years, large language models (LLMs) trained on texts in programming languages as well as natural languages [2]–[7] have shown the potential to support programming in both programming education [8]–[11] and software development [12]–[14].”
BackgroundReentrancy Vulnerability Detection and Localization: A Deep Learning Based Two-phase Approach
Zhang, Z., Lei, Y., Yan, M. L., Yu, Y., Chen, J., Wang, S., & Mao, X. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3560428 · Cited by 70
“There are widely used models such as GPT [52] and BERT [17] for NLP and Scelmo [38], CodeBert [24], C-Bert [8] and GPT-C [57] for programming languages. users value value users sum users i usersBet sender It is trained on the CodeSearchNet dataset[33], which includes 2.3 millions functions of six programming languages…”
BackgroundMethodInfluentialContraBERT: Enhancing Code Pre-trained Models via Contrastive Learning
Liu, S., Wu, B., Xie, X., Meng, G., & Liu, Y. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00207 · Cited by 90
“To address this limitation, some researchers propose to utilize unsupervised techniques with the massive amount of data to pre-train a general model [28], [29], [30], [3], [31], [32], [33], [34], [35] and then fine-tune it for different downstream tasks.”
MethodEfficient Federated Learning for Modern NLP
Cai, D., Wu, Y., Wang, S., Lin, F. X., & Xu, M. (2023) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/3570361.3592505 · Cited by 83
“Examples include sentiment analysis, QA, and auto completion [42, 68, 76, 77, 92].”
BackgroundAn Empirical Study of Code Smells in Transformer-based Code Generation Techniques
Siddiq, M. L., Majumder, S. H., Mim, M. R., Jajodia, S., & Santos, J. C. S. (2022) · 2022 IEEE 22nd International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam55253.2022.00014 · Cited by 70
“These Large Language Learning Models (LLMs) are trained with a large dataset of source code snippets and fine-tuned for a wide range of software engineering applications, such as automated code completion [3]–[5], summarization [6], documentation [7], and generation [8], [9].”
BackgroundThongtanunam, P., Pornprasit, C., & Tantithamthavorn, C. K. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510067 · Cited by 75
“…[35], query reformulation [19]); (2) Text → Code (e.g., code search [24, 42]); (3) Code → Text (e.g., code summarization [25], commit message generation [29, 34]); and (4) Code → Code (e.g., automated program repair [20, 28, 33], programming language translation [48], code completion [54]).”
BackgroundLearning Deep Semantics for Test Completion
Nie, P., Banerjee, R., Li, J. J., Mooney, R. J., & Gligoric, M. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00178 · Cited by 86
“Despite being closely related to code completion [25]–[29], test completion is distinct in that test code has several unique”
BackgroundMethodResultInfluentialLanguage Models for Code Completion: A Practical Evaluation
Izadi, M., Katzy, J., van Dam, T., Otten, M., Popescu, R. M., & Arie Van Deursen, A. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639138 · Cited by 88
A Survey on Machine Learning Techniques for Source Code AnalysisPreprint
Sharma, T., Kechagia, M., Georgiou, S., Tiwari, R. K., Vats, I., Moazen, H., & Sarro, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.09610 · Cited by 56
“[306] presented IntelliCode that uses a Transformer model that predicts sequences of code tokens of arbitrary types, and generates entire lines of syntactically correct code.”
BackgroundMethodThe Current Challenges of Software Engineering in the Era of Large Language Models
Gao, C., Hu, X., Gao, S., Xia, X., & Jin, Z. G. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3712005 · Cited by 91
“Their approach reduces catastrophic forgetting in LLM and achieves superior performance in cross-lingual and low-resource scenarios.”
A survey of strategy-driven evasion methods for PE malware: Transformation, concealment, and attack
Geng, J., Wang, J., Fang, Z., Zhou, Y., Wu, D. J., & Ge, W. (2023) · Computers & Security · doi:10.1016/j.cose.2023.103595 · Cited by 58
Generative AI for Requirements Engineering: A Systematic Literature Review
Cheng, H., Husen, J. H., Lu, Y., Racharak, T., Yoshioka, N., Ubayashi, N., & Washizaki, H. (2025) · Software Practice and Experience · doi:10.1002/spe.70029 · Cited by 125
“The integration of artificial intelligence (AI) into SE has transformed traditional practices, particularly tasks such as code generation, defect prediction, and software testing, which have substantiallyimprovedefficiencyandqualityacrossdevelopment processes ([5–7]).”
Chain-of-Thought in Neural Code Generation: From and for Lightweight Language Models
Yang, G., Zhou, Y., Chen, X., Zhang, X., Zhuo, T. Y., & Chen, T. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3440503 · Cited by 75
“N EURAL code generation, which can automatically generate programs from natural language requirements based on deep learning, has become a promising approach to meet the challenges of the ever-increasing complexity of software and alleviate the burden on programmers [1], [2].”
A survey on large language models for software engineering
Zhang, Q., Fang, C., Xie, Y., Zhang, Y., Yu, S., Sun, W., Yang, Y., & Chen, Z. (2026) · Science China Information Sciences · doi:10.1007/s11432-025-4670-0 · Cited by 124
Hadi, M. A., & Fard, F. H. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10314-x · Cited by 61
“2022) or for tasks related to programming languages such as comment generation (Svyatkovskiy et al. 2020), transfer learning to low resource languages (Chen et al.”
BackgroundOn Multi-Modal Learning of Editing Source Code
Chakraborty, S., & Ray, B. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678559 · Cited by 55
“Transformers’ ability of reasoning about long-range dependency is proved useful for several source code processing task including code completions [22], code generation [23], code summarization [21].”
Compressing Pre-trained Models of Code into 3 MB
Shi, J., Yang, Z., Xu, B., Kang, H. J., & Lo, D. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556964 · Cited by 47
“, CodeT5 [43], GPT-C [39], and CodeGPT [28].”
BackgroundInfluentialCompilable Neural Code Generation with Compiler Feedback
Wang, X., Wang, Y., Wan, Y., Mi, F., Li, Y., Zhou, P., Liu, J., Wu, H., Jiang, X., & Liu, Q. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.2 · Cited by 56
One Adapter for All Programming Languages? Adapter Tuning for Code Search and Summarization
Wang, D., Chen, B., Li, S., Luo, W., Peng, S., Dong, W., & Liao, X. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00013 · Cited by 57
Vulnerabilities in AI Code Generators: Exploring Targeted Data Poisoning Attacks
Cotroneo, D., Improta, C., Liguori, P., & Natella, R. (2024) · Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension · doi:10.1145/3643916.3644416 · Cited by 45
“To assess code correctness, we adopt the Edit Distance (ED) , a metric widely used in the field to compare the similarity of the code generated by models with respect to a ground-truth implementation used as a reference for the evaluation [10, 46, 47].”
MethodGenerating bug-fixes using pretrained transformers
Drain, D., Wu, C. H., Svyatkovskiy, A., & Sundaresan, N. (2021) · MAPS@PLDI · doi:10.1145/3460945.3464951 · Cited by 49
“For instance, GPT-C attains perplexities in the range of 1.3 – 2 for several programming languages, whereas a comparably sized GPT-2 model attains a perplexity of 26 on Wikitext-3 [16, 17].”
BackgroundInfluentialSelf-citationPrompt Tuning in Code Intelligence: An Experimental Evaluation
Wang, C., Yang, Y., Gao, C., Peng, Y., Zhang, H., & Lyu, M. R.-T. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3313881 · Cited by 48
“Svyatkovskiy et al. [61] and Kim et al. [62] train GPT-2 [63] on large scale programming languages for solving code completion task.”
BackgroundExtending the Frontier of ChatGPT: Code Generation and Debugging
Sakib, F. A., Khan, S. H., & Karim, A. (2024) · International Conference on Electrical, Computer and Energy Technologies (ICECET) · doi:10.1109/icecet61485.2024.10698405 · Cited by 80
“Artificial intelligence has achieved remarkable prowess across a plethora of dimensions, encompassing code generation [23]– [27], program explanation [28]–[31], error correction [32]– [35], and more.”
BackgroundOn the transferability of pre-trained language models for low-resource programming languages
Chen, F., Fard, F. H., Lo, D., & Bryksin, T. (2022) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3524610.3527917 · Cited by 59
“IntelliCode Compose is a GPT based model that is trained on Python, C#, JavaScript, and TypeScript for code completion [41].”
Towards Efficient Fine-Tuning of Pre-trained Code Models: An Experimental Study and Beyond
Shi, E., Wang, Y., Zhang, H., Du, L., Han, S., Zhang, D., & Sun, H. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598036 · Cited by 50
“Similar to previous work [34], the performance is measured by EM and Levenshtein edit similarity (Edit sim) [50].”
MethodA survey on machine learning techniques applied to source code
Sharma, T., Kechagia, M., Georgiou, S., Tiwari, R., Vats, I., Moazen, H., & Sarro, F. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111934 · Cited by 159
Code Search is All You Need? Improving Code Suggestions with Code Search
Chen, J., Hu, X., Li, Z., Gao, C., Xia, X., & Lo, D. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639085 · Cited by 64
Shi, J., Yang, Z., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708525 · Cited by 43
“Numerous decoder-only LLMs tailored for code generation have been developed, including CodeGPT [84], GPT-C [117], Codex [15], PolyCoder [143], CodeGen [87, 88], Code Llama [99], StarCoder [73, 82], and Magicoder [137].”
CrystalBLEU: Precisely and Efficiently Measuring the Similarity of Code
Eghbali, A., & Pradel, M. (2022) · Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3551349.3556903 · Cited by 42
“Code prediction techniques include code completion [4, 22, 23, 26, 46], code translation between programming languages [32, 33, 43], automated program repair [11, 12, 16, 24, 49], predicting code from natural language descriptions of the desired functionality [15, 21, 27, 44, 47], injecting bugs and vulnerabilities in…”
BackgroundAljedaani, W., Habib, A., Aljohani, A., Eler, M., & Feng, Y. (2024) · Proceedings of the 21st International Web for All Conference · doi:10.1145/3677846.3677854 · Cited by 42
Ivie: Lightweight Anchored Explanations of Just-Generated Code
Yan, L., Hwang, A., Wu, Z., & Head, A. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642239 · Cited by 55
“These tools, often called “programming assistants,” include GitHub Copilot [22], OpenAI GPT-4 [40], Amazon CodeWhisperer [2], IntelliCode Compose [56], and CodeT5+ [64].”
MethodTeaching Code LLMs to Use Autocompletion Tools in Repository-Level Code Generation
Wang, C., Zhang, J., Feng, Y., Li, T., Sun, W., Liu, Y., & Peng, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3714462 · Cited by 40
“— Edit Similarity (EditSim) [41].”
Making the most of small Software Engineering datasets with modern machine learning
Prenner, J. A., & Robbes, R. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3135465 · Cited by 47
“[60] introduce IntelliCode compose, a system for intelligent code completion based on the GPT-2 Transformer language model.”
MethodPyART: Python API Recommendation in Real-Time
He, X., Xu, L., Zhang, X., Hao, R., Feng, Y., & Xu, B. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00145 · Cited by 40
“API recommendation approaches based on Machine Learning tend to have better performance for Python than most traditional methods [21]–[24].”
MethodDeep Learning Meets Software Engineering: A Survey on Pre-Trained Models of Source Code
Niu, C., Li, C., Luo, B., & Ng, V. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/775 · Cited by 63
Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacities
Ma, W., Liu, S., Zhao, M., Xie, X., Wang, W., Hu, Q., Zhang, J., & Liu, Y. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3664606 · Cited by 57
“Many code models [2, 4, 13, 17, 29, 31, 41, 42, 63, 71] have been proposed to greatly advance the development of code intelligence.”
BackgroundDomain Adaptive Code Completion via Language Models and Decoupled Domain Databases
Tang, Z. J., Ge, J., Liu, S., Zhu, T., Xu, T., Huang, L., & Luo, B. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00076 · Cited by 40
“For example, CodeFill [29] improves GPT-2 [25] by predicting the code type when completing the next token, while Gramm-former [37] generates code completions with “holes” inserted in places where the model is uncertain.”
BackgroundTransformers in health: a systematic review on architectures for longitudinal data analysis
Siebra, C., Kurpicz-Briki, M., & Wac, K. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-023-10677-z · Cited by 44
“Similarly, programming language code generation using transformers (Svyatkovskiy et al. 2020) employs the tokens of the programming language as vocabulary.”
How People Prompt Generative AI to Create Interactive VR Scenes
Manesh, S. A., Zhang, T., Onishi, Y., Hara, K., Bateman, S., Li, J., & Tang, A. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3661547 · Cited by 48
“The code generation ability of current LLMs further powers programming agents that synthesize computer programs supporting complex logic and user interactions from language input (e.g., [53, 66]).”
BackgroundUnveiling ChatGPT’s usage in open source projects: A mining-based study
Tufano, R., Mastropaolo, A., Pepe, F., Dabic, O., Di Penta, M., & Bavota, G. (2024) · Proceedings of the 21st International Conference on Mining Software Repositories · doi:10.1145/3643991.3644918 · Cited by 38
“…the aforementioned problems, but also to perform tasks for which no recommender was previously thought in the past, including generating entire code blocks [56, 103], automatically reviewing source code [76, 104, 107, 108], or generating scenarios for automatically reproducing issues [50, 66, 116].”
BackgroundAgents in software engineering: survey, landscape, and vision
Wang, Y., Zhong, W., Huang, Y., Shi, E., Yang, M., Chen, J., Li, H., Ma, Y., Wang, Q., & Zheng, Z. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00544-2 · Cited by 84
Program merge conflict resolution via neural transformers
Svyatkovskiy, A., Fakhoury, S., Ghorbani, N., Mytkowicz, T., Dinella, E., Bird, C., Jang, J., Sundaresan, N., & Lahiri, S. K. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549163 · Cited by 38
“Neural language models (LMs) have shown great performance in natural language generation [36, 38], and have been successfully applied to the domain of source code [17, 24, 48].”
BackgroundSelf-citationFine-Tuning Large Language Models to Improve Accuracy and Comprehensibility of Automated Code Review
Yu, Y., Rong, G., Shen, H., Zhang, H., Shao, D., Wang, M., Wei, Z., Xu, Y., & Wang, J. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695993 · Cited by 47
“The activities of ACR primarily encompass reviewer recommendation [46, 61], code issue prediction [27, 52], and review comment recommendation [44] and generation [27].”
Pu, K., Lazaro, D., Arawjo, I., Xia, H., Xiao, Z., Grossman, T., & Chen, Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713357 · Cited by 67
“Tools like Github Copilot [4] and Visual Studio’s IntelliCode [83] offer the auto-completion feature, which fills the code line as the user is typing, to alleviate prompt engineering efforts and proactively provide in-situ support.”
Large Language Model-Aware In-Context Learning for Code Generation
Li, J., Li, G., Tao, C., Li, J., Zhang, H., Liu, F., & Jin, Z. (2023) · arXiv · doi:10.48550/arxiv.2310.09748 · Cited by 32
Pre-training by Predicting Program Dependencies for Vulnerability Analysis Tasks
Liu, Z., Tang, Z., Zhang, J., Xia, X., & Yang, X. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639142 · Cited by 31
“Existing pre-trained code models can generally be divided into encoder-only [18, 20, 26, 31, 33, 62], decoder-only [40, 45, 57] and encoder-decoder models [7, 13, 25, 32, 42, 48, 49, 63, 65].”
BackgroundMethodMulti-target Backdoor Attacks for Code Pre-trained Models
Li, Y., Liu, S., Chen, K., Xie, X., Zhang, T., & Liu, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.399 · Cited by 53
How could Neural Networks understand Programs?Preprint
Peng, D., Zheng, S., Li, Y., Ke, G., He, D., & Liu, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.04297 · Cited by 36
“Svyatkovskiy et al. (2020) propose to train a variant of GPT-2 (Radford et al., 2019) from scratch on source code to improve the performance of code completion.”
BackgroundEfficient transformer with code token learner for code clone detection
Zhang, A., Fang, L., Ge, C., Li, P., & Liu, Z. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111557 · Cited by 33
CodeRetriever: A Large Scale Contrastive Pre-Training Method for Code Search
Li, X., Gong, Y., Shen, Y., Qiu, X., Zhang, H., Yao, B., Qi, W., Jiang, D., Chen, W., & Duan, N. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.187 · Cited by 31
Metadata
- Authors
- Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N.
- Affiliations
- Microsoft (United States) (US)
- Microsoft, USA
- ORCID
- Sundaresan, Neel
- Venue
- Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2020-11-08
- Pages
- 1433-1443
- OpenAlex
- W3025993830
- Cited by
- 474
- References
- 38
- Citations by year
- 2026: 42 · 2025: 97 · 2024: 117 · 2023: 103 · 2022: 52 · 2021: 56 · 2020: 7
- FWCI
- 41.33
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- code generation, code completion, transformer models, integrated development environment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
