Improving Code Autocompletion with Transfer Learning
Open accessAbstract
Software language models have achieved promising results predicting code completion usages, and several industry studies have described successful IDE integrations. Recently, accuracy in autocompletion prediction improved 12.8% from training on a real-world dataset collected from programmers' IDE activity. But what if limited examples of IDE autocompletion in the target programming language are available for model training? In this paper, we investigate the efficacy of pretraining autocompletion models on non-IDE, non-autocompletion, and different-language example code sequences. We find that these unsupervised pretrainings improve model accuracy by over 50% on very small fine-tuning datasets and over 10% on 50k labeled examples. We confirm the real-world impact of these pretrainings in an online setting through A/B testing on thousands of IDE autocompletion users, finding that pretraining is responsible for increases of up to 6.63% autocompletion usage.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
DCPU (Daily Completions accepted Per User) has been reported around 20.
“An alternative measure is that of DCPU (Daily Completions accepted Per User) for which a value of around 20 has been reported [3,19].”
From Productivity Assessment of Neural Code Completion (2022) · cites this paper as [19]
The E. Evaluation section defines DCPU as the number of autocompletion suggestions a developer accepts in a day. The Online evaluation section reports percentage changes in DCPU, but neither section reports a DCPU value around 20; searches for that value found none. (LLM, conf=0.88)
Experimental — accuracy not yet measuredAn alternative measure is daily completions accepted per user (DCPU), for which a value of around 20 has been reported.
“An alternative measure is that of daily completions accepted per user (DCPU) for which a value of around 20 has been reported.3,29”
From Measuring GitHub Copilot's Impact on Productivity (2024) · cites this paper as [29]
The Evaluation section defines DCPU as the number of autocompletion suggestions a developer accepts in a day. The Online evaluation reports percentage changes in DCPU, but no value of around 20 accepted completions per user per day; the cited paper therefore supports the measure, not the reported value. (LLM, conf=0.88)
Experimental — accuracy not yet measuredThe empirical software engineering research community has concluded that developer productivity is a multidimensional topic that cannot be summarized by a single metric.
“To sum up, after many studies and many definitions, measurements, and approaches to productivity, the empirical software engineering research community has concluded that developer productivity is a multidimensional topic that cannot be summarized by a single metric.10”
From Measuring GitHub Copilot's Impact on Productivity (2024) · cites this paper as [10]
The Evaluation section describes metrics for code-autocompletion performance and use, including daily completions per user. Searches for developer productivity, multidimensional measurement, and a single metric found no passage saying that the empirical software engineering community reached the attributed conclusion. (LLM, conf=0.74)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · cited 6× · see citing works
“Transfer learning has revolutionized natural language processing (NLP) since the seminal papers on models such as GPT-2 [9], BERT [10] and RoBERTa [22].”
MethodInfluentialRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 1× · see citing works
“Transfer learning has revolutionized natural language processing (NLP) since the seminal papers on models such as GPT-2 [9], BERT [10] and RoBERTa [23].”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824 · cited 2× · see citing works
“Since then several works have been proposed [12], [13], [23]–[28] to apply the idea of transfer learning in the domain of code.”
BackgroundMethodNeural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974 · see citing works
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751 · cited 1× · see citing works
“Code sequences fed to our multilingual model are modified by prepending a control code [15] to indicate the source language.”
Unified Pre-training for Program Understanding and Generation
Ahmad, W., Chakraborty, S., Ray, B., & Chang, K.-W. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.211 · Cited by 642 · see citing works
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.04664 · Cited by 536 · see citing works
The Natural Language Decathlon: Multitask Learning as Question Answering
McCann, B., Keskar, N. S., Xiong, C., & Socher, R. (2018) · arXiv · doi:10.48550/arxiv.1806.08730 · Cited by 448 · see citing works
On the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Devanbu, P. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227135 · Cited by 425 · cited 1× · see citing works
“Earlier attempts at modeling software languages and completion ranking were based on n-gram language models [4], [19], [20] or probabilistic models using the information from ASTs [21].”
How are Java software developers using the Eclipse IDE?
Murphy, G. C., Kersten, M., & Findlater, L. (2006) · IEEE Software · doi:10.1109/ms.2006.105 · Cited by 413 · see citing works
Learning from examples to improve code completion systems
Bruch, M., Monperrus, M., & Mezini, M. (2009) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/1595696.1595728 · Cited by 383 · cited 1× · see citing works
“Significant attention has been given to improving suggestion prediction through machine learning [3]–[6] by feeding code to models as a sequence of tokens or even AST nodes [7].”
BackgroundAre deep neural networks the best choice for modeling source code?
Hellendoorn, V. J., & Devanbu, P. (2017) · Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering · doi:10.1145/3106237.3106290 · Cited by 292 · see citing works
The adverse effects of code duplication in machine learning models of code
Allamanis, M. (2019) · Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software · doi:10.1145/3359591.3359735 · Cited by 284 · see citing works
Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks
Mastropaolo, A., Scalabrino, S., Cooper, N., Palacio, D. N., Poshyvanyk, D., Oliveto, R., & Bavota, G. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00041 · Cited by 271 · see citing works
Code Completion with Neural Attention and Pointer Networks
Li, J., Wang, Y., Lyu, M. R., & King, I. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/578 · Cited by 252 · see citing works
A statistical semantic language model for source code
Nguyen, T. T., Nguyen, A., Nguyen, H., & Nguyen, T. (2013) · ESEC/FSE 2013 · doi:10.1145/2491411.2491458 · Cited by 231 · see citing works
Code Prediction by Feeding Trees to Transformers
Kim, S., Zhao, J., Tian, Y., & Chandra, S. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00026 · Cited by 209 · see citing works
Self-citationGenerative Code Modeling with GraphsPreprint
Brockschmidt, M., Allamanis, M., Gaunt, A. L., & Polozov, O. (2018) · International Conference on Learning Representations · doi:10.48550/arxiv.1805.08490 · Cited by 199 · cited 1× · see citing works
“the-art performance for software language modeling [3, 8].”
Traceability Transformed: Generating More Accurate Links with Pre-Trained BERT Models
Lin, J., Liu, Y., Zeng, Q., Jiang, M., & Cleland‐Huang, J. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00040 · Cited by 167 · cited 1× · see citing works
“Lin et al. [25] apply pretrained BERT models for learning relationships between issues and commits in a software repository.”
Code smell detection by deep direct-learning and transfer-learning
Sharma, T., Efstathiou, V., Λουρίδας, Π., & Spinellis, D. D. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.110936 · Cited by 149 · see citing works
Applying CodeBERT for Automated Program Repair of Java Simple Bugs
Mashhadi, E., & Hemmati, H. (2021) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr52588.2021.00063 · Cited by 144 · see citing works
Trex: Learning Execution Semantics from Micro-Traces for Binary SimilarityPreprint
Pei, K., Zhou, X., Yang, J., Jana, S., & Ray, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.08680 · Cited by 92 · see citing works
Elnaggar, A., Ding, W., Jones, L., Gibbs, T., Fehér, T. B., Angerer, C., Severini, S., Matthes, F., & Rost, B. (2021) · arXiv.org · doi:10.48550/arxiv.2104.02443 · Cited by 86 · cited 1× · see citing works
“Since then several works have been proposed [12], [13], [23]–[28] to apply the idea of transfer learning in the domain of code.”
BackgroundWhen Code Completion Fails: A Case Study on Real-World Completions
Hellendoorn, V. J., Proksch, S., Gall, H. C., & Bacchelli, A. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00101 · Cited by 76 · cited 3× · see citing works
“One solution might be to collect random, non-autocompletion code sequences during code authoring, but several recent studies[2, 7] have shown a myriad of”
BackgroundInfluentialMaybe Deep Neural Networks are the Best Choice for Modeling Source CodePreprint
Karampatsis, R.-M., & Sutton, C. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.05734 · Cited by 41 · see citing works
Sequence Model Design for Code Completion in the Modern IDEPreprint
Aye, G. A., & Kaiser, G. E. (2020) · arXiv.org · doi:10.48550/arxiv.2004.05249 · Cited by 33 · see citing works
Self-citationLearning Autocompletion from Real-World Datasets
Aye, G. A., Kim, S., & Li, H. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip52600.2021.00022 · Cited by 29 · see citing works
Self-citationA Simple Approach for Handling Out-of-Vocabulary Identifiers in Deep Learning for Source Code
Chirkova, N., & Troshin, S. (2020) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.naacl-main.26 · Cited by 12 · see citing works
Flagged first, then by citations
2 of the 32 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- P. Bielik, V. Raychev, and M. Vechev, "Phog: Probabilistic model for code," in Proceedings of the 33rd International Conference on Inter- national Conference on Machine Learning -Volume 48, ser. ICML'16. JMLR.org, 2016, p. 2933-2942.
Cited by
Sorted by citations
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650 · see citing works
“For example, transfer learning has been proposed as a way to improve code completion performance when the volume of training examples for a specific programming language is inadequate [91].”
BackgroundProductivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534864 · Cited by 200 · see citing works
“An alternative measure is that of DCPU (Daily Completions accepted Per User) for which a value of around 20 has been reported [3, 21].”
BackgroundMeasuring GitHub Copilot's Impact on Productivity
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2024) · Communications of the ACM · doi:10.1145/3633453 · Cited by 124 · see citing works
“3,29 To calculate acceptance rate one must, of course, normalize DCPU by the time spent coding each day.”
BackgroundRLCoder: Reinforcement Learning for Repository-Level Code Completion
Wang, Y., Wang, Y., Guo, D., Chen, J., Zhang, R., Ma, Y., & Zheng, Z. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00014 · Cited by 43 · see citing works
AI-Assisted Code Authoring at Scale: Fine-Tuning, Deploying, and Mixed Methods Evaluation
Murali, V., Maddila, C., Ahmad, I., Bolin, M., Cheng, D., Ghorbani, N., Fernandez, R., Nagappan, N., & Rigby, P. C. (2023) · Proc. ACM Softw. Eng. · doi:10.1145/3643774 · Cited by 48 · see citing works
“A strict metric like Exact Match, BLEU score or percentage of accepted suggestions as-is will underrepresent the actual benefits offered by these solutions.”
BackgroundSelf-citationMulti-line AI-Assisted Code Authoring
Dunay, O., Cheng, D., Tait, A., Thakkar, P., Rigby, P. C., Chiu, A., Ahmad, I., Ganesan, A., Maddila, C., Murali, V., et al. (2024) · SIGSOFT FSE Companion · doi:10.1145/3663529.3663836 · Cited by 29 · see citing works
Self-citationIn-IDE Human-AI Experience in the Era of Large Language Models; A Literature Review
Sergeyuk, A., Titov, S., & Izadi, M. (2024) · Ide · doi:10.1145/3643796.3648463 · Cited by 19
“[4, 8, 11, 22, 24, 25, 30, 40, 41] are qualitatively and quantitatively evaluating various aspects of AI assistants’ functionality, ranging from correctness and under-standability of suggested code to investigating security concerns.”
BackgroundCross-lingual Transfer in Programming Languages: An Extensive Empirical Study
Baltaji, R., Pujar, S., Hirzel, M., Mandel, L., Buratti, L., & Varshney, L. (2023) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2310.16937 · Cited by 17
RAMBO: Enhancing RAG-based Repository-Level Method Body Completion
Bui, T.-D., Luu-Van, D.-T., Nguyen, T., Nguyen, T.-T., Nguyen, S., & Vo, H. (2024) · Information and Software Technology · doi:10.48550/arxiv.2409.15204 · Cited by 16
“For practical applications, one should limit the surrounding context to around 50 LOCs, as larger contexts yield only marginal improvements in R AMBO ’s performance while adding complexity and potential noise to the generation process.”
xASTNN: Improved Code Representations for Industrial Practice
Xu, Z., Zhou, M., Zhao, X.-B., Chen, Y., Cheng, X., & Zhang, H.-Y. (2023) · ESEC/SIGSOFT FSE · doi:10.1145/3611643.3613869 · Cited by 11
“Recently, code representation learning has aroused much interest in both academia [25] and industry [18, 43, 55, 56].”
BackgroundAn Analysis of the Costs and Benefits of Autocomplete in IDEs
Jiang, S., & Coblenz, M. J. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660765 · Cited by 6 · see citing works
“Various studies [6, 20, 27, 31, 36] propose improvements to autocomplete techniques, such as designing new algorithms and using better datasets, to make autocomplete work more effectively.”
DCServCG: A data-centric service code generation using deep learning
Alizadehsani, Z., Ghaemi, H., Shahraki, A., González‐Briones, A., & Corchado, J. M. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106304 · Cited by 12 · see citing works
AI-Assisted SQL Authoring at Industry Scale
Maddila, C., Ghorbani, N., Jabre, K., Murali, V., Kim, E., Thakkar, P., Laptev, N. P., Harman, O., Hsu, D., Abreu, R., et al. (2024) · 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00019 · Cited by 7 · see citing works
Self-citationNeural Methods for Programming: A Comprehensive Survey and Future Directions
Maru, G. G., Lee, S., Ji, S., Ko, S.-K., & Im, H. (2025) · Applied Sciences · doi:10.3390/app152212150 · Cited by 6
Completing Function Documentation Comments Using Structural Information
Ciurumelea, A., Alexandru, C. V., Gall, H., & Proksch, S. (2023) · Empirical Software Engineering · doi:10.1007/s10664-022-10284-6 · Cited by 6 · see citing works
“This metric has been used in a paper (Zhou et al. 2022) that evaluates deep learning models for the task of code completion, which is similar to our task.”
MethodSong, L., & Ding, S. H. H. (2023) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac57700.2023.00084 · Cited by 1 · see citing works
“Some popular research areas involve code autocomplete [18, 21], code clone detection [5], code search [6], code documentation generation [8], etc.”
BackgroundPeykani, P., Charles, V., Emrouznejad, A., Ghanidel, S., & Sabour, D. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10720-6 · Cited by 1 · see citing works
- Authors
- Zhou, W., Kim, S., Murali, V., & Aye, G. A.
- Affiliations
- Meta (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2105.05991
- OpenAlex
- W4303711643
- Cited by
- 0
- References
- 32
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- transfer learning, pre-training, IDE integration, fine-tuning, unsupervised pre-training, A/B testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
