Fast and Memory-Efficient Neural Code Completion
Open accessAbstract
Code completion is one of the most widely used features of modern integrated development environments (IDEs). While deep learning has made significant progress in the statistical prediction of source code, state-of-the-art neural network models consume hundreds of megabytes of memory, bloating the development environment. We address this in two steps: first we present a modular neural framework for code completion. This allows us to explore the design space and evaluate different techniques. Second, within this framework we design a novel reranking neural completion model that combines static analysis with granular token encodings. The best neural reranking model consumes just 6 MB of RAM, - 19x less than previous models - computes a single completion in 8 ms, and achieves 90% accuracy in its top five suggestions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Traditional code completion techniques typically involve analyzing code to identify potential suggestions, followed by re-ranking them.
“Traditional code completion techniques typically involve analyzing code to identify potential suggestions, followed by re-ranking them”
From RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation (2023) · cites this paper as Svyatkovskiy et al., 2021
The Candidate Providers section (page unavailable) says static-analysis tools commonly identify plausible completion candidates, and the Approach section describes ranking those candidates. But the paper presents neural reranking as its own novel approach; it does not say traditional code completion techniques typically follow that full sequence. (LLM, conf=0.84)
Experimental — accuracy not yet measuredThe traditional re-ranking approach offers efficient performance but lacks flexibility to generate code at arbitrary granularity.
“While this approach offers efficient performance, it lacks the flexibility to generate code at arbitrary granularity.”
From RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation (2023) · cites this paper as Svyatkovskiy et al., 2021
The Introduction reports fast, memory-efficient neural re-ranking, and the Approach says it ranks completion candidates returned by static analysis. Those passages support the efficiency claim, but the paper does not state that re-ranking cannot generate code at arbitrary granularity; its evaluated implementation is for API completion. (LLM, conf=0.84)
Experimental — accuracy not yet measuredThe evaluation of context-aware API completion for Visual Studio Intelli-Code considered Recall@5, and this metric fell from 90% in offline evaluation to 70% when used online.
“The evaluation of context-aware API completion for Visual Studio Intelli-Code considered Recall@5Ðthe proportion of completions for which the correct method call was in the top 5 suggestions.This metric fell from 90% in offline evaluation to 70% when used online [12].”
From Productivity Assessment of Neural Code Completion (2022) · cites this paper as [12]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
The magical number seven, plus or minus two: Some limits on our capacity for processing information.
Miller, G. A. (1956) · Psychological Review · doi:10.1037/h0043158 · Cited by 17,812 · see citing works
The magical number seven, plus or minus two: Some limits on our capacity for processing information.
Miller, G. A. (1994) · Psychological Review · doi:10.1037/0033-295x.101.2.343 · Cited by 17,469 · see citing works
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831 · see citing works
Character-Aware Neural Language Models
Kim, Y., Jernite, Y., Sontag, D. A., & Rush, A. M. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10362 · Cited by 1,546 · see citing works
Character-Aware Neural Language Models
Kim, Y., Jernite, Y., Sontag, D., & Rush, A. M. (2015) · arXiv · doi:10.48550/arxiv.1508.06615 · Cited by 1,530 · see citing works
Convolutional Neural Networks for Sentence ClassificationPreprint
Kim, Y. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1408.5882 · Cited by 1,112 · see citing works
Feature hashing for large scale multitask learning
Weinberger, K. Q., Dasgupta, A., Langford, J. C., Smola, A. J., & Attenberg, J. (2009) · International Conference on Machine Learning · doi:10.1145/1553374.1553516 · Cited by 928 · see citing works
Introduction to Statistical Decision Theory.
Schervish, M. J., Pratt, J. W., Raiffa, H., & Schlaifer, R. (1996) · Journal of the American Statistical Association · doi:10.2307/2291759 · Cited by 705 · see citing works
Gutmann, M. U., & Hyvärinen, A. (2012) · Edinburgh Research Explorer (University of Edinburgh) · doi:10.5555/2503308.2188396 · Cited by 596 · see citing works
On the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Dévanbu, P. (2012) · doi:10.5555/2337223.2337322 · Cited by 574 · see citing works
A cache-based natural language model for speech recognition
Kühn, R., & Mori, R. D. (1990) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.56193 · Cited by 503 · see citing works
On the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Devanbu, P. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227135 · Cited by 425 · see citing works
How are Java software developers using the Eclipse IDE?
Murphy, G. C., Kersten, M., & Findlater, L. (2006) · IEEE Software · doi:10.1109/ms.2006.105 · Cited by 413 · see citing works
Training deep neural networks with low precision multiplications
Courbariaux, M., Bengio, Y., & David, J.-P. (2014) · arXiv · doi:10.48550/arxiv.1412.7024 · Cited by 407 · see citing works
Mining source code repositories at massive scale using language modeling
Allamanis, M., & Sutton, C. (2013) · IEEE Working Conference on Mining Software Repositories · doi:10.1109/msr.2013.6624029 · Cited by 393 · see citing works
Self-citationLearning from examples to improve code completion systems
Bruch, M., Monperrus, M., & Mezini, M. (2009) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/1595696.1595728 · Cited by 383 · see citing works
Suggesting accurate method and class names
Allamanis, M., Barr, E. T., Bird, C., & Sutton, C. A. (2015) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2786805.2786849 · Cited by 379 · see citing works
Self-citationAn Introduction to Neural Information Retrieval
Mitra, B., & Craswell, N. (2018) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000061 · Cited by 317 · see citing works
Tying Word Vectors and Word Classifiers: A Loss Framework for Language ModelingPreprint
Inan, H., Khosravi, K., & Socher, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01462 · Cited by 314 · see citing works
Are deep neural networks the best choice for modeling source code?
Hellendoorn, V. J., & Devanbu, P. (2017) · Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering · doi:10.1145/3106237.3106290 · Cited by 292 · see citing works
The adverse effects of code duplication in machine learning models of code
Allamanis, M. (2019) · Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software · doi:10.1145/3359591.3359735 · Cited by 284 · see citing works
Self-citationTu, Z., Su, Z., & Devanbu, P. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635875 · Cited by 238 · see citing works
A statistical semantic language model for source code
Nguyen, T. T., Nguyen, A., Nguyen, H., & Nguyen, T. (2013) · ESEC/FSE 2013 · doi:10.1145/2491411.2491458 · Cited by 231 · see citing works
DéjàVu: a map of code duplicates on GitHub
Lopes, C. V., Maj, P., Martins, P., Saini, V., Yang, D., Zitny, J., Sajnani, H., & Vitek, J. (2017) · Proceedings of the ACM on Programming Languages · doi:10.1145/3133908 · Cited by 183 · see citing works
Big code != big vocabulary: open-vocabulary models for source code
Karampatsis, R.-M., Babii, H., Robbes, R., Sutton, C., & Janes, A. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380342 · Cited by 182 · see citing works
Pythia: AI-assisted Code Completion System
Svyatkovskiy, A., Zhao, Y., Fu, S., & Sundaresan, N. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330699 · Cited by 157 · see citing works
Self-citationFast Learning in Multi-Resolution Hierarchies
Moody, J. E. (1988) · Cited by 145 · see citing works
Structured Generative Models of Natural Source CodePreprint
Maddison, C. J., & Tarlow, D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1401.0514 · Cited by 129 · see citing works
PHOG: probabilistic model for code
Bielik, P., Raychev, V., & Vechev, M. (2016) · International Conference on Machine Learning · Cited by 115 · see citing works
Intelligent Code Completion with Bayesian Networks
Proksch, S., Lerch, J., & Mezini, M. (2015) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/2744200 · Cited by 81 · see citing works
A Study of Visual Studio Usage in Practice
Amann, S., Proksch, S., Nadi, S., & Mezini, M. (2016) · 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER) · doi:10.1109/saner.2016.39 · Cited by 77 · see citing works
When Code Completion Fails: A Case Study on Real-World Completions
Hellendoorn, V. J., Proksch, S., Gall, H. C., & Bacchelli, A. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00101 · Cited by 76 · see citing works
An Introduction to Neural Information Retrieval t
Mitra, B., & Craswell, N. (2018) · now publishers, Inc. eBooks · doi:10.1561/9781680835335 · Cited by 57 · see citing works
Sequence Model Design for Code Completion in the Modern IDEPreprint
Aye, G. A., & Kaiser, G. E. (2020) · arXiv.org · doi:10.48550/arxiv.2004.05249 · Cited by 33 · see citing works
Combining Program Analysis and Statistical Language Model for Code Statement Completion
Nguyen, S., Nguyen, T. N., Li, Y., & Wang, S. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00072 · Cited by 32 · see citing works
Learning Autocompletion from Real-World Datasets
Aye, G. A., Kim, S., & Li, H. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip52600.2021.00022 · Cited by 29 · see citing works
Learning Autocompletion from Real-World DatasetsPreprint
Aye, G. A., Kim, S., & Li, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.04542 · Cited by 3 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Research works printed but not linked: 7
- [1]1. M. Allamanis, E. T. Barr, P. Devanbu, and C. Sutton, "A survey of machine learning for big code and naturalness," ACM Computing Surveys (CSUR), vol. 51, no. 4, p. 81, 2018.
- [17]17. M. Allamanis, M. Brockschmidt, and M. Khademi, "Learning to represent programs with graphs," in Pro- ceedings of the International Conference on Learning Representations (ICLR), 2018.
- [19]19. R. Sennrich, B. Haddow, and A. Birch, "Neural machine translation of rare words with subword units," in Pro- ceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2016.
- [20]20. X. Zhang, J. Zhao, and Y. LeCun, "Character-level con- volutional networks for text classification," in Advances in neural information processing systems, 2015, pp. 649- 657.
- [23]23. D. Bahdanau, K. Cho, and Y. Bengio, "Neural machine translation by jointly learning to align and translate," in Proceedings of the International Conference on Learning Representations (ICLR), 2015.
- [42]42. I. Goodfellow, Y. Bengio, and A. Courville, Deep Learn- ing. MIT Press, 2016, www.deeplearningbook.org.
- [46]46. M. Brockschmidt, M. Allamanis, A. L. Gaunt, and O. Polozov, "Generative code modeling with graphs," in Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Cited by
Sorted by citations
Productivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Simister, S., Sittampalam, G., Li, A., Rice, A., Rifkin, D., & Aftandilian, E. (2022) · arXiv · doi:10.48550/arxiv.2205.06537 · Cited by 200 · see citing works
Productivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534864 · Cited by 200 · see citing works
Code Prediction by Feeding Trees to Transformers
Kim, S., Zhao, J., Tian, Y., & Chandra, S. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00026 · Cited by 209 · see citing works
RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation
Zhang, F., Chen, B., Zhang, Y., Keung, J., Liu, J., Zan, D., Mao, Y., Lou, J.-G., & Chen, W. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.151 · Cited by 173 · see citing works
Measuring GitHub Copilot's Impact on Productivity
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2024) · Communications of the ACM · doi:10.1145/3633453 · Cited by 124 · see citing works
“21 Due to the diversity of potential solutions to a multi-line completion task, researchers have used software testing to evaluate the behavior of completions.”
BackgroundMethodAn Empirical Study of Code Smells in Transformer-based Code Generation Techniques
Siddiq, M. L., Majumder, S. H., Mim, M. R., Jajodia, S., & Santos, J. C. S. (2022) · 2022 IEEE 22nd International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam55253.2022.00014 · Cited by 70 · see citing works
Shi, J., Yang, Z., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708525 · Cited by 43 · see citing works
Leveraging automated unit tests for unsupervised code translation
Roziere, B., Zhang, J. M., Charton, F., Harman, M., Synnaeve, G., & Lample, G. (2021) · arXiv · doi:10.48550/arxiv.2110.06773 · Cited by 39 · see citing works
Non-Autoregressive Line-Level Code Completion
Liu, F., Fu, Z., Li, G., Jin, Z., Liu, H., Hao, Y., & Zhang, L. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3649594 · Cited by 22 · see citing works
ReACC: A Retrieval-Augmented Code Completion Framework
Lu, S., Duan, N., Han, H., Guo, D., Hwang, S.-W., & Svyatkovskiy, A. (2022) · arXiv · doi:10.48550/arxiv.2203.07722 · Cited by 96 · see citing works
“Svyatkovskiy et al. (2021) and Clement et al. (2021), on the other hand, investigate ways to improve the efficiency and long-range modeling in the code completion task, respectively.”
BackgroundSelf-citationFull Line Code Completion: Bringing AI to Desktop
Semenkin, A., Bibaev, V., Sokolov, Y., Krylov, K., Kalina, A., Khannanova, A., Savenkov, D., Rovdo, D., Davidenko, I., Karnaukhov, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.08704 · Cited by 8 · see citing works
“In recent years, machine learning techniques were studied and applied for the different aspects of the completion problem [9]–[12], including the generation of the completion sequences itself.”
- Authors
- Svyatkovskiy, A., Lee, S., Hadjitofi, A., Riechert, M., Franco, J. V., & Allamanis, M.
- Affiliations
- Microsoft (United States) (US)
- Venue
- 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR)
- Publisher
- IEEE
- Published
- 2021-05
- Pages
- 329-340
- OpenAlex
- W3021206621
- Cited by
- 95
- References
- 41
- Citations by year
- 2026: 6 · 2025: 19 · 2024: 16 · 2023: 17 · 2022: 21 · 2021: 14 · 2020: 2
- FWCI
- 14.07
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- International Conference on Mining Software Repositories (MSR) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- memory efficiency, static analysis, integrated development environment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
