An Empirical Study on the Usage of Transformer Models for Code Completion
Open accessThe DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Abstract
Code completion aims at speeding up code writing by predicting the next code token(s) the developer is likely to write. Works in this field focused on improving the accuracy of the generated predictions, with substantial leaps forward made possible by deep learning (DL) models. However, code completion techniques are mostly evaluated in the scenario of predicting the next token to type, with few exceptions pushing the boundaries to the prediction of an entire code statement. Thus, little is known about the performance of state-of-the-art code completion approaches in more challenging scenarios in which, for example, an entire code block must be generated. We present a large-scale study exploring the capabilities of state-of-the-art Transformer-based models in supporting code completion at different granularity levels, including single tokens, one or multiple entire statements, up to entire code blocks (e.g., the iterated block of a for loop). We experimented with several variants of two recently proposed Transformer-based models, namely RoBERTa and the Text-To-Text Transfer Transformer (T5), for the task of code completion. The achieved results show that Transformer-based models, and in particular the T5, represent a viable solution for code completion, with perfect predictions ranging from ~29%, obtained when asking the model to guess entire blocks, up to ~69%, reached in the simpler scenario of few tokens masked from the same code statement.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages
McNemar, Q. (1947) · Psychometrika · doi:10.1007/bf02295996 · Cited by 4,754
Code completion with statistical language models
Raychev, V., Vechev, M., & Yahav, E. (2014) · Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2594291.2594321 · Cited by 555
CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
Husain, H., Wu, H.-H., Gazit, T., Allamanis, M., & Brockschmidt, M. (2019) · arXiv · doi:10.48550/arxiv.1909.09436 · Cited by 483 · cited 1×
“The finetuning dataset has been built starting from the CodeSearchNet dataset [41], which features Java methods mined from open source projects.”
MethodIntelliCode compose: code generation using transformer
Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417058 · Cited by 474 · cited 3×
“Last, but not least, Deep Learning (DL) models have been applied to code completion [8], [22], [47], [49], [73], [84], setting new standards in terms of prediction performance.”
On the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Devanbu, P. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227135 · Cited by 425
Research works printed but not linked: 58
- 10.1109/ICSE.2009.5070525 · DOI · Paper page
- 10.1109/ASE.2008.43 · DOI · Paper page
- 10.1145/1321631.1321663 · DOI · Paper page
- 10.1109/ASE.2004.1342740 · DOI
- 10.1109/ICSE.2019.00021 · DOI · Paper page
- 10.1109/ASE.2009.64 · DOI · Paper page
- 10.1145/2491956.2462192 · DOI · Paper page
- 10.1145/2950290.2950334 · DOI · Paper page
- grissom, 2005, Effect Sizes for Research A Broad Practical Approach
- 10.1007/s10515-011-0083-2 · DOI
- 10.1109/ASE.2009.64 · DOI · Paper page
- 10.1109/TSE.2011.84 · DOI · Paper page
- nguyen, 2016, A Large-Scale Study on Repetitiveness, Containment, and Composability of Routines in Open-Source Projects, 2016 IEEE/ACM 13th Conference on Mining Software Repositories (MSR), 362
- 10.1109/ICSE.2012.6227205 · DOI · Paper page
- 10.1145/3196398.3196460 · DOI
- nguyen, 2015, Graph-based statistical language model for code, Proceedings of the International Conference on Software Engineering ICSE'94, 858
- gage, 1994, A new algorithm for data compression, C Users J, 12, 23
- 10.1016/j.jss.2016.07.026 · DOI
- 10.1145/2744200 · DOI · Paper page
- 10.1145/1806799.1806868 · DOI
- 10.1109/MSR.2013.6624031 · DOI
- 0, Hugging Face’s Tokenizer Repositor
- 10.1109/TSE.2019.2940179 · DOI · Paper page
- 10.1109/MSR52588.2021.00024 · DOI · Paper page
- 10.1109/MSR52588.2021.00074 · DOI
- dreyer, 2012, HyTER: Meaning-equivalent semantics for translation evaluation, Proc Conf North Amer Chapter Assoc Comput Linguistics Hum Lang Technol, 162
- 10.18653/v1/D18-2012 · DOI · Paper page
- levenshtein, 1966, Binary codes capable of correcting deletions, insertions and reversals, Sov Phys Doklady, 10, 707
- 10.1109/ICSE43902.2021.00041 · DOI · Paper page
- 10.1145/3196398.3196450 · DOI
- 10.1145/1065010.1065018 · DOI · Paper page
- 10.1109/TSE.2019.2924886 · DOI
- 10.1109/TSE.2020.3018481 · DOI
- 10.24963/ijcai.2018/578 · DOI · Paper page
- 10.1109/ICSME.2014.110 · DOI
- 10.1109/ICSM.2011.6080790 · DOI · Paper page
- 10.1109/ICSME.2017.80 · doi:10.1109/icsme.2017.80
- 10.1145/3360585 · DOI · Paper page
- 10.1145/1176617.1176671 · DOI · Paper page
- 10.1145/3377811.3380429 · DOI · Paper page
- vaswani, 2017, Attention is all you need, Proc Adv Neural Inf Process Syst, 6000
- 10.1109/MSR.2015.38 · DOI · Paper page
- 10.1145/1595696.1595728 · DOI · Paper page
- 10.1109/ICSE43902.2021.00025 · DOI
- 10.1109/TSE.2021.3074309 · DOI
- 0, ScrML Website
- 0, N-gram Cached Model
- 10.1007/978-0-387-32833-1_11 · DOI
- 0, Weights and Biases Website
- 10.2307/3001968 · DOI · Paper page
- alon, 2020, Structural language models of code, Proc Int Conf Mach Learn, 245
- allamanis, 0, CodeSearchNet Deduplication Algorithm
- 10.1109/ICSE.2012.6227136 · DOI
- 10.1145/3290353 · DOI · Paper page
- 10.1093/biomet/30.1-2.81 · DOI · Paper page
- 10.1109/ASE.2019.00062 · DOI
- 10.1145/3196398.3196474 · DOI
- 10.1109/ASE.2017.8115626 · DOI · Paper page
Cited by
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695988 · Cited by 1,029
Large language models for code completion: A systematic literature review
Husein, R. A., Aburajouh, H., & Catal, C. (2025) · Computer Standards & Interfaces · doi:10.1016/j.csi.2024.103917 · Cited by 72
Unveiling ChatGPT’s usage in open source projects: A mining-based study
Tufano, R., Mastropaolo, A., Pepe, F., Dabic, O., Di Penta, M., & Bavota, G. (2024) · Proceedings of the 21st International Conference on Mining Software Repositories · doi:10.1145/3643991.3644918 · Cited by 38
Self-citationNon-Autoregressive Line-Level Code Completion
Liu, F., Fu, Z., Li, G., Jin, Z., Liu, H., Hao, Y., & Zhang, L. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3649594 · Cited by 22
Toward a Theory of Causation for Interpreting Neural Code Models
Nader Palacio, D., Velasco, A., Cooper, N., Rodriguez, A., Moran, K., & Poshyvanyk, D. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3379943 · Cited by 18
Self-citation
Metadata
- Authors
- Ciniselli, M., Cooper, N., Pascarella, L., Mastropaolo, A., Aghajani, E., Poshyvanyk, D., Di Penta, M., & Bavota, G.
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2021
- Pages
- 1-1
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W3189653670
- Cited by
- 94
- References
- 22
- FWCI
- 14.41
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- Funders
- H2020 European Research Council (851720)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: An Empirical Study on the Usage of Transformer Models for Code Completion. https://app.cochat.ai/papers/10.1109/tse.2021.3128234
- The paper
- Ciniselli, M., Cooper, N., Pascarella, L., Mastropaolo, A., Aghajani, E., Poshyvanyk, D., Di Penta, M., & Bavota, G. (2021). An Empirical Study on the Usage of Transformer Models for Code Completion. IEEE Transactions on Software Engineering. https://doi.org/10.1109/tse.2021.3128234
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.