An Empirical Study on the Usage of BERT Models for Code Completion
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
Code completion is one of the main features of modern Integrated Development Environments (IDEs). Its objective is to speed up code writing by predicting the next code token(s) the developer is likely to write. Research in this area has substantially bolstered the predictive performance of these techniques. However, the support to developers is still limited to the prediction of the next few tokens to type. In this work, we take a step further in this direction by presenting a large-scale empirical study aimed at exploring the capabilities of state-of-the-art deep learning (DL) models in supporting code completion at different granularity levels, including single tokens, one or multiple entire statements, up to entire code blocks (e.g., the iterated block of a for loop). To this aim, we train and test several adapted variants of the recently proposed RoBERTa model, and evaluate its predictions from several perspectives, including: (i) metrics usually adopted when assessing DL generative models (i.e., BLEU score and Levenshtein distance); (ii) the percentage of perfect predictions (i.e., the predicted code snippets that match those written by developers); and (iii) the "semantic" equivalence of the generated code as compared to the one written by developers. The achieved results show that BERT models represent a viable solution for code completion, with perfect predictions ranging from ~7%, obtained when asking the model to guess entire blocks, up to ~58%, reached in the simpler scenario of few tokens masked from the same code statement.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages
McNemar, Q. (1947) · Psychometrika · doi:10.1007/bf02295996 · Cited by 4,754
Code completion with statistical language models
Raychev, V., Vechev, M., & Yahav, E. (2014) · Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2594291.2594321 · Cited by 555
CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
Husain, H., Wu, H.-H., Gazit, T., Allamanis, M., & Brockschmidt, M. (2019) · arXiv · doi:10.48550/arxiv.1909.09436 · Cited by 483
IntelliCode compose: code generation using transformer
Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417058 · Cited by 474
On the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Devanbu, P. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227135 · Cited by 425
Research works printed but not linked: 35
- 0, SrcML Website
- 10.1145/3196398.3196460 · DOI
- allamanis, 0, CodeSearchNet Deduplication Algorithm
- 10.1145/3377811.3380429 · DOI · Paper page
- 10.1109/ASE.2017.8115626 · DOI · Paper page
- 10.1145/2950290.2950334 · DOI · Paper page
- 0, Weights and Biases Website
- 10.1109/ICSE.2019.00021 · DOI · Paper page
- 10.1145/1595696.1595728 · DOI · Paper page
- levenshtein, 1966, Binary Codes Capable of Correcting Deletions, Insertions and Reversals, Soviet Physics Doklady, 10, 707
- vaswani, 2017, Attention is all you need, Advances in Neural IInformation Processing Systems, 5998
- wolf, 2019, Huggingface’s transformers: State-of-the-art natural language processing, ArXiv
- 0, Hugging Face’s Tokenizer Repositor
- gage, 1994, A new algorithm for data compression, C Users J, 12, 23
- 10.1109/ASE.2008.43 · DOI · Paper page
- 10.1145/1806799.1806868 · DOI
- 10.1145/2744200 · DOI · Paper page
- 10.1109/ICSE.2012.6227136 · DOI
- 10.1145/2491956.2462192 · DOI · Paper page
- 10.1145/1065010.1065018 · DOI · Paper page
- 10.1109/TSE.2011.84 · DOI · Paper page
- nguyen, 2016, A Large-Scale Study on Repetitiveness, Containment, and Composability of Routines in Open-Source Projects, 2016 IEEE/ACM 13th Conference on Mining Software Repositories (MSR), 362
- 10.1016/j.jss.2016.07.026 · DOI
- 0, N-gram Cached Model
- karampatsis, 2019, Maybe deep neural networks are the best choice for modeling source code, CoRR
- 10.1109/MSR.2015.38 · DOI · Paper page
- 10.1109/ASE.2009.64 · DOI · Paper page
- 10.1007/s10515-011-0083-2 · DOI
- 10.1109/ICSE.2012.6227205 · DOI · Paper page
- 10.1145/1321631.1321663 · DOI · Paper page
- 10.1109/ICSME.2014.110 · DOI
- 10.1145/3360585 · DOI · Paper page
- 10.1109/ICSE.2009.5070525 · DOI · Paper page
- 10.1145/1176617.1176671 · DOI · Paper page
- 10.1109/TSE.2019.2940179 · DOI · Paper page
Cited by
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Investigating Code Generation Performance of ChatGPT with Crowdsourcing Social Data
Feng, Y., Vanam, S., Cherukupally, M., Zheng, W., Qiu, M., & Chen, H. (2023) · 2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC) · doi:10.1109/compsac57700.2023.00117 · Cited by 117
Large language models for code completion: A systematic literature review
Husein, R. A., Aburajouh, H., & Catal, C. (2025) · Computer Standards & Interfaces · doi:10.1016/j.csi.2024.103917 · Cited by 72
Kuhail, M. A., Mathew, S. S., Khalil, A., Berengueres, J., & Shah, S. J. H. (2024) · Science of Computer Programming · doi:10.1016/j.scico.2024.103111 · Cited by 68
How Practitioners Expect Code Completion?
Wang, C., Hu, J., Gao, C., Jin, Y., Xie, T., Huang, H., Lei, Z., & Deng, Y. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3616280 · Cited by 50
Metadata
- Authors
- Ciniselli, M., Cooper, N., Pascarella, L., Poshyvanyk, D., Di Penta, M., & Bavota, G.
- Venue
- 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR)
- Publisher
- IEEE
- Published
- 2021-05
- Pages
- 108-119
- OpenAlex
- W3136309015
- Cited by
- 78
- References
- 20
- FWCI
- 12.03
- Type
- conference paper
- Language
- en
- Open access
- green
- Licence
- Funders
- European Research Council
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: An Empirical Study on the Usage of BERT Models for Code Completion. https://app.cochat.ai/papers/10.1109/msr52588.2021.00024
- The paper
- Ciniselli, M., Cooper, N., Pascarella, L., Poshyvanyk, D., Di Penta, M., & Bavota, G. (2021). An Empirical Study on the Usage of BERT Models for Code Completion. 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR). https://doi.org/10.1109/msr52588.2021.00024
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.