CoChat

An Empirical Study on the Usage of Transformer Models for Code Completion

Open access
Ciniselli, Matteo, Cooper, Nathan, Pascarella, Luca, Mastropaolo, Antonio, Aghajani, Emad, Poshyvanyk, Denys, Di Penta, Massimiliano, Bavota, Gabriele · 2021 · IEEE Transactions on Software Engineering
Cited by 94

The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.

Abstract

Code completion aims at speeding up code writing by predicting the next code token(s) the developer is likely to write. Works in this field focused on improving the accuracy of the generated predictions, with substantial leaps forward made possible by deep learning (DL) models. However, code completion techniques are mostly evaluated in the scenario of predicting the next token to type, with few exceptions pushing the boundaries to the prediction of an entire code statement. Thus, little is known about the performance of state-of-the-art code completion approaches in more challenging scenarios in which, for example, an entire code block must be generated. We present a large-scale study exploring the capabilities of state-of-the-art Transformer-based models in supporting code completion at different granularity levels, including single tokens, one or multiple entire statements, up to entire code blocks (e.g., the iterated block of a for loop). We experimented with several variants of two recently proposed Transformer-based models, namely RoBERTa and the Text-To-Text Transfer Transformer (T5), for the task of code completion. The achieved results show that Transformer-based models, and in particular the T5, represent a viable solution for code completion, with perfect predictions ranging from ~29%, obtained when asking the model to guess entire blocks, up to ~69%, reached in the simpler scenario of few tokens masked from the same code statement.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tse.2021.3128234

Cites

22 of 22 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 22 references

Research works printed but not linked: 58

  1. 10.1109/ICSE.2009.5070525 · DOI · Paper page
  2. 10.1109/ASE.2008.43 · DOI · Paper page
  3. 10.1145/1321631.1321663 · DOI · Paper page
  4. 10.1109/ASE.2004.1342740 · DOI
  5. 10.1109/ICSE.2019.00021 · DOI · Paper page
  6. 10.1109/ASE.2009.64 · DOI · Paper page
  7. 10.1145/2491956.2462192 · DOI · Paper page
  8. 10.1145/2950290.2950334 · DOI · Paper page
  9. grissom, 2005, Effect Sizes for Research A Broad Practical Approach
  10. 10.1007/s10515-011-0083-2 · DOI
  11. 10.1109/ASE.2009.64 · DOI · Paper page
  12. 10.1109/TSE.2011.84 · DOI · Paper page
  13. nguyen, 2016, A Large-Scale Study on Repetitiveness, Containment, and Composability of Routines in Open-Source Projects, 2016 IEEE/ACM 13th Conference on Mining Software Repositories (MSR), 362
  14. 10.1109/ICSE.2012.6227205 · DOI · Paper page
  15. 10.1145/3196398.3196460 · DOI
  16. nguyen, 2015, Graph-based statistical language model for code, Proceedings of the International Conference on Software Engineering ICSE'94, 858
  17. gage, 1994, A new algorithm for data compression, C Users J, 12, 23
  18. 10.1016/j.jss.2016.07.026 · DOI
  19. 10.1145/2744200 · DOI · Paper page
  20. 10.1145/1806799.1806868 · DOI
  21. 10.1109/MSR.2013.6624031 · DOI
  22. 0, Hugging Face’s Tokenizer Repositor
  23. 10.1109/TSE.2019.2940179 · DOI · Paper page
  24. 10.1109/MSR52588.2021.00024 · DOI · Paper page
  25. 10.1109/MSR52588.2021.00074 · DOI
  26. dreyer, 2012, HyTER: Meaning-equivalent semantics for translation evaluation, Proc Conf North Amer Chapter Assoc Comput Linguistics Hum Lang Technol, 162
  27. 10.18653/v1/D18-2012 · DOI · Paper page
  28. levenshtein, 1966, Binary codes capable of correcting deletions, insertions and reversals, Sov Phys Doklady, 10, 707
  29. 10.1109/ICSE43902.2021.00041 · DOI · Paper page
  30. 10.1145/3196398.3196450 · DOI
  31. 10.1145/1065010.1065018 · DOI · Paper page
  32. 10.1109/TSE.2019.2924886 · DOI
  33. 10.1109/TSE.2020.3018481 · DOI
  34. 10.24963/ijcai.2018/578 · DOI · Paper page
  35. 10.1109/ICSME.2014.110 · DOI
  36. 10.1109/ICSM.2011.6080790 · DOI · Paper page
  37. 10.1109/ICSME.2017.80 · doi:10.1109/icsme.2017.80
  38. 10.1145/3360585 · DOI · Paper page
  39. 10.1145/1176617.1176671 · DOI · Paper page
  40. 10.1145/3377811.3380429 · DOI · Paper page
  41. vaswani, 2017, Attention is all you need, Proc Adv Neural Inf Process Syst, 6000
  42. 10.1109/MSR.2015.38 · DOI · Paper page
  43. 10.1145/1595696.1595728 · DOI · Paper page
  44. 10.1109/ICSE43902.2021.00025 · DOI
  45. 10.1109/TSE.2021.3074309 · DOI
  46. 0, ScrML Website
  47. 0, N-gram Cached Model
  48. 10.1007/978-0-387-32833-1_11 · DOI
  49. 0, Weights and Biases Website
  50. 10.2307/3001968 · DOI · Paper page
  51. alon, 2020, Structural language models of code, Proc Int Conf Mach Learn, 245
  52. allamanis, 0, CodeSearchNet Deduplication Algorithm
  53. 10.1109/ICSE.2012.6227136 · DOI
  54. 10.1145/3290353 · DOI · Paper page
  55. 10.1093/biomet/30.1-2.81 · DOI · Paper page
  56. 10.1109/ASE.2019.00062 · DOI
  57. 10.1145/3196398.3196474 · DOI
  58. 10.1109/ASE.2017.8115626 · DOI · Paper page

Cited by

See all 94

Metadata

Authors
Ciniselli, M., Cooper, N., Pascarella, L., Mastropaolo, A., Aghajani, E., Poshyvanyk, D., Di Penta, M., & Bavota, G.
Venue
IEEE Transactions on Software Engineering
Published
2021
Pages
1-1
ISSN
  • 0098-5589 (ISSN-L)
  • 1939-3520
  • 2326-3881
OpenAlex
W3189653670
Cited by
94
References
22
FWCI
14.41
Type
journal article
Language
en
Open access
green
Funders
H2020 European Research Council (851720)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: An Empirical Study on the Usage of Transformer Models for Code Completion. https://app.cochat.ai/papers/10.1109/tse.2021.3128234
The paper
Ciniselli, M., Cooper, N., Pascarella, L., Mastropaolo, A., Aghajani, E., Poshyvanyk, D., Di Penta, M., & Bavota, G. (2021). An Empirical Study on the Usage of Transformer Models for Code Completion. IEEE Transactions on Software Engineering. https://doi.org/10.1109/tse.2021.3128234

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.