A Transformer-Based Approach for Smart Invocation of Automatic Code Completion
Open accessThe DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index; 27 of the 37 sources its claims cite were checked for support.
Abstract
Transformer-based language models are highly effective for code completion, with much research dedicated to enhancing the content of these completions. Despite their effectiveness, these models come with high operational costs and can be intrusive, especially when they suggest too often and interrupt developers who are concentrating on their work. Current research largely overlooks how these models interact with developers in practice and neglects to address when a developer should receive completion suggestions. To tackle this issue, we developed a machine learning model that can accurately predict when to invoke a code completion tool given the code context and available telemetry data. To do so, we collect a dataset of 200k developer interactions with our cross-IDE code completion plugin and train several invocation filtering models. Our results indicate that our small-scale transformer model significantly outperforms the baseline while maintaining low enough latency. We further explore the search space for integrating additional telemetry data into a pre-trained transformer directly and obtain promising results. To further demonstrate our approach’s practical potential, we deployed the model in an online environment with 34 developers and provided real-world insights based on 74k actual invocations.
Overview
CoChat Summary
## TL;DR
The paper proposes JonBERTa, a lightweight transformer-based filter to decide when a code-completion suggestion should be invoked, using code context and IDE telemetry. In offline and online evaluations, the best JonBERTa-head variant outperforms telemetry-only baselines and shows practical viability with low latency.
## Key findings
- On the main offline comparison, telemetry-only logistic regression reached 66.7% average accuracy, while adding textual code-context features raised it to 75.8%, and CodeBERTa using snippet features reached 82.1% average accuracy.
- In the hybrid model comparison, JonBERTa-head (dense) achieved the best reported offline average of 82.7%, compared with 82.1% for CodeBERTa and 82.0% for the JonBERTa-attn 0-layer variant.
- The tokenisation ablation favored the joint prefix-and-suffix strategy with 81.3% average accuracy, versus 77.8% for prefix-only and 68.8% for suffix-only.
- In the online deployment with 34 developers and 74k requests, JonBERTa-head showed the best harmonic-mean score (0.905) with 20.8 ms median latency, versus 0.864 for no filter and 0.609 for the logistic regression filter.
## Methods
- The study trains invocation-filtering classifiers to predict whether a completion should be shown, treating manual invocations and automatic accepted completions as positive, and automatic rejected completions as negative.
- Data come from the authors’ open-source Code4Me plugin: about 1M invocations were collected, filtered to about 200k high-quality samples, and then undersampled to about 10k training/validation samples; the test set contains about 20k samples.
- They compare a telemetry-and-text logistic regression baseline, a CodeBERTa transformer, and two transformer extensions: JonBERTa-head and JonBERTa-attn; offline evaluation uses per-class accuracy and macro average, with bootstrapping over n = 10,000 samples, and the online study uses acceptance rate, CodeBERTScore, harmonic mean, and latency.
## Limitations
- The paper states that the training data are limited to 10k samples for RQ1/RQ2, which is “not optimal” for fully examining hybrid transformer models and increases overfitting risk.
- It also states the plugin has a smaller user base than large production systems, so generalisability to other code-completion tools is limited.
- The authors note their proxy online metric may not fully capture usability, and they explicitly recommend future qualitative studies; they also cannot share the dataset because of GDPR/privacy constraints.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
15 Valid · 7 Not valid · 7 Not checked · 57 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“the frequency and timing of these suggestions is critical for the overall productivity the tools aim to boost [34].”
What the sources say
- Supportedcites [34] [34] James Prather, Brent N. Reeves, Paul Denny, Brett A. Becker, Juho Leinonen, Andrew Luxton-Reilly, Garrett Powell, James Finnie-Ansley, and Eddie Antonio Santos. 2023. "It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers. (April 2023). https://doi.org/10.48550/ ARXIV.2304.02491 Publisher: arXiv Version Number: 1. -
Section 4.1.2 reports that a constant stream of suggestions distracted novice programmers and interrupted their problem-solving. Section 5.4.1 says suggestions shown when students did not need help slowed them down, and recommends adjusting how and when suggestions appear. These passages support the claim about suggestion frequency and timing affecting the productivity the tool aims to improve, in the paper’s novice-programmer setting. Page numbers are not available in our copy. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“CodeBERTScore is a recently-proposed measure that correlates closest with both functional correctness and human preference [47].”
What the sources say
- Supportedcites [47] [47] Shuyan Zhou, Uri Alon, Sumit Agarwal, and Graham Neubig. 2023. CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code. http: //arxiv.org/abs/2302.05527 arXiv:2302.05527 [cs]. - source
The Correlation with human preference section reports that CodeBERTScore achieves the highest correlation with human preference among the compared metrics. The same section reports the highest or comparable correlation with functional correctness across four languages, and the highest average correlation across languages relative to the baselines. This supports the claim in the paper’s evaluated settings. (LLM, conf=0.93)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“For any incoming completion prompt, this library waits about 10ms for to see if it can be batched together with other requests.”
What the sources say
- Could not verifycites [23] [23] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. http://arxiv.org/abs/2309.06180 arXiv:2309.06180 [cs]. - sourceExperimental — accuracy not yet measured
Section 4.3 (pp. 5–6) discusses batch sizes for RoBERTa and BERT training, not batching incoming completion requests. Searches for vLLM, a 10 ms wait, and inference-request batching found no passage addressing the claim. (LLM, conf=0.76)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Moreover, both of these tools aim to maximise the acceptance rate of the shown completions”
What the sources say
- Could not verifycites [32] [32] Hussein Mozannar, Gagan Bansal, Adam Fourney, and Eric Horvitz. 2023. When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming. http://arxiv.org/abs/2306.04930 arXiv:2306.04930 [cs]. - sourceExperimental — accuracy not yet measured
The Single line completion section evaluates Code Llama’s completion accuracy using exact match and BLEU, not whether users accept shown completions. Searches for “acceptance rate,” “Sun et al.,” and “Mozannar” found no passage in our copy addressing the stated aim of the two tools. (LLM, conf=0.76)
- Not checked — no full text of the sourcecites [39] [39] Zhensu Sun, Xiaoning Du, Fu Song, Shangwen Wang, Mingze Ni, and Li Li. 2023. Don't Complete It! Preventing Unhelpful Code Completion for Productive and Sustainable Neural Code Completion Systems. In 2023 IEEE/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSECompanion) . IEEE, Melbourne, Australia, 324-325. https://doi.org/10.1109/ICSECompanion58688.2023.00089 - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Specifically, optimising only acceptance rate results in worse completions [32, 49].”
What the sources say
- Could not verifycites [32] [32] Hussein Mozannar, Gagan Bansal, Adam Fourney, and Eric Horvitz. 2023. When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming. http://arxiv.org/abs/2306.04930 arXiv:2306.04930 [cs]. - sourceExperimental — accuracy not yet measured
The Single line completion section reports that long-context fine-tuning slightly worsens performance on short code completions, but does not connect that result to acceptance-rate optimization. Searches for “acceptance rate,” “acceptance,” and user acceptance of suggested completions found no passage addressing the claimed trade-off. (LLM, conf=0.74)
- Partly supportedcites [49] [49] Albert Ziegler, Eirini Kalliamvakou, Shawn Simister, Ganesh Sittampalam, Alice Li, Andrew Rice, Devon Rifkin, and Edward Aftandilian. 2022. Productivity Assessment of Neural Code Completion. http://arxiv.org/abs/2205.06537 arXiv:2205.06537 [cs]. sourceExperimental — accuracy not yet measured
Section 8 (Broader Impact; page not provided) warns that optimizing acceptance rate alone can encourage changes that improve the metric without improving usefulness, and might even decrease user benefit. It does not establish that doing so necessarily results in worse completions; the citing sentence turns a possible outcome into a categorical result. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
29 more claims · 2 with their cited sources checked
Create a free account to see all 86 claims and to verify the 6 references still open.
- It may state two separate claims as one. 21
- Probably not a substantial claim. 32
- The quote only partly states this claim. 3
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023) · arXiv · doi:10.48550/arxiv.2309.06180 · Cited by 1,802
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
UniXcoder: Unified Cross-Modal Pre-training for Code Representation
Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., & Yin, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.499 · Cited by 673
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.04664 · Cited by 536
Code Llama: Open Foundation Models for Code
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12950 · Cited by 531 · cited 1×
“Recent advancements have expanded the considered context to include not only the subsequent code [3, 15, 19, 26, 35], but also related snippets from other files to enrich the prediction accuracy [27, 45].”
BackgroundCodeT5+: Open Code Large Language Models for Code Understanding and Generation
Wang, Y., Le, H., Gotmare, A. D., Bui, N. D. Q., Li, J., & Hoi, S. C. H. (2023) · arXiv · doi:10.48550/arxiv.2305.07922 · Cited by 459
CodeT5+: Open Code Large Language Models for Code Understanding and Generation
Wang, Y., Le, H., Gotmare, A., Bui, N., Li, J., & Hoi, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.68 · Cited by 459
Cited by
AI-driven innovations in software engineering: a review of current practices and future directions
Alenezi, M., & Akour, M. (2025) · Applied Sciences · doi:10.3390/app15031344 · Cited by 121
Human-AI experience in integrated development environments: a systematic literature review
Sergeyuk, A., Zakharov, I., Koshchenko, E., & Izadi, M. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10793-0 · Cited by 49
Self-citationCan an LLM Find Its Way Around a Spreadsheet?
Lee, C.-T., Neeser, A., Xu, S., Katyan, J., Cross, P., Pathakota, S., Norman, M., Simeone, J., Chandrasekaran, J., & Ramakrishnan, N. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00101 · Cited by 3
Awad, M. N. A., Ivanov, S. V., & Tikhonova, O. B. (2025) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew67777.2025.00032 · Cited by 2
Mundhra, Y., Valk, M., & Izadi, M. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00287 · Cited by 3
Self-citation
Metadata
- Authors
- de Moor, A., van Deursen, A., & Izadi, M.
- Affiliations
- Delft University of Technology, Delft, Netherlands
- Delft University of Technology (NL)
- Venue
- Proceedings of the 1st ACM International Conference on AI-Powered Software
- Publisher
- ACM
- Published
- 2024-07-10
- Pages
- 28-37
- OpenAlex
- W4398859672
- Semantic Scholar
- 4a4917ad9af7f8d352b2ce80afd2527ac7849978
- Cited by
- 7
- References
- 19
- Citations by year
- 2026: 2 · 2025: 5
- FWCI
- 2.66
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on AI-Powered Software (AIware) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- developer interaction, telemetry data, transformer models, developer productivity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Transformer-Based Approach for Smart Invocation of Automatic Code Completion. https://app.cochat.ai/papers/10.1145/3664646.3664760
- The paper
- de Moor, A., van Deursen, A., & Izadi, M. (2024). A Transformer-Based Approach for Smart Invocation of Automatic Code Completion. Proceedings of the 1st ACM International Conference on AI-Powered Software. https://doi.org/10.1145/3664646.3664760
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.