CoChat

A Transformer-Based Approach for Smart Invocation of Automatic Code Completion

Open access
de Moor, Aral, van Deursen, Arie, Izadi, Maliheh · 2024 · Proceedings of the 1st ACM International Conference on AI-Powered Software

The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index; 27 of the 37 sources its claims cite were checked for support.

Abstract

Transformer-based language models are highly effective for code completion, with much research dedicated to enhancing the content of these completions. Despite their effectiveness, these models come with high operational costs and can be intrusive, especially when they suggest too often and interrupt developers who are concentrating on their work. Current research largely overlooks how these models interact with developers in practice and neglects to address when a developer should receive completion suggestions. To tackle this issue, we developed a machine learning model that can accurately predict when to invoke a code completion tool given the code context and available telemetry data. To do so, we collect a dataset of 200k developer interactions with our cross-IDE code completion plugin and train several invocation filtering models. Our results indicate that our small-scale transformer model significantly outperforms the baseline while maintaining low enough latency. We further explore the search space for integrating additional telemetry data into a pre-trained transformer directly and obtain promising results. To further demonstrate our approach’s practical potential, we deployed the model in an online environment with 34 developers and provided real-world insights based on 74k actual invocations.

Overview

CoChat Summary

## TL;DR

The paper proposes JonBERTa, a lightweight transformer-based filter to decide when a code-completion suggestion should be invoked, using code context and IDE telemetry. In offline and online evaluations, the best JonBERTa-head variant outperforms telemetry-only baselines and shows practical viability with low latency.

## Key findings

  • On the main offline comparison, telemetry-only logistic regression reached 66.7% average accuracy, while adding textual code-context features raised it to 75.8%, and CodeBERTa using snippet features reached 82.1% average accuracy.
  • In the hybrid model comparison, JonBERTa-head (dense) achieved the best reported offline average of 82.7%, compared with 82.1% for CodeBERTa and 82.0% for the JonBERTa-attn 0-layer variant.
  • The tokenisation ablation favored the joint prefix-and-suffix strategy with 81.3% average accuracy, versus 77.8% for prefix-only and 68.8% for suffix-only.
  • In the online deployment with 34 developers and 74k requests, JonBERTa-head showed the best harmonic-mean score (0.905) with 20.8 ms median latency, versus 0.864 for no filter and 0.609 for the logistic regression filter.

## Methods

  • The study trains invocation-filtering classifiers to predict whether a completion should be shown, treating manual invocations and automatic accepted completions as positive, and automatic rejected completions as negative.
  • Data come from the authors’ open-source Code4Me plugin: about 1M invocations were collected, filtered to about 200k high-quality samples, and then undersampled to about 10k training/validation samples; the test set contains about 20k samples.
  • They compare a telemetry-and-text logistic regression baseline, a CodeBERTa transformer, and two transformer extensions: JonBERTa-head and JonBERTa-attn; offline evaluation uses per-class accuracy and macro average, with bootstrapping over n = 10,000 samples, and the online study uses acceptance rate, CodeBERTScore, harmonic mean, and latency.

## Limitations

  • The paper states that the training data are limited to 10k samples for RQ1/RQ2, which is “not optimal” for fully examining hybrid transformer models and increases overfitting risk.
  • It also states the plugin has a smaller user base than large production systems, so generalisability to other code-completion tools is limited.
  • The authors note their proxy online metric may not fully capture usability, and they explicitly recommend future qualitative studies; they also cannot share the dataset because of GDPR/privacy constraints.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3664646.3664760

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Reasons for Rejected Completions.
Figure 1 Reasons for Rejected Completions.From the PDF
Figure 2 Telemetry Feature Data in Classification Head.
Figure 2 Telemetry Feature Data in Classification Head.From the PDF
Figure 3 Self-Attention Extended to Telemetry Feature Data
Figure 3 Self-Attention Extended to Telemetry Feature DataFrom the PDF
Image 4
From the PDF
Figure 4 Copilot's Language Map. Higher-scoring languages are more likely to get a completion.
Figure 4 Copilot's Language Map. Higher-scoring languages are more likely to get a completion.From the PDF
Figure 6 Dataset Distributions
Figure 6 Dataset DistributionsFrom the PDF

Verification

Claims the paper makes

15 Valid · 7 Not valid · 7 Not checked · 57 Own results and interpretations

29 more claims · 2 with their cited sources checked

Create a free account to see all 86 claims and to verify the 6 references still open.

See all claims — free account
  • It may state two separate claims as one. 21
  • Probably not a substantial claim. 32
  • The quote only partly states this claim. 3

Create a free account to see these claims.

Citations

Cites

Show all 50 references

Cited by

See all 7

Metadata

Authors
de Moor, A., van Deursen, A., & Izadi, M.
Affiliations
  • Delft University of Technology, Delft, Netherlands
  • Delft University of Technology (NL)
Venue
Proceedings of the 1st ACM International Conference on AI-Powered Software
Publisher
ACM
Published
2024-07-10
Pages
28-37
OpenAlex
W4398859672
Cited by
7
References
19
Citations by year
2026: 2 · 2025: 5
FWCI
2.66
Type
conference paper
Language
en
Open access
Locations
  • ACM International Conference on AI-Powered Software (AIware) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
developer interaction, telemetry data, transformer models, developer productivity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A Transformer-Based Approach for Smart Invocation of Automatic Code Completion. https://app.cochat.ai/papers/10.1145/3664646.3664760
The paper
de Moor, A., van Deursen, A., & Izadi, M. (2024). A Transformer-Based Approach for Smart Invocation of Automatic Code Completion. Proceedings of the 1st ACM International Conference on AI-Powered Software. https://doi.org/10.1145/3664646.3664760

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.