Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index; 19 of the 23 sources its claims cite were checked for support.

Productivity assessment of neural code completion

Open access
Ziegler, Albert, Kalliamvakou, Eirini, Li, X. Alice, Rice, Andrew, Rifkin, Devon, Simister, Shawn, Sittampalam, Ganesh, Aftandilian, Edward · 2022 · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming

Abstract

Neural code synthesis has reached a point where snippet generation is accurate enough to be considered for integration into human software development workflows. Commercial products aim to increase programmers’ productivity, without being able to measure it directly. In this case study, we asked users of GitHub Copilot about its impact on their productivity, and sought to find a reflection of their perception in directly measurable user data. We find that the rate with which shown suggestions are accepted, rather than more specific metrics regarding the persistence of completions in the code over time, drives developers’ perception of productivity.

Overview

CoChat Summary

TL;DR

In a case study of GitHub Copilot users, acceptance rate of shown suggestions was the usage metric most positively correlated with self-reported productivity, outperforming persistence-based measures. The authors conclude acceptance rate is useful for coarse-grained monitoring but not a singular quality criterion.

Key findings

  • Acceptance rate (`accepted_per_shown`) best predicted aggregate perceived productivity: ρ = 0.24, P < 0.0001; all persistence measures were less well correlated. A PLS regression’s first component explained 43.2% of variance, and the second captured the acceptance-rate/change-rate dichotomy, explaining a further 13.1%.
  • In this study GitHub Copilot had an acceptance rate of 27% and a mean DCPU in excess of 31. Common languages were TypeScript (24.7% of shown completions, 21.9% for survey users), JavaScript (21.3%, 24.2%), and Python (14.1%, 14.5%); the latter two had higher acceptance rates.
  • Acceptance rate varied by time regime: weekends averaged 23.5%, weekday non-working hours 23%, and typical working hours 21.2%, all statistically significantly distinct at p < 0.001% by bootstrap re-sampling. A model found the actual time bucket mattered little, but whether it lay in the user’s usual time regime did matter.
  • Incremental feature selection showed that after acceptance rate, adding `shown_per_hour`, `accepted_char_per_hour`, `shown_per_opportunity`, or `accepted_per_hour` each further improved prediction at statistically significant levels (p < 0.05).

Methods

  • Case study pairing a productivity survey with IDE telemetry for GitHub Copilot, which uses OpenAI Codex. The authors emailed 17,420 users and received 2,047 responses they could match to usage measurements during the 4-week period leading up to February 12–March 2022; the Introduction reports analyzing 2,631 survey responses.
  • Survey used the SPACE framework (Satisfaction, Performance, Communication, Efficiency; Activity excluded), with 11 Likert-style statements plus one overall productivity statement, encoded 1–5, and an aggregate productivity score calculated as the mean of 12 individual measures.
  • Usage measurements included shown, accepted, accepted characters, unchanged and mostly unchanged completions at 30/120/300/600 seconds, and active hours, normalized into metrics such as `accepted_per_shown` and `accepted_per_hour`. Analysis used Pearson’s R and p-values, PLS regression, and incremental feature selection predicting residuals of models fit with other metrics.

Limitations

  • Stated: the work investigates correlation, not causation; the best metric has a Pearson coefficient of 0.24, leaving considerable unexplained variance; perceived productivity is not necessarily actual productivity; only a single completion system with a single fixed neural engine was considered; telemetry raises security and privacy concerns; and optimizing acceptance rate can bias toward popular use cases or be gamed by splitting suggestions.
  • Evident from the text: the survey sample came from unpaid technical-preview users who had opted in to communications, and the paper reports different matched-response counts in different sections (2,047 matched responses in Methods vs. 2,631 survey responses in the Introduction).

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3520312.3534864

Cites

19 of 19 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 GitHub Copilot's code completion funnel.
Figure 1 GitHub Copilot's code completion funnel.CC-BY · From the PDF
Figure 3 Correlation between metrics.
Figure 3 Correlation between metrics.CC-BY · From the PDF
Figure 4 Different metrics clustering in latent structures predicting perceived productivity. We color the following groups: fl awless suggestions (anything counting the number of unchanged suggestions), persistence rate (ratio of accepted suggestions that are unchanged), and fuzzy persistence rate (ratio of accepted suggestions that are mostly unchanged).
Figure 4 Different metrics clustering in latent structures predicting perceived productivity. We color the following groups: fl awless suggestions (anything counting the number of unchanged suggestions), persistence rate (ratio of accepted suggestions that are unchanged), and fuzzy persistence rate (ratio of accepted suggestions that are mostly unchanged).CC-BY · From the PDF
Figure 5 Programming language use by survey participants vs. all users.
Figure 5 Programming language use by survey participants vs. all users.CC-BY · From the PDF
Figure 6 Average acceptance rate for hour-long time buckets during the week. Each point represents the average for such a bucket, whereas the shaded ribbon represents the min-max variation for single hours during the observed 4 week period.
Figure 6 Average acceptance rate for hour-long time buckets during the week. Each point represents the average for such a bucket, whereas the shaded ribbon represents the min-max variation for single hours during the observed 4 week period.CC-BY · From the PDF
Figure 7 Acceptance rate depending on whether the user is mostly active on weekdays / typical work hours (x-axis), and whether it is actually a weekday / typical office hour (color).
Figure 7 Acceptance rate depending on whether the user is mostly active on weekdays / typical work hours (x-axis), and whether it is actually a weekday / typical office hour (color).CC-BY · From the PDF

Verification

Claims the paper makes

16 Valid · 0 Not valid · 3 Not checked · 65 Own results and interpretations

25 more claims

Create a free account to see all 84 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 42
  • It may state two separate claims as one. 22
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

23 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 16 references

Research works printed but not linked: 1

  1. 10.1145/3491101.3519665 · DOI · Paper page

Cited by

See all 215

Metadata

Authors
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E.
Affiliations
GitHub, USA
Venue
Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming
Publisher
ACM
Published
2022-06-13
Pages
21-29
OpenAlex
W4281669078
Cited by
200
References
16
Citations by year
2026: 47 · 2025: 56 · 2024: 70 · 2023: 25 · 2022: 2
FWCI
34.20
Type
conference paper
Language
en
Open access
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
Microsoft Research
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
Keywords
GitHub Copilot, developer productivity, code completion, developer perceptions, code generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Productivity assessment of neural code completion. https://app.cochat.ai/papers/10.1145/3520312.3534864
The paper
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2022). Productivity assessment of neural code completion. Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming. https://doi.org/10.1145/3520312.3534864

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.