CoChat

Measuring GitHub Copilot's Impact on Productivity

Open access
Ziegler, Albert, Kalliamvakou, Eirini, Li, X. Alice, Rice, Andrew, Rifkin, Devon, Simister, Shawn, Sittampalam, Ganesh, Aftandilian, Edward · 2024 · Communications of the ACM

The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index; 8 of the 17 sources its claims cite were checked for support.

Abstract

Case study asks Copilot users about its impact on their productivity, and seeks to find their perceptions mirrored in user data.

Overview

CoChat Summary

TL;DR

This study links GitHub Copilot IDE usage telemetry to developers’ self-reported productivity and finds that acceptance rate of shown suggestions is the best usage predictor of perceived productivity. Acceptance rate correlates better than persistence or contribution-volume measures, though it leaves substantial unexplained variance and varies by experience, language, and time.

Key findings

  • Acceptance rate (`accepted_per_shown`) most positively predicts perceived productivity and correlates best with aggregate productivity (ρ = 0.24, P < 0.0001); it is the best-performing measurement for at least one survey question in each SPACE dimension.
  • Persistence measures (unchanged/mostly unchanged after 30, 120, 300, 600 seconds) were less well correlated than acceptance rate; PLS first component explains 43.2% of variance, second 13.1%, both drawing most strongly from acceptance rate.
  • GitHub Copilot had an acceptance rate of 27% and mean DCPU in excess of 312; acceptance rate varied by subgroup and time (weekend 23.5%, typical weekday working hours 21.2%, off hours 23%).
  • Junior developers reported higher productivity gains and tended to accept more suggestions; both proficiency and years of experience negatively correlated with aggregate productivity (ρ = -0.095, P = 0.0001; ρ = -0.161, P < 0.0001), though effects were non-significant in multivariate regressions controlling for demographics.

Methods

  • Survey emailed to 17,420 unpaid technical preview users; the abstract reports 2,631 survey responses, while the methodology reports 2,047 responses matched to usage data from the four-week study period ending around Feb 12, 2022.
  • Usage telemetry from the GitHub Copilot IDE extension: opportunity, shown, accepted, accepted characters, unchanged/mostly unchanged after 30/120/300/600 seconds, active hour; survey used the SPACE framework with 11 statements plus one overall productivity statement, aggregated into a mean productivity score.
  • Analysis: Pearson correlations between each usage metric and perceived productivity, PLS regression across all usage metrics, and linear regressions with experience covariates.

Limitations

  • Stated: notable unexplained variance remains due to confounding and human factors; other approaches are necessary for fine-grained investigation; using PST for U.S. users introduces noise from time-zone inhomogeneity.
  • Evident: outcome is self-reported perceived productivity, not objective productivity; sample is unpaid technical preview users who opted into communications, limiting generalizability; observational correlations cannot establish causation.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3633453

Cites

29 of 29 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

4 Valid · 6 Not valid · 7 Not checked · 47 Own results and interpretations

28 more claims

Create a free account to see all 64 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 24
  • It may state two separate claims as one. 10
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

5 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 29 references

Research works printed but not linked: 1

  1. Vaithilingam, P., Zhang, T., and Glassman, E. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In Proceedings of the 2022 Conf. on Human Factors in Computing Systems.

Cited by

Items 101–126 of 126

Metadata

Authors
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E.
Affiliations
  • GitHub, Inc., San Francisco, CA, USA
  • GitHub, San Francisco, CA, USA
Venue
Communications of the ACM
Published
2024-02-22
Volume
67
Issue
3
Pages
54-63
ISSN
  • 0001-0782 (ISSN-L)
  • 1557-7317
OpenAlex
W4392023437
Cited by
124
References
26
Citations by year
2026: 58 · 2025: 53 · 2024: 13
FWCI
46.42
Type
journal article
Language
en
Open access
Locations
Communications of the ACM (publishedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
Keywords
GitHub Copilot, AI-assisted programming, developer productivity, user perception, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.