Measuring GitHub Copilot's Impact on Productivity
Open accessThe DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index; 8 of the 17 sources its claims cite were checked for support.
Abstract
Case study asks Copilot users about its impact on their productivity, and seeks to find their perceptions mirrored in user data.
Overview
CoChat Summary
TL;DR
This study links GitHub Copilot IDE usage telemetry to developers’ self-reported productivity and finds that acceptance rate of shown suggestions is the best usage predictor of perceived productivity. Acceptance rate correlates better than persistence or contribution-volume measures, though it leaves substantial unexplained variance and varies by experience, language, and time.
Key findings
- Acceptance rate (`accepted_per_shown`) most positively predicts perceived productivity and correlates best with aggregate productivity (ρ = 0.24, P < 0.0001); it is the best-performing measurement for at least one survey question in each SPACE dimension.
- Persistence measures (unchanged/mostly unchanged after 30, 120, 300, 600 seconds) were less well correlated than acceptance rate; PLS first component explains 43.2% of variance, second 13.1%, both drawing most strongly from acceptance rate.
- GitHub Copilot had an acceptance rate of 27% and mean DCPU in excess of 312; acceptance rate varied by subgroup and time (weekend 23.5%, typical weekday working hours 21.2%, off hours 23%).
- Junior developers reported higher productivity gains and tended to accept more suggestions; both proficiency and years of experience negatively correlated with aggregate productivity (ρ = -0.095, P = 0.0001; ρ = -0.161, P < 0.0001), though effects were non-significant in multivariate regressions controlling for demographics.
Methods
- Survey emailed to 17,420 unpaid technical preview users; the abstract reports 2,631 survey responses, while the methodology reports 2,047 responses matched to usage data from the four-week study period ending around Feb 12, 2022.
- Usage telemetry from the GitHub Copilot IDE extension: opportunity, shown, accepted, accepted characters, unchanged/mostly unchanged after 30/120/300/600 seconds, active hour; survey used the SPACE framework with 11 statements plus one overall productivity statement, aggregated into a mean productivity score.
- Analysis: Pearson correlations between each usage metric and perceived productivity, PLS regression across all usage metrics, and linear regressions with experience covariates.
Limitations
- Stated: notable unexplained variance remains due to confounding and human factors; other approaches are necessary for fine-grained investigation; using PST for U.S. users introduces noise from time-zone inhomogeneity.
- Evident: outcome is self-reported perceived productivity, not objective productivity; sample is unpaid technical preview users who opted into communications, limiting generalizability; observational correlations cannot establish causation.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 6 Not valid · 7 Not checked · 47 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Offline evaluation, where the system is shown a partial snippet of code and then asked to complete it, is difficult not least because for longer completions there are many acceptable alternatives and no straightforward mechanism for labeling them automatically.5”
§ Case study asks Copilot users > Abstract
What the sources say
- Partly supportedcites [5] 5. Wagner, S. and Ruhe, M. A systematic review of productivity factors in software development. arXiv preprint arXiv:1801.06475 (2018). sourceExperimental — accuracy not yet measured
Section 2.1 (p. 2) says reference-matching metrics cannot account for the large space of functionally equivalent programs, supporting the point that many completions may be acceptable. It proposes unit tests for automatically evaluating docstring-based code generation, but does not establish that longer completions of partial code snippets lack a straightforward automatic labeling mechanism. (LLM, conf=0.83)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, proficiency positively predicts developers agreeing that Copilot helps them stay in the flow, focus on more satisfying work, spend less effort on repetitive tasks, and perform repetitive tasks faster.”
§ Case study asks Copilot users > Developer Productivity and the SPACE Framework
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All persistence measures were less well correlated than acceptance rate.”
§ Case study asks Copilot users > Productivity survey.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, in multivariate regressions predicting productivity from usage metrics while controlling for demographics, proficiency had a non-significant positive effect ( coeff = 0 .021 , P = 0 .213 ), while years of experience had a non-significant negative effect ( coeff = -0 .032 , P = 0 .122 ).”
§ Case study asks Copilot users > Developer Productivity and the SPACE Framework
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A vast majority of survey users (more than 80%) filled out the survey within the first two days, on or before February 12, 2022.”
§ Case study asks Copilot users > Productivity survey.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
28 more claims
Create a free account to see all 64 claims and every source verdict.
- Probably not a substantial claim. 24
- It may state two separate claims as one. 10
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Their findings suggest that the AI assistant had a significant positive impact on perceived productivity across multiple dimensions, e.g., task completion time, quality, cognitive load, enjoyment, and learning.”
From Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability (2026) · cites this paper as Ziegler et al. 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Ziegler et al. (2024) surveyed over 17,000 GitHub Copilot users and reported a large positive impact across different dimensions.”
From Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability (2026) · cites this paper as Ziegler et al. (2024)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One reason is that only the tool vendors have access to fine-grained usage data [4] that allows them to determine which code suggestions were accepted and hence which code was co-authored by GenAI tools.”
From Self-Admitted GenAI Usage in Open-Source Software (2026) · cites this paper as [4]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Based on surveys and usage telemetry, they found that developers perceived productivity improvements when using Copilot, particularly for repetitive and routine coding activities, with the perceived magnitude of improvement varying considerably based on task complexity and context.”
From Self-Admitted GenAI Usage in Open-Source Software (2026) · cites this paper as [4]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Further, GenAI tools can increase developers' perceived productivity [Ziegler et al. 2024] and level of understanding [Prather et al. 2024] without delivering actual improvements, potentially leading to a false sense of confidence [Lee et al. 2025] among novices.”
From How Do Programming Students Use Generative AI? (2025) · cites this paper as Ziegler et al. 2024
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Case study asks Copilot users > Developer Productivity and the SPACE Framework
“with options starting with "Student" and ranging from "0-2 years" to "16+ years" in twoyear intervals.”
§ Case study asks Copilot users > Abstract
“0-2 Years Prof. Experience 3-5 Years Prof. Experience 6-10 Years Prof. Experience 11-15 Years Prof. Experience 16+ Years Prof. Experience”
Citations
Cites
PLS-regression: a basic tool of chemometrics
Wold, S., Sjöström, M., & Eriksson, L. (2001) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/s0169-7439(01)00155-1 · Cited by 10,084 · cited 1×
“The choice of PLS, which captures the common variation of these variables as is linearly connected to the aggregate productivity, 28 is due to the high collinearity of the single metrics.”
BackgroundEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
IntelliCode compose: code generation using transformer
Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417058 · Cited by 474 · cited 1×
“20 An alternative measure is that of daily completions accepted per user (DCPU) for which a value of around 20 has been report-ed.”
BackgroundOn the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Devanbu, P. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227135 · Cited by 425 · cited 1×
“Code completion in IDEs using language models was first proposed in Hindle et al., 9 and today neural synthesis tools such as GitHub Copilot, CodeWhisperer, and TabNine suggest code snippets within an IDE with the explicitly stated intention to increase a user’s productivity. a Alternatively, we…”
MethodProgram Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388
The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023) · arXiv · doi:10.48550/arxiv.2302.06590 · Cited by 287
Self-citationWang, D., Churchill, E., Maes, P., Fan, X., Shneiderman, B., Shi, Y., & Wang, Q. (2020) · Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3334480.3381069 · Cited by 271
Measuring knowledge worker productivity: A taxonomy
Ramírez, Y. W., & Nembhard, D. A. (2004) · Journal of Intellectual Capital · doi:10.1108/14691930410567040 · Cited by 255
What makes a good conversation? How controllable attributes affect human judgments
See, A., Roller, S., Kiela, D., & Weston, J. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1170 · Cited by 247
Research works printed but not linked: 1
- Vaithilingam, P., Zhang, T., and Glassman, E. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In Proceedings of the 2022 Conf. on Human Factors in Computing Systems.
Cited by
Items 101–126 of 126
From RSE to AI4RSE: A Quadrant Model for AI-Augmented Research Software
Farshidi, S., Bennin, K. E., & Tekinerdoğan, B. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3786582.3786838
“Tools such as GitHub Copilot and large language models offer on-demand code generation, automated refactoring, and documentation synthesis [24, 25].”
Kerim, A., Meglić, F., Volarević, A., & Dejanović, D. (2026) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro70003.2026.11591660
“Studies report significant productivity gains, with developers completing repetitive tasks faster when assisted by AI tools [15].”
Huo, B., Li, F., Xin, S., et al. (2026) · Higher education and practice. · doi:10.62381/h261407
Gonçalves, V. F., Santo, J. P. T. E., Silva, G. F., Tome, C. F. A., Vasconcellos, J., & Matsubara, P. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3786580.3786993
“Recent studies on generative AI and its effects on development environments have adopted SPACE to examine how AI assistants, such as Copilot, affect developers’ productivity [38].”
Meryem, G., AATIF, Y., & Abdelkamel, A. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1861067
Lifecycle-Aware GenAI Assistance with MCP via Context Refinement Loops: A Reference Architecture
Elsisi, O., Santos, F., & Melo, G. (2026) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3803437.3805617 · Cited by 1
Grasping AI Reliance in Program Comprehension and Coding through the AIRELI Persona Taxonomy
Alakmeh, T., Anderson, N. B., Jackson, V., Pereira, G. V., Akırmak, Ü., Estey, A., Prikladnicki, R., van der Hoek, A., Storey, M., & Fritz, T. (2026) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3794763.3794804 · Cited by 5
“With modern AI assistants now capable of solving such isolated snippet tasks with ease [3, 21, 37, 53], there is an increasing need to design comprehension experiments that involve larger, more complex, and realistic codebases.”
When AI Is Wrong on Purpose: How Students Respond to Buggy GenAI Code
Pădurean, V.-A., Riegel, K., Gotovos, A., Mahapatra, J., Ghosh, A., Denny, P. C., Leinonen, J., Prather, J. E., & Singla, A. K. (2026) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3765964.3811667 · Cited by 4
Prompt Writing and Program Quality: An Investigation from a Programming Education Perspective
Nakamura, Y., Ito, W., Nishigata, Y., & Kuramitsu, K. (2026) · Journal of Information Processing · doi:10.2197/ipsjjip.34.648
Prause, M. (2026) · Systems · doi:10.3390/systems14081018 · Cited by 1
“For example, developers using Copilot finish tasks up to 55% faster and experience a lighter cognitive workload [1].”
Security Analysis of LLM-Generated Web API Backends
Khan, A. A., Rauti, S., & Mäkilä, T. (2026) · IEEE Access · doi:10.1109/access.2026.3727682
Jung, S., & Ahn, H. (2026) · The Journal of Korean Institute of Information Technology · doi:10.14801/jkiit.2026.24.8.231
The DX+ conceptual framework for AI-supported inclusive block programming in higher education
Маҳмудова, Д. (2026) · Frontiers in Education · doi:10.3389/feduc.2026.1902352
Skill-Based Architecture for Reusable, AI-Native Enterprise Modules: An Empirical Study
Jathar, N. (2026) · IEEE International Conference on Information Reuse and Integration (IRI) · doi:10.1109/iri69576.2026.00012
“LLMs translate natural language into executable code [4], reach competition-level programming performance [5], and measurably improve developer productivity [6].”
Beyond the Hype: The Efficiency-Throughput Gap with GitHub Copilot
Liu, F., & McKinnon, T. (2026) · Communications of the ACM · doi:10.1145/3797488 · Cited by 1
Dugyala, R. (2026) · International Conference on Modern Sustainable Systems (CMSS) · doi:10.1109/cmss69636.2026.11689630 · Cited by 1
Đambić, G., Maurovic, A., Biškupić, I. O., & Radovan, A. (2026) · Information · doi:10.3390/info17090918
Empowering the Manufacturing Workforce to Integrate AI Coding Assistants
Leberruyer, N., Ahlskog, M., Bruch, J., & Backhans, E. (2026) · IFIP advances in information and communication technology · doi:10.1007/978-3-032-38618-2_5
Unpacking AI Agent Participation in Issue-Centered Collaboration in Open-Source Software Development
Zhi, K., Fan, G., & Chen, W. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832206
Liu, J., Zhong, C., Meng, F., Shank, D. B., & Wunsch, D. C. (2026) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2026.103140
Beyond Code Generation: Reassessing Software Quality in AI-Assisted Development
Cernau, L. D. (2026) · Acta Universitatis Sapientiae Informatica · doi:10.1007/s44427-026-00046-3
“Ziegler et al. [11] extended this analysis using large-scale telemetry data, demonstrating that Copilot’s impact on productivity depends on task type and developer experience, with junior developers benefiting most from code-completion support.”
Command Coding: A Methodological Framework for AI-Accelerated Software Development
Villani, T. S., & Kellogg, M. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834536
FlightRecorder: Tracing Fine-Grained Development with AI Assistants
Dick, T., Bliem, B., Weis, K., Welter, A., & Apel, S. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834607
“Most existing studies on how developers use AI coding assistants rely on behavioral telemetry – tracking IDE focus, clipboard activity, and pasting patterns – rather than analyzing code evolution directly [9, 16].”
Qarkaxhija, J. (2026) · doi:10.31235/osf.io/m75hn_v1
“A survey of 2,047 matched Copilot users found that perceived productivity tracked suggestion-acceptance rate more closely than it tracked any other usage measure [7].”
InfluentialHelwig, M. (2026) · arXiv · doi:10.48550/arxiv.2609.05510
“The genre runs from population-scale telemetry [Ziegler 2024; Peng 2023] through org-scale deployment reports (25 months of GenAI telemetry at WhatsApp [WhatsCode 2026], LLM-driven migrations at Google [GoogleMigrate 2025]) to RCTs whose disagreement about productivity [METR-RCT 2025] underlines…”
Making AI Visible, Not Vanished: How AI Policies Reshape Developer Experience on GitHub
Chen, Y., Zimmermann, T., & Trinkenreich, B. (2026) · arXiv · doi:10.48550/arxiv.2608.03329
Metadata
- Authors
- Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E.
- Affiliations
- GitHub, Inc., San Francisco, CA, USA
- GitHub, San Francisco, CA, USA
- Venue
- Communications of the ACM
- Published
- 2024-02-22
- Volume
- 67
- Issue
- 3
- Pages
- 54-63
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W4392023437
- Cited by
- 124
- References
- 26
- Citations by year
- 2026: 58 · 2025: 53 · 2024: 13
- FWCI
- 46.42
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Communications of the ACM (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
- Keywords
- GitHub Copilot, AI-assisted programming, developer productivity, user perception, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.