The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index; 8 of the 17 sources its claims cite were checked for support.
Measuring GitHub Copilot's Impact on Productivity
Open accessAbstract
Case study asks Copilot users about its impact on their productivity, and seeks to find their perceptions mirrored in user data.
Overview
CoChat Summary
TL;DR
This study links GitHub Copilot IDE usage telemetry to developers’ self-reported productivity and finds that acceptance rate of shown suggestions is the best usage predictor of perceived productivity. Acceptance rate correlates better than persistence or contribution-volume measures, though it leaves substantial unexplained variance and varies by experience, language, and time.
Key findings
- Acceptance rate (`accepted_per_shown`) most positively predicts perceived productivity and correlates best with aggregate productivity (ρ = 0.24, P < 0.0001); it is the best-performing measurement for at least one survey question in each SPACE dimension.
- Persistence measures (unchanged/mostly unchanged after 30, 120, 300, 600 seconds) were less well correlated than acceptance rate; PLS first component explains 43.2% of variance, second 13.1%, both drawing most strongly from acceptance rate.
- GitHub Copilot had an acceptance rate of 27% and mean DCPU in excess of 312; acceptance rate varied by subgroup and time (weekend 23.5%, typical weekday working hours 21.2%, off hours 23%).
- Junior developers reported higher productivity gains and tended to accept more suggestions; both proficiency and years of experience negatively correlated with aggregate productivity (ρ = -0.095, P = 0.0001; ρ = -0.161, P < 0.0001), though effects were non-significant in multivariate regressions controlling for demographics.
Methods
- Survey emailed to 17,420 unpaid technical preview users; the abstract reports 2,631 survey responses, while the methodology reports 2,047 responses matched to usage data from the four-week study period ending around Feb 12, 2022.
- Usage telemetry from the GitHub Copilot IDE extension: opportunity, shown, accepted, accepted characters, unchanged/mostly unchanged after 30/120/300/600 seconds, active hour; survey used the SPACE framework with 11 statements plus one overall productivity statement, aggregated into a mean productivity score.
- Analysis: Pearson correlations between each usage metric and perceived productivity, PLS regression across all usage metrics, and linear regressions with experience covariates.
Limitations
- Stated: notable unexplained variance remains due to confounding and human factors; other approaches are necessary for fine-grained investigation; using PST for U.S. users introduces noise from time-zone inhomogeneity.
- Evident: outcome is self-reported perceived productivity, not objective productivity; sample is unpaid technical preview users who opted into communications, limiting generalizability; observational correlations cannot establish causation.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 6 Not valid · 7 Not checked · 47 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Offline evaluation, where the system is shown a partial snippet of code and then asked to complete it, is difficult not least because for longer completions there are many acceptable alternatives and no straightforward mechanism for labeling them automatically.5”
§ Case study asks Copilot users > Abstract
What the sources say
- Partly supportedcites [5] 5. Wagner, S. and Ruhe, M. A systematic review of productivity factors in software development. arXiv preprint arXiv:1801.06475 (2018). sourceExperimental — accuracy not yet measured
Section 2.1 (p. 2) says reference-matching metrics cannot account for the large space of functionally equivalent programs, supporting the point that many completions may be acceptable. It proposes unit tests for automatically evaluating docstring-based code generation, but does not establish that longer completions of partial code snippets lack a straightforward automatic labeling mechanism. (LLM, conf=0.83)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, proficiency positively predicts developers agreeing that Copilot helps them stay in the flow, focus on more satisfying work, spend less effort on repetitive tasks, and perform repetitive tasks faster.”
§ Case study asks Copilot users > Developer Productivity and the SPACE Framework
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All persistence measures were less well correlated than acceptance rate.”
§ Case study asks Copilot users > Productivity survey.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, in multivariate regressions predicting productivity from usage metrics while controlling for demographics, proficiency had a non-significant positive effect ( coeff = 0 .021 , P = 0 .213 ), while years of experience had a non-significant negative effect ( coeff = -0 .032 , P = 0 .122 ).”
§ Case study asks Copilot users > Developer Productivity and the SPACE Framework
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A vast majority of survey users (more than 80%) filled out the survey within the first two days, on or before February 12, 2022.”
§ Case study asks Copilot users > Productivity survey.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
28 more claims
Create a free account to see all 64 claims and every source verdict.
- Probably not a substantial claim. 24
- It may state two separate claims as one. 10
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Their findings suggest that the AI assistant had a significant positive impact on perceived productivity across multiple dimensions, e.g., task completion time, quality, cognitive load, enjoyment, and learning.”
From Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability (2026) · cites this paper as Ziegler et al. 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Ziegler et al. (2024) surveyed over 17,000 GitHub Copilot users and reported a large positive impact across different dimensions.”
From Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability (2026) · cites this paper as Ziegler et al. (2024)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One reason is that only the tool vendors have access to fine-grained usage data [4] that allows them to determine which code suggestions were accepted and hence which code was co-authored by GenAI tools.”
From Self-Admitted GenAI Usage in Open-Source Software (2026) · cites this paper as [4]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Based on surveys and usage telemetry, they found that developers perceived productivity improvements when using Copilot, particularly for repetitive and routine coding activities, with the perceived magnitude of improvement varying considerably based on task complexity and context.”
From Self-Admitted GenAI Usage in Open-Source Software (2026) · cites this paper as [4]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Further, GenAI tools can increase developers' perceived productivity [Ziegler et al. 2024] and level of understanding [Prather et al. 2024] without delivering actual improvements, potentially leading to a false sense of confidence [Lee et al. 2025] among novices.”
From How Do Programming Students Use Generative AI? (2025) · cites this paper as Ziegler et al. 2024
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Case study asks Copilot users > Developer Productivity and the SPACE Framework
“with options starting with "Student" and ranging from "0-2 years" to "16+ years" in twoyear intervals.”
§ Case study asks Copilot users > Abstract
“0-2 Years Prof. Experience 3-5 Years Prof. Experience 6-10 Years Prof. Experience 11-15 Years Prof. Experience 16+ Years Prof. Experience”
Citations
Cites
PLS-regression: a basic tool of chemometrics
Wold, S., Sjöström, M., & Eriksson, L. (2001) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/s0169-7439(01)00155-1 · Cited by 10,084 · cited 1×
“The choice of PLS, which captures the common variation of these variables as is linearly connected to the aggregate productivity, 28 is due to the high collinearity of the single metrics.”
BackgroundEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
IntelliCode compose: code generation using transformer
Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417058 · Cited by 474 · cited 1×
“20 An alternative measure is that of daily completions accepted per user (DCPU) for which a value of around 20 has been report-ed.”
BackgroundOn the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Devanbu, P. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227135 · Cited by 425 · cited 1×
“Code completion in IDEs using language models was first proposed in Hindle et al., 9 and today neural synthesis tools such as GitHub Copilot, CodeWhisperer, and TabNine suggest code snippets within an IDE with the explicitly stated intention to increase a user’s productivity. a Alternatively, we…”
MethodProgram Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388
The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023) · arXiv · doi:10.48550/arxiv.2302.06590 · Cited by 287
Self-citationWang, D., Churchill, E., Maes, P., Fan, X., Shneiderman, B., Shi, Y., & Wang, Q. (2020) · Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3334480.3381069 · Cited by 271
Measuring knowledge worker productivity: A taxonomy
Ramírez, Y. W., & Nembhard, D. A. (2004) · Journal of Intellectual Capital · doi:10.1108/14691930410567040 · Cited by 255
What makes a good conversation? How controllable attributes affect human judgments
See, A., Roller, S., Kiela, D., & Weston, J. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1170 · Cited by 247
Software developers' perceptions of productivity
Meyer, A. N., Fritz, T., Murphy, G. C., & Zimmermann, T. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635892 · Cited by 224 · cited 1×
“˲ In addition, developers’ perspective on what affects their productivity 12 and what metrics might reflect it 14 differs from the inputs/ outputs view.”
BackgroundThe Work Life of Developers: Activities, Switches and Perceived Productivity
Meyer, A. N., Barton, L. E., Murphy, G. C., Zimmermann, T., & Fritz, T. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2656886 · Cited by 170 · cited 1×
“13 Given this, the view of developer productivity as inputs and outputs, or using metrics that strictly focus on coding, ignores the reality of the work developers do.”
BackgroundThe SPACE of Developer Productivity: There's more to it than you think.
Forsgren, N., Storey, M.-A., Maddila, C., Zimmermann, T., Houck, B., & Butler, J. (2021) · Queue · doi:10.1145/3454122.3454124 · Cited by 164
Measuring Coding Challenge Competence With APPS
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.09938 · Cited by 142
Perfection Not Required? Human-AI Partnerships in Code Translation
Weisz, J. D., Muller, M., Houde, S., Richards, J., Ross, S. I., Martinez, F., Agarwal, M., & Talamadupula, K. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450656 · Cited by 129 · cited 1×
“26 Measuring developer productivity through activity counts over time (a typical definition of productivity bor-rowed from economics) disregards the complexity of software development, as they account for only a subset of developer outputs.”
BackgroundToday was a good day: The daily life of software developers
Meyer, A. N., Barr, E. T., Bird, C., & Zimmermann, T. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2904957 · Cited by 123
A Systematic Review of Productivity Factors in Software Development
Wagner, S., & Ruhe, M. (2018) · arXiv · doi:10.48550/arxiv.1801.06475 · Cited by 114
SPoC: Search-based Pseudocode to Code
Kulal, S., Pasupat, P., Chandra, K., Lee, M., Padon, O., Aiken, A., & Liang, P. (2019) · arXiv · doi:10.48550/arxiv.1906.04908 · Cited by 103
What Predicts Software Developers’ Productivity?
Murphy-Hill, E., Jaspan, C., Sadowski, C., Shepherd, D., Phillips, M., Winter, C., Knight, A., Smith, E., & Jorde, M. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2900308 · Cited by 100 · cited 1×
“We point readers to excellent presentations of the existing discourse in the community in Meyer et al. 12 and Murphy-Hill et al.; 15”
BackgroundFast and Memory-Efficient Neural Code Completion
Svyatkovskiy, A., Lee, S., Hadjitofi, A., Riechert, M., Franco, J. V., & Allamanis, M. (2021) · 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr52588.2021.00045 · Cited by 95 · cited 2×
“21 Due to the diversity of potential solutions to a multi-line completion task, researchers have used software testing to evaluate the behavior of completions.”
BackgroundMethodTowards a Theory of Software Developer Job Satisfaction and Perceived Productivity
Storey, M.-A., Zimmermann, T., Bird, C., Czerwonka, J., Murphy, B., & Kalliamvakou, E. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2944354 · Cited by 90
Self-citationA Study of Visual Studio Usage in Practice
Amann, S., Proksch, S., Nadi, S., & Mezini, M. (2016) · 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER) · doi:10.1109/saner.2016.39 · Cited by 77
When Code Completion Fails: A Case Study on Real-World Completions
Hellendoorn, V. J., Proksch, S., Gall, H. C., & Bacchelli, A. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00101 · Cited by 76 · cited 1×
“7 The evaluation of context-aware API completion for Visual Studio IntelliCode considered Re-call@5—the proportion of completions for which the correct method call was in the top five suggestions.”
MethodSoftware Engineering at Google : Lessons Learned from Programming Over Time
Winters, T. (2020) · Cited by 56
Learning Autocompletion from Real-World Datasets
Aye, G. A., Kim, S., & Li, H. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip52600.2021.00022 · Cited by 29 · cited 2×
“3,29 To calculate acceptance rate one must, of course, normalize DCPU by the time spent coding each day.”
BackgroundThe SPACE of developer productivity
Forsgren, N., Storey, M.-A., Maddila, C., Zimmermann, T., Houck, B., & Butler, J. (2021) · Communications of the ACM · doi:10.1145/3453928 · Cited by 21
No Single Metric Captures Productivity
Jaspan, C., & Sadowski, C. (2019) · Rethinking Productivity in Software Engineering · doi:10.1007/978-1-4842-4221-6_2 · Cited by 20
Mind the Gap: On the Relationship Between Automatically Measured and Self-Reported Productivity
Beller, M., Orgovan, V., Buja, S., & Zimmermann, T. (2021) · IEEE Software · doi:10.1109/ms.2020.3048200 · Cited by 16 · cited 1×
“4 We used the SPACE framework 6 to design a survey that captures self-reported productivity and paired the self-reported data with usage telemetry.”
MethodImproving Code Autocompletion with Transfer Learning
Zhou, W., Kim, S., Murali, V., & Aye, G. A. (2021) · arXiv · doi:10.48550/arxiv.2105.05991 · cited 2×
“3,29 To calculate acceptance rate one must, of course, normalize DCPU by the time spent coding each day.”
Background
Research works printed but not linked: 1
- Vaithilingam, P., Zhang, T., and Glassman, E. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In Proceedings of the 2022 Conf. on Human Factors in Computing Systems.
Cited by
The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B. (2024) · Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3632620.3671116 · Cited by 196
“Third, a study in Communications of the ACM in March, 2024 by Ziegler et al. investigated how GitHub Copilot is impacting programmer productivity [73].”
Prather, J. E., Leinonen, J., Kiesler, N., Benario, J. G., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., et al. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3689187.3709614 · Cited by 208
“Typically, productivity is measured in terms of the acceptance rates of coding suggestions [163].”
The future of AI-driven software engineering
Terragni, V., Vella, A., Roop, P., & Blincoe, K. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715003 · Cited by 62
Copiloting the future: How generative AI transforms Software Engineering
Banh, L., Holldack, F., & Strobel, G. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107751 · Cited by 67
Weisz, J. D., Kumar, S. V., Muller, M., Browne, K.-E., Goldberg, A., Heintze, K. E., & Bajpai, S. (2025) · Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706599.3706670 · Cited by 37
“In addition, researchers have found correlations between subjective and objective productivity metrics, such as the acceptance rate of suggested code [62] and the number of source code files owned by a developer [40] being correlated with perceived productivity.”
Influential
Metadata
- Authors
- Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E.
- Affiliations
- GitHub, Inc., San Francisco, CA, USA
- GitHub, San Francisco, CA, USA
- Venue
- Communications of the ACM
- Published
- 2024-02-22
- Volume
- 67
- Issue
- 3
- Pages
- 54-63
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W4392023437
- Cited by
- 124
- References
- 26
- Citations by year
- 2026: 58 · 2025: 53 · 2024: 13
- FWCI
- 46.42
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Communications of the ACM (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
- Keywords
- GitHub Copilot, AI-assisted programming, developer productivity, user perception, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.