The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index; 19 of the 23 sources its claims cite were checked for support.
Productivity assessment of neural code completion
Open accessAbstract
Neural code synthesis has reached a point where snippet generation is accurate enough to be considered for integration into human software development workflows. Commercial products aim to increase programmers’ productivity, without being able to measure it directly. In this case study, we asked users of GitHub Copilot about its impact on their productivity, and sought to find a reflection of their perception in directly measurable user data. We find that the rate with which shown suggestions are accepted, rather than more specific metrics regarding the persistence of completions in the code over time, drives developers’ perception of productivity.
Overview
CoChat Summary
TL;DR
In a case study of GitHub Copilot users, acceptance rate of shown suggestions was the usage metric most positively correlated with self-reported productivity, outperforming persistence-based measures. The authors conclude acceptance rate is useful for coarse-grained monitoring but not a singular quality criterion.
Key findings
- Acceptance rate (`accepted_per_shown`) best predicted aggregate perceived productivity: ρ = 0.24, P < 0.0001; all persistence measures were less well correlated. A PLS regression’s first component explained 43.2% of variance, and the second captured the acceptance-rate/change-rate dichotomy, explaining a further 13.1%.
- In this study GitHub Copilot had an acceptance rate of 27% and a mean DCPU in excess of 31. Common languages were TypeScript (24.7% of shown completions, 21.9% for survey users), JavaScript (21.3%, 24.2%), and Python (14.1%, 14.5%); the latter two had higher acceptance rates.
- Acceptance rate varied by time regime: weekends averaged 23.5%, weekday non-working hours 23%, and typical working hours 21.2%, all statistically significantly distinct at p < 0.001% by bootstrap re-sampling. A model found the actual time bucket mattered little, but whether it lay in the user’s usual time regime did matter.
- Incremental feature selection showed that after acceptance rate, adding `shown_per_hour`, `accepted_char_per_hour`, `shown_per_opportunity`, or `accepted_per_hour` each further improved prediction at statistically significant levels (p < 0.05).
Methods
- Case study pairing a productivity survey with IDE telemetry for GitHub Copilot, which uses OpenAI Codex. The authors emailed 17,420 users and received 2,047 responses they could match to usage measurements during the 4-week period leading up to February 12–March 2022; the Introduction reports analyzing 2,631 survey responses.
- Survey used the SPACE framework (Satisfaction, Performance, Communication, Efficiency; Activity excluded), with 11 Likert-style statements plus one overall productivity statement, encoded 1–5, and an aggregate productivity score calculated as the mean of 12 individual measures.
- Usage measurements included shown, accepted, accepted characters, unchanged and mostly unchanged completions at 30/120/300/600 seconds, and active hours, normalized into metrics such as `accepted_per_shown` and `accepted_per_hour`. Analysis used Pearson’s R and p-values, PLS regression, and incremental feature selection predicting residuals of models fit with other metrics.
Limitations
- Stated: the work investigates correlation, not causation; the best metric has a Pearson coefficient of 0.24, leaving considerable unexplained variance; perceived productivity is not necessarily actual productivity; only a single completion system with a single fixed neural engine was considered; telemetry raises security and privacy concerns; and optimizing acceptance rate can bias toward popular use cases or be gamed by splitting suggestions.
- Evident from the text: the survey sample came from unpaid technical-preview users who had opted in to communications, and the paper reports different matched-response counts in different sections (2,047 matched responses in Methods vs. 2,631 survey responses in the Introduction).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
16 Valid · 0 Not valid · 3 Not checked · 65 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Offline evaluation where the system is shown a partial snippet of code and then asked to complete it is difficult not least because for longer completions there are many acceptable alternatives and no straightforward mechanism for labeling them automatically [5].”
§ Productivity Assessment of Neural Code Completion > 1 Introduction
What the sources say
- Partly supportedcites [5] 5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suc sourceExperimental — accuracy not yet measured
Section 2.1 (p. 2) says that match-based metrics cannot account for the large space of functionally equivalent programs, supporting the point that many completions may be acceptable. It does not specifically address longer completions of partial code snippets or say that automatic labeling lacks a straightforward mechanism; instead, it describes unit tests as a way to assess functional correctness. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, however, 38% of the variance of the different levels for the languages' average perceived productivity can be explained by the acceptance rates in Figure 5 above (using a linear model factoring through acceptance rate).”
§ Productivity Assessment of Neural Code Completion > 5.1 Language Use
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Typical non-working hours during the week: evenings after 4 pm PST until mornings 7 am PST, where the average acceptance rate is also rather high at 23%.”
§ Productivity Assessment of Neural Code Completion > 5.2 Circadian and Weekly Rhythms
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The weekend: Saturdays and Sundays (day boundaries taken from PST), where the average acceptance rate is comparatively high at 23.5%.”
§ Productivity Assessment of Neural Code Completion > 5.2 Circadian and Weekly Rhythms
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The first component, to which every metric under consideration contributes positively, explains 43.2% of the variance.”
§ Productivity Assessment of Neural Code Completion > 4 What Drives Perceived Productivity?
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
25 more claims
Create a free account to see all 84 claims and every source verdict.
- Probably not a substantial claim. 42
- It may state two separate claims as one. 22
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“There is prior work on multi-language evaluation [7], but it uses perplexity as a proxy for performance, instead of benchmarks that check correctness.”
From MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023) · cites this paper as [7]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“Github Copilot user's acceptance rate of suggestions is 6.6% per hour.”
From Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022) · cites this paper as [22]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“They found both that developers felt more productive using Copilot and that these self-reported perceptions were reasonably correlated with suggestion acceptance rates.”
From Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming (2024) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, most of these studies focus on AI-assisted tool use in the context of understanding, generating, or modifying source code, providing little insight into the role AI plays in developer information seeking and its impact.”
From The Evolution of Information Seeking in Software Development: Understanding the Role and Impact of AI Assistants (2025) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While our participants reported, and prior work suggests [14,33,38,40], AI tools demonstrably increase short-term productivity through faster task completion and reduced blocked time”
From The Evolution of Information Seeking in Software Development: Understanding the Role and Impact of AI Assistants (2025) · cites this paper as [40]
Experimental — accuracy not yet measured
23 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Productivity Assessment of Neural Code Completion > 1 Introduction
“We analyze 2,631 survey responses from developers using GitHub Copilot and match their responses to measurements collected from the IDE.”
§ Productivity Assessment of Neural Code Completion > 3.2 Productivity Survey
“we received 2,047 responses we could match to usage measurements during the 4 week period leading up to February 12 th March 2022.”
Citations
Cites
PLS-regression: a basic tool of chemometrics
Wold, S., Sjöström, M., & Eriksson, L. (2001) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/s0169-7439(01)00155-1 · Cited by 10,084
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
IntelliCode compose: code generation using transformer
Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417058 · Cited by 474 · cited 1×
“The IntelliCode Compose system uses the term CTR (Click Through Rate) for this and reports a value of 10% in online trials [12].”
MethodProgram Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388
What makes a good conversation? How controllable attributes affect human judgments
See, A., Roller, S., Kiela, D., & Weston, J. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1170 · Cited by 247
Best practices for the human evaluation of automatically generated text
van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S., & Krahmer, E. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8643 · Cited by 205 · cited 1×
“In future work, we wish to further explore this analogy, borrowing ideas [16] from the evaluation of chatbots and natural language text generation.”
BackgroundThe SPACE of Developer Productivity: There's more to it than you think.
Forsgren, N., Storey, M.-A., Maddila, C., Zimmermann, T., Houck, B., & Butler, J. (2021) · Queue · doi:10.1145/3454122.3454124 · Cited by 164
In-IDE Code Generation from Natural Language: Promise and Challenges
Xu, F. F., Vasilescu, B., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2101.11149 · Cited by 153
Measuring Coding Challenge Competence With APPS
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.09938 · Cited by 142
Research works printed but not linked: 1
- 10.1145/3491101.3519665 · DOI · Paper page
Cited by
Augmenting large language models with chemistry tools
M. Bran, A., Cox, S., Schilter, O., Baldassari, C., White, A. D., & Schwaller, P. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00832-8 · Cited by 794
GitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417
Using an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487
CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models
Lemieux, C., Inala, J. P., Lahiri, S. K., & Sen, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00085 · Cited by 306
The Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493
Metadata
- Authors
- Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E.
- Affiliations
- GitHub, USA
- Venue
- Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming
- Publisher
- ACM
- Published
- 2022-06-13
- Pages
- 21-29
- OpenAlex
- W4281669078
- Cited by
- 200
- References
- 16
- Citations by year
- 2026: 47 · 2025: 56 · 2024: 70 · 2023: 25 · 2022: 2
- FWCI
- 34.20
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Microsoft Research
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
- Keywords
- GitHub Copilot, developer productivity, code completion, developer perceptions, code generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Productivity assessment of neural code completion. https://app.cochat.ai/papers/10.1145/3520312.3534864
- The paper
- Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2022). Productivity assessment of neural code completion. Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming. https://doi.org/10.1145/3520312.3534864
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.