Productivity assessment of neural code completion
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index; 19 of the 23 sources its claims cite were checked for support.
Abstract
Neural code synthesis has reached a point where snippet generation is accurate enough to be considered for integration into human software development workflows. Commercial products aim to increase programmers’ productivity, without being able to measure it directly. In this case study, we asked users of GitHub Copilot about its impact on their productivity, and sought to find a reflection of their perception in directly measurable user data. We find that the rate with which shown suggestions are accepted, rather than more specific metrics regarding the persistence of completions in the code over time, drives developers’ perception of productivity.
Overview
CoChat Summary
TL;DR
In a case study of GitHub Copilot users, acceptance rate of shown suggestions was the usage metric most positively correlated with self-reported productivity, outperforming persistence-based measures. The authors conclude acceptance rate is useful for coarse-grained monitoring but not a singular quality criterion.
Key findings
- Acceptance rate (`accepted_per_shown`) best predicted aggregate perceived productivity: ρ = 0.24, P < 0.0001; all persistence measures were less well correlated. A PLS regression’s first component explained 43.2% of variance, and the second captured the acceptance-rate/change-rate dichotomy, explaining a further 13.1%.
- In this study GitHub Copilot had an acceptance rate of 27% and a mean DCPU in excess of 31. Common languages were TypeScript (24.7% of shown completions, 21.9% for survey users), JavaScript (21.3%, 24.2%), and Python (14.1%, 14.5%); the latter two had higher acceptance rates.
- Acceptance rate varied by time regime: weekends averaged 23.5%, weekday non-working hours 23%, and typical working hours 21.2%, all statistically significantly distinct at p < 0.001% by bootstrap re-sampling. A model found the actual time bucket mattered little, but whether it lay in the user’s usual time regime did matter.
- Incremental feature selection showed that after acceptance rate, adding `shown_per_hour`, `accepted_char_per_hour`, `shown_per_opportunity`, or `accepted_per_hour` each further improved prediction at statistically significant levels (p < 0.05).
Methods
- Case study pairing a productivity survey with IDE telemetry for GitHub Copilot, which uses OpenAI Codex. The authors emailed 17,420 users and received 2,047 responses they could match to usage measurements during the 4-week period leading up to February 12–March 2022; the Introduction reports analyzing 2,631 survey responses.
- Survey used the SPACE framework (Satisfaction, Performance, Communication, Efficiency; Activity excluded), with 11 Likert-style statements plus one overall productivity statement, encoded 1–5, and an aggregate productivity score calculated as the mean of 12 individual measures.
- Usage measurements included shown, accepted, accepted characters, unchanged and mostly unchanged completions at 30/120/300/600 seconds, and active hours, normalized into metrics such as `accepted_per_shown` and `accepted_per_hour`. Analysis used Pearson’s R and p-values, PLS regression, and incremental feature selection predicting residuals of models fit with other metrics.
Limitations
- Stated: the work investigates correlation, not causation; the best metric has a Pearson coefficient of 0.24, leaving considerable unexplained variance; perceived productivity is not necessarily actual productivity; only a single completion system with a single fixed neural engine was considered; telemetry raises security and privacy concerns; and optimizing acceptance rate can bias toward popular use cases or be gamed by splitting suggestions.
- Evident from the text: the survey sample came from unpaid technical-preview users who had opted in to communications, and the paper reports different matched-response counts in different sections (2,047 matched responses in Methods vs. 2,631 survey responses in the Introduction).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
16 Valid · 0 Not valid · 3 Not checked · 65 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Offline evaluation where the system is shown a partial snippet of code and then asked to complete it is difficult not least because for longer completions there are many acceptable alternatives and no straightforward mechanism for labeling them automatically [5].”
§ Productivity Assessment of Neural Code Completion > 1 Introduction
What the sources say
- Partly supportedcites [5] 5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suc sourceExperimental — accuracy not yet measured
Section 2.1 (p. 2) says that match-based metrics cannot account for the large space of functionally equivalent programs, supporting the point that many completions may be acceptable. It does not specifically address longer completions of partial code snippets or say that automatic labeling lacks a straightforward mechanism; instead, it describes unit tests as a way to assess functional correctness. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, however, 38% of the variance of the different levels for the languages' average perceived productivity can be explained by the acceptance rates in Figure 5 above (using a linear model factoring through acceptance rate).”
§ Productivity Assessment of Neural Code Completion > 5.1 Language Use
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Typical non-working hours during the week: evenings after 4 pm PST until mornings 7 am PST, where the average acceptance rate is also rather high at 23%.”
§ Productivity Assessment of Neural Code Completion > 5.2 Circadian and Weekly Rhythms
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The weekend: Saturdays and Sundays (day boundaries taken from PST), where the average acceptance rate is comparatively high at 23.5%.”
§ Productivity Assessment of Neural Code Completion > 5.2 Circadian and Weekly Rhythms
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The first component, to which every metric under consideration contributes positively, explains 43.2% of the variance.”
§ Productivity Assessment of Neural Code Completion > 4 What Drives Perceived Productivity?
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
25 more claims
Create a free account to see all 84 claims and every source verdict.
- Probably not a substantial claim. 42
- It may state two separate claims as one. 22
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“There is prior work on multi-language evaluation [7], but it uses perplexity as a proxy for performance, instead of benchmarks that check correctness.”
From MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023) · cites this paper as [7]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“Github Copilot user's acceptance rate of suggestions is 6.6% per hour.”
From Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022) · cites this paper as [22]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“They found both that developers felt more productive using Copilot and that these self-reported perceptions were reasonably correlated with suggestion acceptance rates.”
From Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming (2024) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, most of these studies focus on AI-assisted tool use in the context of understanding, generating, or modifying source code, providing little insight into the role AI plays in developer information seeking and its impact.”
From The Evolution of Information Seeking in Software Development: Understanding the Role and Impact of AI Assistants (2025) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While our participants reported, and prior work suggests [14,33,38,40], AI tools demonstrably increase short-term productivity through faster task completion and reduced blocked time”
From The Evolution of Information Seeking in Software Development: Understanding the Role and Impact of AI Assistants (2025) · cites this paper as [40]
Experimental — accuracy not yet measured
23 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Productivity Assessment of Neural Code Completion > 1 Introduction
“We analyze 2,631 survey responses from developers using GitHub Copilot and match their responses to measurements collected from the IDE.”
§ Productivity Assessment of Neural Code Completion > 3.2 Productivity Survey
“we received 2,047 responses we could match to usage measurements during the 4 week period leading up to February 12 th March 2022.”
Citations
Cites
PLS-regression: a basic tool of chemometrics
Wold, S., Sjöström, M., & Eriksson, L. (2001) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/s0169-7439(01)00155-1 · Cited by 10,084 · cited 1×
“In order to assess the different metrics in a single model in a way that is still robust against their strong collinearity and unaffected by the decision whether or not to include highly similar metrics, we ran a regression using projection on latent structures (PLS), which captures the common variation of these varia…”
MethodEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
IntelliCode compose: code generation using transformer
Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417058 · Cited by 474 · cited 1×
“The IntelliCode Compose system uses the term CTR (Click Through Rate) for this and reports a value of 10% in online trials [12].”
MethodProgram Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388
What makes a good conversation? How controllable attributes affect human judgments
See, A., Roller, S., Kiela, D., & Weston, J. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1170 · Cited by 247 · cited 1×
“And there are clear programming parallels to factors such as specificity and repetition that have been identified to affect human judgements of conversation quality [11].”
BackgroundBest practices for the human evaluation of automatically generated text
van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S., & Krahmer, E. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8643 · Cited by 205 · cited 1×
“In future work, we wish to further explore this analogy, borrowing ideas [16] from the evaluation of chatbots and natural language text generation.”
BackgroundThe SPACE of Developer Productivity: There's more to it than you think.
Forsgren, N., Storey, M.-A., Maddila, C., Zimmermann, T., Houck, B., & Butler, J. (2021) · Queue · doi:10.1145/3454122.3454124 · Cited by 164
In-IDE Code Generation from Natural Language: Promise and Challenges
Xu, F. F., Vasilescu, B., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2101.11149 · Cited by 153
Measuring Coding Challenge Competence With APPS
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.09938 · Cited by 142 · cited 1×
“Competitive programming sites have been used as a source of such data [8, 9] as well as hand-written programming problems [5].”
Method
Research works printed but not linked: 1
- 14. Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Ex- pectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In CHI Conference on Hu- man Factors in Computing Systems Extended Abstracts (New Orleans, LA, USA) (CHI EA '22). Association for Computing Machinery, New York, NY, USA, Article 332, 7 pages. https://doi.org/10.1145/3491101. 3519665 · DOI · Paper page
Cited by
Augmenting large language models with chemistry tools
M. Bran, A., Cox, S., Schilter, O., Baldassari, C., White, A. D., & Schwaller, P. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00832-8 · Cited by 794
“These specialized tools provide exact answers, thereby compensating for the inherent deficiencies of LLMs in specific domains and enhancing their overall performance and applicability.”
BackgroundGitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417
GitHub Copilot AI pair programmer: Asset or Liability?
Dakhel, A. M., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., Ming, Z., & Jiang (2022) · arXiv · doi:10.48550/arxiv.2206.15331 · Cited by 417
“Authors in [37] surveyed 2631 developers about the impact of Copilot on their productivity”
BackgroundUsing an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487
CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models
Lemieux, C., Inala, J. P., Lahiri, S. K., & Sen, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00085 · Cited by 306
The Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493
The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023) · arXiv · doi:10.48550/arxiv.2302.06590 · Cited by 287
Self-citationCodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Shen, L., Wang, Z., Wang, A., Li, Y., et al. (2023) · Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3580305.3599790 · Cited by 563
“Research studies [46] also show that 88% of users of GitHub Copilot—a paid service powered by Codex—feel more productive when coding with it.”
BackgroundRoss, S. I., Martinez, F., Houde, S., Muller, M., & Weisz, J. D. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584037 · Cited by 258
Do Users Write More Insecure Code with AI Assistants?
Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2023) · Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3576915.3623157 · Cited by 259
A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
Liang, J. T., Yang, C., & Myers, B. A. (2023) · arXiv · doi:10.48550/arxiv.2303.17125 · Cited by 206
“While some recent studies do not find any significant difference in using AI programming assistants in terms of task completion [52, 56] and code quality [27], other studies find these tools are positively associated with developers’ self-perceived productivity [58].”
BackgroundResultInfluentialA Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
Liang, J. T., Yang, C., & Myers, B. A. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3608128 · Cited by 203
“Study participants also echoed prior work [62] by describing anAI programming assistant’s ability to "speed up.”
BackgroundResultInfluentialPrompting Is All You Need: Automated Android Bug Replay with Large Language Models
Feng, S., & Chen, C. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3608137 · Cited by 176
“Additionally, some studies have examined the impact of LLMs on software development [38, 55], developer productivity [21, 69, 84], and security vulnerabilities [36, 59, 60].”
BackgroundRetrieval-Based Prompt Selection for Code-Related Few-Shot Learning
Nashid, N., Sintaha, M., & Mesbah, A. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00205 · Cited by 178
Taking Flight with Copilot: Early insights and opportunities of AI-powered pair-programming tools
Bird, C., Ford, D., Zimmermann, T., Forsgren, N., Kalliamvakou, E., Lowdermilk, T., & Gazit, I. (2022) · Queue · doi:10.1145/3582083 · Cited by 158
Self-citationNavigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
Benbya, H., Deakin University, Strich, F., Deakin University, Tamm, T., & Deakin University (2024) · Journal of the Association for Information Systems · doi:10.17705/1jais.00861 · Cited by 150
Trust in Generative AI among students: An Exploratory Study
Amoozadeh, M., Daniels, D., Nam, D., Kumar, A., Chen, S., Hilton, M., Ragavan, S. S., & Alipour, M. A. (2023) · arXiv · doi:10.48550/arxiv.2310.04631 · Cited by 143
Trust in Generative AI among Students: An exploratory study
Amoozadeh, M., Daniels, D., Nam, D., Kumar, A., Chen, S., Hilton, M., Srinivasa Ragavan, S., & Alipour, M. A. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630842 · Cited by 143
On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot
Mastropaolo, A., Pascarella, L., Guglielmi, E., Ciniselli, M., Scalabrino, S., Oliveto, R., & Bavota, G. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00181 · Cited by 138
Is GitHub’s Copilot as bad as humans at introducing vulnerabilities in code?
Asare, O., Nagappan, M., & Asokan, N. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10380-1 · Cited by 128
“Ziegler et al. (2022) compared the results of a Copilot user survey with data directly measured from Copilot usage.”
ResultA Review on Code Generation with LLMs: Application and Evaluation
Wang, J., & Chen, Y. (2023) · IEEE International Conference on Medical Artificial Intelligence (MedAI) · doi:10.1109/medai59581.2023.00044 · Cited by 258
LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation
Fakhoury, S., Naik, A., Sakkas, G., Chakraborty, S., & Lahiri, S. K. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3428972 · Cited by 120
“In the case of LLM-based code generation, the generated tests not only help make natural language intent more precise and prune incorrect suggestions generated by the LLM, but can also serves as debugging aid for remaining suggestions and regression tests for future code edits [7].”
BackgroundMultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., et al. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3267446 · Cited by 394
“New products such as GitHub Copilot 1 , Amazon CodeWhisperer 2 , and Tabnine 3 built on code generation models are growing in popularity with developers [7].”
BackgroundUsing AI-based coding assistants in practice: State of affairs, perceptions, and ways forward
Sergeyuk, A., Golubev, Y., Bryksin, T., & Ahmed, I. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2024.107610 · Cited by 101
“Ziegler et al. [25] discuss the use of neural code synthesis in software development and study perceived productivity by investigating whether usage measurements of developer interactions with GitHub Copilot can predict perceived productivity as reported by developers in a survey.”
BackgroundMethodReading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641936 · Cited by 114
Investigating and Designing for Trust in AI-powered Code Generation Tools
Wang, R., Cheng, R., Ford, D., & Zimmermann, T. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658984 · Cited by 87
On the Design of AI-powered Code Assistants for Notebooks
Mcnutt, A. M., Wang, C., Deline, R. A., & Drucker, S. M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580940 · Cited by 77
Kuhail, M. A., Mathew, S. S., Khalil, A., Berengueres, J., & Shah, S. J. H. (2024) · Science of Computer Programming · doi:10.1016/j.scico.2024.103111 · Cited by 68
How Beginning Programmers and Code LLMs (Mis)read Each Other
Nguyen, S., Babe, H. M., Zi, Y., Guha, A., Anderson, C. J., & Feldman, M. Q. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642706 · Cited by 66
Generative Artificial Intelligence for Software Engineering—A Research Agenda
Nguyen‐Duc, A., Cabrero‐Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K., et al. (2025) · Software: Practice and Experience · doi:10.1002/spe.70005 · Cited by 60
“The impacts of the utilization of GenAI tools have also been discussed, primarily from the point of view of productivity [113].”
Significant Productivity Gains through Programming with Large Language Models
Weber, T., Brandmaier, M., Schmidt, A., & Mayer, S. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3661145 · Cited by 64
“Thus, the presentation and interaction are essential components to ensure that developers can use the LLMs productively in their work [4, 25, 70].”
BackgroundMethodInfluentialLanguage Models for Code Completion: A Practical Evaluation
Izadi, M., Katzy, J., van Dam, T., Otten, M., Popescu, R. M., & van Deursen, A. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639138 · Cited by 131
Language Models for Code Completion: A Practical Evaluation
Izadi, M., Katzy, J., van Dam, T., Otten, M., Popescu, R. M., & van Deursen, A. (2024) · arXiv · doi:10.48550/arxiv.2402.16197 · Cited by 57
How Readable is Model-generated Code? Examining Readability and Visual Inspection of GitHub Copilot
Al Madi, N. (2022) · Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3551349.3560438 · Cited by 50
“Although the productivity benefits of Copilot remain questionable [28], our results suggest that reading and inspecting modelgenerated code is important.”
BackgroundCheng, R., Wang, R., Zimmermann, T., & Ford, D. (2024) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3651990 · Cited by 47
“…have been considered revolutionary for the workflow of software developers that are long considered complex and highly demanding [ 23 , 52 ], as they are designed to augment developers’ productivity by auto-completing lines of code in real-time, and/or generating code based on prompts [ 5 , 73 ].”
BackgroundStudent-AI Interaction: A Case Study of CS1 students
Amoozadeh, M., Nam, D., Prol, D., Alfageeh, A., Prather, J. E., Hilton, M. L., Ragavan, S. S., & Alipour, M. A. (2024) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3699538.3699567 · Cited by 90
How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study
Gu, K., Grunde-McLaughlin, M., McNutt, A., Heer, J., & Althoff, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641891 · Cited by 40
“Effective planning suggestions can not be achieved by simply feeding the current analysis context into an LLM as it requires selective context, higher-level scaffolding, targeted prompts, and sensitive triggers.”
BackgroundUnveiling ChatGPT’s usage in open source projects: A mining-based study
Tufano, R., Mastropaolo, A., Pepe, F., Dabic, O., Di Penta, M., & Bavota, G. (2024) · Proceedings of the 21st International Conference on Mining Software Repositories · doi:10.1145/3643991.3644918 · Cited by 38
“These investigations include studies on the impact of the recommender on developers [71, 93, 109, 118], robustness assessment [82, 113, 115], empirical evaluation of correctness [88, 115], and scrutiny of security aspects [47, 92, 97].”
BackgroundBird, C., Ford, D., Zimmermann, T., Forsgren, N., Kalliamvakou, E., Lowdermilk, T., & Gazit, I. (2023) · Communications of the ACM · doi:10.1145/3589996 · Cited by 40
Self-citationQuality Assessment of ChatGPT Generated Code and their Use by Developers
Siddiq, M. L., Roney, L., Zhang, J., & Santos, J. C. S. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3645071 · Cited by 104
Weisz, J. D., Kumar, S. V., Muller, M., Browne, K.-E., Goldberg, A., Heintze, K. E., & Bajpai, S. (2025) · Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706599.3706670 · Cited by 37
Fuzzing JavaScript Interpreters with Coverage-Guided Reinforcement Learning for LLM-Based Mutation
Eom, J., Jeong, S., & Kwon, T. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680389 · Cited by 89
How Do Analysts Understand and Verify AI-Assisted Data Analyses?
Gu, K., Shang, R., Althoff, T., Wang, C., & Drucker, S. M. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642497 · Cited by 37
Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants
Sandoval, G., Pearce, H., Nys, T., Karri, R., Garg, S., & Dolan-Gavitt, B. (2022) · arXiv · doi:10.48550/arxiv.2208.09727 · Cited by 52
Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMs
Cassano, F., Gouwar, J., Lucchetti, F., Schlesinger, C., Freeman, A., Anderson, C. J., Feldman, M. Q., Greenberg, M., Jangda, A., & Guha, A. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689735 · Cited by 149
AI-assisted Code Authoring at Scale: Fine-tuning, deploying, and mixed methods evaluation
Murali, V., Maddila, C., Ahmad, I., Bolin, M., Cheng, D., Ghorbani, N., Fernandez, R., Nagappan, N., & Rigby, P. C. (2023) · arXiv · doi:10.48550/arxiv.2305.12050 · Cited by 33
Arakawa, R., Yakura, H., & Goto, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581133 · Cited by 29
Nazzal, M., Khalil, I. M., Khreishah, A. A., & Phan, N. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3690298 · Cited by 95
Effects of LLM-based Search on Decision Making: Speed, Accuracy, and Overreliance
Spatharioti, S. E., Rothschild, D., Goldstein, D. G., & Hofman, J. M. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714082 · Cited by 36
Exploring and Characterizing Large Language Models for Embedded System Development and Debugging
Englhardt, Z., Li, R. C., Nissanka, D., Zhang, Z., Narayanswamy, G., Breda, J., Liu, X., Patel, S., & Iyer, V. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650764 · Cited by 67
SALLM: Security Assessment of Generated Code
Siddiq, M. L., Santos, J. C. S., Devareddy, S., & Muller, A. (2024) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1145/3691621.3694934 · Cited by 124
Calibration and Correctness of Language Models for Code
Spieß, C. K., Gros, D., Pai, K. S., Pradel, M., Rabin, M. R. I., Alipour, M. A., Jha, S., Dévanbu, P., & Ahmed, T. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00040 · Cited by 122
Using GitHub Copilot for Test Generation in Python: An Empirical Study
Haji, K. E., Brandt, C., & Zaidman, A. (2024) · doi:10.1145/3644032.3644443 · Cited by 58
Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts
Liang, J. T., Lin, M. M., Rao, N., & Myers, B. A. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729342 · Cited by 75
Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions
Vasconcelos, H., Bansal, G., Fourney, A., Liao, Q. V., & Vaughan, J. W. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3702320 · Cited by 91
AI-Assisted Code Authoring at Scale: Fine-Tuning, Deploying, and Mixed Methods Evaluation
Murali, V., Maddila, C., Ahmad, I., Bolin, M., Cheng, D., Ghorbani, N., Fernandez, R., Nagappan, N., & Rigby, P. C. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3643774 · Cited by 114
When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2306.04930 · Cited by 22
“…performance of code recommendation models, but these typically assess how well the model can complete code in an offline setting without developer input rather than evaluating how well it assists programmers in situ (Ziegler et al. 2022; Li et al. 2022; Evtikhiev et al. 2022; Dakhel et al. 2022).”
BackgroundGeneration Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions
Vasconcelos, H., Bansal, G., Fourney, A., Liao, Q. V., & Vaughan, J. W. (2023) · arXiv · doi:10.48550/arxiv.2302.07248 · Cited by 23
Rocks Coding, Not Development: A Human-Centric, Experimental Evaluation of LLM-Supported SE Tasks
Wang, W., Ning, H., Zhang, G., Liu, L., & Wang, Y. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3643758 · Cited by 54
The extended hollowed mind: why foundational knowledge is indispensable in the age of AI
Klein, C. R., & Klein, R. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1719019 · Cited by 41
Take It, Leave It, or Fix It: Measuring Productivity and Trust in Human-AI Collaboration
Qian, C., & Wexler, J. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640543.3645198 · Cited by 63
ChatGPT Code Detection: Techniques for Uncovering the Source of Code
Oedingen, M., Engelhardt, R. C., Denz, R., Hammer, M., & Konen, W. (2024) · AI · doi:10.3390/ai5030053 · Cited by 37
Non-Expert Programmers in the Generative AI Future
Feldman, M. Q., & Anderson, C. J. (2024) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3663384.3663393 · Cited by 51
When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i9.28878 · Cited by 23
(2022) · doi:10.2139/ssrn.4496380 · Cited by 20
How much does AI impact development speed? An enterprise-based randomized controlled trial
Paradis, E., Grey, K., Madison, Q., Nam, D., Macvean, A., Meimand, V., Zhang, N., Ferrari-Church, B., & Chandra, S. (2024) · arXiv · doi:10.48550/arxiv.2410.12944 · Cited by 20
Identifying the Factors That Influence Trust in AI Code Completion
Brown, A., D'Angelo, S., Murillo, A., Jaspan, C., & Green, C. (2024) · Proceedings of the 1st ACM International Conference on AI-Powered Software · doi:10.1145/3664646.3664757 · Cited by 20
Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2022) · arXiv · doi:10.48550/arxiv.2210.14306 · Cited by 114
“Finally, participants completed a post-study questionnaire about their experience mimicking the one in Ziegler et al. [2022]. The entire experiment was designed to last 60 minutes.”
MethodThe Impact of Generative AI Coding Assistants on Developers Who Are Visually Impaired
Flores-Saviaga, C., Hanrahan, B. V., Imteyaz, K., Clarke, S., & Savage, S. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714008 · Cited by 39
Developer Behaviors in Validating and Repairing LLM-Generated Code Using IDE and Eye Tracking
Tang, N., Chen, M., Ning, Z., Bansal, A., Huang, Y., McMillan, C., & Li, T. J.-J. (2024) · 2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc60511.2024.00015 · Cited by 18
COLDECO: An End User Spreadsheet Inspection Tool for AI-Generated Code
Ferdowsi, K., Williams, J., Drosos, I., Gordon, A. D., Negreanu, C., Polikarpova, N., Sarkar, A., & Zorn, B. (2023) · 2023 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc57772.2023.00017 · Cited by 17
Toward Effective AI Support for Developers: A survey of desires and concerns.
Khemka, M., & Houck, B. (2024) · Communications of the ACM · doi:10.1145/3690928 · Cited by 32
A User-centered Security Evaluation of Copilot
Asare, O., Nagappan, M., & Asokan, N. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639154 · Cited by 32
Hao, H., Hasan, K. A., Qin, H., Macedo, M., Tian, Y., Ding, S. H. H., & Hassan, A. E. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10540-x · Cited by 51
Show It or Tell It? Text, Visualization, and Their Combination
Hearst, M. A. (2023) · Communications of the ACM · doi:10.1145/3593580 · Cited by 29
Generative AI for Software Development: A Family of Studies on Code Generation
Dakhel, A. M., Nikanjam, A., Khomh, F., Desmarais, M. C., & Washizaki, H. (2024) · Springer eBooks · doi:10.1007/978-3-031-55642-5_7 · Cited by 20
Hliš, T., Četina, L., Beranič, T., & Pavlič, L. (2023) · Applied Sciences · doi:10.3390/app132413061 · Cited by 23
"You're on a bicycle with a little motor": Benefits and Challenges of Using AI Code Assistants
Mendes, W., Souza, S., & de Souza, C. R. B. (2024) · International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1145/3641822.3641882 · Cited by 43
Sun, Z., Du, X., Song, F., Wang, S., & Li, L. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639120 · Cited by 29
How Much Does AI Impact Development Speed? an Enterprise-Based Randomized Controlled Trial
Paradis, E., Grey, K., Madison, Q., Nam, D., Macvean, A., Meimand, V., Zhang, N., Ferrari-Church, B., & Chandra, S. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00060 · Cited by 20
“For instance, Ziegler et al. [18] analyzed telemetry data to investigate the productivity benefit of GitHub Copilot, and reported over one-fifth of suggestions were accepted by developers, which correlated with their perceived productivity.”
How Scientists Use Large Language Models to Program
O'Brien, G. (2025) · arXiv · doi:10.48550/arxiv.2502.17348 · Cited by 14
Don’t settle for the first! How many GitHub Copilot solutions should you check?
Oertel, J., Klünder, J., & Hebig, R. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107737 · Cited by 22
Piloting Copilot, Codex, and StarCoder2: Hot temperature, cold prompts, or black magic?
Döderlein, J.-B., Kouadio, N. H., Acher, M., Khelladi, D. E., & Combemale, B. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112562 · Cited by 70
How Scientists Use Large Language Models to Program
O’Brien, G. E. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713668 · Cited by 42
Zero-shot Prompting for Code Complexity Prediction Using GitHub Copilot
Siddiq, M. L., Samee, A., Azgor, S. R., Haider, M. A., Sawraz, S. I., & Santos, J. C. S. (2023) · IEEE/ACM International Workshop on Natural Language-Based Software Engineering (NLBSE) · doi:10.1109/nlbse59153.2023.00018 · Cited by 18
Insights from the Frontline: GenAI Utilization Among Software Engineering Students
Choudhuri, R., Ramakrishnan, A., Chatterjee, A., Trinkenreich, B., Steinmacher, I., Gerosa, M. A., & Sarma, A. (2025) · IEEE International Conference on Software Engineering Education and Training (CSEE&T) · doi:10.1109/cseet66350.2025.00007 · Cited by 38
Human-AI experience in integrated development environments: a systematic literature review
Sergeyuk, A., Zakharov, I., Koshchenko, E., & Izadi, M. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10793-0 · Cited by 51
ChatGPT vs. Stack Overflow: An Exploratory Comparison of Programming Assistance Tools
Liu, J., Tang, X., Li, L., Chen, P., & Liu, Y. (2023) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c60940.2023.00105 · Cited by 19
The Fast and Spurious: Developer Productivity with GenAI
Afroz, S., Feng, Z., Menezes, T., Kimura, K., Trinkenreich, B., Steinmacher, I., & Sarma, A. (2025) · arXiv · doi:10.48550/arxiv.2510.24265 · Cited by 11
Qiao, H., Vermeulen, J., Fitzmaurice, G., & Matejka, J. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714103 · Cited by 11
Basha, M., & Rodríguez-Pérez, G. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107804 · Cited by 19
Lessons from Building StackSpot AI: A Contextualized AI Coding Assistant
Pinto, G., De Souza, C., Neto, J. B., Souza, A., Gotto, T., & Monteiro, E. (2024) · Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice · doi:10.1145/3639477.3639751 · Cited by 12
Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories
Bouzenia, I., & Pradel, M. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00234 · Cited by 142
Al Haque, E., Brown, C., LaToza, T. D., & Johnson, B. (2025) · Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering · doi:10.1145/3696630.3731665 · Cited by 9
An Empirical Study on Leveraging LLMs for Metamodels and Code Co-evolution.
Kebaili, Z. K., Khelladi, D. E., Acher, M., & Barais, O. (2024) · The Journal of Object Technology · doi:10.5381/jot.2024.23.3.a6 · Cited by 17
Sun, Z., Du, X., Song, F., Wang, S., Ni, M., & Li, L. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion58688.2023.00089 · Cited by 8
Siddiq, M. L., Zhang, J., & Santos, J. C. S. (2024) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3643916.3644424 · Cited by 19
Sun, Z., Du, X., Song, F., Wang, S., Ni, M., Li, L., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3688831 · Cited by 8
“According to another study [41], among the Copilot completions displayed to users, around 70% were not accepted.”
BackgroundMethodResultInfluential
Metadata
- Authors
- Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E.
- Affiliations
- GitHub, USA
- Venue
- Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming
- Publisher
- ACM
- Published
- 2022-06-13
- Pages
- 21-29
- OpenAlex
- W4281669078
- Cited by
- 200
- References
- 16
- Citations by year
- 2026: 47 · 2025: 56 · 2024: 70 · 2023: 25 · 2022: 2
- FWCI
- 34.20
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Microsoft Research
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
- Keywords
- GitHub Copilot, developer productivity, code completion, developer perceptions, code generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.