Productivity assessment of neural code completion
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index; 19 of the 23 sources its claims cite were checked for support.
Abstract
Neural code synthesis has reached a point where snippet generation is accurate enough to be considered for integration into human software development workflows. Commercial products aim to increase programmers’ productivity, without being able to measure it directly. In this case study, we asked users of GitHub Copilot about its impact on their productivity, and sought to find a reflection of their perception in directly measurable user data. We find that the rate with which shown suggestions are accepted, rather than more specific metrics regarding the persistence of completions in the code over time, drives developers’ perception of productivity.
Overview
CoChat Summary
TL;DR
In a case study of GitHub Copilot users, acceptance rate of shown suggestions was the usage metric most positively correlated with self-reported productivity, outperforming persistence-based measures. The authors conclude acceptance rate is useful for coarse-grained monitoring but not a singular quality criterion.
Key findings
- Acceptance rate (`accepted_per_shown`) best predicted aggregate perceived productivity: ρ = 0.24, P < 0.0001; all persistence measures were less well correlated. A PLS regression’s first component explained 43.2% of variance, and the second captured the acceptance-rate/change-rate dichotomy, explaining a further 13.1%.
- In this study GitHub Copilot had an acceptance rate of 27% and a mean DCPU in excess of 31. Common languages were TypeScript (24.7% of shown completions, 21.9% for survey users), JavaScript (21.3%, 24.2%), and Python (14.1%, 14.5%); the latter two had higher acceptance rates.
- Acceptance rate varied by time regime: weekends averaged 23.5%, weekday non-working hours 23%, and typical working hours 21.2%, all statistically significantly distinct at p < 0.001% by bootstrap re-sampling. A model found the actual time bucket mattered little, but whether it lay in the user’s usual time regime did matter.
- Incremental feature selection showed that after acceptance rate, adding `shown_per_hour`, `accepted_char_per_hour`, `shown_per_opportunity`, or `accepted_per_hour` each further improved prediction at statistically significant levels (p < 0.05).
Methods
- Case study pairing a productivity survey with IDE telemetry for GitHub Copilot, which uses OpenAI Codex. The authors emailed 17,420 users and received 2,047 responses they could match to usage measurements during the 4-week period leading up to February 12–March 2022; the Introduction reports analyzing 2,631 survey responses.
- Survey used the SPACE framework (Satisfaction, Performance, Communication, Efficiency; Activity excluded), with 11 Likert-style statements plus one overall productivity statement, encoded 1–5, and an aggregate productivity score calculated as the mean of 12 individual measures.
- Usage measurements included shown, accepted, accepted characters, unchanged and mostly unchanged completions at 30/120/300/600 seconds, and active hours, normalized into metrics such as `accepted_per_shown` and `accepted_per_hour`. Analysis used Pearson’s R and p-values, PLS regression, and incremental feature selection predicting residuals of models fit with other metrics.
Limitations
- Stated: the work investigates correlation, not causation; the best metric has a Pearson coefficient of 0.24, leaving considerable unexplained variance; perceived productivity is not necessarily actual productivity; only a single completion system with a single fixed neural engine was considered; telemetry raises security and privacy concerns; and optimizing acceptance rate can bias toward popular use cases or be gamed by splitting suggestions.
- Evident from the text: the survey sample came from unpaid technical-preview users who had opted in to communications, and the paper reports different matched-response counts in different sections (2,047 matched responses in Methods vs. 2,631 survey responses in the Introduction).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
16 Valid · 0 Not valid · 3 Not checked · 65 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Offline evaluation where the system is shown a partial snippet of code and then asked to complete it is difficult not least because for longer completions there are many acceptable alternatives and no straightforward mechanism for labeling them automatically [5].”
§ Productivity Assessment of Neural Code Completion > 1 Introduction
What the sources say
- Partly supportedcites [5] 5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suc sourceExperimental — accuracy not yet measured
Section 2.1 (p. 2) says that match-based metrics cannot account for the large space of functionally equivalent programs, supporting the point that many completions may be acceptable. It does not specifically address longer completions of partial code snippets or say that automatic labeling lacks a straightforward mechanism; instead, it describes unit tests as a way to assess functional correctness. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, however, 38% of the variance of the different levels for the languages' average perceived productivity can be explained by the acceptance rates in Figure 5 above (using a linear model factoring through acceptance rate).”
§ Productivity Assessment of Neural Code Completion > 5.1 Language Use
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Typical non-working hours during the week: evenings after 4 pm PST until mornings 7 am PST, where the average acceptance rate is also rather high at 23%.”
§ Productivity Assessment of Neural Code Completion > 5.2 Circadian and Weekly Rhythms
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The weekend: Saturdays and Sundays (day boundaries taken from PST), where the average acceptance rate is comparatively high at 23.5%.”
§ Productivity Assessment of Neural Code Completion > 5.2 Circadian and Weekly Rhythms
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The first component, to which every metric under consideration contributes positively, explains 43.2% of the variance.”
§ Productivity Assessment of Neural Code Completion > 4 What Drives Perceived Productivity?
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
25 more claims
Create a free account to see all 84 claims and every source verdict.
- Probably not a substantial claim. 42
- It may state two separate claims as one. 22
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“There is prior work on multi-language evaluation [7], but it uses perplexity as a proxy for performance, instead of benchmarks that check correctness.”
From MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023) · cites this paper as [7]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“Github Copilot user's acceptance rate of suggestions is 6.6% per hour.”
From Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022) · cites this paper as [22]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“They found both that developers felt more productive using Copilot and that these self-reported perceptions were reasonably correlated with suggestion acceptance rates.”
From Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming (2024) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, most of these studies focus on AI-assisted tool use in the context of understanding, generating, or modifying source code, providing little insight into the role AI plays in developer information seeking and its impact.”
From The Evolution of Information Seeking in Software Development: Understanding the Role and Impact of AI Assistants (2025) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While our participants reported, and prior work suggests [14,33,38,40], AI tools demonstrably increase short-term productivity through faster task completion and reduced blocked time”
From The Evolution of Information Seeking in Software Development: Understanding the Role and Impact of AI Assistants (2025) · cites this paper as [40]
Experimental — accuracy not yet measured
23 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Productivity Assessment of Neural Code Completion > 1 Introduction
“We analyze 2,631 survey responses from developers using GitHub Copilot and match their responses to measurements collected from the IDE.”
§ Productivity Assessment of Neural Code Completion > 3.2 Productivity Survey
“we received 2,047 responses we could match to usage measurements during the 4 week period leading up to February 12 th March 2022.”
Citations
Cites
PLS-regression: a basic tool of chemometrics
Wold, S., Sjöström, M., & Eriksson, L. (2001) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/s0169-7439(01)00155-1 · Cited by 10,084 · cited 1×
“In order to assess the different metrics in a single model in a way that is still robust against their strong collinearity and unaffected by the decision whether or not to include highly similar metrics, we ran a regression using projection on latent structures (PLS), which captures the common variation of these varia…”
MethodEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
IntelliCode compose: code generation using transformer
Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417058 · Cited by 474 · cited 1×
“The IntelliCode Compose system uses the term CTR (Click Through Rate) for this and reports a value of 10% in online trials [12].”
MethodProgram Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388
What makes a good conversation? How controllable attributes affect human judgments
See, A., Roller, S., Kiela, D., & Weston, J. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1170 · Cited by 247 · cited 1×
“And there are clear programming parallels to factors such as specificity and repetition that have been identified to affect human judgements of conversation quality [11].”
BackgroundBest practices for the human evaluation of automatically generated text
van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S., & Krahmer, E. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8643 · Cited by 205 · cited 1×
“In future work, we wish to further explore this analogy, borrowing ideas [16] from the evaluation of chatbots and natural language text generation.”
BackgroundThe SPACE of Developer Productivity: There's more to it than you think.
Forsgren, N., Storey, M.-A., Maddila, C., Zimmermann, T., Houck, B., & Butler, J. (2021) · Queue · doi:10.1145/3454122.3454124 · Cited by 164
In-IDE Code Generation from Natural Language: Promise and Challenges
Xu, F. F., Vasilescu, B., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2101.11149 · Cited by 153
Measuring Coding Challenge Competence With APPS
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.09938 · Cited by 142 · cited 1×
“Competitive programming sites have been used as a source of such data [8, 9] as well as hand-written programming problems [5].”
Method
Research works printed but not linked: 1
- 14. Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Ex- pectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In CHI Conference on Hu- man Factors in Computing Systems Extended Abstracts (New Orleans, LA, USA) (CHI EA '22). Association for Computing Machinery, New York, NY, USA, Article 332, 7 pages. https://doi.org/10.1145/3491101. 3519665 · DOI · Paper page
Cited by
Items 101–200 of 218
A Transformer-Based Approach for Smart Invocation of Automatic Code Completion
de Moor, A., van Deursen, A., & Izadi, M. (2024) · Proceedings of the 1st ACM International Conference on AI-Powered Software · doi:10.1145/3664646.3664760 · Cited by 7
Evaluating Copilot on CS1 Code Writing Problems with Suppressed Specifications
Venkatesh, V., Venkatesh, V., & Kumar, V. (2023) · Bangalore Annual Compute Conference · doi:10.1145/3627217.3627235 · Cited by 8
HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agent
Wu, J. J., & Fard, F. H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715109 · Cited by 55
Source Code Recommender Systems: The Practitioners' Perspective
Ciniselli, M., Pascarella, L., Aghajani, E., Scalabrino, S., Oliveto, R., & Bavota, G. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00182 · Cited by 22
Kudriavtseva, A., Hotak, N. A., & Gadyatskaya, O. (2025) · Proceedings of the 40th ACM/SIGAPP Symposium on Applied Computing · doi:10.1145/3672608.3707778 · Cited by 9
Assessing the Use of GitHub Copilot on Students of Engineering of Information Systems
Cirett-Galán, F., Peralta, R. T., Navarro-Hernández, R., Ochoa, J. L., Contreras-Rivera, S., Estrada-Ríos, L. A., & Machado-Encinas, G. (2024) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194024500335 · Cited by 8
Human-Machine Teaming and Team Effectiveness in AI Tools for Software Engineering
Rauf, I., Sharp, H., López, T., & Wermelinger, M. (2025) · International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1109/chase66643.2025.00017 · Cited by 18
Stray, V., Barbala, A., & Wivestad, V. T. (2025) · Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering · doi:10.1145/3696630.3730566 · Cited by 19
LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptions
Liao, Z., Antoniak, M., Cheong, I., Cheng, E. Y.-Y., Lee, A.-H., Lo, K., Chang, J. C., & Zhang, A. X. (2024) · arXiv · doi:10.48550/arxiv.2411.05025 · Cited by 7
An Analysis of the Costs and Benefits of Autocomplete in IDEs
Jiang, S., & Coblenz, M. J. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660765 · Cited by 11
One Size Does Not Fit All: Investigating Efficacy of Perplexity in Detecting LLM-Generated Code
Xu, J., Zhang, H., Yang, Y., Yang, L., Cheng, Z., Lyu, J., Liu, B., Zhou, X., Bacchelli, A., Chiam, Y. K., et al. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3748506 · Cited by 26
Durán, F., Martínez, M., Lago, P., & Martínez‐Fernández, S. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112574 · Cited by 18
Hicks, C. M., Lee, C. S., & Foster-Marks, K. L. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/2gej5 · Cited by 10
The Recent Trends of Research on GitHub Copilot: A Systematic Review
Ani, Z. C., Hamid, Z. A., & Zhamri, N. N. (2024) · Communications in Computer and Information Science · doi:10.1007/978-981-99-9589-9_27 · Cited by 5
Toward Effective AI Support for Developers: A survey of desires and concerns
Khemka, M., & Houck, B. (2024) · Queue · doi:10.1145/3675416 · Cited by 7
Guiding ChatGPT for Better Code Generation: An Empirical Study
Liu, C., Bao, X., Zhang, H., Zhang, N., Hu, H., Zhang, X., & Yan, M. L. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00018 · Cited by 18
Learning Program Behavioral Models from Synthesized Input–Output Pairs
Mammadov, T., Klakow, D., Koller, A., & Zeller, A. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3748720 · Cited by 5
Threats to scientific software from over-reliance on AI code assistants
O’Brien, G. (2025) · Nature Computational Science · doi:10.1038/s43588-025-00845-2 · Cited by 6
Cracks in The Stack: Hidden Vulnerabilities and Licensing Risks in LLM Pre-Training Datasets
Jahanshahi, M., & Mockus, A. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00018 · Cited by 16
Re(gEx|DoS)Eval: Evaluating Generated Regular Expressions and their Proneness to DoS Attacks
Siddiq, M. L., Zhang, J., Roney, L., & Santos, J. C. S. (2024) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1145/3639476.3639757 · Cited by 11
On the Generalizability of Deep Learning-based Code Completion Across Programming Language Versions
Ciniselli, M., Martin-Lopez, A., & Bavota, G. (2024) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3643916.3644411 · Cited by 9
Martínez, M. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3728704 · Cited by 14
"My productivity is boosted, but …" Demystifying Users’ Perception on AI Coding Assistants
Lyu, Y., Yang, Z., Shi, J., Chang, J., Liu, Y., & Lo, D. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00024 · Cited by 25
The Fault in our Stars: Quality Assessment of Code Generation Benchmarks
Siddiq, M. L., Dristi, S., Saha, J., & Santos, J. C. S. (2024) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam63643.2024.00028 · Cited by 44
Ansible Lightspeed: A Code Generation Service for IT Automation
Sahoo, P., Pujar, S., Nalawade, G., Genhardt, R., Mandel, L., & Buratti, L. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695277 · Cited by 18
Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMs
Cheng, C.-A., Nie, A., & Swaminathan, A. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-2287 · Cited by 4
“Computational workflows that integrate large language models (LLMs), machine learning (ML) models, orchestration, retrievers, tools, etc., power many state-of-the-art AI applications [1]: from chatbots [2], coding assistants [3], robots [4], to multi-agent systems [5].”
BackgroundMethodAssessing GitHub Copilot in Solidity Development: Capabilities, Testing, and Bug Fixing
Baralla, G., Ibba, G., & Tonelli, R. (2024) · IEEE Access · doi:10.1109/access.2024.3486365 · Cited by 13
Advancing Secure and Standard Source Code Generation Techniques
Siddiq, M. L. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion66252.2025.00023 · Cited by 5
Responsible AI in Agile Software Engineering - An Industry Perspective
Ulfsnes, R., Moe, N. B., Emmerhoff, J., Floryan, M., Griva, A., Gundelsby, J. H., Barbala, A., & Conboy, K. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-031-72781-8_4 · Cited by 14
How much SPACE do metrics have in GenAI assisted software development?
Sikand, S., Phokela, K. K., Sharma, V. S., Singi, K., Kaulgud, V., Tung, T., Sharma, P., & Burden, A. P. (2024) · Innovations in Software Engineering Conference (ISEC) · doi:10.1145/3641399.3641419 · Cited by 8
Bringing Context-Aware Completion Suggestions to Arbitrary Text Entry Interfaces
Aveni, T. J., Fox, A., & Hartmann, B. (2023) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3586182.3615825 · Cited by 6
EvoC2Rust: A Skeleton-guided Framework for Project-Level C-to-Rust Translation
Wang, C., Yu, T., Shen, B., Wang, J., Chen, D., Zhang, W., Shi, Y., Xie, C., & Gu, X. (2026) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3786583.3786856 · Cited by 4
PALM: Synergizing Program Analysis and LLMs to Enhance Rust Unit Test Coverage
Chu, B., Feng, Y., Liu, K., Shi, H., Nan, Z., Guo, Z., & Xu, B. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00223 · Cited by 12
Challenges to Using Large Language Models in Code Generation and Repair
Pasquale, L., Sabetta, A., d’Amorim, M., Hegedűs, P., Mirakhorli, M. T., Okhravi, H., Payer, M., Rashid, A., Santos, J. C. S., Spring, J., et al. (2025) · IEEE Security & Privacy · doi:10.1109/msec.2025.3530488 · Cited by 11
Blasquez, I. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729050 · Cited by 5
Agentic Much? Adoption of Coding Agents on GitHub
Robbes, R., Matricon, T., Degueule, T., Hora, A., & Zacchiroli, S. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3822180 · Cited by 72
Artificial Intelligence in Software Documentation: Embracing the Documentation as Code Paradigm
Nafz, F., Krajinovic, M., & Ley, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-92196-4_14 · Cited by 3
Organizational Ohm's Law in Practice: Measuring Engineering Productivity
Liu, F., & McKinnon, T. (2024) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2024.3421892 · Cited by 3
Cracking CodeWhisperer: Analyzing Developers’ Interactions and Patterns During Programming Tasks
Javahar, J., Budhrani, T., Basha, M., de Souza, R. B. C., Beschastnikh, I., & Rodríguez-Pérez, G. (2025) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc65237.2025.00051 · Cited by 2
Code Generation of Smart Contracts with LLMs: A Case Study on Hyperledger Fabric
Olivieri, L., Beste, D., Negrini, L., Schönherr, L., Ciná, A. E., & Ferrara, P. (2025) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre66568.2025.00034 · Cited by 3
Exploring the Use and Misuse of Large Language Models
Valdez, H. P. D., Abri, F., Webb, J. B., & Austin, T. H. (2025) · Information · doi:10.3390/info16090758 · Cited by 6
Refuting LLM-generated Code with Reactive Task Comprehension
Hebbar, S. V., Sasmita Harini, S., & Kumar, V. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729100 · Cited by 5
Understanding and Enhancing CS Students’ Interaction Experience with AI Coding Assistant Tools
Long, X., Tan, X. L., Zhu, Y., Jiang, J., & Zhang, L. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3785479 · Cited by 10
AI Copilot for the Modern Developer : Leveraging GenAI in Software Development
Nagpal, A. (2024) · International Journal of Scientific Research in Computer Science Engineering and Information Technology · doi:10.32628/cseit241051056 · Cited by 2
Brandebusemeyer, C. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion66252.2025.00043 · Cited by 3
“No Free Lunch” when using Large Language Models to Verify Self-Generated Programs
Zilberman, S., & Cheng, H. C. B. (2024) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw60967.2024.00018 · Cited by 4
Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows
Chen, V., Talwalkar, A., Brennan, R., & Neubig, G. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790850 · Cited by 53
The Emerging Artifacts of Centralized Open-Code
Choksi, M. Z., Mandel, I., Widder, D. G., & Shvartzshnaider, Y. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3659019 · Cited by 5
Katzy, J., Huang, Y., Panchu, G.-R., Ziemlewski, M., Loizides, P., Vermeulen, S., van Deursen, A., & Izadi, M. (2025) · International Conference on Predictive Models in Software Engineering · doi:10.1145/3727582.3728683 · Cited by 9
Full Line Code Completion: Bringing AI to Desktop
Semenkin, A., Bibaev, V., Sokolov, Y., Krylov, K., Kalina, A., Khannanova, A., Savenkov, D., Rovdo, D., Davidenko, I., Karnaukhov, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.08704 · Cited by 8
“This metric can be defined differently [62], and it accounts for the ratio of code that stays in the file for some time after accepting code completion.”
Alakmeh, T., D’Angelo, S., & Fritz, T. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3774748.3787608 · Cited by 1
Martin-Lopez, A., Tufano, R., Guglielmi, E., Sánchez, A. B., Sanz, A. L., Scalabrino, S., Oliveto, R., Segura, S., & Bavota, G. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3679627 · Cited by 7
APILOT: Improving the Security and Usability of LLM Code Suggestions via Outdated API Mitigation
Bai, W., Ke, X., Huang, P., Wu, Q., Wen, J., Wu, J., & Lu, K. (2025) · Annual Computer Security Applications Conference · doi:10.1109/acsac67867.2025.00094 · Cited by 17
Exploring the Challenges and Opportunities of AI-assisted Codebase Generation
Eibl, P., Sabouri, S., & Chattopadhyay, S. (2025) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc65237.2025.00034 · Cited by 15
AI IDEs or Autonomous Agents? Measuring the Impact of Coding Agents on Software Development
Agarwal, S., He, H., & Vasilescu, B. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793589 · Cited by 28
Can we trust the source? A systematic review of watermarking and attribution for AI-generated code
Fahad, I. A., & Fuad, M. M. N. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108157 · Cited by 4
CodeVoyager: Integrating Interactive Visual Aids with LLMs for Code Comprehension
Kim, Y., Jang, K., Song, H., Shin, M., & Suh, B. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789057 · Cited by 2
An Eye for AI: Eye-Tracking the Micro-Interruptions of GenAI Code Suggestions
Alakmeh, T., D’Angelo, S., & Fritz, T. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773132 · Cited by 2
Patel, T., Bayyavarapu, S. R. K. V., Soni, V. R., Purushothaman, R., Thokala, G. B., & Ranganathan, V. (2026) · International Conference on Advances in Artificial Intelligence and Machine Learning (AAIML) · doi:10.1109/aaiml67890.2026.11498111 · Cited by 1
Human-AI Interaction, Moral Responsibility and Epistemic Credit
Largo, J. (2026) · Philosophy & Technology · doi:10.1007/s13347-026-01104-6 · Cited by 1
Using AI-Based Code Completion for Domain-Specific Languages
Piereder, C., Fleck, G., Geist, V., Moser, M., & Pichler, J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-49266-2_16 · Cited by 1
A Systematic Approach to Measuring Developer Productivity: The Prismetrix Method
Kawalerowicz, M., & Pietranik, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-09318-9_23 · Cited by 2
AI-Assisted SQL Authoring at Industry Scale
Maddila, C., Ghorbani, N., Jabre, K., Murali, V., Kim, E., Thakkar, P., Laptev, N., Harman, O., Hsu, D., Abreu, R., et al. (2025) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00019 · Cited by 9
Exploring Direct Instruction and Summary-Mediated Prompting in LLM-Assisted Code Modification
Tang, N., Smith, E., Huang, Y., Collin, M., & Li, T. J.-J. (2025) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc65237.2025.00017 · Cited by 5
Wivestad, V. T., Barbala, A., & Stray, V. G. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-031-72781-8_13 · Cited by 6
Bishop, G., Chakraborty, S., Zhou, H., Dehlinger, J., Deng, L., Lin, J., & Kist, B. (2026) · Automated Software Engineering · doi:10.1007/s10515-026-00617-w · Cited by 2
How do Copilot Suggestions Impact Developers' Frustration and Productivity?
Guglielmi, E., Arnoudova, V., Bavota, G., Oliveto, R., & Scalabrino, S. (2025) · arXiv · doi:10.48550/arxiv.2504.06808 · Cited by 1
“These tools promise to improve developer productivity by reducing cognitive effort and speeding up task completion [3]–[5].”
InfluentialAdoption and Usability of AI Tools in Bangladesh: Opportunities and Challenges
Hossain, D. T., & Sakib, N. (2025) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit68739.2025.11491700
Do Language Models Prefer Vulnerable Code? A Probabilistic Study of Insecure Code Preference
Melo, R., Reis, S., Catarino, A., & Abreu, R. (2026) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst69053.2026.00075 · Cited by 2
Software Vulnerability Management in the Era of Artificial Intelligence: An Industry Perspective
Kholoosi, M. M., Le, T. H. M., & Babar, M. A. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3787781 · Cited by 1
Programming Language Confusion: When Code LLMs Can't Keep Their Languages Straight
Moumoula, M. B., Nikiema, S. L., Kaboré, A. K., Klein, J., & Bissyandé, T. F. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner67736.2026.00087 · Cited by 7
Software Engineering Reliability in the Era of Generic and Agentic AI
Nwodo, K., & Stavrou, A. (2026) · Studies in computational intelligence · doi:10.1007/978-3-032-18585-3_9
Automated Update of Android Deprecated API Usages With Large Language Models
Mahmud, T., Duan, B., Che, M., Yasmin, A., Ngu, A. H. H., & Yang, G. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3627897 · Cited by 9
Dey, R. K., Nath, S., Mukherjee, D., Ray, B., Mahapatra, K., & Swastik, B. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-3751-6.ch003
Github Copilot's Impact on Developer Productivity : A Review of Early Evidence
Adepoju, S. A. (2023) · International Journal of Scientific Research in Science and Technology · doi:10.32628/ijsrst2221192 · Cited by 3
Suryanarayanan, S. (2026) · International Conference on Inventive Computing and Informatics (ICICI) · doi:10.1109/icici68773.2026.11581171
Das, P., Reddy, Y. R., Potdar, S., & Warudkar, H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36590-3_28
Understanding and Predicting Accepted Code Suggestions in AI-Assisted Programming
Jiang, J., Li, L., Hou, J., Tan, X. L., & Zhang, L. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808125
GeoMind: An Autonomous Multi-Agent Framework for Verified Geospatial AnalysisPreprint
Habib, J., Srabon, M. I., Nahar, M. L., Lutfy, M., Khatun, M. A., Shammo, M. B. A. Z., & Hasan, M. A. (2026) · Research Square · doi:10.21203/rs.3.rs-10034367/v1
Comparing AI Coding Agents: A Task-Stratified Analysis of Pull Request Acceptance
Pinna, G., Gong, J., Williams, D., & Sarro, F. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793576 · Cited by 13
The Watermark Paradox: How Provenance Verification Paves the Road to Camouflaged Backdoors
Zhang, H., Ran, H., Kim, K., & Tang, X. (2026) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3803437.3805553
Inoue, S., Morisaki, S., Wada, T., & Hattori, Y. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3702933
Brandebusemeyer, C., Schimmer, T., & Arnrich, B. (2026) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3786583.3786870 · Cited by 6
Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability
Borg, M., Hewett, D., Hagatulah, N., Couderc, N., Söderberg, E., Graham, D., Kini, U., & Farley, D. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10889-1 · Cited by 4
Test Coverage of Code Changes in AI-Generated Pull Requests
Alves, T., & Teixeira, L. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793618 · Cited by 1
Testing Android Third Party Libraries with LLMs to Detect Incompatible APIs
Mahmud, T., Duan, B., Che, M., Ngu, A. H. H., & Yang, G. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00039
Anderson, A. (2026) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse72781.2026.00048
The Art of Programming: Challenges in Generating Code for Creative Applications
Cook, M. L. (2024) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1145/3664646.3664774
A Journey Through SPACE: Unpacking the Perceived Productivity of GitHub Copilot
Wivestad, V. T., & Ulfsnes, R. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-031-72781-8_5 · Cited by 2
PROBE: Benchmarking code generation in large language models
Nogueira, R. P., Vieira, M., & Campos, J. R. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10904-5 · Cited by 2
The Automation Fallacy: Why AI Doesn’t Function as a Simple “Job Killer”
Yadav, P. S. (2026) · Future of business and finance · doi:10.1007/978-3-032-11748-9_1
One More Prompt: LLM Coding is a Game You Can’t Quite Put Down - A Provocation
Beale, R. (2026) · Designing Interactive Systems Conference · doi:10.1145/3802974.3809412
Beyond Correctness: Measuring Hallucination Impact in AI-Assisted Development Workflows
Gupta, S., Vanteru, K., Reddy, P. K., & Sunkireddy, S. (2026) · International Conference on Innovations in Emerging Technologies for Sustainable Development (ICIETSD) · doi:10.1109/icietsd68684.2026.11584879
Schadow, F., Gerling, K., Kretzer, F., & Maedche, A. (2026) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3805029.3818286
AthenaLLM: Supporting Experiments with Large Language Models in Software Development
de Oliveira, B., & Castor, F. (2024) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3643916.3644437 · Cited by 2
Colaboração com Assistente de Codificação Baseado em IA: Benefícios e Desafios
Mendes, W., Souza, S., & de Souza, C. R. B. (2024) · Simpósio Brasileiro de Sistemas Colaborativos (SBSC) · doi:10.5753/sbsc.2024.237964
Wang, C., Zhang, Z., Xu, X., Zheng, M., Wei, G., & Zhang, X. (2026) · Foundations and Trends® in Programming Languages · doi:10.1108/ftpgl-11-2025-0068
Nogueira, R. P., Vieira, M. P. A., & Campos, J. R. (2025) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre66568.2025.00036 · Cited by 5
Garg, A., Ishaani, M., & DeLaPena, R. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3798372 · Cited by 1
Metadata
- Authors
- Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E.
- Affiliations
- GitHub, USA
- Venue
- Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming
- Publisher
- ACM
- Published
- 2022-06-13
- Pages
- 21-29
- OpenAlex
- W4281669078
- Cited by
- 200
- References
- 16
- Citations by year
- 2026: 47 · 2025: 56 · 2024: 70 · 2023: 25 · 2022: 2
- FWCI
- 34.20
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Microsoft Research
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
- Keywords
- GitHub Copilot, developer productivity, code completion, developer perceptions, code generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.