The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index; 8 of the 17 sources its claims cite were checked for support.
Measuring GitHub Copilot's Impact on Productivity
Open accessAbstract
Case study asks Copilot users about its impact on their productivity, and seeks to find their perceptions mirrored in user data.
Overview
CoChat Summary
TL;DR
This study links GitHub Copilot IDE usage telemetry to developers’ self-reported productivity and finds that acceptance rate of shown suggestions is the best usage predictor of perceived productivity. Acceptance rate correlates better than persistence or contribution-volume measures, though it leaves substantial unexplained variance and varies by experience, language, and time.
Key findings
- Acceptance rate (`accepted_per_shown`) most positively predicts perceived productivity and correlates best with aggregate productivity (ρ = 0.24, P < 0.0001); it is the best-performing measurement for at least one survey question in each SPACE dimension.
- Persistence measures (unchanged/mostly unchanged after 30, 120, 300, 600 seconds) were less well correlated than acceptance rate; PLS first component explains 43.2% of variance, second 13.1%, both drawing most strongly from acceptance rate.
- GitHub Copilot had an acceptance rate of 27% and mean DCPU in excess of 312; acceptance rate varied by subgroup and time (weekend 23.5%, typical weekday working hours 21.2%, off hours 23%).
- Junior developers reported higher productivity gains and tended to accept more suggestions; both proficiency and years of experience negatively correlated with aggregate productivity (ρ = -0.095, P = 0.0001; ρ = -0.161, P < 0.0001), though effects were non-significant in multivariate regressions controlling for demographics.
Methods
- Survey emailed to 17,420 unpaid technical preview users; the abstract reports 2,631 survey responses, while the methodology reports 2,047 responses matched to usage data from the four-week study period ending around Feb 12, 2022.
- Usage telemetry from the GitHub Copilot IDE extension: opportunity, shown, accepted, accepted characters, unchanged/mostly unchanged after 30/120/300/600 seconds, active hour; survey used the SPACE framework with 11 statements plus one overall productivity statement, aggregated into a mean productivity score.
- Analysis: Pearson correlations between each usage metric and perceived productivity, PLS regression across all usage metrics, and linear regressions with experience covariates.
Limitations
- Stated: notable unexplained variance remains due to confounding and human factors; other approaches are necessary for fine-grained investigation; using PST for U.S. users introduces noise from time-zone inhomogeneity.
- Evident: outcome is self-reported perceived productivity, not objective productivity; sample is unpaid technical preview users who opted into communications, limiting generalizability; observational correlations cannot establish causation.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 6 Not valid · 7 Not checked · 47 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Offline evaluation, where the system is shown a partial snippet of code and then asked to complete it, is difficult not least because for longer completions there are many acceptable alternatives and no straightforward mechanism for labeling them automatically.5”
§ Case study asks Copilot users > Abstract
What the sources say
- Partly supportedcites [5] 5. Wagner, S. and Ruhe, M. A systematic review of productivity factors in software development. arXiv preprint arXiv:1801.06475 (2018). sourceExperimental — accuracy not yet measured
Section 2.1 (p. 2) says reference-matching metrics cannot account for the large space of functionally equivalent programs, supporting the point that many completions may be acceptable. It proposes unit tests for automatically evaluating docstring-based code generation, but does not establish that longer completions of partial code snippets lack a straightforward automatic labeling mechanism. (LLM, conf=0.83)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, proficiency positively predicts developers agreeing that Copilot helps them stay in the flow, focus on more satisfying work, spend less effort on repetitive tasks, and perform repetitive tasks faster.”
§ Case study asks Copilot users > Developer Productivity and the SPACE Framework
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All persistence measures were less well correlated than acceptance rate.”
§ Case study asks Copilot users > Productivity survey.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, in multivariate regressions predicting productivity from usage metrics while controlling for demographics, proficiency had a non-significant positive effect ( coeff = 0 .021 , P = 0 .213 ), while years of experience had a non-significant negative effect ( coeff = -0 .032 , P = 0 .122 ).”
§ Case study asks Copilot users > Developer Productivity and the SPACE Framework
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A vast majority of survey users (more than 80%) filled out the survey within the first two days, on or before February 12, 2022.”
§ Case study asks Copilot users > Productivity survey.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
28 more claims
Create a free account to see all 64 claims and every source verdict.
- Probably not a substantial claim. 24
- It may state two separate claims as one. 10
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Their findings suggest that the AI assistant had a significant positive impact on perceived productivity across multiple dimensions, e.g., task completion time, quality, cognitive load, enjoyment, and learning.”
From Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability (2026) · cites this paper as Ziegler et al. 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Ziegler et al. (2024) surveyed over 17,000 GitHub Copilot users and reported a large positive impact across different dimensions.”
From Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability (2026) · cites this paper as Ziegler et al. (2024)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One reason is that only the tool vendors have access to fine-grained usage data [4] that allows them to determine which code suggestions were accepted and hence which code was co-authored by GenAI tools.”
From Self-Admitted GenAI Usage in Open-Source Software (2026) · cites this paper as [4]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Based on surveys and usage telemetry, they found that developers perceived productivity improvements when using Copilot, particularly for repetitive and routine coding activities, with the perceived magnitude of improvement varying considerably based on task complexity and context.”
From Self-Admitted GenAI Usage in Open-Source Software (2026) · cites this paper as [4]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Further, GenAI tools can increase developers' perceived productivity [Ziegler et al. 2024] and level of understanding [Prather et al. 2024] without delivering actual improvements, potentially leading to a false sense of confidence [Lee et al. 2025] among novices.”
From How Do Programming Students Use Generative AI? (2025) · cites this paper as Ziegler et al. 2024
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Case study asks Copilot users > Developer Productivity and the SPACE Framework
“with options starting with "Student" and ranging from "0-2 years" to "16+ years" in twoyear intervals.”
§ Case study asks Copilot users > Abstract
“0-2 Years Prof. Experience 3-5 Years Prof. Experience 6-10 Years Prof. Experience 11-15 Years Prof. Experience 16+ Years Prof. Experience”
Citations
Cites
PLS-regression: a basic tool of chemometrics
Wold, S., Sjöström, M., & Eriksson, L. (2001) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/s0169-7439(01)00155-1 · Cited by 10,084 · cited 1×
“The choice of PLS, which captures the common variation of these variables as is linearly connected to the aggregate productivity, 28 is due to the high collinearity of the single metrics.”
BackgroundEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
IntelliCode compose: code generation using transformer
Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417058 · Cited by 474 · cited 1×
“20 An alternative measure is that of daily completions accepted per user (DCPU) for which a value of around 20 has been report-ed.”
BackgroundOn the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Devanbu, P. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227135 · Cited by 425 · cited 1×
“Code completion in IDEs using language models was first proposed in Hindle et al., 9 and today neural synthesis tools such as GitHub Copilot, CodeWhisperer, and TabNine suggest code snippets within an IDE with the explicitly stated intention to increase a user’s productivity. a Alternatively, we…”
MethodProgram Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388
The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023) · arXiv · doi:10.48550/arxiv.2302.06590 · Cited by 287
Self-citationWang, D., Churchill, E., Maes, P., Fan, X., Shneiderman, B., Shi, Y., & Wang, Q. (2020) · Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3334480.3381069 · Cited by 271
Measuring knowledge worker productivity: A taxonomy
Ramírez, Y. W., & Nembhard, D. A. (2004) · Journal of Intellectual Capital · doi:10.1108/14691930410567040 · Cited by 255
What makes a good conversation? How controllable attributes affect human judgments
See, A., Roller, S., Kiela, D., & Weston, J. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1170 · Cited by 247
Research works printed but not linked: 1
- Vaithilingam, P., Zhang, T., and Glassman, E. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In Proceedings of the 2022 Conf. on Human Factors in Computing Systems.
Cited by
The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B. (2024) · Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3632620.3671116 · Cited by 196
“Third, a study in Communications of the ACM in March, 2024 by Ziegler et al. investigated how GitHub Copilot is impacting programmer productivity [73].”
Prather, J. E., Leinonen, J., Kiesler, N., Benario, J. G., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., et al. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3689187.3709614 · Cited by 208
“Typically, productivity is measured in terms of the acceptance rates of coding suggestions [163].”
The future of AI-driven software engineering
Terragni, V., Vella, A., Roop, P., & Blincoe, K. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715003 · Cited by 62
Copiloting the future: How generative AI transforms Software Engineering
Banh, L., Holldack, F., & Strobel, G. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107751 · Cited by 67
Weisz, J. D., Kumar, S. V., Muller, M., Browne, K.-E., Goldberg, A., Heintze, K. E., & Bajpai, S. (2025) · Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706599.3706670 · Cited by 37
“In addition, researchers have found correlations between subjective and objective productivity metrics, such as the acceptance rate of suggested code [62] and the number of source code files owned by a developer [40] being correlated with perceived productivity.”
InfluentialHow Do Programming Students Use Generative AI?
Rahe, C., & Maalej, W. (2025) · Proceedings of the ACM on Software Engineering · doi:10.1145/3715762 · Cited by 45
“Further, GenAI tools can increase developers’ perceived productivity [Ziegler et al. 2024] and level of understanding [Prather et al. 2024] without delivering actual improvements, potentially leading to a false sense of confidence [Lee et al. 2025] among novices.”
Zhou, X., Liang, P., Zhang, B., Li, Z., Ahmad, A., Shahin, M., & Waseem, M. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112204 · Cited by 30
Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts
Liang, J. T., Lin, M. M., Rao, N., & Myers, B. A. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729342 · Cited by 71
“AI programming assistants that generate code (e.g., GitHub Copilot [21]) have aided developers across a variety of tasks such as writing significant portions of code, learning new APIs and programming languages, writing tests [47], and information seeking behaviors [33] thereby improving the…”
Benchmarks and Metrics for Evaluations of Code Generation: A Critical Review
Paul, D. G., Zhu, H., & Bayley, I. (2024) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest62860.2024.00019 · Cited by 73
“…in their study of GitHub Copilot, Ziegler et al. found that “while suggestion correctness is important, the driving factor for these improvements [in productivity] appears to be not correctness as such, but whether the suggestions are useful as a starting point for further development” [43].”
BackgroundMethodInfluentialLLMs: A game-changer for software engineers?
Haque, M. A. (2025) · BenchCouncil Transactions on Benchmarks Standards and Evaluations · doi:10.1016/j.tbench.2025.100204 · Cited by 53
The Impact of Generative AI on Creativity in Software Development: A Research Agenda
Jackson, V., Vasilescu, B., Russo, D., Ralph, P., Prikladnicki, R., Izadi, M., D’Angelo, S., Inman, S., Andrade, A., & van der Hoek, A. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708523 · Cited by 23
“…although most research, studies, and discussions in the media have concentrated on GenAI and programmer productivity (e.g., [14, 140]), understanding the short-term, medium-term, and long-term impacts and connections between GenAI and creativity in software development warrants equal, if…”
Detecting Code Smells using ChatGPT: Initial Insights
Silva, L. L., da Silva, J. R., Montandon, J. E., Andrade, M., & Valente, M. T. (2024) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3674805.3690742 · Cited by 25
Clear, T., Cajander, Å., Clear, A., McDermott, R., Daniels, M., Divitini, M., Forshaw, M., Humble, N., Kasinidou, M., Kleanthous, S., et al. (2025) · 2024 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3689187.3709607 · Cited by 19
Herath, S., Bashardoust, A., Bole, Y., & Shrestha, Y. R. (2026) · European Journal of Information Systems · doi:10.1080/0960085x.2026.2616042 · Cited by 16
“For instance, in coding, GenAI assists less-experienced programmers in producing efficient, clean code (Ziegler et al., 2024), while in fashion design, it supports novice designers in exploring design spaces and generating innovative design concepts, enabling them to achieve results comparable to…”
What Guides Our Choices? Modeling Developers' Trust and Behavioral Intentions Towards Genai
Choudhuri, R., Trinkenreich, B., Pandita, R., Kalliamvakou, E., Steinmacher, I., Gerosa, M., Sanchez, C., & Sarma, A. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) · doi:10.1109/icse55347.2025.00087 · Cited by 17
“Prior work highlights that developers’ expectations of clear advantages from using AI tools (e.g., increased productivity, improved code quality) contribute to their trust in using these tools [46, 101].”
Self-citationWhat Needs Attention? Prioritizing Drivers of Developers' Trust and Adoption of Generative AI
Choudhuri, R., Trinkenreich, B., Pandita, R., Kalliamvakou, E., Steinmacher, I., Gerosa, M., Sanchez, C., & Sarma, A. (2025) · arXiv · doi:10.48550/arxiv.2505.17418 · Cited by 14
Self-citationRequirements Are All You Need: The Final Frontier for End-User Software Engineering
Robinson, D., Cabrera, C., Gordon, A. D., Lawrence, N. D., & Mennen, L. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708524 · Cited by 12
“GitHub’s Copilot, launched in mid-2021, empowers professional developers by using an LLM to generate snippets of code from textual prompts, to explain code, to diagnose errors and to repair them [106].”
Statically Contextualizing Large Language Models with Typed Holes
Blinn, A., Li, X., Kim, J. H., & Omar, C. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689728 · Cited by 16
“For example, one study reports a 50% increase in productivity when using Copilot [88].”
BackgroundMethodSelf-citationQuality In, Quality Out: Investigating Training Data's Role in AI Code Generation
Improta, C., Tufano, R., Liguori, P., Cotroneo, D., & Bavota, G. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00056 · Cited by 21
“Several studies demonstrated the benefits brought by the usage of AI-based coding assistants [9], [10], for example in terms of increased perceived and actual productivity [11].”
Gemini at Work: Knowledge Workers' Perceptions and Assessment of Productivity Gains
Sun, N., & Kalar, D. J. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715336.3735679 · Cited by 17
LLM Assistance for Memory Safety
Mohammed, N., Lal, A., Rastogi, A., Sharma, R., & Roy, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00023 · Cited by 20
“Recently, Large Language Models (LLMs) have shown promise in improving the productivity of software developers [4].”
Qiao, Y., Shihab, M. I. H., & Hundhausen, C. D. (2025) · ACM Transactions on Computing Education · doi:10.1145/3785366 · Cited by 31
Challenges and Paths Towards AI for Software EngineeringPreprint
Gu, A., Jain, N., Li, W., Shetty, M., Shao, Y., Li, Z., Yang, D., Ellis, K., Sen, K., & Solar-Lezama, A. (2025) · Qeios · doi:10.32388/vv1661 · Cited by 49
Meske, C., Hermanns, T., Weiden, E. V. D., Loser, K., & Berger, T. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5378201 · Cited by 15
Time Warp: The Gap Between Developers' Ideal vs Actual Workweeks in an AI-Driven Era
Kumar, S., Goel, D., Zimmermann, T., Houck, B., Ashok, B., & Bansal, C. (2025) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00007 · Cited by 7
“There have been works [29]–[32] that analyze how different AI tools impact developer productivity.”
Ferino, S., Hoda, R., Grundy, J., & Treude, C. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3800580 · Cited by 39
LLM-Assistance for Quality Control of LLM Output
Sandkuhl, K. (2024) · Lecture notes in business information processing · doi:10.1007/978-3-031-71333-0_3 · Cited by 5
Ko, E., & Kang, P. (2025) · IEEE Access · doi:10.1109/access.2025.3553870 · Cited by 8
“The code they generate should be considered a starting point, not a finished product [6].”
On the Need to Rethink Trust in AI Assistants for Software Development: A Critical Review
Baltes, S., Speith, T., Chiteri, B., Mohsenimofidi, S., Chakraborty, S., & Buschek, D. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3659804 · Cited by 5
“” 1 The IDE integration of GitHub Copilot and similar AI development assistants allows tool vendors to collect fine-grained usage data to understand which suggestions developers accept, which they reject, and which they modify before committing a change [176].”
The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B., Kimmel, B., Wright, J., & Briggs, B. (2024) · arXiv · Cited by 196
“Third, a study in Communications of the ACM in March, 2024 by Ziegler et al. investigated how GitHub Copilot is impacting programmer productivity [73].”
Developer Confidence, Competence, and Self-Efficacy in the Age of AI
Gupta, D., Xusniddin, R., Jumaniyazova, N., Sherbek, S., Rajabova, N., Shavkat, R., & Dilorom, O. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-8450-4.ch002 · Cited by 5
Reaching new frontiers in nanoelectronics through artificial intelligence
Sivasubramani, S., & Prodromakis, T. (2025) · Frontiers in Nanotechnology · doi:10.3389/fnano.2025.1627210 · Cited by 8
Blasquez, I. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729050 · Cited by 5
LLM Support for Domain Experts in Enterprise Modeling: Experiences and Implications
Müller, E., Nast, B., & Sandkuhl, K. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-032-04375-7_8 · Cited by 5
Yılmaz, F. G. K., Olpak, Y. Z., & Yılmaz, R. (2026) · Journal of Research on Technology in Education · doi:10.1080/15391523.2026.2683793 · Cited by 6
Infusing Artificial Intelligence Into Software Engineering and the DevSecOps Continuum
Bannon, T. (2024) · Computer · doi:10.1109/mc.2024.3423108 · Cited by 5
Calyam, P. P., Clemm, A., Pandey, A., Roy, U., Keller, A., Das, S. K., & Calvert, K. L. (2024) · IEEE Internet Computing · doi:10.1109/mic.2024.3485954 · Cited by 3
Fuzzing microservices: A series of user studies in industry on industrial systems with EvoMaster
Zhang, M., Arcuri, A., Li, Y., Liu, Y., Xue, K., Wang, Z., Huo, J., & Huang, W. (2025) · Science of Computer Programming · doi:10.1016/j.scico.2025.103322 · Cited by 31
“Some exceptions exist, like the study on Copilot made by the researchers at GitHub [84].”
Advancing research software engineering with AI: a research framework
Farshidi, S., Bennin, K. E., Babur, Ö., Sallou, J., Kassahun, A., & Tekinerdoğan, B. (2026) · Automated Software Engineering · doi:10.1007/s10515-026-00621-0 · Cited by 9
Moving Faster and Reducing Risk: Using LLMs in Release Deployment
Abreu, R., Murali, V., Rigby, P. C., Maddila, C., Sun, W., Ge, J., Chinniah, K., Mockus, A., Mehta, M., & Nagappan, N. (2025) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00045 · Cited by 18
“Generative LLMs: The recent rise of generative models has demonstrated their ability to supercharge various software engineering activities [17], [18], [19], [20], [21], [22], [23], [24], [25], [26].”
An Abstraction Is Worth a Thousand Vibes
Steghöfer, J.-P., & Borg, M. (2025) · IEEE Software · doi:10.1109/ms.2025.3621786 · Cited by 2
An Empirical Study of Knowledge Transfer in AI Pair Programming
Welter, A., Schneider, N., Dick, T., Weis, K., Tinnes, C., Wyrich, M., & Apel, S. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00022 · Cited by 17
“The advent of G IT H UB C OPILOT [3] in late 2021, followed by other coding assistants, has noticeably changed how code is written [4, 5].”
Building Software by Rolling the Dice: A Qualitative Study of Vibe Coding
Chou, Y.-H., Jiang, B., Chen, Y. W., Weng, M., Jackson, V., Zimmermann, T., & Jones, J. A. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3797105 · Cited by 22
AI IDEs or Autonomous Agents? Measuring the Impact of Coding Agents on Software Development
Agarwal, S., He, H., & Vasilescu, B. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793589 · Cited by 20
“Research on Copilot spans productivity [22, 39, 48], code quality [32, 36, 46], and developer experience [29, 43], yet the impacts of autonomous, repository-level agents remain largely unexplored.”
Black-Box Reconstruction Attacks on LLMs: A Preliminary Study in Code Summarization
Russodivito, M., Spina, A., Scalabrino, S., & Oliveto, R. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-70245-7_27 · Cited by 5
Concepts for Teaching Software Development in the Age of AI-Tools
Böttcher, A., Thurner, V., & Zönnchen, B. (2025) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon62633.2025.11016505 · Cited by 5
“Since there is strong evidence that GenAI tools can significantly accelerate the coding process, thus enhancing productivity in industrial settings [1], [2], it is highly probable that the usage of GenAI tools in programming will be the new industry standard in the field of software development –…”
CodeWatcher: IDE Telemetry Data Extraction Tool for Understanding Coding Interactions with LLMs
Basha, M., Ribeiro, A. M., Javahar, J., de Souza, C. R. B., & Rodríguez-Pérez, G. (2025) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme64153.2025.00110 · Cited by 2
“Prior research has shown that developer telemetry, which is data about interactions with editors, compilers, and version control systems, can identify productivity bottlenecks and support developer success [9], [10], [11], [12].”
Chen, N., Qiu, L. K., Wang, A. Z., Wang, Z., & Yang, Y. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790726 · Cited by 16
Wivestad, V. T., Barbala, A., & Stray, V. G. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-031-72781-8_13 · Cited by 1
Exploring the Role of Large Language Models as Artificial Tutors
Zönnchen, B., Hobelsberger, M., Socher, G., Thurner, V., & Ottinger, S. (2025) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon62633.2025.11016571 · Cited by 4
AI in Software Maintenance: An Empirical Multi-source Approach
Ancán, Ó., Cares, C., Peral, J., et al. (2025) · Springer proceedings in complexity · doi:10.1007/978-3-031-78623-5_9 · Cited by 1
The Good, the Bad, and the Potential of AI-based Systems in Computing Education
Heffelman, A., Zuber, W., Deb, M., Manjunath, A., Aggabao, H., Magsi, H., Galley, M., Tairin, M., & Sharmin, M. (2025) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac65507.2025.00329 · Cited by 1
“Knowledge and skill in the effective use of AI-based systems are especially critical for CS students, as it is believed to be an essential skill for software developers, one of the most common career choices for CS students [21].”
Improving JavaScript Test Quality with Large Language Models: Lessons from Test Smell Refactoring
Amaral, G., Gomes, H., Figueiredo, E., Bezerra, C. I. M., & Rocha, L. (2025) · Simpósio Brasileiro de Engenharia de Software (SBES) · doi:10.5753/sbes.2025.11568 · Cited by 1
A Systematic Approach to Measuring Developer Productivity: The Prismetrix Method
Kawalerowicz, M., & Pietranik, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-09318-9_23 · Cited by 2
Uncovering Code Insights: Leveraging GitHub Artifacts for Deeper Code Understanding
Nevo, Z., Raz, O., & Yorav, K. (2025) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew67777.2025.00011 · Cited by 1
“Offline evaluation methods often fail to capture the diversity of valid explanations, especially when multiple interpretations are possible [3].”
Awad, M. N. A., Ivanov, S. V., & Tikhonova, O. B. (2025) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew67777.2025.00032 · Cited by 2
“However, despite their power, LLM-driven code suggestions often fail to align with developer intent in real-time, leading to low acceptance rates, disrupted workflows, and wasted computational resources [1].”
AI and the democratization of knowledge work
Daepp, M. I. G., Tomlinson, K., Counts, S., & Suri, S. (2026) · Nature Computational Science · doi:10.1038/s43588-026-00985-z · Cited by 3
“Similar productivity benefits are observed for other knowledge-work tasks spanning from programming 20,21 to advertisement creation 22 .”
Beyond the Commit: Developer Perspectives on Productivity with AI Coding Assistants
Chen, V., He, J., Williams, B., Valentino, J., & Talwalkar, A. (2026) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3786583.3786848 · Cited by 13
Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMs
Al-Kaswan, A., Spieß, C. K., Dévanbu, P., Arie Van Deursen, A., & Izadi, M. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793341 · Cited by 1
“For instance, models like GitHub Copilot, built on top of LLMs, have been widely adopted by developers, with millions of users leveraging them daily for productivity gains [56].”
VOOL: A modular insight-based framework for vocalizing OLAP sessions
Francia, M., Gallinucci, E., Golfarelli, M., & Rizzi, S. (2024) · Information Systems · doi:10.1016/j.is.2024.102496 · Cited by 2
The fools are certain; the wise are doubtful: Exploring LLM confidence in code completion
Kotti, Z., Dritsa, K., Spinellis, D. D., & Λουρίδας, Π. (2026) · Automated Software Engineering · doi:10.21203/rs.3.rs-7302683/v1 · Cited by 3
“Empirical research largely focuses on Copilot (Wang and Chen, 2023), examining its applications (Moroz et al, 2022), impact on productivity (Ziegler et al, 2024), and developer interaction (Barke et al, 2023).”
Self-Admitted GenAI Usage in Open-Source Software
Xiao, T., Fan, Y., Calefato, F., Treude, C., Kula, R. G., Hata, H., & Baltes, S. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3681886 · Cited by 1
Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability
Borg, M., Hewett, D., Hagatulah, N., Couderc, N., Söderberg, E., Graham, D., Kini, U., & Farley, D. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10889-1 · Cited by 4
Wivestad, V. T., & Ulfsnes, R. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-031-72781-8_5
Predicting ChatGPT’s Ability to Solve Complex Programming Challenges
Ho, N., May, J., Ngo, B., Formato, J., Ngo, L., Ho, V. L., & Bui, H. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825139
Quantifying AI’s Impact on Software Infrastructure: Beyond the Hype
Agrawal, V. (2025) · Journal of Artificial Intelligence Machine Learning and Data Science · doi:10.51219/jaimld/vishakha-agrawal/545
Interview Study to Determine the Potential of Large Language Model Usage in IT-Consultancy
Jehnert, M., Meyer, C., & Sandkuhl, K. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-031-94193-1_7
Rouabhia, D., & Hadjadj, I. (2025) · Research Square · doi:10.21203/rs.3.rs-6793290/v1 · Cited by 9
Evaluating the Development Experience in Open Science Tools: A Case Study of the dataWASHES API
de Souza, A. A., Viana, D., dos Santos, R. P., & Araújo, A. A. (2025) · doi:10.5753/washes.2025.8006
AI and Work Engagement: A Study of IT Professionals Through the Lens of Self-Determination Theory
Cajander, Å., Bergqvist, A., Clear, T., Daniels, M., Humble, N., Larusdottir, M. K., Normark, M., & Ouhbi, S. (2025) · IFIP advances in information and communication technology · doi:10.1007/978-3-031-95334-7_9
Predicting multidimensional cubes through intentional analytics
Francia, M., Rizzi, S., Golfarelli, M., & Marcel, P. (2025) · Information Systems · doi:10.1016/j.is.2025.102628
“Despite the advancements in AI and LLMs, it is also recognized that applications such as Data Scientist, Analyst, and GTP-4o can, at the state of the art, support data scientists but not automatically pilot the extraction of insights [6,7].”
SLIDE: A machine-learning based method for forced dynamic response estimation of multibody systems
Manzl, P., Humer, A., Khadim, Q., & Gerstmayr, J. (2025) · Mechanics Based Design of Structures and Machines · doi:10.1080/15397734.2025.2559325 · Cited by 1
“…to multibody system dynamics was shown in [20], where natural language is used to create simulation models to assist and speed up the development of the models – similar to tools like Github Copilot or Visual Studio IntelliCode, which were shown to have a big impact on productivity [21].”
Künstliche Intelligenz in der Softwareentwicklung
Steghöfer, J.-P., Amslinger, R., & Nordsieck, R. (2025) · Springer eBooks · doi:10.1007/978-3-662-71567-3_20
The Impact of Generative AI on Code Expertise Models: An Exploratory Study
Cury, O., & Avelino, G. A. (2025) · Simpósio Brasileiro de Engenharia de Software (SBES) · doi:10.5753/sbes.2025.11074
Twelve quick tips for AI-assisted coding in science
Bridgeford, E., Campbell, I. D., Chen, Z., Lin, Z., Ritz, H., Vandekerckhove, J., & Poldrack, R. A. (2026) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1014428 · Cited by 11
Cunningham, A., Barbotin, N., Drogemuller, A., & Fraser, J. (2025) · IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) · doi:10.1109/ismar-adjunct68609.2025.00042
“For example, GitHub Copilot 1 is a conversational agent that allows developers to “auto-complete” code while programming within an IDE and has seen enabled productivity gains from developers utilising the tool [34].”
Peeking inside the black box: Training data exposure in code language models
Spina, A., Russodivito, M., Scalabrino, S., & Oliveto, R. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112729 · Cited by 3
HiLDE: Intentional Code Generation via Human-in-the-Loop Decoding
Gonzalez, E. A., Rothkopf, R., Lerner, S., & Polikarpova, N. (2025) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc65237.2025.00032 · Cited by 2
The Prompt–Refactor–Verify (PRV) Cycle: A Human-Centered Framework for AI-Assisted Programming
Yanev, N., Getova, I. D., Mihaylova, E., Kostadinova, I. S., Bankovska, M., & Dimitrov, G. (2025) · International Conference on Communications, Information, Electronic and Energy Systems (CIEES) · doi:10.1109/ciees66347.2025.11300239 · Cited by 2
Lessons Learned from the M365 AI Copilot Trial
Bano, M., Zowghi, D., Whittle, J., & Zhu, L. (2026) · IEEE Software · doi:10.1109/ms.2026.3654785
The Expert Validation Framework (EVF): Enabling Domain Expert Control in AI Engineering
Gren, L., & Dobslaw, F. (2026) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1145/3793653.3793796 · Cited by 1
Context-Aware CodeLLM Eviction for AI-assisted Coding
Thangarajah, K., Chen, B., Chang, S., & Hassan, A. E. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00271 · Cited by 4
“AI-assisted coding with CodeLLMs has significantly reshaped software development, boosting productivity by reducing both coding time and errors [3]–[8].”
The Productivity Paradox of AI-Powered Development
Fortes, L., Pereira, G. V., Coelho, A., Prikladnicki, R., & Kohl, K. (2026) · IEEE Software · doi:10.1109/ms.2026.3658651 · Cited by 1
Matejić, T., Mihailović, Đ., & Dukić, A. (2025) · Limes plus. · doi:10.69899/limes-plus25222046m
Esnaola, L. M., Ramón, H. D., & Lanzarini, L. C. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10813-7 · Cited by 2
“…ecological validity and feasibility: (i) a group of non-programmers without AI would likely exhibit a predictably failing performance, pro - viding only a redundant baseline, and (ii) a group of programmers with AI has already been extensively studied with consistent findings (Ziegler et al. 2024).”
InfluentialGenerating REST API Tests With Descriptive Names: Practitioner Surveys and Industrial Case StudyPreprint
Garrett, P., Galeotti, J. P., Arcuri, A., Poth, A., & Rrjolli, O. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6257117
AI-Assisted Collaboration: Exploring Developer Experience with GitHub Copilot and Windsurf
Winckler, S. C., Ribeiro, J. F., Machado, L. R., & Kroll, J. (2026) · IEEE Software · doi:10.1109/ms.2026.3671677
Gaikwad, P. (2026) · International Journal of Advanced Research in Science Communication and Technology · doi:10.48175/ijarsct-31627
Singh, P., Gaspari, M., Meratnia, N., et al. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.115800
The Coming Commoditization of Computational Thinking
Schmidt, D. C. (2026) · Communications of the ACM · doi:10.1145/3764667
“The value of LLMs goes beyond programming speed by enabling developers to work at a higher level of abstraction, focusing more on intent and design rather than slogging through boilerplate [10].”
Xue, Q., Yu, Y., yan, Y., Zhang, J., & Li, Z. (2026) · Applied Soft Computing · doi:10.1016/j.asoc.2026.115112
The Supervisory Paradox: How AI Delegation Transforms Productive Labor into Cognitive VigilancePreprint
Tripp, J. F. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6503239 · Cited by 1
de Oliveira, L. A. F., & de Oliveira, C. C. (2026) · Research Society and Development · doi:10.33448/rsd-v15i4.50966 · Cited by 1
CodeWatcher: IDE Telemetry Data Extraction Tool for Understanding Coding Interactions with LLMsPreprint
Rodríguez-Pérez, G., & Basha, M. (2025) · OSF Preprints (OSF Preprints)
Jiang, T. (2026) · Journal of Intelligence · doi:10.3390/jintelligence14050089 · Cited by 1
How students use generative AI for software testing: An observational study
Ardıç, B., Dilavrec, Q. L., & Zaidman, A. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10898-0 · Cited by 5
Yeşilyayla, H. (2026) · Journal of Business Innovation and Governance · doi:10.54472/jobig.1921341
Herbold, S., Trautsch, A., Kikteva, Z., & Hautli-Janisz, A. (2026) · PLoS ONE · doi:10.1371/journal.pone.0347757 · Cited by 11
“Modern Generative Artificial Intelligence models (GenAI) like GPT [1], Claude [2], and Gemini [3] have been shown to generate high-quality textual content including source code [4], persuasive student essays [5], and legal analysis [6].”
A Needle in a Haystack: Developers’ Information Foraging on the Wild Web
Kuttal, S. K., Sthapit, S., Phillips, R., & Rahal, P. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29175-2_17
Metadata
- Authors
- Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E.
- Affiliations
- GitHub, Inc., San Francisco, CA, USA
- GitHub, San Francisco, CA, USA
- Venue
- Communications of the ACM
- Published
- 2024-02-22
- Volume
- 67
- Issue
- 3
- Pages
- 54-63
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W4392023437
- Cited by
- 124
- References
- 26
- Citations by year
- 2026: 58 · 2025: 53 · 2024: 13
- FWCI
- 46.42
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Communications of the ACM (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
- Keywords
- GitHub Copilot, AI-assisted programming, developer productivity, user perception, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.