The Impact of AI on Developer Productivity: Evidence from GitHub Copilot

Empirical measurements often show more modest improvements than self-reported productivity gains from AI assistants.

“While developers consistently report significant productivity gains from these tools, empirical measurements often show more modest improvements.”

Human-AI collaboration consistently improves measurable output quality in professional writing contexts.

“human-AI collaboration consistently improves measurable output quality in professional writing contexts”

A GitHub experiment with 95 participants found task completion was reduced by 55.8% in the Copilot condition.

“GitHub performed such an experiment where 95 participants were split into two groups and asked to write a web server.The study concluded by fnding that task completion was reduced by 55.8% in the Copilot condition [28].”

Evidence from existing laboratory experiments generally suggests significant productivity effects of generative AI tools for software developers.

“Evidence from existing (laboratory) experiments generally suggests significant productivity effects of generative AI tools for software developers.”

Peng et al. (2023) found that professional programmers using GitHub Copilot completed an HTTP server task 55.8% faster, with no effect on whether the task was completed.

“Peng et al. (2023) hired 95 professional programmers to develop an HTTP server in JavaScript.The group that used GitHub Copilot completed the task 55.8% faster, yet there was no effect on whether the task was completed.”

AI tools demonstrably increase short-term productivity through faster task completion and reduced blocked time.

“While our participants reported, and prior work suggests [14,33,38,40], AI tools demonstrably increase short-term productivity through faster task completion and reduced blocked time”

Novice programmers may place undue trust in AI-generated code without fully understanding its broader design or integration implications, increasing the likelihood of downstream quality and integration challenges.

“However, both Dakhel et al. (2023) and Peng et al. (2023) caution that novice programmers may place undue trust in AI-generated code without fully understanding its broader design or integration implications, increasing the likelihood of downstream quality and integration challenges.”

The introduction of GitHub Copilot leads to higher development activity at both the repository and individual levels, measured by commits and PR.

“First, the introduction of GitHub Copilot leads to higher development activity at both the repository and individual levels, measured by commits and PR”

Prior works have presented mixed results on whether AI helps or hinders coding productivity.

“Prior works have presented mixed results on whether AI helps or hinders coding productivity [Peng et al., 2023, Becker et al., 2025]; our study differs from prior results in that it is designed to study how AI affects skill formation while performing a task requiring new knowledge.”

For software engineering, AI assistance significantly improves productivity in daily tasks.

“Software engineering, in particular, has been identified as a profession in which AI tools can be readily applied and AI assistance significantly improves productivity in daily tasks [Peng et al., 2023, Cui et al., 2024].”

AI assistance produces significant productivity gains across professional domains, particularly for novice workers.

“AI assistance produces significant productivity gains across professional domains, particularly for novice workers.”

Cui et al. (2024) estimate a 26% productivity gain from GitHub Copilot among software developers, and Peng et al. (2023) report a 58% increase.

“Cui et al. (2024) estimate a 26% productivity gain from GitHub Copilot among software developers, while Peng et al. (2023) report a 58% increase.”

Existing empirical work on the micro-level effects of generative AI tools tends to find that access to these tools benefits less experienced workers more, compressing performance distributions.

“Existing empirical work on the micro-level effects of generative AI tools tends to find that access to these tools benefits less experienced workers more, compressing performance distributions [22; 23; 10; 24].”

Less experienced developers may rely more heavily on AI assistance.

“Prior work suggests that less experienced developers may rely more heavily on AI assistance [9]”

LLM-based code generation shows significant productivity improvements (e.g., GitHub Copilot improves productivity 55.8%).

“LLM-based code generation shows significant productivity improvements (e.g., GitHub Copilot improves productivity 55.8% (Peng et al., 2023))”

The findings offer only partial support for the productivity gain hypothesis previously reported in technical writing contexts: while human-AI collaboration produces consistently acceptable prose with low variance, it does not demonstrably outperform original human writing in perceived quality, and gains from AI assistance in professional and technical domains do not translate straightforwardly to

“Our findings offer only partial support for the 'productivity gain' hypothesis previously reported in technical writing contexts (Noy and Zhang, 2023; Peng et al. 2023): while human-AI collaboration produces consistently acceptable prose with low variance, it does not demonstrably outperform original human writing in perceived quality”

Peng et al. (2023) found that software developers using GitHub Copilot completed entry-level tasks 55% faster than the control group.

“Peng et al. (2023) found comparable effects among software developers using GitHub Copilot, who completed entry-level tasks 55% faster than the control group.”

LLMs can streamline research processes, enhance productivity, and allow experts to focus on higher-level analytical tasks.

“Dell’Acqua et al. [1] and Peng et al. [2] demonstrated that these tools can streamline research processes, enhance productivity, and allow experts to focus on higher-level analytical tasks.”

Prior studies show that generative AI can enhance individual developer productivity, measured as the amount of code generated per developer.

“Prior studies show that generative AI can enhance individual developer productivity, measured as the amount of code generated per developer (i.e., intensive margin)”

AI pair programmers reduce the expected costs of code contributions by lowering the time and effort required to complete development tasks.

“while reducing expected costs by lowering the time and effort required to complete development tasks.”

Prior studies on generative AI generally show that lower skilled individuals benefit more from AI tools because AI reduces task completion costs.

“Prior studies on generative AI generally show that lower skilled individuals benefit more from AI tools because AI reduces task completion costs (e.g., Peng et al. 2023, Cui et al. 2026).”

A recent GitHub study found Copilot could potentially reduce task completion time by a factor of two.

“In fact, a recent study from GitHub, found that Copilot could potentially reduce task completion time by a factor of two [28].”

The finding of no statistically significant difference after adopting Copilot differs from prior studies (Al-Ahmad et al., 2024; Cui et al., 2024; Peng et al., 2023; Rajbhoj et al., 2024).

“For Copilot users, our findings suggest a relatively small increase in average weekly activity; however, there is no statistically significant difference after adopting Copilot, which differs from prior studies (Al-Ahmad et al., 2024; Cui et al., 2024; Peng et al., 2023; Rajbhoj et al., 2024).”

Prior studies focus on short-term gains in controlled or semi-controlled settings, either by measuring task completion time or by simulating software projects.

“Instead, they focus on short-term gains in controlled or semi-controlled settings, either by measuring task completion time (Peng et al., 2023), or by simulating software projects to illustrate how GenAI tools can assist developers during different phases of the development process (Al-Ahmad et al., 2024; Rajbhoj et al., 2024).”

Peng et al. (2023) found that participants with access to GitHub Copilot completed standardized programming tasks 55.8% faster.

“For example, Peng et al. (2023) found that participants with access to GitHub Copilot completed standardized programming tasks 55.8% faster.”

GitHub Copilot increases software developer productivity.

“GenAI solutions like GitHub Copilot have been shown to increase the productivity of software developers.”

AI tools lower the skill barrier for writing code.

“Because AI tools can lower the skill barrier for writing code (Dakhel et al. 2023)”

Less experienced programmers appear to benefit more from the use of Copilot.

“These maintenance-related concerns are amplified by evidence that less experienced programmers appear to benefit more from the use of Copilot.”

GenAI coding assistants have demonstrated considerable potential to enhance developer productivity.

“GenAI coding assistants have demonstrated considerable potential to enhance developer productivity.”

Cui et al. (2025), Gambacorta, Qiu, Shan, and Rees (2024), and Peng, Kalliamvakou, Cihon, and Demirer (2023) document strong productivity increases for software development tasks, with larger gains among less experienced developers.

“Cui et al. (2025); Gambacorta, Qiu, Shan, and Rees (2024); Peng, Kalliamvakou, Cihon, and Demirer (2023) document strong productivity increases for software development tasks, again with larger gains among less experienced developers.”

Contrary to previous work, our results do not show a significant improvement in productivity if we only look at the total completion time across the treatment and control groups.

“Contrary to previous work finding significant uplift or speedup of AI assistance for coding [Peng et al., 2023, Cui et al., 2024], our results do not show a significant improvement in productivity if we only look at the total completion time across the treatment and control groups.”

Randomized trials find productivity gains above 15% in the occupations studied here.

“significant productivity gains-often exceeding 15%-documented by RCTs in our study occupations”

Reported time savings are highest in occupations covered by RCTs, including customer and IT support, marketing and human resources, and software development.

“First, our reported time savings are highest precisely in the occupations covered by RCTs, such as customer and IT support (Brynjolfsson, Li and Raymond, 2025), marketing and human resources (Noy and Zhang, 2023), and software development (Peng et al., 2023).”

Peng et al. found 56% speedup on coding tasks when using AI assistance.

“Peng et al. [10] and Paradis et al. [11] find 56% and 21% speedups on coding tasks when using AI assistance”

Frontier AI systems demonstrate impressive capabilities on a wide range of software benchmarks and in experiments measuring AI's impact on developer productivity when completing synthetic tasks.

“Frontier AI systems demonstrate impressive capabilities on a wide range of software benchmarks [3; 4; 5; 6; 7; 8; 9] and in experiments measuring AI's impact on developer productivity when completing synthetic tasks [10; 11].”

Peng et al.'s task is unrepresentative of most software development work and likely similar to a large amount of LLM training data, which may unfairly advantage AI systems relative to humans.

“For example, Peng et al. [10] asks developers to implement a very basic HTTP server in JavaScript to satisfy several automatic test cases that are shown to the developers-this task is a) unrepresentative of most software development work, and b) likely to be similar to a large amount of LLM training data, which may unfairly advantage AI systems relative to humans.”

Peng et al. [10] found AI made developers 56% faster.

“Peng et al. [10] | ↑ 56% faster”

AI helps the study's developers less than in existing literature because the developers have substantially more experience on their repositories (5 years and 1,500 commits on average).

“We hypothesize that analogously, AI helps our developers less compared to existing literature [10; 11] because our developers have substantially more experience on their respective repositories (5 years and 1,500 commits on average).”

Limited token spend would not explain the difference between the study's results and other work that finds positive speedup for programming tasks.

“We note that limited token spend would not explain the difference between our results and other work that find positive speedup for programming tasks [10; 17].”

AI broadly helps the study's developers less than in existing randomized controlled trials because of the overall size and complexity of the included repositories.

“We hypothesize that analogously to these size and complexity effects within our study, AI broadly helps our developers less compared to existing randomized controlled trials (RCTs) measuring speedup from AI tools because of the overall size and complexity of the repositories included”

In a controlled experiment with 95 professional developers, access to GitHub Copilot led to a 55.8% reduction in task completion time for implementing an HTTP server, with less experienced and older developers benefiting the most.

“In a controlled experiment with 95 professional developers, Peng et.al. [9] found that access to GitHub Copilot led to a 55.8% reduction in task completion time for implementing an HTTP server, with less experienced and older developers benefiting the most.”

The 34.9% reduction in task completion time and 26.2% improvement in test pass rates align with previous studies of GenAI programming assistants on professional programmers and undergraduate students in greenfield scenarios.

“The 34.9% reduction in task completion time and 26.2% improvement in test pass rates align with previous studies of the impact of GenAI programming assistants on both professional programmers [26, 28] and undergraduate students [13, 32] in greenfield programming scenarios.”

Peng et al. (2023) found that the treatment group using GitHub Copilot completed the task 55.8% faster on average.

“The authors report that the treatment group completed the task 55.8% faster on average.”

GitHub's research (Peng et al. 2023) reported a 55% improvement in completion time from working with AI.

“GitHub's research (Peng et al. 2023), which reported a 55% improvement.”

Peng et al. (2023) reported a 55.8% speedup when implementing an HTTP server in JavaScript.

“For example, Peng et al. (2023) reported a 55.8% speedup when implementing an HTTP server in JavaScript;”

Peng et al. (2023) reported that junior developers benefited the most from GitHub Copilot.

“The early work by Peng et al. (2023) reported that junior developers benefited the most from GitHub Copilot,”

Developers who used Copilot completed a JavaScript-based task 55.8% faster than those who did not have access to the tool.

“Developers who used Copilot completed a JavaScript-based task 55.8% faster than those who did not have access to the tool”

A controlled experiment by Peng et al. (2023) found that GitHub Copilot significantly boosted productivity: the group using Copilot completed programming tasks approximately 56% faster than the control group without access to the tool.

“For example, a controlled experiment by Peng et al. (2023) found that GitHub Copilot, a generative AI programming aid, significantly boosted productivity: the group using Copilot completed programming tasks approximately 56% faster than the control group without access to the tool.”

Current studies have small sample sizes.

“Other shortcomings include missing nuance or understanding of the impact of AI tools by developer- or task-level contexts (e.g., [13]), limited tool and task complexity in experimental settings (e.g., using GitHub Classroom rather than a more naturalistic developer environment [12]), and small sample sizes (e.g., only 32 participants in [15], 24 participants in [16], or 21 participants in [17]).”

The operational and environmental costs of these models pose significant challenges.

“their operational and environmental costs pose significant challenges [6, 30].”

Our results are consistent with those of Peng et al. (2023).

“Our results are consistent with those of Peng et al. (2023)”

Copilot's suggestions are perceived as invasive by experienced developers.

“Peng et al. [12] showed that Copilot's suggestions are perceived as invasive by experienced developer.”