The DOI matches its DataCite record; 16 of the 16 references it cites were found in a registry or an index; 2 of the 5 sources its claims cite were checked for support.
The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
Open accessAbstract
Generative AI tools hold promise to increase human productivity. This paper presents results from a controlled experiment with GitHub Copilot, an AI pair programmer. Recruited software developers were asked to implement an HTTP server in JavaScript as quickly as possible. The treatment group, with access to the AI pair programmer, completed the task 55.8% faster than the control group. Observed heterogenous effects show promise for AI pair programmers to help people transition into software development careers.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
6 Own method · 6 Own interpretation · 6 Own result · 1 Partly supported · 1 Not checked yet · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“there has been little research on productivity impacts of AI-powered tools in professional contexts, cf.[Mozannar et al., 2022, Vaithilingam et al., 2022, Ziegler et al., 2022].”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Introduction
What the sources say
- Partly supportedcites Mozannar et al., 2022 sourceExperimental — accuracy not yet measured
The Introduction (no page supplied) says existing evaluations of AI code completion leave insufficient detail about productivity gains and possible inefficiencies. That supports a narrower research gap in AI-assisted programming, but not the broader claim that little research has examined productivity impacts of AI tools across professional contexts. The Background and Related Work section, in fact, describes several productivity studies. (LLM, conf=0.78)
- Not checked — no full text of the sourcecites Vaithilingam et al., 2022 source
- Partly supportedcites Ziegler et al., 2022 sourceExperimental — accuracy not yet measured
The Background section (page unavailable) describes this as the first study, to the authors’ knowledge, linking code-suggestion-tool usage measurements with developer productivity or happiness, and identifies two earlier studies. That supports limited prior research on AI code tools, but not the broader claim about AI-powered tools across professional contexts. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The treatment group, with access to the AI pair programmer, completed the task 55.8% faster than the control group.”
- Also stated in: § Introduction · plain
“the treated group completed the task 55.8% faster (95% confidence interval: 21-89%).”
- Also stated in: § Results · plain
“This represents a 55.8% reduction in completion time.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Introduction · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The p-value for the t-test is 0.0017, and a 95% confidence interval for the improvement is between [21%, 89%].”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This result suggests that Copilot increases average productivity significantly in our experiment population.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Results
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In this way, we accurately measured the timing and completion for each participant.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Study Design
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This result provides indirect evidence that treated group benefited from Copilot during their task as their willingness to pay is significantly higher than the control group.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Results
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Observed heterogenous effects show promise for AI pair programmers to help people transition into software development careers.”
- Also stated in: § Introduction · hedged
“These heterogeneous effects point towards promise for AI-pair programmers in support of expanding access to careers in software development.”
- Also stated in: § Discussion · hedged
“the productivity benefits for novice programmers and programmers of older age point to important possibilities for skill initiatives that support job transitions into software development.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Abstract
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The treatment effect is a linear function of the covariates of interest.
- Also stated in: § Introduction · hedged
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We verify from telemetry after the experiment that all participants from the treated group have configured GitHub Copilot and accepted recommendations other than five who did not finish the sign up and thus started the experiment without the GitHub Copilot.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Study Design
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 6
- The quote only partly states this claim. 1
- Probably not a substantial claim. 7
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On average, participants in both treated and control groups estimated a 35% increase in productivity, which is an underestimation compared with the 55.8% increase in their revealed productivity.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Conditioning on completing the task, the average completion time from the treated group is 71.17 minutes and 160.89 minutes for the control group.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results show that less experienced developers (years of professional coding), developers with heavy coding load (hours of coding per day), and older developers (developers aged between 25 and 44) benefit more from Copilot.”
- Also stated in: § Introduction · plain
“Developers with less programming experience, older programmers, and those who program more hours per day benefited the most.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The treatment effect is a linear function of the covariates of interest.
- Also stated in: § Introduction · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also find that the treated group's success rate is 7 percentage points higher than the control group, but the estimate is not statistically significant, with a 95% confidence interval of [-0.11, 0.25].”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If this result persists in further studies, the productivity benefits for novice programmers and programmers of older age point to important possibilities for skill initiatives that support job transitions into software development.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
- The authors state this as an assumption: The treatment effect is a linear function of the covariates of interest.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Once participants signed the contract, they were randomly split into control and treatment groups.”
- Also stated in: § Results · plain
“The 95 developers were randomly assigned into control and treated groups, with 45 in the treated group and 50 in control.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Study Design
plainn = 50- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Also stated in: § Results · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If the results of this study were to be extrapolated to the population level, a 55.8% increase in productivity would imply a significant amount of cost savings in the economy and have a notable impact on GDP growth.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants could see the tests but were unable to alter them.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Study Design
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“The potential productivity impacts of AI have major implications for the labor market and firms, including changes in employment, skills, and firm organization [Raj andSeamans, 2018, Agrawal et al., 2019].”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Introduction
What the sources say
- Not matchedcites Raj andSeamans, 2018
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites Agrawal et al., 2019 source
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Thirty-five developers from both the treated and control groups completed the task and survey.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Productivity benefits may vary across specific tasks and programming languages, so more research is needed to understand how our results generalizes to other tasks.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Using a standardized task provides us with precise measures of performance as it is difficult to measure productivity of software developers.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Study Design
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The authors state this as an assumption.
The treatment effect is a linear function of the covariates of interest.
“We assume the treatment effect is a linear function of the covariates of interest.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“While developers consistently report significant productivity gains from these tools, empirical measurements often show more modest improvements.”
From Towards Decoding Developer Cognition in the Age of AI Assistants (2025) · cites this paper as [9]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“human-AI collaboration consistently improves measurable output quality in professional writing contexts”
From How Creative Is AI Writing? Generative and Collaborative AI in Japanese Fiction (2026) · cites this paper as Peng et al. (2023)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GitHub performed such an experiment where 95 participants were split into two groups and asked to write a web server.The study concluded by fnding that task completion was reduced by 55.8% in the Copilot condition [28].”
From Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming (2024) · cites this paper as [28]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Evidence from existing (laboratory) experiments generally suggests significant productivity effects of generative AI tools for software developers.”
From The Productivity Effects of Generative AI: Evidence from a Field Experiment with GitHub Copilot (2024) · cites this paper as Peng et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Peng et al. (2023) hired 95 professional programmers to develop an HTTP server in JavaScript.The group that used GitHub Copilot completed the task 55.8% faster, yet there was no effect on whether the task was completed.”
From The Productivity Effects of Generative AI: Evidence from a Field Experiment with GitHub Copilot (2024) · cites this paper as Peng et al. (2023)
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While our participants reported, and prior work suggests [14,33,38,40], AI tools demonstrably increase short-term productivity through faster task completion and reduced blocked time”
From The Evolution of Information Seeking in Software Development: Understanding the Role and Impact of AI Assistants (2025) · cites this paper as [33]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, both Dakhel et al. (2023) and Peng et al. (2023) caution that novice programmers may place undue trust in AI-generated code without fully understanding its broader design or integration implications, increasing the likelihood of downstream quality and integration challenges.”
From AI-Assisted Programming Decreases the Productivity of Experienced Developers by Increasing the Technical Debt and Maintenance Burden (2025) · cites this paper as Peng et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“First, the introduction of GitHub Copilot leads to higher development activity at both the repository and individual levels, measured by commits and PR”
From AI-Assisted Programming Decreases the Productivity of Experienced Developers by Increasing the Technical Debt and Maintenance Burden (2025) · cites this paper as Peng et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prior works have presented mixed results on whether AI helps or hinders coding productivity [Peng et al., 2023, Becker et al., 2025]; our study differs from prior results in that it is designed to study how AI affects skill formation while performing a task requiring new knowledge.”
From How AI Impacts Skill Formation (2026) · cites this paper as Peng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Software engineering, in particular, has been identified as a profession in which AI tools can be readily applied and AI assistance significantly improves productivity in daily tasks [Peng et al., 2023, Cui et al., 2024].”
From How AI Impacts Skill Formation (2026) · cites this paper as Peng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI assistance produces significant productivity gains across professional domains, particularly for novice workers.”
From How AI Impacts Skill Formation (2026) · cites this paper as Peng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Cui et al. (2024) estimate a 26% productivity gain from GitHub Copilot among software developers, while Peng et al. (2023) report a 58% increase.”
From Still Waters, Rapid Currents: Early Labor Market Transformation under Generative AI (2025) · cites this paper as Peng et al. (2023)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing empirical work on the micro-level effects of generative AI tools tends to find that access to these tools benefits less experienced workers more, compressing performance distributions [22; 23; 10; 24].”
From Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prior work suggests that less experienced developers may rely more heavily on AI assistance [9]”
From Towards Decoding Developer Cognition in the Age of AI Assistants (2025) · cites this paper as [9]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLM-based code generation shows significant productivity improvements (e.g., GitHub Copilot improves productivity 55.8% (Peng et al., 2023))”
From CodeCRDT: Observation-Driven Coordination for Multi-Agent LLM Code Generation (2025) · cites this paper as Peng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Our findings offer only partial support for the 'productivity gain' hypothesis previously reported in technical writing contexts (Noy and Zhang, 2023; Peng et al. 2023): while human-AI collaboration produces consistently acceptable prose with low variance, it does not demonstrably outperform original human writing in perceived quality”
From How Creative Is AI Writing? Generative and Collaborative AI in Japanese Fiction (2026) · cites this paper as Peng et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Peng et al. (2023) found comparable effects among software developers using GitHub Copilot, who completed entry-level tasks 55% faster than the control group.”
From How Creative Is AI Writing? Generative and Collaborative AI in Japanese Fiction (2026) · cites this paper as Peng et al. (2023)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Dell’Acqua et al. [1] and Peng et al. [2] demonstrated that these tools can streamline research processes, enhance productivity, and allow experts to focus on higher-level analytical tasks.”
From Integrating large language models in biostatistical workflows for clinical and translational research (2025) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prior studies show that generative AI can enhance individual developer productivity, measured as the amount of code generated per developer (i.e., intensive margin)”
From The Impact of Generative AI on Collaborative Open-Source Software Development: Evidence from GitHub Copilot (2024) · cites this paper as Peng et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“while reducing expected costs by lowering the time and effort required to complete development tasks.”
From The Impact of Generative AI on Collaborative Open-Source Software Development: Evidence from GitHub Copilot (2024) · cites this paper as Peng et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prior studies on generative AI generally show that lower skilled individuals benefit more from AI tools because AI reduces task completion costs (e.g., Peng et al. 2023, Cui et al. 2026).”
From The Impact of Generative AI on Collaborative Open-Source Software Development: Evidence from GitHub Copilot (2024) · cites this paper as Peng et al. 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In fact, a recent study from GitHub, found that Copilot could potentially reduce task completion time by a factor of two [28].”
From Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming (2024) · cites this paper as [28]
The cited source says what the claim attributes to it.
In the citing paper
“For Copilot users, our findings suggest a relatively small increase in average weekly activity; however, there is no statistically significant difference after adopting Copilot, which differs from prior studies (Al-Ahmad et al., 2024; Cui et al., 2024; Peng et al., 2023; Rajbhoj et al., 2024).”
From Developer Productivity With and Without GitHub Copilot: A Longitudinal Mixed-Methods Case Study (2026) · cites this paper as Peng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Instead, they focus on short-term gains in controlled or semi-controlled settings, either by measuring task completion time (Peng et al., 2023), or by simulating software projects to illustrate how GenAI tools can assist developers during different phases of the development process (Al-Ahmad et al., 2024; Rajbhoj et al., 2024).”
From Developer Productivity With and Without GitHub Copilot: A Longitudinal Mixed-Methods Case Study (2026) · cites this paper as Peng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“For example, Peng et al. (2023) found that participants with access to GitHub Copilot completed standardized programming tasks 55.8% faster.”
From Developer Productivity With and Without GitHub Copilot: A Longitudinal Mixed-Methods Case Study (2026) · cites this paper as Peng et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“GenAI solutions like GitHub Copilot have been shown to increase the productivity of software developers.”
From AI-Assisted Programming Decreases the Productivity of Experienced Developers by Increasing the Technical Debt and Maintenance Burden (2025) · cites this paper as Peng et al. 2023
The cited source says what the claim attributes to it.
In the citing paper
“Because AI tools can lower the skill barrier for writing code (Dakhel et al. 2023)”
From AI-Assisted Programming Decreases the Productivity of Experienced Developers by Increasing the Technical Debt and Maintenance Burden (2025) · cites this paper as Peng et al. 2023
The cited source says what the claim attributes to it.
In the citing paper
“These maintenance-related concerns are amplified by evidence that less experienced programmers appear to benefit more from the use of Copilot.”
From AI-Assisted Programming Decreases the Productivity of Experienced Developers by Increasing the Technical Debt and Maintenance Burden (2025) · cites this paper as Peng et al. 2023
The cited source says what the claim attributes to it.
In the citing paper
“Controlled experiments measure productivity gains of roughly 56% on isolated coding tasks (Peng et al., 2023)”
From Indie at Scale: How Large Language Models Make Community-Centered Game Development Economically Viable (2026) · cites this paper as Peng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“GenAI coding assistants have demonstrated considerable potential to enhance developer productivity.”
From The Effects of GitHub Copilot on Computing Students' Programming Effectiveness, Efficiency, and Processes in Brownfield Coding Tasks (2025) · cites this paper as [28]
The cited source says what the claim attributes to it.
In the citing paper
“Cui et al. (2025); Gambacorta, Qiu, Shan, and Rees (2024); Peng, Kalliamvakou, Cihon, and Demirer (2023) document strong productivity increases for software development tasks, again with larger gains among less experienced developers.”
From Labor Market Consequences of Generative AI: Early Evidence from Norway (2026) · cites this paper as Peng, Kalliamvakou, Cihon, and Demirer (2023)
The cited source says what the claim attributes to it.
In the citing paper
“Contrary to previous work finding significant uplift or speedup of AI assistance for coding [Peng et al., 2023, Cui et al., 2024], our results do not show a significant improvement in productivity if we only look at the total completion time across the treatment and control groups.”
From How AI Impacts Skill Formation (2026) · cites this paper as Peng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“significant productivity gains-often exceeding 15%-documented by RCTs in our study occupations”
From Still Waters, Rapid Currents: Early Labor Market Transformation under Generative AI (2025) · cites this paper as Peng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“First, our reported time savings are highest precisely in the occupations covered by RCTs, such as customer and IT support (Brynjolfsson, Li and Raymond, 2025), marketing and human resources (Noy and Zhang, 2023), and software development (Peng et al., 2023).”
From Still Waters, Rapid Currents: Early Labor Market Transformation under Generative AI (2025) · cites this paper as Peng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Peng et al. [10] and Paradis et al. [11] find 56% and 21% speedups on coding tasks when using AI assistance”
From Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025) · cites this paper as [10]
The cited source says what the claim attributes to it.
In the citing paper
“Frontier AI systems demonstrate impressive capabilities on a wide range of software benchmarks [3; 4; 5; 6; 7; 8; 9] and in experiments measuring AI's impact on developer productivity when completing synthetic tasks [10; 11].”
From Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025) · cites this paper as [10]
The cited source says what the claim attributes to it.
In the citing paper
“For example, Peng et al. [10] asks developers to implement a very basic HTTP server in JavaScript to satisfy several automatic test cases that are shown to the developers-this task is a) unrepresentative of most software development work, and b) likely to be similar to a large amount of LLM training data, which may unfairly advantage AI systems relative to humans.”
From Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025) · cites this paper as [10]
The cited source says what the claim attributes to it.
In the citing paper
“Peng et al. [10] | ↑ 56% faster”
From Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025) · cites this paper as Peng et al. [10]
The cited source says what the claim attributes to it.
In the citing paper
“We hypothesize that analogously, AI helps our developers less compared to existing literature [10; 11] because our developers have substantially more experience on their respective repositories (5 years and 1,500 commits on average).”
From Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025) · cites this paper as [10]
The cited source says what the claim attributes to it.
In the citing paper
“We note that limited token spend would not explain the difference between our results and other work that find positive speedup for programming tasks [10; 17].”
From Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025) · cites this paper as [10]
The cited source says what the claim attributes to it.
In the citing paper
“We hypothesize that analogously to these size and complexity effects within our study, AI broadly helps our developers less compared to existing randomized controlled trials (RCTs) measuring speedup from AI tools because of the overall size and complexity of the repositories included”
From Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025) · cites this paper as [10]
The cited source says what the claim attributes to it.
In the citing paper
“In a controlled experiment with 95 professional developers, Peng et.al. [9] found that access to GitHub Copilot led to a 55.8% reduction in task completion time for implementing an HTTP server, with less experienced and older developers benefiting the most.”
From Towards Decoding Developer Cognition in the Age of AI Assistants (2025) · cites this paper as [9]
The cited source says what the claim attributes to it.
In the citing paper
“The 34.9% reduction in task completion time and 26.2% improvement in test pass rates align with previous studies of the impact of GenAI programming assistants on both professional programmers [26, 28] and undergraduate students [13, 32] in greenfield programming scenarios.”
From The Effects of GitHub Copilot on Computing Students' Programming Effectiveness, Efficiency, and Processes in Brownfield Coding Tasks (2025) · cites this paper as [28]
The cited source says what the claim attributes to it.
In the citing paper
“The authors report that the treatment group completed the task 55.8% faster on average.”
From Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability (2026) · cites this paper as Peng et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“GitHub's research (Peng et al. 2023), which reported a 55% improvement.”
From Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability (2026) · cites this paper as Peng et al. 2023
The cited source says what the claim attributes to it.
In the citing paper
“For example, Peng et al. (2023) reported a 55.8% speedup when implementing an HTTP server in JavaScript;”
From Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability (2026) · cites this paper as Peng et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“The early work by Peng et al. (2023) reported that junior developers benefited the most from GitHub Copilot,”
From Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability (2026) · cites this paper as Peng et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“Developers who used Copilot completed a JavaScript-based task 55.8% faster than those who did not have access to the tool”
From How do Copilot Suggestions Impact Developers' Frustration and Productivity? (2025) · cites this paper as Peng et al. [12]
The cited source says what the claim attributes to it.
In the citing paper
“For example, a controlled experiment by Peng et al. (2023) found that GitHub Copilot, a generative AI programming aid, significantly boosted productivity: the group using Copilot completed programming tasks approximately 56% faster than the control group without access to the tool.”
From Large language models at work in China’s labor market (2025) · cites this paper as Peng et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“Other shortcomings include missing nuance or understanding of the impact of AI tools by developer- or task-level contexts (e.g., [13]), limited tool and task complexity in experimental settings (e.g., using GitHub Classroom rather than a more naturalistic developer environment [12]), and small sample sizes (e.g., only 32 participants in [15], 24 participants in [16], or 21 participants in [17]).”
From How Much Does AI Impact Development Speed? an Enterprise-Based Randomized Controlled Trial (2025) · cites this paper as [16]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“their operational and environmental costs pose significant challenges [6, 30].”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as [30]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Our results are consistent with those of Peng et al. (2023)”
From Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability (2026) · cites this paper as Peng et al. 2023
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Peng et al. [12] showed that Copilot's suggestions are perceived as invasive by experienced developer.”
From How do Copilot Suggestions Impact Developers' Frustration and Productivity? (2025) · cites this paper as [12]
Experimental — accuracy not yet measured
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 2
- [4]4. Athey, S. and Imbens, G. W. (2015). Machine learning methods for estimating heterogeneous causal effects. stat, 1050(5):1-26.
- [24]24. Raj, M. and Seamans, R. (2018). Artificial intelligence, labor, pro- ductivity, and the need for firm-level data. In The economics of artificial intelligence: An agenda, pages 553-565. University of Chicago Press.
Cited by
Sorted by citations
Items 301–340 of 340
Proxy-Based Interaction Logging for Traceability in AI-Assisted Software Development
Otsuka, A., Luo, J., Hasegawa, M., Bąk, A., & Ito, A. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-27927-9_32
Zare, A., & Assal, H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29723-5_35
Computational Approaches and the Future of Urban Crime Research
Campedelli, G. M., Jelveh, Z., Chalfin, A. J., Semenza, D. C., Piza, E. L., Zumel, A. A., Lepri, B., & Sharkey, P. T. (2026) · Nature · doi:10.1038/s41586-026-10622-4
Brandebusemeyer, C., Schimmer, T., & Bert Arnrich, A. (2026) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3786583.3786870 · Cited by 4
Who Tells the Documentation Team? How AI Agents Disrupt Documentation Feedback Loops
Bhat, A., & Guo, J. (2026) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3803437.3806708
Lifecycle-Aware GenAI Assistance with MCP via Context Refinement Loops: A Reference Architecture
Elsisi, O., Santos, F., & Melo, G. (2026) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3803437.3805617
Jamil, H. M. (2026) · ACM Transactions on Computing Education · doi:10.1145/3833217
Alexander, R., Carnegie, L., & Moon, N. (2026) · Journal of Statistics and Data Science Education · doi:10.1080/26939169.2026.2708860
Rapid Prototyping Adaptive Electron Microscopy Workflows with AI-Assisted Programming
Geurts, R. (2026) · Microscopy and Microanalysis · doi:10.1093/mam/ozag053.1011
Grasping AI Reliance in Program Comprehension and Coding through the AIRELI Persona Taxonomy
Alakmeh, T., Anderson, N. B., Jackson, V., Pereira, G. V., Akırmak, Ü., Estey, A., Prikladnicki, R., van der Hoek, A., Storey, M., & Fritz, T. (2026) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3794763.3794804
LGTM! Characteristics of Auto-Merged LLM-based Agentic PRs
Branco, R., Santos, P. H., Gamboa, C., & Fonseca, A. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793621
Gondauri, D. (2026) · Springer proceedings in business and economics · doi:10.1007/978-3-032-18962-2_8
Generative AI in the Workplace: Workers’ Experiences and Competence Development
Larsen, A. G., Skjuve, M. B., & Følstad, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-26717-7_17
Operationalizing AI in Regional Development: Evidence, Contexts, and Policy Implications
Chaniotou, N., Schenk, B., Lainevuo, A., Bartzanas, T., Kallio, A., Vilhu, E., Gercane, L., Kokko, H., Kallasvaara, H., Räty, J. P., et al. (2026) · Engineering cyber-physical systems and critical infrastructures · doi:10.1007/978-3-032-18275-3_1
RL–LLMfuzzer: reinforcement learning–guided dual–model differential fuzzing for compiler testing
Li, D., Li, Y., Zhao, B., Xu, J., Yao, J., Wang, N., & Liu, L. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-01115-5
From Design to Code: Exploring LLM-Supported Workflows in Figma Dev Mode
Rajpal, S., & Riener, A. (2026) · Mensch und Computer (MuC) · doi:10.1145/3820253.3820266
Digital nudging in software development: a review and research agenda
Pribik, I., & Felfernig, A. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1898557
Steinerová, J. (2026) · Acta Informatica Pragensia · doi:10.18267/j.aip.316
Siebertz, K. (2026) · Springer eBooks · doi:10.1007/978-3-662-74590-8_3
Assimilating generative AI into ideation work: seeding, search and human contribution
Sabbah, J., Banerjee, A., & Li, F. (2026) · Information Technology and People · doi:10.1108/itp-12-2025-1861
Siebertz, K. (2026) · Springer eBooks · doi:10.1007/978-3-662-74590-8_5
Berg, C. (2026) · Palgrave studies in classical liberalism · doi:10.1007/978-3-032-22310-4_4
Fink‐Lamotte, J., Günther, F., Haaf, J. M., Specht, J., Cludius, B., Hilbert, K., & Saulin, A. (2026) · Psychologische Rundschau · doi:10.1026/0033-3042/a000788
Command Coding: A Methodological Framework for AI-Accelerated Software Development
Villani, T. S., & Kellogg, M. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834536
Wittig, A., Mailach, A., & Siegmund, N. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834386
Shehral, M. A., Ravi, K., Trivedi, N., & Bayat, A. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834535
Ma, M., Wang, R., Bansal, C., Rajmohan, S., Dang, Y., & Zhang, H. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3837473
Beyond Productivity: AI, Community, and Sustainability in Open Source Software
Min, E., Frluckaj, H., Gaughan, M., Champion, K., Crowston, K., Hill, B. M., & Howison, J. (2026) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3785651.3818817
Choi, S. (2026) · The Journal of Korean Association of Computer Education · doi:10.32431/kace.2026.29.9.005
Başer, O. (2026) · Journal of health economics and outcomes research · doi:10.36469/001c.171942
Waiting is not Idle Time: Human Work in AI-Assisted Programming
Tian, F., Sun, Z., 英二, 柳., & Tong, S. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-6916-7_14
Exploring the Role of AI in Hybrid Work Within Software Engineering
Broomandi, F., & Paasivaara, M. (2026) · Lecture notes in business information processing · doi:10.1007/978-3-032-39205-3_13
Making AI Visible, Not Vanished: How AI Policies Reshape Developer Experience on GitHub
Chen, Y., Zimmermann, T., & Trinkenreich, B. (2026) · arXiv · doi:10.48550/arxiv.2608.03329
Who Pays the Review Cost? Triage, Fairness, and Accountability in AI-authored Pull Requests
Rahman, M. S., Alam, K., Roy, B., & Roy, C. K. (2026) · arXiv · doi:10.48550/arxiv.2610.11179
Plausible Is Not Enough: Trust and Evidence in AI-Authored Pull Request Review
Rahman, M. S., Alam, K., Roy, B., & Roy, C. K. (2026) · arXiv · doi:10.48550/arxiv.2610.21144
Autor, D., Rodchenko, T., Martin, J., Iscenko, Z., Strand, S., Pearl, D., & Ferere, M. (2026) · doi:10.3386/w35720
The Evolution of Information Seeking in Software Development: Understanding the Role and Impact of AI Assistants
Haque, E. A., Brown, C., LaToza, T. D., & Johnson, B. (2024) · arXiv
“””
"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2023) · arXiv
“Finally, we offer design implications based on all of the findings and insights we have presented.”
A Transformer-Based Approach for Smart Invocation of Automatic Code Completion
de Moor, A., van Deursen, A., & Izadi, M. (2024) · arXiv
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Metadata
- Authors
- Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M.
- Affiliations
- Microsoft (United States) (US)
- Massachusetts Institute of Technology (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2302.06590
- OpenAlex
- W4320854981
- Semantic Scholar
- 15abedb29536d50afeeec739a25358255cbda3e8
- Cited by
- 287
- References
- 0
- Citations by year
- 2026: 52 · 2025: 88 · 2024: 96 · 2023: 50
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
- Keywords
- GitHub Copilot, generative AI, developer productivity, software development, JavaScript, treatment effect heterogeneity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
