The DOI matches its DataCite record; 16 of the 16 references it cites were found in a registry or an index; 2 of the 5 sources its claims cite were checked for support.
The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
Open accessAbstract
Generative AI tools hold promise to increase human productivity. This paper presents results from a controlled experiment with GitHub Copilot, an AI pair programmer. Recruited software developers were asked to implement an HTTP server in JavaScript as quickly as possible. The treatment group, with access to the AI pair programmer, completed the task 55.8% faster than the control group. Observed heterogenous effects show promise for AI pair programmers to help people transition into software development careers.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
6 Own method · 6 Own interpretation · 6 Own result · 1 Partly supported · 1 Not checked yet · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“there has been little research on productivity impacts of AI-powered tools in professional contexts, cf.[Mozannar et al., 2022, Vaithilingam et al., 2022, Ziegler et al., 2022].”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Introduction
What the sources say
- Partly supportedcites Mozannar et al., 2022 sourceExperimental — accuracy not yet measured
The Introduction (no page supplied) says existing evaluations of AI code completion leave insufficient detail about productivity gains and possible inefficiencies. That supports a narrower research gap in AI-assisted programming, but not the broader claim that little research has examined productivity impacts of AI tools across professional contexts. The Background and Related Work section, in fact, describes several productivity studies. (LLM, conf=0.78)
- Not checked — no full text of the sourcecites Vaithilingam et al., 2022 source
- Partly supportedcites Ziegler et al., 2022 sourceExperimental — accuracy not yet measured
The Background section (page unavailable) describes this as the first study, to the authors’ knowledge, linking code-suggestion-tool usage measurements with developer productivity or happiness, and identifies two earlier studies. That supports limited prior research on AI code tools, but not the broader claim about AI-powered tools across professional contexts. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The treatment group, with access to the AI pair programmer, completed the task 55.8% faster than the control group.”
- Also stated in: § Introduction · plain
“the treated group completed the task 55.8% faster (95% confidence interval: 21-89%).”
- Also stated in: § Results · plain
“This represents a 55.8% reduction in completion time.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Introduction · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The p-value for the t-test is 0.0017, and a 95% confidence interval for the improvement is between [21%, 89%].”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This result suggests that Copilot increases average productivity significantly in our experiment population.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Results
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In this way, we accurately measured the timing and completion for each participant.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Study Design
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This result provides indirect evidence that treated group benefited from Copilot during their task as their willingness to pay is significantly higher than the control group.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Results
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Observed heterogenous effects show promise for AI pair programmers to help people transition into software development careers.”
- Also stated in: § Introduction · hedged
“These heterogeneous effects point towards promise for AI-pair programmers in support of expanding access to careers in software development.”
- Also stated in: § Discussion · hedged
“the productivity benefits for novice programmers and programmers of older age point to important possibilities for skill initiatives that support job transitions into software development.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Abstract
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The treatment effect is a linear function of the covariates of interest.
- Also stated in: § Introduction · hedged
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We verify from telemetry after the experiment that all participants from the treated group have configured GitHub Copilot and accepted recommendations other than five who did not finish the sign up and thus started the experiment without the GitHub Copilot.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Study Design
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 6
- The quote only partly states this claim. 1
- Probably not a substantial claim. 7
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On average, participants in both treated and control groups estimated a 35% increase in productivity, which is an underestimation compared with the 55.8% increase in their revealed productivity.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Conditioning on completing the task, the average completion time from the treated group is 71.17 minutes and 160.89 minutes for the control group.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results show that less experienced developers (years of professional coding), developers with heavy coding load (hours of coding per day), and older developers (developers aged between 25 and 44) benefit more from Copilot.”
- Also stated in: § Introduction · plain
“Developers with less programming experience, older programmers, and those who program more hours per day benefited the most.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The treatment effect is a linear function of the covariates of interest.
- Also stated in: § Introduction · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also find that the treated group's success rate is 7 percentage points higher than the control group, but the estimate is not statistically significant, with a 95% confidence interval of [-0.11, 0.25].”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If this result persists in further studies, the productivity benefits for novice programmers and programmers of older age point to important possibilities for skill initiatives that support job transitions into software development.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
- The authors state this as an assumption: The treatment effect is a linear function of the covariates of interest.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Once participants signed the contract, they were randomly split into control and treatment groups.”
- Also stated in: § Results · plain
“The 95 developers were randomly assigned into control and treated groups, with 45 in the treated group and 50 in control.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Study Design
plainn = 50- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Also stated in: § Results · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If the results of this study were to be extrapolated to the population level, a 55.8% increase in productivity would imply a significant amount of cost savings in the economy and have a notable impact on GDP growth.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants could see the tests but were unable to alter them.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Study Design
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“The potential productivity impacts of AI have major implications for the labor market and firms, including changes in employment, skills, and firm organization [Raj andSeamans, 2018, Agrawal et al., 2019].”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Introduction
What the sources say
- Not matchedcites Raj andSeamans, 2018
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites Agrawal et al., 2019 source
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Thirty-five developers from both the treated and control groups completed the task and survey.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Productivity benefits may vary across specific tasks and programming languages, so more research is needed to understand how our results generalizes to other tasks.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Using a standardized task provides us with precise measures of performance as it is difficult to measure productivity of software developers.”
§ The Impact of AI on Developer Productivity: Evidence from GitHub Copilot > Study Design
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The authors state this as an assumption.
The treatment effect is a linear function of the covariates of interest.
“We assume the treatment effect is a linear function of the covariates of interest.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“While developers consistently report significant productivity gains from these tools, empirical measurements often show more modest improvements.”
From Towards Decoding Developer Cognition in the Age of AI Assistants (2025) · cites this paper as [9]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“human-AI collaboration consistently improves measurable output quality in professional writing contexts”
From How Creative Is AI Writing? Generative and Collaborative AI in Japanese Fiction (2026) · cites this paper as Peng et al. (2023)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GitHub performed such an experiment where 95 participants were split into two groups and asked to write a web server.The study concluded by fnding that task completion was reduced by 55.8% in the Copilot condition [28].”
From Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming (2024) · cites this paper as [28]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Evidence from existing (laboratory) experiments generally suggests significant productivity effects of generative AI tools for software developers.”
From The Productivity Effects of Generative AI: Evidence from a Field Experiment with GitHub Copilot (2024) · cites this paper as Peng et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Peng et al. (2023) hired 95 professional programmers to develop an HTTP server in JavaScript.The group that used GitHub Copilot completed the task 55.8% faster, yet there was no effect on whether the task was completed.”
From The Productivity Effects of Generative AI: Evidence from a Field Experiment with GitHub Copilot (2024) · cites this paper as Peng et al. (2023)
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While our participants reported, and prior work suggests [14,33,38,40], AI tools demonstrably increase short-term productivity through faster task completion and reduced blocked time”
From The Evolution of Information Seeking in Software Development: Understanding the Role and Impact of AI Assistants (2025) · cites this paper as [33]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, both Dakhel et al. (2023) and Peng et al. (2023) caution that novice programmers may place undue trust in AI-generated code without fully understanding its broader design or integration implications, increasing the likelihood of downstream quality and integration challenges.”
From AI-Assisted Programming Decreases the Productivity of Experienced Developers by Increasing the Technical Debt and Maintenance Burden (2025) · cites this paper as Peng et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“First, the introduction of GitHub Copilot leads to higher development activity at both the repository and individual levels, measured by commits and PR”
From AI-Assisted Programming Decreases the Productivity of Experienced Developers by Increasing the Technical Debt and Maintenance Burden (2025) · cites this paper as Peng et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prior works have presented mixed results on whether AI helps or hinders coding productivity [Peng et al., 2023, Becker et al., 2025]; our study differs from prior results in that it is designed to study how AI affects skill formation while performing a task requiring new knowledge.”
From How AI Impacts Skill Formation (2026) · cites this paper as Peng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Software engineering, in particular, has been identified as a profession in which AI tools can be readily applied and AI assistance significantly improves productivity in daily tasks [Peng et al., 2023, Cui et al., 2024].”
From How AI Impacts Skill Formation (2026) · cites this paper as Peng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI assistance produces significant productivity gains across professional domains, particularly for novice workers.”
From How AI Impacts Skill Formation (2026) · cites this paper as Peng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Cui et al. (2024) estimate a 26% productivity gain from GitHub Copilot among software developers, while Peng et al. (2023) report a 58% increase.”
From Still Waters, Rapid Currents: Early Labor Market Transformation under Generative AI (2025) · cites this paper as Peng et al. (2023)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing empirical work on the micro-level effects of generative AI tools tends to find that access to these tools benefits less experienced workers more, compressing performance distributions [22; 23; 10; 24].”
From Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prior work suggests that less experienced developers may rely more heavily on AI assistance [9]”
From Towards Decoding Developer Cognition in the Age of AI Assistants (2025) · cites this paper as [9]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLM-based code generation shows significant productivity improvements (e.g., GitHub Copilot improves productivity 55.8% (Peng et al., 2023))”
From CodeCRDT: Observation-Driven Coordination for Multi-Agent LLM Code Generation (2025) · cites this paper as Peng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Our findings offer only partial support for the 'productivity gain' hypothesis previously reported in technical writing contexts (Noy and Zhang, 2023; Peng et al. 2023): while human-AI collaboration produces consistently acceptable prose with low variance, it does not demonstrably outperform original human writing in perceived quality”
From How Creative Is AI Writing? Generative and Collaborative AI in Japanese Fiction (2026) · cites this paper as Peng et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Peng et al. (2023) found comparable effects among software developers using GitHub Copilot, who completed entry-level tasks 55% faster than the control group.”
From How Creative Is AI Writing? Generative and Collaborative AI in Japanese Fiction (2026) · cites this paper as Peng et al. (2023)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Dell’Acqua et al. [1] and Peng et al. [2] demonstrated that these tools can streamline research processes, enhance productivity, and allow experts to focus on higher-level analytical tasks.”
From Integrating large language models in biostatistical workflows for clinical and translational research (2025) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prior studies show that generative AI can enhance individual developer productivity, measured as the amount of code generated per developer (i.e., intensive margin)”
From The Impact of Generative AI on Collaborative Open-Source Software Development: Evidence from GitHub Copilot (2024) · cites this paper as Peng et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“while reducing expected costs by lowering the time and effort required to complete development tasks.”
From The Impact of Generative AI on Collaborative Open-Source Software Development: Evidence from GitHub Copilot (2024) · cites this paper as Peng et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prior studies on generative AI generally show that lower skilled individuals benefit more from AI tools because AI reduces task completion costs (e.g., Peng et al. 2023, Cui et al. 2026).”
From The Impact of Generative AI on Collaborative Open-Source Software Development: Evidence from GitHub Copilot (2024) · cites this paper as Peng et al. 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In fact, a recent study from GitHub, found that Copilot could potentially reduce task completion time by a factor of two [28].”
From Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming (2024) · cites this paper as [28]
The cited source says what the claim attributes to it.
In the citing paper
“For Copilot users, our findings suggest a relatively small increase in average weekly activity; however, there is no statistically significant difference after adopting Copilot, which differs from prior studies (Al-Ahmad et al., 2024; Cui et al., 2024; Peng et al., 2023; Rajbhoj et al., 2024).”
From Developer Productivity With and Without GitHub Copilot: A Longitudinal Mixed-Methods Case Study (2026) · cites this paper as Peng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Instead, they focus on short-term gains in controlled or semi-controlled settings, either by measuring task completion time (Peng et al., 2023), or by simulating software projects to illustrate how GenAI tools can assist developers during different phases of the development process (Al-Ahmad et al., 2024; Rajbhoj et al., 2024).”
From Developer Productivity With and Without GitHub Copilot: A Longitudinal Mixed-Methods Case Study (2026) · cites this paper as Peng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“For example, Peng et al. (2023) found that participants with access to GitHub Copilot completed standardized programming tasks 55.8% faster.”
From Developer Productivity With and Without GitHub Copilot: A Longitudinal Mixed-Methods Case Study (2026) · cites this paper as Peng et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“GenAI solutions like GitHub Copilot have been shown to increase the productivity of software developers.”
From AI-Assisted Programming Decreases the Productivity of Experienced Developers by Increasing the Technical Debt and Maintenance Burden (2025) · cites this paper as Peng et al. 2023
The cited source says what the claim attributes to it.
In the citing paper
“Because AI tools can lower the skill barrier for writing code (Dakhel et al. 2023)”
From AI-Assisted Programming Decreases the Productivity of Experienced Developers by Increasing the Technical Debt and Maintenance Burden (2025) · cites this paper as Peng et al. 2023
The cited source says what the claim attributes to it.
In the citing paper
“These maintenance-related concerns are amplified by evidence that less experienced programmers appear to benefit more from the use of Copilot.”
From AI-Assisted Programming Decreases the Productivity of Experienced Developers by Increasing the Technical Debt and Maintenance Burden (2025) · cites this paper as Peng et al. 2023
The cited source says what the claim attributes to it.
In the citing paper
“Controlled experiments measure productivity gains of roughly 56% on isolated coding tasks (Peng et al., 2023)”
From Indie at Scale: How Large Language Models Make Community-Centered Game Development Economically Viable (2026) · cites this paper as Peng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“GenAI coding assistants have demonstrated considerable potential to enhance developer productivity.”
From The Effects of GitHub Copilot on Computing Students' Programming Effectiveness, Efficiency, and Processes in Brownfield Coding Tasks (2025) · cites this paper as [28]
The cited source says what the claim attributes to it.
In the citing paper
“Cui et al. (2025); Gambacorta, Qiu, Shan, and Rees (2024); Peng, Kalliamvakou, Cihon, and Demirer (2023) document strong productivity increases for software development tasks, again with larger gains among less experienced developers.”
From Labor Market Consequences of Generative AI: Early Evidence from Norway (2026) · cites this paper as Peng, Kalliamvakou, Cihon, and Demirer (2023)
The cited source says what the claim attributes to it.
In the citing paper
“Contrary to previous work finding significant uplift or speedup of AI assistance for coding [Peng et al., 2023, Cui et al., 2024], our results do not show a significant improvement in productivity if we only look at the total completion time across the treatment and control groups.”
From How AI Impacts Skill Formation (2026) · cites this paper as Peng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“significant productivity gains-often exceeding 15%-documented by RCTs in our study occupations”
From Still Waters, Rapid Currents: Early Labor Market Transformation under Generative AI (2025) · cites this paper as Peng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“First, our reported time savings are highest precisely in the occupations covered by RCTs, such as customer and IT support (Brynjolfsson, Li and Raymond, 2025), marketing and human resources (Noy and Zhang, 2023), and software development (Peng et al., 2023).”
From Still Waters, Rapid Currents: Early Labor Market Transformation under Generative AI (2025) · cites this paper as Peng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Peng et al. [10] and Paradis et al. [11] find 56% and 21% speedups on coding tasks when using AI assistance”
From Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025) · cites this paper as [10]
The cited source says what the claim attributes to it.
In the citing paper
“Frontier AI systems demonstrate impressive capabilities on a wide range of software benchmarks [3; 4; 5; 6; 7; 8; 9] and in experiments measuring AI's impact on developer productivity when completing synthetic tasks [10; 11].”
From Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025) · cites this paper as [10]
The cited source says what the claim attributes to it.
In the citing paper
“For example, Peng et al. [10] asks developers to implement a very basic HTTP server in JavaScript to satisfy several automatic test cases that are shown to the developers-this task is a) unrepresentative of most software development work, and b) likely to be similar to a large amount of LLM training data, which may unfairly advantage AI systems relative to humans.”
From Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025) · cites this paper as [10]
The cited source says what the claim attributes to it.
In the citing paper
“Peng et al. [10] | ↑ 56% faster”
From Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025) · cites this paper as Peng et al. [10]
The cited source says what the claim attributes to it.
In the citing paper
“We hypothesize that analogously, AI helps our developers less compared to existing literature [10; 11] because our developers have substantially more experience on their respective repositories (5 years and 1,500 commits on average).”
From Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025) · cites this paper as [10]
The cited source says what the claim attributes to it.
In the citing paper
“We note that limited token spend would not explain the difference between our results and other work that find positive speedup for programming tasks [10; 17].”
From Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025) · cites this paper as [10]
The cited source says what the claim attributes to it.
In the citing paper
“We hypothesize that analogously to these size and complexity effects within our study, AI broadly helps our developers less compared to existing randomized controlled trials (RCTs) measuring speedup from AI tools because of the overall size and complexity of the repositories included”
From Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025) · cites this paper as [10]
The cited source says what the claim attributes to it.
In the citing paper
“In a controlled experiment with 95 professional developers, Peng et.al. [9] found that access to GitHub Copilot led to a 55.8% reduction in task completion time for implementing an HTTP server, with less experienced and older developers benefiting the most.”
From Towards Decoding Developer Cognition in the Age of AI Assistants (2025) · cites this paper as [9]
The cited source says what the claim attributes to it.
In the citing paper
“The 34.9% reduction in task completion time and 26.2% improvement in test pass rates align with previous studies of the impact of GenAI programming assistants on both professional programmers [26, 28] and undergraduate students [13, 32] in greenfield programming scenarios.”
From The Effects of GitHub Copilot on Computing Students' Programming Effectiveness, Efficiency, and Processes in Brownfield Coding Tasks (2025) · cites this paper as [28]
The cited source says what the claim attributes to it.
In the citing paper
“The authors report that the treatment group completed the task 55.8% faster on average.”
From Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability (2026) · cites this paper as Peng et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“GitHub's research (Peng et al. 2023), which reported a 55% improvement.”
From Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability (2026) · cites this paper as Peng et al. 2023
The cited source says what the claim attributes to it.
In the citing paper
“For example, Peng et al. (2023) reported a 55.8% speedup when implementing an HTTP server in JavaScript;”
From Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability (2026) · cites this paper as Peng et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“The early work by Peng et al. (2023) reported that junior developers benefited the most from GitHub Copilot,”
From Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability (2026) · cites this paper as Peng et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“Developers who used Copilot completed a JavaScript-based task 55.8% faster than those who did not have access to the tool”
From How do Copilot Suggestions Impact Developers' Frustration and Productivity? (2025) · cites this paper as Peng et al. [12]
The cited source says what the claim attributes to it.
In the citing paper
“For example, a controlled experiment by Peng et al. (2023) found that GitHub Copilot, a generative AI programming aid, significantly boosted productivity: the group using Copilot completed programming tasks approximately 56% faster than the control group without access to the tool.”
From Large language models at work in China’s labor market (2025) · cites this paper as Peng et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“Other shortcomings include missing nuance or understanding of the impact of AI tools by developer- or task-level contexts (e.g., [13]), limited tool and task complexity in experimental settings (e.g., using GitHub Classroom rather than a more naturalistic developer environment [12]), and small sample sizes (e.g., only 32 participants in [15], 24 participants in [16], or 21 participants in [17]).”
From How Much Does AI Impact Development Speed? an Enterprise-Based Randomized Controlled Trial (2025) · cites this paper as [16]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“their operational and environmental costs pose significant challenges [6, 30].”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as [30]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Our results are consistent with those of Peng et al. (2023)”
From Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability (2026) · cites this paper as Peng et al. 2023
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Peng et al. [12] showed that Copilot's suggestions are perceived as invasive by experienced developer.”
From How do Copilot Suggestions Impact Developers' Frustration and Productivity? (2025) · cites this paper as [12]
Experimental — accuracy not yet measured
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 2
- [4]4. Athey, S. and Imbens, G. W. (2015). Machine learning methods for estimating heterogeneous causal effects. stat, 1050(5):1-26.
- [24]24. Raj, M. and Seamans, R. (2018). Artificial intelligence, labor, pro- ductivity, and the need for firm-level data. In The economics of artificial intelligence: An agenda, pages 553-565. University of Chicago Press.
Cited by
Sorted by citations
Experimental evidence on the productivity effects of generative artificial intelligence
Noy, S., & Zhang, W. (2023) · Science · doi:10.1126/science.adh2586 · Cited by 2,150
Brynjolfsson, E., Li, D., & Raymond, L. (2025) · The Quarterly Journal of Economics · doi:10.1093/qje/qjae044 · Cited by 1,083
Brynjolfsson, E., Li, D., & Raymond, L. (2023) · National Bureau of Economic Research · doi:10.3386/w31161 · Cited by 995
Dell'Acqua, F., McFowland III, E., Mollick, E. R., Lifshitz-Assaf, H., Kellogg, K., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. R. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4573321 · Cited by 794
GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models
Eloundou, T., Manning, S., Mishkin, P., & Rock, D. (2023) · arXiv · doi:10.48550/arxiv.2303.10130 · Cited by 2,330
“This could lower the cost and engineering expertise required to create such tools, potentially accelerating LLM adoption and integration even further (Chen et al., 2021; Peng et al., 2023).”
BackgroundLarge language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
“Similarly, Peng et al. [39] reported that programmers could complete a non-trivial task (implementing an HTTP server in JavaScript) 56% faster when paired with GitHub Copilot, compared to the control group that did not receive any such support.”
MethodGenerative artificial intelligence, human creativity, and art
Zhou, E., & Lee, D. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae052 · Cited by 459
The impact of generative artificial intelligence on socioeconomic inequalities and policy making
Capraro, V., Lentsch, A., Acemoglu, D., Akgun, S., Akhmedova, A., Bilancini, E., Bonnefon, J.-F., Brañas-Garza, P., Butera, L., Douglas, K. M., et al. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae191 · Cited by 371
Computing Education in the Era of Generative AI
Denny, P., Prather, J., Becker, B. A., Finnie-Ansley, J., Hellas, A., Leinonen, J., Luxton-Reilly, A., Reeves, B. N., Santos, E. A., & Sarsa, S. (2024) · Communications of the ACM · doi:10.1145/3624720 · Cited by 316
“…source code automatically from natural language descriptions of problems is likely to greatly improve the productivity of professional developers [28], and is being actively explored by well-funded entities such as OpenAI (ChatGPT, Codex, GPT-x ), Amazon (CodeWhisperer), and Google (AlphaCode,…”
BackgroundGenerative AI for economic research: use cases and implications for economists
Korinek, A. (2023) · Journal of Economic Literature · doi:10.1257/jel.20231736 · Cited by 293
AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103599 · Cited by 671
“Moreover, applications are no longer confined to narrow domains like simulations or logistics, but now extend to broad practical settings demanding real-time reasoning and adaptive control.”
Artificial Intelligence & Creativity: A Manifesto for Collaboration
Vinchon, F., Lubart, T., Bartolotta, S., Gironnay, V., Botella, M., Bourgeois-Bougrine, S., Burkhardt, J., Bonnardel, N., Corazza, G. E., Glăveanu, V., et al. (2023) · The Journal of Creative Behavior · doi:10.1002/jocb.597 · Cited by 273
Liang, W., Zhang, Y., Cao, H., Wang, B., Ding, D., Yang, X., Vodrahalli, K., He, S., Smith, D., Yin, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.01783 · Cited by 243
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256
Lau, S., & Guo, P. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600138 · Cited by 241
Navigating the Complexity of Generative AI Adoption in Software Engineering
Russo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3652154 · Cited by 217
“The integration of AI not only reduces the time commitment for these activities but also enhances the overall efficiency and effectiveness of the software development process [62].”
BackgroundMethodThe Rapid Adoption of Generative AI
Bick, A., Blandin, A., & Deming, D. (2024) · NBER Working Paper · doi:10.3386/w32966 · Cited by 200
Knowledge Management Perspective of Generative Artificial Intelligence
Alavi, M., Leidner, D. E., & Mousavi, R. (2024) · Journal of the Association for Information Systems · doi:10.17705/1jais.00859 · Cited by 272
New Frontiers: The Origins and Content of New Work, 1940–2018
Autor, D., Chin, C., Salomons, A., & Seegmiller, B. (2022) · The Quarterly Journal of Economics · doi:10.1093/qje/qjae008 · Cited by 300
The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B. (2024) · Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3632620.3671116 · Cited by 196
AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i3.15161 · Cited by 186
Dell’Acqua, F., McFowland, E., Mollick, E., Lifshitz, H., Kellogg, K. C., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. R. (2026) · Organization Science · doi:10.1287/orsc.2025.21838 · Cited by 1,189
“ChatGPT seems too good to be true”: College students’ use and perceptions of generative AI
Baek, C., Tate, T., & Warschauer, M. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100294 · Cited by 306
The Uneven Impact of Generative AI on Entrepreneurial PerformancePreprint
Otis, N. G., Clarke, R. P., Delecourt, S., Holtz, D., & Koning, R. M. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/hdjpk · Cited by 168
Chuang, Y., Chiang, H.-L., & Lin, A.-P. (2025) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2025.102887 · Cited by 209
Measuring GitHub Copilot's Impact on Productivity
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2024) · Communications of the ACM · doi:10.1145/3633453 · Cited by 124
Self-citationGenerative artificial intelligence enhances creativity but reduces the diversity of novel content
Doshi, A., & Hauser, O. (2023) · doi:10.2139/ssrn.4535536 · Cited by 1,350
Addressing 6 challenges in generative AI for digital health: A scoping review
Templin, T., Perez, M. W., Sylvia, S., Leek, J. T., & Sinnott-Armstrong, N. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000503 · Cited by 129
Fan, G., Liu, D., Zhang, R., & Pan, L. (2025) · International Journal of STEM Education · doi:10.1186/s40594-025-00537-3 · Cited by 109
“This finding is consistent with recent research on the benefits of AI-assisted coding (Barke et al., 2023; Peng et al., 2023).”
A Survey of Generative AI Applications
Gozalo-Brizuela, R., & Merchan, E. E. G. (2024) · Journal of Computer Science · doi:10.3844/jcssp.2024.801.818 · Cited by 101
Managing a ChatGPT-empowered workforce: Understanding its affordances and side effects
Retkowsky, J., Hafermalz, E., & Huysman, M. (2024) · Business Horizons · doi:10.1016/j.bushor.2024.04.009 · Cited by 90
Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641936 · Cited by 114
An Autoethnographic Case Study of Generative Artificial Intelligence's Utility for Accessibility
Glazko, K. S., Yamagami, M., Desai, A., Mack, K. A., Potluri, V., Xu, X., & Mankoff, J. (2023) · The 25th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3597638.3614548 · Cited by 80
“As GAI is being taken up by everyday users in a wide variety of domains from communication [1, 19, 34] to computer programming [35], it is imperative that we understand how GAI can be helpful and useful specifically to disabled people.”
BackgroundMethodPinski, M., Hofmann, T. F., & Benlian, A. (2024) · Electronic Markets · doi:10.1007/s12525-024-00707-1 · Cited by 100
“…30% of firms, according to the global executive study, have already found value-generating use cases of AI, and more use cases are continuously being developed, such as recent applications of generative AI in customer service or software development (Brynjolfsson et al., 2023; Peng et al., 2023).”
Cui, Z., Demirer, M., Jaffe, S., Musolff, L., Peng, S., & Salz, T. (2024) · doi:10.2139/ssrn.4945566 · Cited by 73
Self-citationThe Notorious GPT: science communication in the age of artificial intelligence
Schäfer, M. S. (2023) · Journal of Science Communication · doi:10.22323/2.22020402 · Cited by 67
“…fundamentally impact, even “disrupt” many realms of life [e.g. Johnson, 2022]: the economy [Eloundou, Manning, Mishkin & Rock, 2023], programming [Peng, Kalliamvakou, Cihon & Demirer, 2023], journalism [Pavlik, 2023], art [Eisikovits & Stubbs, 2023], music [Nicolaou, 2023], sports [Goldberg,…”
ChatGPT in the Classroom. Exploring Its Potential and Limitations in a Functional Programming Course
Popovici, M.-D. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2023.2269006 · Cited by 70
“A more recent study conducted by Peng et al. (2023) involving 95 participants has yielded a very different outcome.”
Initial policy considerations for generative artificial intelligence
Lorenz, P., Perset, K., & Berryhill, J. (2023) · OECD artificial intelligence papers · doi:10.1787/fae2d1e6-en · Cited by 123
Kuhail, M. A., Mathew, S. S., Khalil, A., Berengueres, J., & Shah, S. J. H. (2024) · Science of Computer Programming · doi:10.1016/j.scico.2024.103111 · Cited by 68
Boguslawski, S., Deer, R., & Dawson, M. G. (2025) · Information and Learning Sciences · doi:10.1108/ils-10-2023-0163 · Cited by 66
“Peng et al. (2023) found that less experienced programmers bene fi ted more, but there was nonetheless a productivity gain for developers across all experience levels, highlighting a need for further research in this area.”
BackgroundSignificant Productivity Gains through Programming with Large Language Models
Weber, T., Brandmaier, M., Schmidt, A., & Mayer, S. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3661145 · Cited by 64
“In the domain of auto-complete systems, Peng et al. [40] conducted a between-groups experiment where participants had to create an HTTP server using JavaScript.”
MethodResultEconomic impacts of AI-augmented R&D
Besiroglu, T., Emery-Xu, N., & Thompson, N. (2024) · Research Policy · doi:10.1016/j.respol.2024.105037 · Cited by 82
Large language models reduce public knowledge sharing on online Q&A platforms
del Rio-Chanona, R. M., Laurentsyeva, N., & Wachs, J. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae400 · Cited by 87
“Because ChatGPT performs relatively well on software programming tasks (15), we study Stack Overflow, the largest online Q&A platform for software development and programming.”
Are Large Language Models a Threat to Digital Public Goods? Evidence from Activity on Stack Overflow
del Rio-Chanona, M., Laurentsyeva, N., & Wachs, J. (2023) · arXiv · doi:10.48550/arxiv.2307.07367 · Cited by 62
McNeese, N. J., Flathmann, C., O’Neill, T., & Salas, E. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2023.107874 · Cited by 68
Obschonka, M., Grégoire, D. A., Nikolaev, B., Ooms, F., Lévesque, M., Pollack, J. M., & Behrend, T. S. (2024) · Entrepreneurship Theory and Practice · doi:10.1177/10422587241304676 · Cited by 76
“Academic work has started documenting substantial productivity gains in a wide range of creative and knowledge intensive tasks such as coding (Peng et al., 2023), creative writing (Doshi & Hauser, 2024), professional business tasks (Noy & Zhang, 2023), ideation (Girotra et al., 2023), strategic…”
Kobiella, C., Flores López, Y. S., Waltenberger, F., Draxler, F., & Schmidt, A. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641964 · Cited by 57
“Large language models (LLMs) and specifically ChatGPT are incorporated into knowledge work because they promise increased productivity [28, 34, 70, 74].”
BackgroundCopiloting the future: How generative AI transforms Software Engineering
Banh, L., Holldack, F., & Strobel, G. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107751 · Cited by 78
Chen, J., Lu, X., Du, Y., Rejtig, M., Bagley, R., Horn, M., & Wilensky, U. J. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642377 · Cited by 66
“As the field just started exploring this direction, previous studies mostly focused on general user impressions[102], orconductedbehavioraltasksonpre-scripted,close-endedtasks[62].”
BackgroundChatGPT for PLC/DCS Control Logic Generation
Koziolek, H., Grüner, S., & Ashiwal, V. (2023) · IEEE International Conference on Emerging Technologies and Factory Automation (ETFA) · doi:10.1109/etfa54631.2023.10275411 · Cited by 50
“GitHub Copilot uses GPT-3 and has provided a 55 percent developer productivity boost in experiments [2].”
MethodLarge language models can help boost food production, but be mindful of their risks
Clercq, D. D., Nehring, E., Mayne, H., & Mahdi, A. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1326153 · Cited by 65
“One study revealed that developers using Copilot completed their coding tasks over 50% faster than their counterparts who did not have access to the tool [37].”
BackgroundHarnessing generative artificial intelligence to support nature‐based solutions
Richards, D. R., Worden, D., Song, X. P., & Lavorel, S. (2024) · People and Nature · doi:10.1002/pan3.10622 · Cited by 59
New Paradigm for Economic and Financial Research With Generative AI: Impact and Perspective
Zheng, X., Li, J., Lu, M., & Wang, F. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2023.3334306 · Cited by 51
“Nevertheless, in terms of tasks that require repetitive work, GAI can also complement human labor by enhancing productivity and ef fi ciency [10].”
BackgroundCan LLMs Replace Manual Annotation of Software Engineering Artifacts?
Ahmed, T., Dévanbu, P., Treude, C., & Pradel, M. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00086 · Cited by 105
Generative Artificial Intelligence and Large Language Models in Primary Care Medical Education
Parente, D. J. (2024) · Family Medicine · doi:10.22454/fammed.2024.775525 · Cited by 40
“57 I call on the primary care research community to develop pilot LLM-based simulations, validate themwithrespecttohallucinationandbias,andsubjectthemto economiccost-benefitanalysistodeterminetheirperformance compared to (a) traditional simulation, and (b) other teaching methods (eg, didactic…”
Liu, J., & Li, S. (2024) · Journal of Educational Computing Research · doi:10.1177/07356331241240460 · Cited by 53
Winners and losers of generative AI: Early Evidence of Shifts in Freelancer Demand
Teutloff, O., Einsiedler, J., Kässi, O., Braesemann, F., Mishkin, P., & del Rio-Chanona, R. M. (2025) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2024.106845 · Cited by 39
Navigating software development in the ChatGPT and GitHub copilot era
France, S. L. (2024) · Business Horizons · doi:10.1016/j.bushor.2024.05.009 · Cited by 38
ChatGPT and Beyond: Exploring the Responsible Use of Generative AI in the Workplace
Söllner, M., Arnold, T., Benlian, A., Bretschneider, U., Knight, C., Ohly, S., Rudkowski, L., Schreiber, G., & Wendt, D. H. (2025) · Business & Information Systems Engineering · doi:10.1007/s12599-025-00932-8 · Cited by 61
“With the use of GitHub’s Copilot, the average completion time of a standard programming task decreased by more than 50%, with no significant difference in success rate.”
Using ChatGPT and Other Large Language Model (LLM) Applications for Academic Paper Assignments
Jungherr, A. (2023) · doi:10.31235/osf.io/d84q6 · Cited by 37
The impact of Artificial Intelligence on productivity, distribution and growth
Filippucci, F., Gál, P., Jona‐Lasinio, C., Leandro, Á., & Nicoletti, G. (2024) · OECD artificial intelligence papers · doi:10.1787/8d900037-en · Cited by 123
Generative AI: driving productivity and scientific breakthroughs in pharmaceutical R&D
Doron, G., Genway, S., Roberts, M., & Jasti, S. (2024) · Drug Discovery Today · doi:10.1016/j.drudis.2024.104272 · Cited by 39
Generative AI at WorkPreprint
Brynjolfsson, E., Li, D., & Raymond, L. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4426942 · Cited by 45
Still Waters, Rapid Currents: Early Labor Market Transformation under Generative AI
Humlum, A., & Vestergaard, E. (2025) · doi:10.3386/w33777 · Cited by 68
ChatGPT Hallucinates Non-existent Citations: Evidence from Economics
Buchanan, J., Hill, S., & Shapoval, O. (2024) · The American Economist · doi:10.1177/05694345231218454 · Cited by 40
“Economists predict that Arti fi cial Intelligence (AI) tools like OpenAI ’ s ChatGPTwill make human workers more productive (Eloundou et al., 2023; Goldfarb et al., 2023; Peng et al., 2023).”
BackgroundArtificial intelligence, tasks, skills, and wages: Worker-level evidence from Germany
Engberg, E., Koch, M. R., Lodefalk, M., & Schroeder, S. (2025) · Research Policy · doi:10.1016/j.respol.2025.105285 · Cited by 46
The Uneven Impact of Generative AI on Entrepreneurial PerformancePreprint
Otis, N. G., Clarke, R. P., Delecourt, S., Holtz, D., & Koning, R. M. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4671369 · Cited by 38
Zhou, X., Liang, P., Zhang, B., Li, Z., Ahmad, A., Shahin, M., & Waseem, M. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112204 · Cited by 30
“Some prior research investigated the quality of the code generated by Copilot (Yetistiren et al., 2022) (Nguyen and Nadi, 2022), while others examined its performance in practical software development (Imai, 2022) (Barke et al., 2023) (Peng et al., 2023).”
Song, F., Agarwal, A., & Wen, W. (2024) · SSRN Electronic Journal (preprint) · doi:10.2139/ssrn.4856935 · Cited by 30
Shifting Work Patterns with Generative AI
Dillon, E. W., Jaffe, S., Immorlica, N., & Stanton, C. (2025) · National Bureau of Economic Research · doi:10.3386/w33795 · Cited by 47
The power of generative marketing: Can generative AI create superhuman visual marketing content?Preprint
Hartmann, J., Exner, Y., & Domdey, S. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4597899 · Cited by 34
The Consequences of Generative AI for UGC and Online Community EngagementPreprint
Burtch, G., Lee, D., & Chen, Z. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4521754 · Cited by 33
“Here, it is worth noting that both Noy and Zhang (2023) and Peng et al. (2023) observed heterogeneity in the treatment effects, such that systematically greater productivity benefits were observed among lesser-skilled or lower-quality workers.”
BackgroundShihab, M. I. H., Hundhausen, C., Tariq, A., Haque, S., Qiao, Y., & Mulanda, B. W. (2025) · Proceedings of the 2025 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3702652.3744219 · Cited by 28
Should Users Trust Advanced AI Assistants? Justified Trust As a Function of Competence and Alignment
Manzini, A., Keeling, G., Marchal, N., McKee, K. R., Rieser, V., & Gabriel, I. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658964 · Cited by 28
The Impact of Generative AI Images on Consumer Attitudes in Advertising
Zhang, L., & Hur, C. (2025) · Administrative Sciences · doi:10.3390/admsci15100395 · Cited by 28
Knowledge Management Perspective of Generative Artificial Intelligence (GenAI)Preprint
Alavi, M., Leidner, D. E., & Mousavi, R. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4782875 · Cited by 27
Large Language Models, Small Labor Market Effects
Humlum, A., & Vestergaard, E. (2025) · doi:10.2139/ssrn.5219933 · Cited by 27
Does generative AI facilitate investor Trading? Early evidence from ChatGPT outages
Cheng, Q., Lin, P., & Zhao, Y. (2025) · Journal of Accounting & Economics · doi:10.1016/j.jacceco.2025.101821 · Cited by 26
Will Artificial Intelligence Get in the Way of Achieving Gender Equality?
Carvajal, D., Franco, C., & Isaksson, S. (2024) · doi:10.2139/ssrn.4759218 · Cited by 32
Haslberger, M., Gingrich, J. R., & Bhatia, J. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4594466 · Cited by 30
Alam, M. F., Lentsch, A., Yu, N., Barmack, S., Kim, S., Acemoğlu, D., Hart, A. J., Johnson, S., & Ahmed, F. (2024) · doi:10.21428/e4baedd9.e39b392d · Cited by 31
"Generate" the Future of Work through AI: Empirical Evidence from Online Labor MarketsPreprint
Liu, J., Xu, X.-C., Li, Y.-J., & Tan, Y. (2023) · Social Science Research Network · doi:10.2139/ssrn.4529739 · Cited by 33
The Productivity Effects of Generative AI: Evidence from a Field Experiment with GitHub Copilot
Cui, K. Z., Demirer, M., Jaffe, S., Musolff, L., Peng, S., & Salz, T. (2024) · An MIT Exploration of Generative AI · doi:10.21428/e4baedd9.3ad85f1c · Cited by 22
Self-citationThe impact of generative artificial intelligence on socioeconomic inequalities and policy makingPreprint
Capraro, V., Lentsch, A., Acemoğlu, D., Akgün, S., Akhmedova, A., Bilancini, E., Bonnefon, J., Brañas‐Garza, P., Butera, L., Douglas, K. M., et al. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/6fd2y · Cited by 47
The Labor Market Effects of Generative Artificial Intelligence
Hartley, J., Jolevski, F., Melo, V., & Moore, B. (2025) · doi:10.2139/ssrn.5136877 · Cited by 22
Data Authenticity, Consent, and Provenance for AI Are All Broken: What Will It Take to Fix Them?
Longpre, S., Mahari, R., Obeng-Marnu, N., Brannon, W., South, T., Kabbara, J., & Pentland, A. (2024) · doi:10.21428/e4baedd9.a650f77d · Cited by 28
Rasheed, Z., Waseem, M., Ahmad, A., Kemell, K., Wang, X., NguyenDuc, A., & Abrahamsson, P. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5149683 · Cited by 23
The Impact of Large Language Models on Open-source Innovation: Evidence from GitHub Copilot
Yeverechyahu, D., Mayya, R., & Oestreicher-Singer, G. (2024) · doi:10.2139/ssrn.4684662 · Cited by 20
“Most research, including Peng et al. (2023), notes that LLMs enhance an individual’s ability to complete guided tasks.”
BackgroundInfluentialCui, K. Z., Demirer, M., Jaffe, S., Musolff, L., Peng, S., & Salz, T. (2026) · Management Science · doi:10.1287/mnsc.2025.00535 · Cited by 28
Self-citationGenerative AI, Human Creativity, and Art
Zhou, E., & Lee, D. (2023) · doi:10.2139/ssrn.4594824 · Cited by 19
Kemell, K., Saarikallio, M., Nguyen‐Duc, A., & Abrahamsson, P. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107805 · Cited by 27
Generative AI in Higher Education: Evidence from an Elite College
Contractor, Z., & Reyes, G. (2025) · doi:10.2139/ssrn.5392712 · Cited by 19
What an AI-Embracing Software Engineering Curriculum Should Look Like: An Empirical Study
Randall, N., Wäckerle, D., Stein, N., Goßler, D., & Bente, S. (2024) · IEEE Software · doi:10.1109/ms.2023.3344682 · Cited by 18
How effective is AI augmentation in human–AI collaboration? Evidence from a field experiment
Liao, C., Wen, X., Li, S., & Du, P. (2024) · Information Technology and People · doi:10.1108/itp-11-2022-0859 · Cited by 24
Developer Behaviors in Validating and Repairing LLM-Generated Code Using IDE and Eye Tracking
Tang, N., Chen, M., Ning, Z., Bansal, A., Huang, Y., McMillan, C., & Li, T. J.-J. (2024) · 2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc60511.2024.00015 · Cited by 18
Large language models at work in China’s labor market
Chen, Q., Ge, J., Xie, H., Xu, X., & Yang, Y. (2025) · China Economic Review · doi:10.1016/j.chieco.2025.102413 · Cited by 17
“…disruptive labor market impacts of LLMs is emerging quickly, however, it predominantly focuses attention on the labor market in developed economy, in particularly U.S. (Eloundou et al. 2023; Peng et al. 2023; Noy and Zhang 2023; Brynjolfsson, Li, and Raymond 2023; Felten, Raj, and Seamans 2023).”
Who Is AI Replacing? The Impact of Generative AI on Online Freelancing PlatformsPreprint
Demirci, O., Hannane, J., & Zhu, X. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4602944 · Cited by 24
Bringing Worker Voice into Generative AI
Kochan, T. A., Armstrong, B., Shah, J., Castilla, E. J., Likis, B., & Mangelsdorf, M. E. (2024) · doi:10.21428/e4baedd9.0d255ab6 · Cited by 17
Generative AI and Copyright: A Dynamic PerspectivePreprint
Yang, S. A., & Zhang, A. H. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4716233 · Cited by 50
“In the business and management literature, an emerging research stream specifically focuses on how the adoption of generative AI affects user productivity (e.g., Peng et al. 2023, Noy and Zhang 2023, Dell’Acqua et al. 2023) and creativity (e.g., Zhou and Lee 2023).”
What Guides Our Choices? Modeling Developers' Trust and Behavioral Intentions Towards Genai
Choudhuri, R., Trinkenreich, B., Pandita, R., Kalliamvakou, E., Steinmacher, I., Gerosa, M., Sanchez, C., & Sarma, A. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) · doi:10.1109/icse55347.2025.00087 · Cited by 17
Self-citation
Metadata
- Authors
- Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M.
- Affiliations
- Microsoft (United States) (US)
- Massachusetts Institute of Technology (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2302.06590
- OpenAlex
- W4320854981
- Semantic Scholar
- 15abedb29536d50afeeec739a25358255cbda3e8
- Cited by
- 287
- References
- 0
- Citations by year
- 2026: 52 · 2025: 88 · 2024: 96 · 2023: 50
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
- Keywords
- GitHub Copilot, generative AI, developer productivity, software development, JavaScript, treatment effect heterogeneity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
