Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 22Software Engineering ResearchSoftware Engineering Techniques and PracticesArtificial Intelligence Applications

The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index; 4 of the 8 sources its claims cite were checked for support.

The Productivity Effects of Generative AI: Evidence from a Field Experiment with GitHub Copilot

Open access
Cui, Kevin Zheyuan, Demirer, Mert, Jaffe, Sonia, Musolff, Leon, Peng, Sida, Salz, Tobias·2024·An MIT Exploration of Generative AI

Abstract

We are providing a preview of a project that analyzes two field experiments with 1,974 software developers at Microsoft and Accenture to evaluate the productivity impact of Generative AI. As part of our study, a random subset of developers was given access to GitHub Copilot, an AI-based coding assistant that intelligently suggests ‘completions’ for code. Our preliminary results provide suggestive evidence that these developers became more productive, completing 12.92% to 21.83% more pull requests per week at Microsoft and 7.51% to 8.69% at Accenture (depending on specification). Due to low compliance in the Microsoft experiment and internal organizational changes at Accenture, our estimates are not very precise and only reach statistical significance if we weight more heavily periods in which Copilot uptake differs across control and treatment.

Overview

CoChat Summary

## TL;DR

This paper reports preliminary field-experiment evidence that giving software developers access to GitHub Copilot increased productivity at Microsoft and Accenture. The estimated effects are positive but imprecise because compliance was low at Microsoft and the Accenture analysis was affected by organizational changes.

## Key findings

  • In the abstract, the authors report that treated developers completed 12.92% to 21.83% more pull requests per week at Microsoft and 7.51% to 8.69% more at Accenture, depending on specification.
  • In the Microsoft experiment, the paper says the primary effects of interest are on successful builds, with estimated increases ranging from 84% to 107% across specifications.
  • For Accenture, the paper reports that the SLATE specification gives an 8.69% increase in pull requests, which is statistically significant at the 5% level; the IV1 and IV2 estimates are 7.75% and 7.51% and are not statistically significant.
  • The paper also reports an 11% increase in lines of code changed under SLATE at Accenture, suggesting developers were not merely splitting the same work into more pull requests.

## Methods

  • The study analyzes two randomized field experiments with software developers at Microsoft and Accenture, where one group was randomly assigned access to GitHub Copilot and a control group was not.
  • Sample sizes stated in the text are 1,749 developers at Microsoft and 311 developers at Accenture after exclusions; the abstract describes 1,974 software developers across both experiments.
  • Main analysis uses instrumental variables / encouragement-design regressions rather than simple treatment-control comparisons, including specifications with time fixed effects, developer fixed effects, pre-experimental data, and a SLATE weighting approach that upweights periods with larger treatment-control differences in Copilot uptake.

## Limitations

  • The paper states that both experiments were poorly powered because of low initial Copilot uptake at Microsoft and internal organizational changes at Accenture.
  • The authors note that their productivity measures are not a silver bullet for true developer productivity and that they cannot measure productivity per unit of time because work hours may have changed after adopting Copilot.
  • Evident from the text: the Accenture data are described as still running / still being collected, so the reported results are explicitly preliminary.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.21428/e4baedd9.3ad85f1c

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Cumulative adoption of copilot for Microsoft experiment. This graph shows the cumulative rate of adoption across time for software developers at Microsoft in both the treatment and control group over the course of the experiment.
Figure 1 Cumulative adoption of copilot for Microsoft experiment. This graph shows the cumulative rate of adoption across time for software developers at Microsoft in both the treatment and control group over the course of the experiment.CC-BY-NC · From the PDF
Figure 2 Cumulative adoption of Copilot for Accenture experiment. This graph shows the cumulative adoption rate across time for software developers at Accenture in both the treatment and control over the course of the experiment.
Figure 2 Cumulative adoption of Copilot for Accenture experiment. This graph shows the cumulative adoption rate across time for software developers at Accenture in both the treatment and control over the course of the experiment.CC-BY-NC · From the PDF

Verification

Claims the paper makes

3 Valid · 0 Not valid · 3 Not checked · 24 Own results and interpretations

2 more claims

Create a free account to see all 30 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 12
  • It may state two separate claims as one. 17
  • The quote only partly states this claim. 3

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 11 references

Cited by

See all 22

Metadata

Authors
Cui, K. Z., Demirer, M., Jaffe, S., Musolff, L., Peng, S., & Salz, T.
Affiliations
  • Princeton University (US)
  • Massachusetts Institute of Technology (US)
  • Microsoft (United States) (US)
  • University of Pennsylvania (US)
Venue
An MIT Exploration of Generative AI
Publisher
PubPub
Published
2024-03-27
OpenAlex
W4393234096
Cited by
22
References
11
Citations by year
2026: 13 · 2025: 8 · 2024: 1
FWCI
8.24
Type
journal article
Language
en
Open access
Licence
cc-by-nc
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
generative AI, GitHub Copilot, AI-assisted programming, code completion, pull requests, field experiments, technology adoption, software development

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.