The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index; 4 of the 8 sources its claims cite were checked for support.
The Productivity Effects of Generative AI: Evidence from a Field Experiment with GitHub Copilot
Open accessAbstract
We are providing a preview of a project that analyzes two field experiments with 1,974 software developers at Microsoft and Accenture to evaluate the productivity impact of Generative AI. As part of our study, a random subset of developers was given access to GitHub Copilot, an AI-based coding assistant that intelligently suggests âcompletionsâ for code. Our preliminary results provide suggestive evidence that these developers became more productive, completing 12.92% to 21.83% more pull requests per week at Microsoft and 7.51% to 8.69% at Accenture (depending on specification). Due to low compliance in the Microsoft experiment and internal organizational changes at Accenture, our estimates are not very precise and only reach statistical significance if we weight more heavily periods in which Copilot uptake differs across control and treatment.
Overview
CoChat Summary
## TL;DR
This paper reports preliminary field-experiment evidence that giving software developers access to GitHub Copilot increased productivity at Microsoft and Accenture. The estimated effects are positive but imprecise because compliance was low at Microsoft and the Accenture analysis was affected by organizational changes.
## Key findings
- In the abstract, the authors report that treated developers completed 12.92% to 21.83% more pull requests per week at Microsoft and 7.51% to 8.69% more at Accenture, depending on specification.
- In the Microsoft experiment, the paper says the primary effects of interest are on successful builds, with estimated increases ranging from 84% to 107% across specifications.
- For Accenture, the paper reports that the SLATE specification gives an 8.69% increase in pull requests, which is statistically significant at the 5% level; the IV1 and IV2 estimates are 7.75% and 7.51% and are not statistically significant.
- The paper also reports an 11% increase in lines of code changed under SLATE at Accenture, suggesting developers were not merely splitting the same work into more pull requests.
## Methods
- The study analyzes two randomized field experiments with software developers at Microsoft and Accenture, where one group was randomly assigned access to GitHub Copilot and a control group was not.
- Sample sizes stated in the text are 1,749 developers at Microsoft and 311 developers at Accenture after exclusions; the abstract describes 1,974 software developers across both experiments.
- Main analysis uses instrumental variables / encouragement-design regressions rather than simple treatment-control comparisons, including specifications with time fixed effects, developer fixed effects, pre-experimental data, and a SLATE weighting approach that upweights periods with larger treatment-control differences in Copilot uptake.
## Limitations
- The paper states that both experiments were poorly powered because of low initial Copilot uptake at Microsoft and internal organizational changes at Accenture.
- The authors note that their productivity measures are not a silver bullet for true developer productivity and that they cannot measure productivity per unit of time because work hours may have changed after adopting Copilot.
- Evident from the text: the Accenture data are described as still running / still being collected, so the reported results are explicitly preliminary.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
3 Valid · 0 Not valid · 3 Not checked · 24 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Evidence from existing (laboratory) experiments generally suggests significant productivity effects of generative AI tools for software developers.”
§ An MIT Exploration of Generative AI • From Novel Chemicals to Opera The Productivity E ects of Generative AI: Evidence from a Field Experiment with GitHub Copilot > 1.1. Related Literature
What the sources say
- Partly supportedcites Peng et al. 2023 sourceExperimental — accuracy not yet measured
The Introduction reports a statistically significant productivity effect in a controlled trial: developers using GitHub Copilot completed a JavaScript task 55.8% faster. The Discussion describes this as one experiment on a standardized task and cautions that benefits may vary across tasks and languages. It does not establish what laboratory experiments generally find across generative AI tools. (LLM, conf=0.83)
- Partly supportedcites Campero et al. 2022 sourceExperimental — accuracy not yet measured
Section 4.2 (p. 9) reports a statistically significant speed improvement of about 30% when programmers used GPT-3 for a software-generation task. That supports a productivity effect in one experiment, but not the broader claim that existing experiments generally find such effects for generative AI tools and software developers; the paper says the experiments reviewed in Study 1 did not use GPT-3. (LLM, conf=0.85)
- Not checked — no full text of the sourcecites Noy and Zhang 2023 source
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“However, both experiments were poorly powered.”
§ An MIT Exploration of Generative AI • From Novel Chemicals to Opera The Productivity E ects of Generative AI: Evidence from a Field Experiment with GitHub Copilot > Opera
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To summarize, we find suggestive evidence that GitHub Copilot raises the productivity of software engineers.”
§ An MIT Exploration of Generative AI • From Novel Chemicals to Opera The Productivity E ects of Generative AI: Evidence from a Field Experiment with GitHub Copilot > 5. Conclusion
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The first-stage regressions of these instrumental variable specifications are poorly powered because of the low initial take-up rate in the treatment group.”
§ An MIT Exploration of Generative AI • From Novel Chemicals to Opera The Productivity E ects of Generative AI: Evidence from a Field Experiment with GitHub Copilot > 3.1. Microsoft Experiment
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As this is a single rejection across six outcomes in the balance table, we cautiously interpret this failure of balance not as a failure to randomize (indeed, randomization occurred without access to these variables) but rather as an unlucky draw.”
§ An MIT Exploration of Generative AI • From Novel Chemicals to Opera The Productivity E ects of Generative AI: Evidence from a Field Experiment with GitHub Copilot > 2.2. Accenture Experiment
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
2 more claims
Create a free account to see all 30 claims and every source verdict.
- Probably not a substantial claim. 12
- It may state two separate claims as one. 17
- The quote only partly states this claim. 3
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ An MIT Exploration of Generative AI • From Novel Chemicals to Opera The Productivity E ects of Generative AI: Evidence from a Field Experiment with GitHub Copilot > 2.1. Microsoft Experiment
“The experiment at Microsoft started in the first week of September 2022, involving a sample size of 1,749 developers”
§ An MIT Exploration of Generative AI • From Novel Chemicals to Opera The Productivity E ects of Generative AI: Evidence from a Field Experiment with GitHub Copilot > 1. Introduction and Motivation
“The Microsoft experiment started in September 2022 and included 1,663 developers.”
Citations
Cites
Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence
Noy, S., & Zhang, W. (2023) · doi:10.2139/ssrn.4375283 · Cited by 2,157
Experimental evidence on the productivity effects of generative artificial intelligence
Noy, S., & Zhang, W. (2023) · Science · doi:10.1126/science.adh2586 · Cited by 2,150
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526
What can machine learning do? Workforce implications
Brynjolfsson, E., & Mitchell, T. (2017) · Science · doi:10.1126/science.aap8062 · Cited by 1,131
Brynjolfsson, E., Li, D., & Raymond, L. (2025) · The Quarterly Journal of Economics · doi:10.1093/qje/qjae044 · Cited by 1,083
Dell'Acqua, F., McFowland III, E., Mollick, E. R., Lifshitz-Assaf, H., Kellogg, K., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. R. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4573321 · Cited by 794
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
The Impact of Artificial Intelligence on the Labor Market
Webb, M. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3482150 · Cited by 531
The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023) · arXiv · doi:10.48550/arxiv.2302.06590 · Cited by 287
Self-citationReading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2022) · arXiv · doi:10.48550/arxiv.2210.14306 · Cited by 114
Cited by
Generative AI Vibe Coding for Prototyping Industrial Systems
Silva, D. D., Mills, N., Issadeen, Z., Moraliyage, H., Jennings, A., & Manic, M. (2025) · IEEE International Symposium on Industrial Electronics (ISIE) · doi:10.1109/isie62713.2025.11124737 · Cited by 13
ElSayary, A. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-7332-3.ch015 · Cited by 3
Challenges in Managing the Relationship Between Agentic AI Systems and Humans in Organizations
Wissuchek, C., & Zschech, P. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-031-94193-1_1 · Cited by 4
CodeWatcher: IDE Telemetry Data Extraction Tool for Understanding Coding Interactions with LLMs
Basha, M., Ribeiro, A. M., Javahar, J., de Souza, C. R. B., & Rodríguez-Pérez, G. (2025) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme64153.2025.00110 · Cited by 2
AI’s Hidden Price: AI Tools Reduce Donor Engagement Through Extrinsic Motivation Inferences
Zhao, Y., Zhou, P. P., & Chen, Z. (2026) · Journal of Business Ethics · doi:10.1007/s10551-026-06247-2 · Cited by 3
Metadata
- Authors
- Cui, K. Z., Demirer, M., Jaffe, S., Musolff, L., Peng, S., & Salz, T.
- Affiliations
- Princeton University (US)
- Massachusetts Institute of Technology (US)
- Microsoft (United States) (US)
- University of Pennsylvania (US)
- Venue
- An MIT Exploration of Generative AI
- Publisher
- PubPub
- Published
- 2024-03-27
- OpenAlex
- W4393234096
- Cited by
- 22
- References
- 11
- Citations by year
- 2026: 13 · 2025: 8 · 2024: 1
- FWCI
- 8.24
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-nc
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- generative AI, GitHub Copilot, AI-assisted programming, code completion, pull requests, field experiments, technology adoption, software development
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.