CoChat
Sign Up
PreprintCited by 201Software Engineering ResearchSoftware Engineering Techniques and PracticesOpen Source Software Innovations

Productivity Assessment of Neural Code Completion

Open access
Ziegler, Albert, Kalliamvakou, Eirini, Simister, Shawn, Sittampalam, Ganesh, Li, Alice, Rice, Andrew, Rifkin, Devon, Aftandilian, Edward·2022·arXiv
DOI 10.48550/arxiv.2205.06537arXiv 2205.06537
Open this paper in CoChat — free accountOpen PDF

Abstract

Neural code synthesis has reached a point where snippet generation is accurate enough to be considered for integration into human software development workflows. Commercial products aim to increase programmers' productivity, without being able to measure it directly. In this case study, we asked users of GitHub Copilot about its impact on their productivity, and sought to find a reflection of their perception in directly measurable user data. We find that the rate with which shown suggestions are accepted, rather than more specific metrics regarding the persistence of completions in the code over time, drives developers' perception of productivity.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2205.06537

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The proxy metrics (harmonic mean of acceptance rate and CodeBERTScore) might not fully capture the usability of the interaction without potentially compromising it in some other way.

    “Consistent with earlier research [32, 49], we acknowledge that these proxy metrics might not fully capture the usability of the interaction without potentially compromising it in some other way.”

    From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as [49]

    The Broader Impact section warns that optimizing acceptance rate as a proxy for usefulness can increase the metric while potentially decreasing user benefit. The paper does not assess CodeBERTScore or a harmonic mean of the two metrics, so it supports the caution about acceptance rate but not the specific combined proxy. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • About two-thirds of shown completions are ignored by their end-user.

    “Copilot's authors themselves reported that about two-thirds of shown completions are ignored by their end-user [49].”

    From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as [49]

    The Background section defines acceptance as the fraction of shown completions accepted and reports a 27% acceptance rate for GitHub Copilot. That implies 73% were not accepted—roughly, though not exactly, two-thirds. The paper does not establish that every unaccepted completion was ignored by its user. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Optimising only acceptance rate results in worse completions.

    “Specifically, optimising only acceptance rate results in worse completions [32, 49].”

    From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as [49]

    Section 8 (Broader Impact; page not provided) warns that optimizing acceptance rate alone can encourage changes that improve the metric without improving usefulness, and might even decrease user benefit. It does not establish that doing so necessarily results in worse completions; the citing sentence turns a possible outcome into a categorical result. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Ziegler et al. propose the acceptance rate of suggestions as a proxy for developers' perceived productivity.

    “Ziegler et al. [49] proposes the acceptance rate of suggestions as a proxy for developers' perceived productivity”

    From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as Ziegler et al. [49]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 7

  1. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. CoRR abs/2108.07732 (2021). arXiv:2108.07732 https://arxiv.org/abs/2108.07732 · arXiv
  2. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, An- drew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCan- dlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. CoRR abs/2107.03374 (2021). arXiv:2107.03374 https://arxiv.org/abs/2107.03374 · arXiv
  3. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring Coding Challenge Com- petence With APPS. CoRR abs/2105.09938 (2021). arXiv:2105.09938 https://arxiv.org/abs/2105.09938 · arXiv
  4. Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. 2019. SPoC: Search-based Pseudocode to Code. In Advances in Neural Information Process- ing Systems 32: Annual Conference on Neural Information Process- ing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.). 11883ś11894. https://proceedings.neurips.cc/paper/2019/hash/ 7298332f04ac004a0ca44cc69ecf6f6b-Abstract.html
  5. Priyan Vaithilingam, Tianyi Zhang, and Elena Glassman. 2022. Ex- pectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In CHI '22 Late-Breaking Work: Proceedings of the 2022 Conference on Human Factors in Comput- ing Systems.
  6. Frank F. Xu, Bogdan Vasilescu, and Graham Neubig. 2021. In-IDE Code Generation from Natural Language: Promise and Challenges. CoRR abs/2101.11149 (2021). arXiv:2101.11149 https://arxiv.org/abs/ 2101.11149 · arXiv
  7. Wen Zhou, Seohyun Kim, Vijayaraghavan Murali, and Gareth Ari Aye. 2021. Improving Code Autocompletion with Transfer Learning. CoRR abs/2105.05991 (2021). arXiv:2105.05991 https://arxiv.org/abs/2105. 05991 · arXiv

Cited by

Sorted by citations

Related

Authors
Ziegler, A., Kalliamvakou, E., Simister, S., Sittampalam, G., Li, A., Rice, A., Rifkin, D., & Aftandilian, E.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
Pages
21-29
DOI
10.48550/arxiv.2205.06537
arXiv
2205.06537
OpenAlex
W4281669078
Cited by
200
References
12
Citations by year
2026: 47 · 2025: 56 · 2024: 70 · 2023: 25 · 2022: 2
FWCI
34.20
Type
conference paper
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
Microsoft Research
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
Keywords
GitHub Copilot, developer productivity, code completion, developer perceptions, code generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review