Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index; 6 of the 16 sources its claims cite were checked for support.

Grounded Copilot: How Programmers Interact with Code-Generating Models

Open access
Barke, Shraddha, James, Michael B., Polikarpova, Nadia · 2023 · Proceedings of the ACM on Programming Languages

Abstract

Powered by recent advances in code-generating models, AI assistants like Github Copilot promise to change the face of programming forever. But what is this new face of programming? We present the first grounded theory analysis of how programmers interact with Copilot, based on observing 20 participants—with a range of prior experience using the assistant—as they solve diverse programming tasks across four languages. Our main finding is that interactions with programming assistants are bimodal : in acceleration mode , the programmer knows what to do next and uses Copilot to get there faster; in exploration mode , the programmer is unsure how to proceed and uses Copilot to explore their options. Based on our theory, we provide recommendations for improving the usability of future AI programming assistants.

Overview

CoChat Summary

TL;DR

This paper presents a grounded theory of how programmers interact with GitHub Copilot, based on observing 20 participants across four programming languages. Its central finding is that Copilot-assisted programming is bimodal: programmers use Copilot either in acceleration mode, to execute a known plan faster, or in exploration mode, to discover how to proceed.

Key findings

  • Interactions with Copilot are bimodal: acceleration mode occurs when the programmer already knows what to do next and uses Copilot to get there faster; exploration mode occurs when the programmer is unsure how to proceed and uses Copilot to explore options or get a starting point.
  • Across all participants, total study time in exploration mode was 248.6 minutes, more than twice the 104.7 minutes in acceleration mode.
  • Prompting strategies differed by mode: in acceleration mode, code prompts were most common (71.4%) and context prompts second (15.2%); in exploration mode, comment prompts were much more common than in acceleration mode (57.2% vs 13.1%).
  • Validation also differed: exploration involved more careful examination and documentation reading, while acceleration more often used execution for rapid feedback; participants also used five livestream videos to confirm behavior consistent with the theory.

Methods

  • The authors conducted a Straussian grounded theory study with 20 participants (15 from academia, 5 from industry; 9 had prior Copilot usage) who solved programming tasks with Copilot across Python, Rust, Haskell, and Java.
  • Participants completed a training task, a core task lasting about 20–40 minutes while talking through their interactions, and a semi-structured interview; sessions were recorded and transcribed.
  • Analysis interleaved data collection and qualitative coding, with open, axial, and selective coding, theoretical sampling, and theoretical saturation at the 20th participant; the authors then re-coded videos quantitatively and qualitatively analyzed five livestream videos from YouTube and Twitch.

Limitations

  • Stated in the paper: participants worked on tasks of the authors’ design rather than their own projects, and the tasks focused only on code authorship, not refactoring, testing, debugging, or other software engineering activities.
  • Stated in the paper: the 20-participant sample skews toward academia and is not representative of all programmers; 11 participants had not used Copilot before the study, and prior users may have formed usage patterns during the technical preview.
  • Stated in the paper: the authors could not observe programmers over a longer period to study changing usage patterns, and rapid progress in code-generating models could render the findings obsolete.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3586030

Cites

59 of 59 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Image 1
CC-BY · From the PDF
Fig. 3 Timeline of observed activities in each interaction mode for the 20 study participants. The qualitative codes include different prompting strategies, validation strategies and outcomes of Copilot's suggestions (accept, reject or repair)
Fig. 3 Timeline of observed activities in each interaction mode for the 20 study participants. The qualitative codes include different prompting strategies, validation strategies and outcomes of Copilot's suggestions (accept, reject or repair)CC-BY · From the PDF
Fig. 4 Median time spent in acceleration vs exploration mode for different participant groups.
Fig. 4 Median time spent in acceleration vs exploration mode for different participant groups.CC-BY · From the PDF
Fig. 5 Median time spent in acceleration vs exploration mode, grouped by task.
Fig. 5 Median time spent in acceleration vs exploration mode, grouped by task.CC-BY · From the PDF
Fig. 6 Prevalence of prompting strategy as percentage of total prompting time.
Fig. 6 Prevalence of prompting strategy as percentage of total prompting time.CC-BY · From the PDF
Image 6
CC-BY · From the PDF
(b) Median time, grouped by task.
(b) Median time, grouped by task.CC-BY · From the PDF

Verification

Claims the paper makes

4 Valid · 0 Not valid · 8 Not checked · 13 Own results and interpretations

20 more claims

Create a free account to see all 25 claims and every source verdict.

See all claims — free account

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

16 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

4 of 4

Citations

Cites

Show all 46 references

Research works printed but not linked: 8

  1. Mohammad Bavarian Heewoo Jun Nikolas Tezak John Schulman Christine McLeavey Jerry Tworek and Mark Chen. 2022. Efficient Training of Language Models to Fill in the Middle. Jul https://doi.org/10.48550/arXiv.2207.14255 arXiv:2207.14255 [cs]. · DOI · Paper page
  2. Daniel Fried Armen Aghajanyan Jessy Lin Sida Wang Eric Wallace Freda Shi Ruiqi Zhong Wen-tau Yih Luke Zettlemoyer and Mike Lewis. 2022. InCoder: A Generative Model for Code Infilling and Synthesis. Apr https://doi.org/10.48550/arXiv.2204.05999 arXiv:2204.05999 [cs]. · DOI · Paper page
  3. Daniel Kahneman . 2011. Thinking , fast and slow . Penguin Books . isbn:978-0-14-103357-0 Daniel Kahneman. 2011. Thinking, fast and slow. Penguin Books. isbn:978-0-14-103357-0
  4. Kite. 2020 . Kite: AI-Powered Completions for JupyterLab. https://www.kite.com/integrations/jupyter/ Kite. 2020. Kite: AI-Powered Completions for JupyterLab. https://www.kite.com/integrations/jupyter/
  5. Yujia Li David Choi Junyoung Chung Nate Kushman Julian Schrittwieser Rémi Leblond Tom Eccles James Keeling Felix Gimeno Agustin Dal Lago Thomas Hubert Peter Choy Cyprien de Masson d'Autume Igor Babuschkin Xinyun Chen Po-Sen Huang Johannes Welbl Sven Gowal Alexey Cherepanov James Molloy Daniel J. Mankowitz Esme Sutherland Robson Pushmeet Kohli Nando de Freitas Koray Kavukcuoglu and Oriol Vinyals. 2022. Competition-Level Code Generation with AlphaCode. https://doi.org/10.48550/ARXIV.2203.07814 · DOI · Paper page
  6. Anselm L. Strauss and Juliet Corbin . 1990 . Basics of Qualitative Reseach: Grounded Theory Procedures and Techniques. SAGE Publications, Inc .. isbn:0-8039-3250-2 Anselm L. Strauss and Juliet Corbin. 1990. Basics of Qualitative Reseach: Grounded Theory Procedures and Techniques. SAGE Publications, Inc.. isbn:0-8039-3250-2
  7. Steven J. Taylor and Robert Bogdan . 1998 . Introduction to qualitative research methods: A guidebook and resource, 3 rd ed. John Wiley and Sons Inc . isbn:978-0-471-16868-3 Steven J. Taylor and Robert Bogdan. 1998. Introduction to qualitative research methods: A guidebook and resource, 3rd ed. John Wiley and Sons Inc. isbn:978-0-471-16868-3
  8. Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser and Illia Polosukhin. 2017. Attention Is All You Need. Dec https://doi.org/10.48550/arXiv.1706.03762 arXiv:1706.03762 [cs]. · DOI · Paper page

Cited by

Metadata

Authors
Barke, S., James, M. B., & Polikarpova, N.
Affiliations
University of California San Diego (US)
Venue
Proceedings of the ACM on Programming Languages
Published
2023-04-06
Volume
7
Issue
OOPSLA1
Pages
85-111
ISSN
2475-1421 (ISSN-L)
OpenAlex
W4362659486
Cited by
436
References
46
Citations by year
2026: 147 · 2025: 131 · 2024: 119 · 2023: 39
FWCI
102.07
Type
journal article
Language
en
Open access
Locations
Proceedings of the ACM on Programming Languages (publishedVersion, cc-by)
Funders
National Science Foundation (2107397, 1955457)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
Keywords
GitHub Copilot, acceleration mode, exploration mode, usability, human-AI collaboration, programming tasks

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-27 (prompt extract-v2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.