Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 105 of the 106 references it cites were found in a registry or an index; 15 of the 30 sources its claims cite were checked for support.

“It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers

Open access

Abstract

Recent developments in deep learning have resulted in code-generation models that produce source code from natural language and code-based prompts with high accuracy. This is likely to have profound effects in the classroom, where novices learning to code can now use free tools to automatically suggest solutions to programming exercises and assignments. However, little is currently known about how novices interact with these tools in practice. We present the first study that observes students at the introductory level using one such code auto-generating tool, Github Copilot, on a typical introductory programming (CS1) assignment. Through observations and interviews we explore student perceptions of the benefits and pitfalls of this technology for learning, present new observed interaction patterns, and discuss cognitive and metacognitive difficulties faced by students. We consider design implications of these findings, specifically in terms of how tools like Copilot can better support and scaffold the novice programming experience.

Overview

CoChat Summary

TL;DR

This paper reports the first observational and interview study of novice CS1 students using GitHub Copilot on a typical introductory programming assignment. It finds that novices often saw Copilot as fast and helpful, but also struggled to understand its suggestions, worried about over-reliance, and exhibited two new interaction patterns: shepherding and drifting.

Key findings

  • Most students perceived Copilot as helping them write code faster and avoid syntax errors, but many also worried about not understanding generated code and becoming dependent on it; “Outsourcing” concerns appeared in 46 interview excerpts from 17 unique participants, and “Speed” in 37 excerpts from 17 unique participants.
  • The study identifies two novel interaction patterns: *shepherding*, where students spent most of their time coercing Copilot toward a solution, and *drifting*, where students moved from one incorrect suggestion to the next, backtracking and deleting code without direction.
  • Students showed cognitive and metacognitive difficulties: “Confused” was observed 65 times across 13 unique participants, and “Metacognitive” appeared 24 times in observations and 17 times in interviews, each across 10 unique participants.
  • Some students attributed intelligence or agency to Copilot, e.g., “It like read my mind!” and “I thought it was weird that it knows what I want,” while others raised fears about future job impacts, privacy, plagiarism, and over-reliance.

Methods

  • Nineteen novice programmers (5 women, 14 men, ages 18–22) enrolled in a CS1 course at a Midwestern private research university in the USA were observed one at a time in April 2022 while solving a Minesweeper-style C++ assignment in Visual Studio Code with Copilot enabled.
  • Each participant had 30 minutes, followed a think-aloud protocol, and then completed a short interview about whether Copilot helped them understand the problem, whether they would use it, its advantages, and their fears or worries.
  • Data from observations and interviews were analyzed together using reflexive thematic analysis following Braun and Clarke’s six phases; themes included Interactions, Cognitive, Purpose, and Speculation, with saturation treated as the point where additional data produced no new emergent themes.

Limitations

  • The paper states that screens and audio were not recorded due to IRB considerations, that study conditions were somewhat lab-like rather than normal in-class assignment conditions, that only Copilot was studied, and that data collection occurred in April 2022 shortly after Copilot’s release, so novelty effects may have shaped results.
  • Evident from the text: the sample is small (19 students), from a single institution and single CS1 course, with only a 30-minute exposure to Copilot, so findings may not generalize to other populations, courses, or longer-term use.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3617367

Cites

105 of 106 references verified1 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

11 Valid · 3 Not valid · 11 Not checked · 109 Own results and interpretations

56 more claims · 1 with their cited sources checked

Create a free account to see all 134 claims and to verify the 2 references still open.

See all claims — free account
  • Probably not a substantial claim. 46
  • It may state two separate claims as one. 38

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

7 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 91 references

Research works printed but not linked: 3

  1. Emily M. Bender. 2019. A Typology of Ethical Risks in Language Technology with an eye towards where transparent documentation can help. Future of Artificial Intelligence: Language, Ethics, Technology Workshop.
  2. 10.1109/SP46215.2023.00001 · doi:10.1109/sp46215.2023.00001
  3. Ashley D. Rittmayer and Margaret E. Beier. 2008. Overview: Self-efficacy in STEM. SWE-AWE CASEE Overviews 1, 3 (2008), 12.

Cited by

Items 201–247 of 247

Metadata

Authors
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A.
Affiliations
  • Abilene Christian University (US)
  • University of Auckland (NZ)
  • University College Dublin (IE)
Venue
ACM Transactions on Computer-Human Interaction
Published
2023-11-29
Volume
31
Issue
1
Pages
1-31
ISSN
  • 1073-0516 (ISSN-L)
  • 1557-7325
OpenAlex
W4386099272
Cited by
256
References
91
Citations by year
2026: 89 · 2025: 85 · 2024: 66 · 2023: 16
FWCI
57.98
Type
journal article
Language
en
Open access
Locations
  • ACM Transactions on Computer-Human Interaction (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Social Sciences › Education › Artificial Intelligence in Education
Keywords
GitHub Copilot, code generation, novice programmers, introductory programming, CS1, programming education, student perceptions, metacognition, cognitive load, scaffolding, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.