Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index; 15 of the 27 sources its claims cite were checked for support.

Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming

Open access
Mozannar, Hussein, Bansal, Gagan, Fourney, Adam, Horvitz, Eric · 2024 · Proceedings of the CHI Conference on Human Factors in Computing Systems

Abstract

Code-recommendation systems, such as Copilot and CodeWhisperer, have the potential to improve programmer productivity by suggesting and auto-completing code. However, to fully realize their potential, we must understand how programmers interact with these systems and identify ways to improve that interaction. To seek insights about human-AI collaboration with code recommendations systems, we studied GitHub Copilot, a code-recommendation system used by millions of programmers daily. We developed CUPS, a taxonomy of common programmer activities when interacting with Copilot. Our study of 21 programmers, who completed coding tasks and retrospectively labeled their sessions with CUPS, showed that CUPS can help us understand how programmers interact with code-recommendation systems, revealing inefficiencies and time costs. Our insights reveal how programmers interact with Copilot and motivate new interface designs and metrics.

Overview

CoChat Summary

TL;DR

The paper introduces CUPS, a 12-state taxonomy of programmer activities during AI-assisted coding, and uses a 21-participant Copilot study with retrospective self-labeling of telemetry segments to profile where time goes. It finds that Copilot-specific activities consume over half of session time on average and that deferring verification inflates acceptance metrics and hides later verification costs.

Key findings

  • Across 21 developers and 3137 CUPS labels, “Verifying Suggestion” was the largest time state at 22.4% (SD=12.97) of session time, followed by “Writing New Functionality” at 14.05% (SD=8.36); Copilot-specific states together accounted for 51.5% (SD=19.3) of average session duration. The conclusion reports 34.3% of total session time on double-checking/editing Copilot suggestions.
  • Deferring thought was frequent (61 labeled states). Acceptance probability was 0.98 ± 0.02 in “Deferring Thought For Later” vs 0.12 ± 0.04 in “Thinking About New Code To Write” (average acceptance 0.34). Accepted suggestions were edited with probability 0.53 if deferred vs 0.18 if verified before (chi-square p=0).
  • Post-acceptance verification is substantial: when participants deferred thought and accepted, 53.2% verified immediately afterward; adjusted mean verification time rose from 3.25s (SD=3.33) to 15.21s (SD=20.68), median from 1.99s to 6.48s. For “Thinking/Verifying Suggestion” before acceptance, 19% post-hoc verified, raising mean from 3.96s to 7.03s.
  • Prompt crafting: among 234 suggestions shown while prompt crafting, only 10.7% were accepted; across 59 prompt-crafting attempts, 81.3% ended with an accepted suggestion. Common transitions included Writing New Functionality → Verifying Suggestion (0.59), Prompt Crafting → Verifying Suggestion (0.54), and Deferring Thought → Verifying Suggestion (0.54).

Methods

  • Mixed-methods: authors iteratively developed CUPS via pilots, then ran a user study with 21 participants recruited from a large organization; 7 used Copilot more than a few times/week, 3 once/month or less, 11 never. Participants used VS Code + Copilot on a VM, solved one of eight pre-selected coding tasks within a 20-minute block (~60-minute session), then retrospectively labeled their own telemetry segments with CUPS immediately after coding.
  • Data/analysis: 3137 CUPS labels; Copilot showed 1024 suggestions in task sessions, 34.0% accepted (average participant acceptance 36.5%, median 33.8%). Analyses included time-in-state aggregation, finite-state-machine transition probabilities/CUPS diagrams, n-gram pattern mining, acceptance probabilities by state, t-tests/chi-square tests, and entropy rate; IDK labels (353 uses, 6.5% of session time) were removed.

Limitations

  • Stated: participants solved time-limited, preselected tasks rather than real-world tasks; task selection was limited; mostly Python (19/21; one C++, one JavaScript); the taxonomy assumes at most one state per telemetry segment; longer-term costs (e.g., security vulnerabilities, overreliance) were not captured; no baseline condition without Copilot; self-labeling may cause inconsistent labeling across participants; small participant numbers limited comparisons by expertise/Copilot experience.
  • Evident from the text: the study is small (n=21), single-organization, and tied to Copilot as of August 2022, so generalization to other CodeRec systems, languages, and settings is uncertain; IDK segments were removed rather than relabeled, and the paper notes labeled data were insufficient for highly accurate CUPS-state prediction models.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3613904.3641936

Cites

41 of 41 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Image 1
CC-BY · From the PDF
Image 2
CC-BY · From the PDF
Figure 2 Schematic of interaction telemetry with Copilot as a timeline. For a given coding session, the telemetry contains a sequence of timestamps and actions with associated prompt and suggestion features (not shown).
Figure 2 Schematic of interaction telemetry with Copilot as a timeline. For a given coding session, the telemetry contains a sequence of timestamps and actions with associated prompt and suggestion features (not shown).CC-BY · From the PDF
Figure 3 Taxonomy of programmer's activities when interacting with CodeRecCUPS.
Figure 3 Taxonomy of programmer's activities when interacting with CodeRecCUPS.CC-BY · From the PDF
Figure 4 Screenshot of retrospective labeling tool for coding sessions. Left: Navigation panel for telemetry segments. Right: Video player for reviewing video of a coding session. Bottom: Buttons and text box for labeling states.
Figure 4 Screenshot of retrospective labeling tool for coding sessions. Left: Navigation panel for telemetry segments. Right: Video player for reviewing video of a coding session. Bottom: Buttons and text box for labeling states.CC-BY · From the PDF
(a) Individual CUPS timelines for 5/21 study participants for the frst 180 secs show the richness of and variance in programmer-CodeRec interaction.
(a) Individual CUPS timelines for 5/21 study participants for the frst 180 secs show the richness of and variance in programmer-CodeRec interaction.CC-BY · From the PDF
(b) The percentage of total session time spent in each state during a coding session. On average, verifying Copilot suggestions occupies a large portion of session time.
(b) The percentage of total session time spent in each state during a coding session. On average, verifying Copilot suggestions occupies a large portion of session time.CC-BY · From the PDF
Figure 6 Myriad of CUPS patterns observed in our study.
Figure 6 Myriad of CUPS patterns observed in our study.CC-BY · From the PDF
Figure 8 Illustration of one of the adjustments required for measuring the total time a programmer spends to verify a suggestion. Here, when a programmer defers thought for a suggestion, they spend time verifying it after accepting it and may also have to wait beforehand for the suggestion to be shown.
Figure 8 Illustration of one of the adjustments required for measuring the total time a programmer spends to verify a suggestion. Here, when a programmer defers thought for a suggestion, they spend time verifying it after accepting it and may also have to wait beforehand for the suggestion to be shown.CC-BY · From the PDF

Verification

Claims the paper makes

12 Valid · 0 Not valid · 7 Not checked · 118 Own results and interpretations

56 more claims

Create a free account to see all 137 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 47
  • It may state two separate claims as one. 43
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

1 more claim about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 23 references

Cited by

See all 114

Metadata

Authors
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E.
Affiliations
  • MIT Computer Science and Artificial Intelligence Laboratory (US)
  • Microsoft (United States) (US)
Venue
Proceedings of the CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2024-05-11
Pages
1-16
OpenAlex
W4396833115
Cited by
114
References
23
Citations by year
2026: 36 · 2025: 44 · 2024: 28
FWCI
41.12
Type
conference paper
Language
en
Open access
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
  • arXiv (Cornell University)
Funders
Microsoft Research
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
Keywords
GitHub Copilot, AI-assisted programming, human-AI collaboration, programmer behavior, code completion, interface design, developer productivity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming. https://app.cochat.ai/papers/10.1145/3613904.3641936
The paper
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2024). Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming. Proceedings of the CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3613904.3641936

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.