The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index; 15 of the 27 sources its claims cite were checked for support.
Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Open accessAbstract
Code-recommendation systems, such as Copilot and CodeWhisperer, have the potential to improve programmer productivity by suggesting and auto-completing code. However, to fully realize their potential, we must understand how programmers interact with these systems and identify ways to improve that interaction. To seek insights about human-AI collaboration with code recommendations systems, we studied GitHub Copilot, a code-recommendation system used by millions of programmers daily. We developed CUPS, a taxonomy of common programmer activities when interacting with Copilot. Our study of 21 programmers, who completed coding tasks and retrospectively labeled their sessions with CUPS, showed that CUPS can help us understand how programmers interact with code-recommendation systems, revealing inefficiencies and time costs. Our insights reveal how programmers interact with Copilot and motivate new interface designs and metrics.
Overview
CoChat Summary
TL;DR
The paper introduces CUPS, a 12-state taxonomy of programmer activities during AI-assisted coding, and uses a 21-participant Copilot study with retrospective self-labeling of telemetry segments to profile where time goes. It finds that Copilot-specific activities consume over half of session time on average and that deferring verification inflates acceptance metrics and hides later verification costs.
Key findings
- Across 21 developers and 3137 CUPS labels, “Verifying Suggestion” was the largest time state at 22.4% (SD=12.97) of session time, followed by “Writing New Functionality” at 14.05% (SD=8.36); Copilot-specific states together accounted for 51.5% (SD=19.3) of average session duration. The conclusion reports 34.3% of total session time on double-checking/editing Copilot suggestions.
- Deferring thought was frequent (61 labeled states). Acceptance probability was 0.98 ± 0.02 in “Deferring Thought For Later” vs 0.12 ± 0.04 in “Thinking About New Code To Write” (average acceptance 0.34). Accepted suggestions were edited with probability 0.53 if deferred vs 0.18 if verified before (chi-square p=0).
- Post-acceptance verification is substantial: when participants deferred thought and accepted, 53.2% verified immediately afterward; adjusted mean verification time rose from 3.25s (SD=3.33) to 15.21s (SD=20.68), median from 1.99s to 6.48s. For “Thinking/Verifying Suggestion” before acceptance, 19% post-hoc verified, raising mean from 3.96s to 7.03s.
- Prompt crafting: among 234 suggestions shown while prompt crafting, only 10.7% were accepted; across 59 prompt-crafting attempts, 81.3% ended with an accepted suggestion. Common transitions included Writing New Functionality → Verifying Suggestion (0.59), Prompt Crafting → Verifying Suggestion (0.54), and Deferring Thought → Verifying Suggestion (0.54).
Methods
- Mixed-methods: authors iteratively developed CUPS via pilots, then ran a user study with 21 participants recruited from a large organization; 7 used Copilot more than a few times/week, 3 once/month or less, 11 never. Participants used VS Code + Copilot on a VM, solved one of eight pre-selected coding tasks within a 20-minute block (~60-minute session), then retrospectively labeled their own telemetry segments with CUPS immediately after coding.
- Data/analysis: 3137 CUPS labels; Copilot showed 1024 suggestions in task sessions, 34.0% accepted (average participant acceptance 36.5%, median 33.8%). Analyses included time-in-state aggregation, finite-state-machine transition probabilities/CUPS diagrams, n-gram pattern mining, acceptance probabilities by state, t-tests/chi-square tests, and entropy rate; IDK labels (353 uses, 6.5% of session time) were removed.
Limitations
- Stated: participants solved time-limited, preselected tasks rather than real-world tasks; task selection was limited; mostly Python (19/21; one C++, one JavaScript); the taxonomy assumes at most one state per telemetry segment; longer-term costs (e.g., security vulnerabilities, overreliance) were not captured; no baseline condition without Copilot; self-labeling may cause inconsistent labeling across participants; small participant numbers limited comparisons by expertise/Copilot experience.
- Evident from the text: the study is small (n=21), single-organization, and tied to Copilot as of August 2022, so generalization to other CodeRec systems, languages, and settings is uncertain; IDK segments were removed rather than relabeled, and the paper notes labeled data were insufficient for highly accurate CUPS-state prediction models.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures









Verification
Claims the paper makes
12 Valid · 0 Not valid · 7 Not checked · 118 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, in each case, these metrics test how well the model can complete code in an ofine setting without developer input rather than evaluating how well such recommendations assist programmers in situ.”
§ Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming > 2 BACKGROUND AND RELATED WORK
What the sources say
- Not checked — no full text of the sourcecites [9] 9. Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, and Zhen Ming Jack Jiang. 2023. Github copilot ai pair programmer: Asset or liability? Journal of Systems and Software 203 (2023), 111734. source
- Supportedcites [11] 11. Mikhail Evtikhiev, Egor Bogomolov, Yaroslav Sokolov, and Timofey Bryksin. 2023. Out of the bleu: how should we assess quality of the code generation models? Journal of Systems and Software 203 (2023), 111741. source
Section 2.2 (p. 3) describes code-generation metrics as automated measures of model output, including comparisons with reference code and tests of generated code. The Abstract (p. 2) says developers separately rated whether generated snippets would help solve a posed problem. These are evaluations of generated outputs, not measurements of how recommendations assist programmers during their work. (LLM, conf=0.86)
- Supportedcites [16] 16. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. 2021. Measuring Coding Challenge Competence With APPS. In Thirty-ffth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). ., ., 1-2. source
The Metrics section (p. 7) describes automatic evaluation of generated code against test cases, without manual analysis. The Introduction (p. 2) likewise says APPS assesses programs with test cases rather than human feedback. This supports the claim that APPS measures standalone code-generation performance, not how recommendations assist programmers in situ. (LLM, conf=0.88)
- Partly supportedcites [21] 21. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-level code generation with alphacode. arXiv preprint arXiv:2203.07814 1, 1 (2022), 1-2. sourceExperimental — accuracy not yet measured
Section 7.4 (p. 29) describes metrics that assess generated code against reference code or by executing programs; Section 2.2 (p. 6) describes a benchmark based on problems solved by model-generated submissions. These support the point that the paper's metrics assess model output rather than how recommendations help programmers, but the paper does not establish the broader claim about code-generation metrics and datasets generally, nor does it explicitly contrast them with assistance in situ. (LLM, conf=0.78)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, the second-most probable transition from the 'Prompt Crafting' state is 'Prompt Crafting 0 --.25 → Waiting for Suggestion'.”
§ Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming > 6.2 Patterns in Behavior as Transitions Between CUPS States
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Likewise, Another probable transition was 'Deferring Thought 0 --.54 →Verifying Suggestion', indicating that if a programmer previously deferred their thought for an accepted suggestion, they would, with high probability, return to verify that suggestion.”
§ Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming > 6.2 Patterns in Behavior as Transitions Between CUPS States
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The leaf nodes of the fnalized taxonomy represent 12 distinct states that programmers can fnd themselves in.”
§ Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming > 4.2 Taxonomy of Telemetry Segments
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The second-lengthiest state is writing new functionality' 14.05% ( = 8.36), all but 6 participants spend more than 9% of session time in this state.”
§ Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming > Metric Suggestion: Time spent in CUPS states as a highlevel diagnosis of the interaction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
56 more claims
Create a free account to see all 137 claims and every source verdict.
- Probably not a substantial claim. 47
- It may state two separate claims as one. 43
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Previous research shows that developers spend almost 50% of coding time in interaction with the LLM, with 35% dedicated to double-checking suggestions, since the generated code is limited in meeting both functional and non-functional requirements [17].”
From Using AI-based coding assistants in practice: State of affairs, perceptions, and ways forward (2025) · cites this paper as [17]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Despite growing evidence that AI coding assistants introduce new cognitive demands around reviewing and validating suggestions, there has been limited investigation of the cognitive load imposed by programming with AI assistance.”
From Towards Decoding Developer Cognition in the Age of AI Assistants (2025) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The time saved in writing code is often balanced by new cognitive demands as developers must carefully review and validate AI-generated suggestions [8].”
From Towards Decoding Developer Cognition in the Age of AI Assistants (2025) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While developers consistently report significant productivity gains from these tools, empirical measurements often show more modest improvements.”
From Towards Decoding Developer Cognition in the Age of AI Assistants (2025) · cites this paper as [8]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Mozannar et al. [8] found that while developers perceive AI tools to be beneficial, they introduce new bottlenecks for reading, understanding and validating AIgenerated code suggestion.”
From Towards Decoding Developer Cognition in the Age of AI Assistants (2025) · cites this paper as [8]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming > 1 INTRODUCTION
“The study participants labeled a total of 3137 coding segments and interacted with 1096 suggestions.”
§ Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming > 5.2 Participants
“During the coding tasks, Copilot showed participants a total of 1024 suggestions, out of which they accepted 34.0%.”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
A survey on semi-supervised learning
van Engelen, J. E., & Hoos, H. H. (2020) · Machine Learning · doi:10.1007/s10994-019-05855-6 · Cited by 2,666
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
The keystroke-level model for user performance time with interactive systems
Card, S. K., Moran, T. P., & Newell, A. (1980) · Communications of the ACM · doi:10.1145/358886.358895 · Cited by 1,174 · cited 2×
“This taxonomy also bears re-semblance to the keystroke level model in that it assigns a time cost to mental processes as well as typing [7, 18].”
Competition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043
The GOMS family of user interface analysis techniques: comparison and contrast
John, B. E., & Kieras, D. E. (1996) · ACM Transactions on Computer-Human Interaction · doi:10.1145/235833.236054 · Cited by 687
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions
Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833571 · Cited by 476
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436
Cited by
Using an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487
Navigating the Complexity of Generative AI Adoption in Software Engineering
Russo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3652154 · Cited by 217
The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B. (2024) · Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3632620.3671116 · Cited by 196
LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation
Fakhoury, S., Naik, A., Sakkas, G., Chakraborty, S., & Lahiri, S. K. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3428972 · Cited by 120
Using AI-based coding assistants in practice: State of affairs, perceptions, and ways forward
Sergeyuk, A., Golubev, Y., Bryksin, T., & Ahmed, I. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2024.107610 · Cited by 101
Metadata
- Authors
- Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E.
- Affiliations
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Microsoft (United States) (US)
- Venue
- Proceedings of the CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2024-05-11
- Pages
- 1-16
- OpenAlex
- W4396833115
- Cited by
- 114
- References
- 23
- Citations by year
- 2026: 36 · 2025: 44 · 2024: 28
- FWCI
- 41.12
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
- arXiv (Cornell University)
- Funders
- Microsoft Research
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- GitHub Copilot, AI-assisted programming, human-AI collaboration, programmer behavior, code completion, interface design, developer productivity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming. https://app.cochat.ai/papers/10.1145/3613904.3641936
- The paper
- Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2024). Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming. Proceedings of the CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3613904.3641936
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.