The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index; 6 of the 16 sources its claims cite were checked for support.
Grounded Copilot: How Programmers Interact with Code-Generating Models
Open accessAbstract
Powered by recent advances in code-generating models, AI assistants like Github Copilot promise to change the face of programming forever. But what is this new face of programming? We present the first grounded theory analysis of how programmers interact with Copilot, based on observing 20 participants—with a range of prior experience using the assistant—as they solve diverse programming tasks across four languages. Our main finding is that interactions with programming assistants are bimodal : in acceleration mode , the programmer knows what to do next and uses Copilot to get there faster; in exploration mode , the programmer is unsure how to proceed and uses Copilot to explore their options. Based on our theory, we provide recommendations for improving the usability of future AI programming assistants.
Overview
CoChat Summary
TL;DR
This paper presents a grounded theory of how programmers interact with GitHub Copilot, based on observing 20 participants across four programming languages. Its central finding is that Copilot-assisted programming is bimodal: programmers use Copilot either in acceleration mode, to execute a known plan faster, or in exploration mode, to discover how to proceed.
Key findings
- Interactions with Copilot are bimodal: acceleration mode occurs when the programmer already knows what to do next and uses Copilot to get there faster; exploration mode occurs when the programmer is unsure how to proceed and uses Copilot to explore options or get a starting point.
- Across all participants, total study time in exploration mode was 248.6 minutes, more than twice the 104.7 minutes in acceleration mode.
- Prompting strategies differed by mode: in acceleration mode, code prompts were most common (71.4%) and context prompts second (15.2%); in exploration mode, comment prompts were much more common than in acceleration mode (57.2% vs 13.1%).
- Validation also differed: exploration involved more careful examination and documentation reading, while acceleration more often used execution for rapid feedback; participants also used five livestream videos to confirm behavior consistent with the theory.
Methods
- The authors conducted a Straussian grounded theory study with 20 participants (15 from academia, 5 from industry; 9 had prior Copilot usage) who solved programming tasks with Copilot across Python, Rust, Haskell, and Java.
- Participants completed a training task, a core task lasting about 20–40 minutes while talking through their interactions, and a semi-structured interview; sessions were recorded and transcribed.
- Analysis interleaved data collection and qualitative coding, with open, axial, and selective coding, theoretical sampling, and theoretical saturation at the 20th participant; the authors then re-coded videos quantitatively and qualitatively analyzed five livestream videos from YouTube and Twitch.
Limitations
- Stated in the paper: participants worked on tasks of the authors’ design rather than their own projects, and the tasks focused only on code authorship, not refactoring, testing, debugging, or other software engineering activities.
- Stated in the paper: the 20-participant sample skews toward academia and is not representative of all programmers; 11 participants had not used Copilot before the study, and prior users may have formed usage patterns during the technical preview.
- Stated in the paper: the authors could not observe programmers over a longer period to study changing usage patterns, and rapid progress in code-generating models could render the findings obsolete.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures







Verification
Claims the paper makes
4 Valid · 0 Not valid · 8 Not checked · 13 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“OpenAI's Codex model [Chen et al. 2021], which contains 12 billion model parameters and is trained on 54 million software repositories on GitHub, is able to correctly solve 30-70% of novel Python problems”
§ Grounded Copilot: How Programmers Interact with Code-Generating Models > 1 INTRODUCTION
What the sources say
- Partly supportedcites [Chen et al. 2021] sourceExperimental — accuracy not yet measured
Section 3.1 (p. 4) says the training data came from 54 million public GitHub repositories, and Section 3 (p. 4) describes Codex models with up to 12 billion parameters. The Abstract (p. 1) reports 28.8% of HumanEval problems solved with one sample and 70.2% with 100 samples per problem. The claimed 30–70% range is approximately right, but it omits that the upper end required repeated sampling and evaluation on a specific problem set. (LLM, conf=0.86)
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Copilot is powered by the OpenAI Codex family of models [Chen et al. 2021], which are derived by fine-tuning GPT-3 [Brown et al. 2020] on publicly available Github repositories.”
§ Grounded Copilot: How Programmers Interact with Code-Generating Models > 2 COPILOT-ASSISTED PROGRAMMING, BY EXAMPLE
What the sources say
- Supportedcites [Chen et al. 2021] source
The Introduction (p. 1) says descendants of the Codex models described in this paper power GitHub Copilot. The Abstract (p. 1) identifies Codex as fine-tuned on publicly available GitHub code, and Methods (p. 4) describes fine-tuning from the GPT-3 model family. (LLM, conf=0.94)
- Supportedcites [Brown et al. 2020] source
The Brown et al. citation is attached to “GPT-3,” identifying the model used in the citing sentence. The Abstract (p. 1) states that the paper trains GPT-3. The separate claims about Copilot, Codex, and GitHub repositories are not attributed to this citation. (LLM, conf=0.93)
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“has recently been adopted to study phenomena in software engineering [Stol et al. 2016] and programming languages [Lubin and Chasins 2021].”
§ Grounded Copilot: How Programmers Interact with Code-Generating Models > 1 INTRODUCTION
What the sources say
- Supportedcites [Stol et al. 2016] source
The citation to Stol et al. is attached to the software-engineering portion of the sentence, not the programming-languages portion. The Conclusion (p. 11) describes grounded theory as an increasingly popular research method in software engineering and reports its use in SE studies. (LLM, conf=0.94)
- Supportedcites [Lubin and Chasins 2021] source
The Lubin and Chasins citation is attached to the programming-languages portion of the sentence. Section 8.1 describes their work as a grounded theory analysis of programming sessions by expert programmers, and Section 4 presents the resulting theory. The separate software-engineering portion is attributed to Stol et al., not this source. (LLM, conf=0.94)
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“We follow the framework laid out by Strauss and Corbin [Strauss and Corbin 1990], commonly called Straussian Grounded Theory [Stol et al. 2016].”
§ Grounded Copilot: How Programmers Interact with Code-Generating Models > Grounded Theory Process.
What the sources say
- Not matchedcites [Strauss and Corbin 1990]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites [Stol et al. 2016] source
The citation to Stol et al. supports the name of the framework, not the citing authors’ statement that they follow it. Section 2.3 (p. 4) explicitly identifies Strauss and Corbin’s grounded theory as “Straussian GT.” (LLM, conf=0.95)
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our main finding is that interactions with programming assistants are bimodal: in acceleration mode, the programmer knows what to do next and uses Copilot to get there faster; in exploration mode, the programmer is unsure how to proceed and uses Copilot to explore their options.”
§ Grounded Copilot: How Programmers Interact with Code-Generating Models > Abstract
20 more claims
Create a free account to see all 25 claims and every source verdict.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Earlier research noted the difficulties users faced in effectively communicating intent and matching their abstraction level to the model's capabilities, often relying on explicit comments (Sarkar et al., 2022a; Barke et al., 2023).”
From Vibe coding: programming through conversation with artificial intelligence (2025) · cites this paper as Barke et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While a number of researchers have looked at how current developers may use LLM-based tools [32,52,81] and how LLM-tools may be incorporated into professional software development tools [84], the participants in these studies have been programmers who learned to code initially without using LLMs.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [32]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These two activities align with the grounded theory proposed by Barke et al. [18], which describes two developers' interaction modes with AI assistants -acceleration (supported by code generation) and exploration (supported by code summarization).”
From Using AI-based coding assistants in practice: State of affairs, perceptions, and ways forward (2025) · cites this paper as [18]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This focus on augmenting the content quality of completions has inadvertently overshadowed a vital aspect of the user experience: the interaction dynamics between the developers and the AI tools [2, 34, 42].”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This introduces a workflow where the programmer may intentionally allow the AI to 'drive' more consistently, even for familiar tasks, creating an fluid interaction style that is distinct from the segmented modes previously identified (Barke et al., 2023).”
From Vibe coding: programming through conversation with artificial intelligence (2025) · cites this paper as Barke et al., 2023
Experimental — accuracy not yet measured
16 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Grounded Copilot: How Programmers Interact with Code-Generating Models > Usability of Copilot.
“only one participant in our study was an undergraduate student”
§ Grounded Copilot: How Programmers Interact with Code-Generating Models > PCU
“P9 Rust 1 Occasional No Undergraduate Student Benford's law P10 Python Occasional No Undergraduate Student Chat Client”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Grounded Copilot: How Programmers Interact with Code-Generating Models > 4.2.5 Programmers Validate Suggestions Explicitly.
“All participants in the Advent of Code task ran their code to check whether they parsed the input file correctly.”
§ Grounded Copilot: How Programmers Interact with Code-Generating Models > Static analysis.
“P17 did not run their code even once for the Advent of Code task”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Grounded Copilot: How Programmers Interact with Code-Generating Models > 5.1.2 Prompting Strategies across Interaction Modes.
“Fig. 6 shows the aggregate percentage of times the 20 participants invoked Copilot using the four different prompting strategies.”
§ Grounded Copilot: How Programmers Interact with Code-Generating Models > 8 LIMITATIONS AND THREATS TO VALIDITY
“Fig. 6.Prevalence of prompting strategy as percentage of total prompting time.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Grounded Copilot: How Programmers Interact with Code-Generating Models > 4.1.2 Programmers Focus on Small Logical Units.
“when P1 wanted to send a message to a client connection object in the Chat Client task”
§ Grounded Copilot: How Programmers Interact with Code-Generating Models > PCU
“P1 Python Professional Yes Professor Chat Server”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
The Discovery of Grounded Theory; Strategies for Qualitative Research
Glaser, B. G., Strauss, A. L., & Strutzel, E. (1968) · Nursing Research · doi:10.1097/00006199-196807000-00014 · Cited by 39,449
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
The SAGE Handbook of Grounded Theory
Bryant, A., & Charmaz, K. (2007) · SAGE · doi:10.4135/9781848607941 · Cited by 2,798
Flow and the Foundations of Positive Psychology: The Collected Works of Mihaly Csikszentmihalyi
Csikszentmihalyi, M. (2014) · doi:10.1007/978-94-017-9088-8 · Cited by 1,653
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Competition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043
(2000) · doi:10.1145/351240.351266 · Cited by 753
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Research works printed but not linked: 8
- Mohammad Bavarian Heewoo Jun Nikolas Tezak John Schulman Christine McLeavey Jerry Tworek and Mark Chen. 2022. Efficient Training of Language Models to Fill in the Middle. Jul https://doi.org/10.48550/arXiv.2207.14255 arXiv:2207.14255 [cs]. · DOI · Paper page
- Daniel Fried Armen Aghajanyan Jessy Lin Sida Wang Eric Wallace Freda Shi Ruiqi Zhong Wen-tau Yih Luke Zettlemoyer and Mike Lewis. 2022. InCoder: A Generative Model for Code Infilling and Synthesis. Apr https://doi.org/10.48550/arXiv.2204.05999 arXiv:2204.05999 [cs]. · DOI · Paper page
- Daniel Kahneman . 2011. Thinking , fast and slow . Penguin Books . isbn:978-0-14-103357-0 Daniel Kahneman. 2011. Thinking, fast and slow. Penguin Books. isbn:978-0-14-103357-0
- Kite. 2020 . Kite: AI-Powered Completions for JupyterLab. https://www.kite.com/integrations/jupyter/ Kite. 2020. Kite: AI-Powered Completions for JupyterLab. https://www.kite.com/integrations/jupyter/
- Yujia Li David Choi Junyoung Chung Nate Kushman Julian Schrittwieser Rémi Leblond Tom Eccles James Keeling Felix Gimeno Agustin Dal Lago Thomas Hubert Peter Choy Cyprien de Masson d'Autume Igor Babuschkin Xinyun Chen Po-Sen Huang Johannes Welbl Sven Gowal Alexey Cherepanov James Molloy Daniel J. Mankowitz Esme Sutherland Robson Pushmeet Kohli Nando de Freitas Koray Kavukcuoglu and Oriol Vinyals. 2022. Competition-Level Code Generation with AlphaCode. https://doi.org/10.48550/ARXIV.2203.07814 · DOI · Paper page
- Anselm L. Strauss and Juliet Corbin . 1990 . Basics of Qualitative Reseach: Grounded Theory Procedures and Techniques. SAGE Publications, Inc .. isbn:0-8039-3250-2 Anselm L. Strauss and Juliet Corbin. 1990. Basics of Qualitative Reseach: Grounded Theory Procedures and Techniques. SAGE Publications, Inc.. isbn:0-8039-3250-2
- Steven J. Taylor and Robert Bogdan . 1998 . Introduction to qualitative research methods: A guidebook and resource, 3 rd ed. John Wiley and Sons Inc . isbn:978-0-471-16868-3 Steven J. Taylor and Robert Bogdan. 1998. Introduction to qualitative research methods: A guidebook and resource, 3rd ed. John Wiley and Sons Inc. isbn:978-0-471-16868-3
- Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser and Illia Polosukhin. 2017. Attention Is All You Need. Dec https://doi.org/10.48550/arXiv.1706.03762 arXiv:1706.03762 [cs]. · DOI · Paper page
Cited by
Items 101–200 of 434
Towards Understanding the Characteristics of Code Generation Errors Made by Large Language Models
Wang, Z., Zhou, Z., Song, D., Huang, Y., Chen, S., Ma, L., & Zhang, T. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00180 · Cited by 118
“Existing studies show that some incorrect code can still serve as a useful starting point for developers [9], [10].”
Can GPT-4 Replicate Empirical Software Engineering Research?
Liang, J. T., Badea, C., Bird, C., DeLine, R., Ford, D., Forsgren, N., & Zimmermann, T. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660767 · Cited by 19
“Therefore, applying directly tying each assumption, analysis plan, and line of code to specific sections in the methodology text could be beneficial to users.”
MethodStudents' Use of GitHub Copilot for Working with Large Code Bases
Shah, A., Chernova, A., Tomson, E., Porter, L., Griswold, W. G., & Soosai Raj, A. G. (2025) · Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3641554.3701800 · Cited by 19
Identifying the Factors That Influence Trust in AI Code Completion
Brown, A., D'Angelo, S., Murillo, A., Jaspan, C., & Green, C. (2024) · Proceedings of the 1st ACM International Conference on AI-Powered Software · doi:10.1145/3664646.3664757 · Cited by 20
“Many early results appear to support these hypotheses, either in terms of objective measurements (e.g., time to complete a development task) or self-reported productivity [5, 7, 25, 28].”
Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2022) · arXiv · doi:10.48550/arxiv.2210.14306 · Cited by 114
“[Barke et al., 2022], who showed that interaction with Copilot falls into two broad categories: the programmer is either in “acceleration mode” where they know what they want to do, and Copilot serves to make them faster; or they are in “exploration mode”, where they are unsure what code to write and Copilot helps the…”
BackgroundGuo, J., Mohanty, V., Ono, J. P., Hao, H., Gou, L., & Ren, L. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3651042 · Cited by 42
Students' Perspectives on AI Code Completion: Benefits and Challenges
Takerngsaksiri, W., Warusavitarne, C., Yaacoub, C., Keng Hou, M. H., & Tantithamthavorn, C. (2024) · 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00252 · Cited by 18
“Similarly, Barke et al. [5] present the first grounded theory analysis of how programmers interact with GitHub Copilot.”
BackgroundCompositional API Recommendation for Library-Oriented Code Generation
Ma, Z., An, S., Xie, B., & Lin, Z. (2024) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3643916.3644403 · Cited by 42
Developer Behaviors in Validating and Repairing LLM-Generated Code Using IDE and Eye Tracking
Tang, N., Chen, M., Ning, Z., Bansal, A., Huang, Y., McMillan, C., & Li, T. J.-J. (2024) · 2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc60511.2024.00015 · Cited by 18
Wang, C., Lee, B., Drucker, S. M., Marshall, D., & Gao, J. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713296 · Cited by 70
“Because data transformation can be difcult to learn and execute, many AI-powered tools have been developed [2, 10, 31, 36, 57, 58].”
COLDECO: An End User Spreadsheet Inspection Tool for AI-Generated Code
Ferdowsi, K., Williams, J. M., Drosos, I., Gordon, A. D., Negreanu, C., Polikarpova, N., Sarkar, A., & Zorn, B. G. (2023) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc57772.2023.00017 · Cited by 26
“These studies show that programmers spend a significant proportion of their time inspecting generated code [22], [24] and often have trouble understanding [21] and debugging [22] such code, or evaluating its correctness [26].”
BackgroundSelf-citationPinto, G., De Souza, C., Rocha, T., Steinmacher, I., Souza, A., & Monteiro, E. (2024) · Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering - Software Engineering for AI · doi:10.1145/3644815.3644949 · Cited by 17
Generating Programs Trivially: Student Use of Large Language Models
Prasad, S., Greenman, B., Nelson, T., & Krishnamurthi, S. (2023) · ACM Global Computing Education Conference (CompEd) · doi:10.1145/3576882.3617921 · Cited by 35
Toward Effective AI Support for Developers: A survey of desires and concerns.
Khemka, M., & Houck, B. (2024) · Communications of the ACM · doi:10.1145/3690928 · Cited by 32
A User-centered Security Evaluation of Copilot
Asare, O., Nagappan, M., & Asokan, N. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639154 · Cited by 32
An assessment of large language models for OpenMP-based code parallelization: a user perspective
Mišić, M. J., & Dodović, M. (2024) · Journal Of Big Data · doi:10.1186/s40537-024-01019-z · Cited by 20
The Impact of Prompt Programming on Function-Level Code Generation
Khojah, R., Neto, F. G. D. O., Mohamad, M., & Leitner, P. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3587794 · Cited by 76
Song, F., Agarwal, A., & Wen, W. (2024) · SSRN Electronic Journal (preprint) · doi:10.2139/ssrn.4856935 · Cited by 30
“First, it contributes to the literature on generative AI in software development (Imai 2022, Barke et al. 2023, Biswas 2023, Peng et al. 2023).”
BackgroundConversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities
Chopra, B., Singha, A., Fariha, A., Gulwani, S., Parnin, C., Tiwari, A., & Henley, A. Z. (2023) · arXiv · doi:10.48550/arxiv.2310.16164 · Cited by 16
Exploring Early Adopters' Perceptions of ChatGPT as a Code Generation Tool
Scoccia, G. L. (2023) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew60602.2023.00016 · Cited by 28
Assessing the Performance of AI-Generated Code: A Case Study on GitHub Copilot
Li, S., Cheng, Y., Chen, J.-F., Xuan, J.-F., He, S., & Shang, W.-Y. (2024) · IEEE International Symposium on Software Reliability Engineering · doi:10.1109/issre62328.2024.00030 · Cited by 29
Generative AI for Software Development: A Family of Studies on Code Generation
Dakhel, A. M., Nikanjam, A., Khomh, F., Desmarais, M. C., & Washizaki, H. (2024) · Springer eBooks · doi:10.1007/978-3-031-55642-5_7 · Cited by 20
Reasoning and Planning with Large Language Models in Code Development
Ding, H., Fan, Z., Guehring, I., Gupta, G. P., Ha, W., Huan, J., Liu, L. Q., Omidvar-Tehrani, B., Wang, S., & Zhou, H. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671452 · Cited by 40
Hao, H., Hasan, K. A., Qin, H., Macedo, M., Tian, Y., Ding, S. H. H., & Hassan, A. E. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10540-x · Cited by 51
Autoethnographic Insights from Neurodivergent GAI “Power Users”
Glazko, K. S., Cha, J., Lewis, A., Kosa, B., Wimer, B. L., Zheng, A., Zheng, Y., & Mankoff, J. C. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713670 · Cited by 32
Tab to Autocomplete: The Effects of AI Coding Assistants on Web Accessibility
Mowar, P., Peng, Y.-H., Steinfeld, A. M., & Bigham, J. P. (2024) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663548.3688513 · Cited by 23
LLM-mediated domain-specific voice agents: the case of TextileBot
Zhong, S., Gatti, E., Hardwick, J., Ribul, M., Cho, Y., & Obrist, M. (2025) · Behaviour and Information Technology · doi:10.1080/0144929x.2025.2456667 · Cited by 22
Analyzing Prompt Influence on Automated Method Generation: An Empirical Study with Copilot
Fagadau, I. D., Mariani, L., Micucci, D., & Riganelli, O. (2024) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3643916.3644409 · Cited by 38
Hliš, T., Četina, L., Beranič, T., & Pavlič, L. (2023) · Applied Sciences · doi:10.3390/app132413061 · Cited by 23
Haase, J., Kremser, W., Leopold, H., Mendling, J., Onnasch, L., Technische Universität Berlin, G., & Plattfaut, R. (2024) · Communications of the Association for Information Systems · doi:10.17705/1cais.05421 · Cited by 30
"You're on a bicycle with a little motor": Benefits and Challenges of Using AI Code Assistants
Mendes, W., Souza, S., & de Souza, C. R. B. (2024) · International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1145/3641822.3641882 · Cited by 41
Multi-line AI-Assisted Code Authoring
Dunay, O., Cheng, D., Tait, A., Thakkar, P., Rigby, P. C., Chiu, A., Ahmad, I., Ganesan, A., Maddila, C., Murali, V., et al. (2024) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3663529.3663836 · Cited by 24
Lyu, W., Wang, Y., Sun, Y., & Zhang, Y. (2025) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3698205.3729544 · Cited by 35
How Scientists Use Large Language Models to Program
O'Brien, G. (2025) · arXiv · doi:10.48550/arxiv.2502.17348 · Cited by 14
Kellogg, K. C., Lifshitz‐Assaf, H., Randazzo, S., Mollick, E., Dell’Acqua, F., McFowland, E., Candelon, F., & Lakhani, K. R. (2025) · Information and Organization · doi:10.1016/j.infoandorg.2025.100559 · Cited by 26
BISCUIT: Scaffolding LLM-Generated Code with Ephemeral UIs in Computational Notebooks
Cheng, R., Barik, T., Leung, A., Hohman, F., & Nichols, J. W. (2024) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc60511.2024.00012 · Cited by 43
Dawson, M. G., Deer, R., & Boguslawski, S. (2025) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2025.100724 · Cited by 29
Don’t settle for the first! How many GitHub Copilot solutions should you check?
Oertel, J., Klünder, J., & Hebig, R. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107737 · Cited by 22
ChatGPT in Data Visualization Education: A Student Perspective
Kim, N. W., Ko, H.-K., Myers, G., & Bach, B. (2024) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc60511.2024.00022 · Cited by 30
Do LLMs Meet the Needs of Software Tutorial Writers? Opportunities and Design Implications
Bhat, A., Shrivastava, D., & Guo, J. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3660692 · Cited by 19
MAPE-K Loop-Based Goal Model Generation Using Generative AI
Nakagawa, H., & Honiden, S. (2023) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew57809.2023.00050 · Cited by 27
Davila, N., Melegati, J., & Wiese, I. (2024) · IEEE Software · doi:10.1109/ms.2024.3428439 · Cited by 25
Building Your Own Product Copilot: Challenges, Opportunities, and Needs
Parnin, C., Soares, G., Pandita, R., Gulwani, S., Rich, J., & Henley, A. Z. (2023) · arXiv · doi:10.48550/arxiv.2312.14231 · Cited by 13
Assessing the Effectiveness and Security Implications of AI Code Generators
Taeb, M., Chi, H., & Bernadin, S. (2024) · Journal of The Colloquium for Information Systems Security Education · doi:10.53735/cisse.v11i1.180 · Cited by 27
Stone, I. (2024) · Conference on United Kingdom & Ireland Computing Education Research (UKICER) · doi:10.1145/3689535.3689553 · Cited by 21
Insights from the Frontline: GenAI Utilization Among Software Engineering Students
Choudhuri, R., Ramakrishnan, A., Chatterjee, A., Trinkenreich, B., Steinmacher, I., Gerosa, M. A., & Sarma, A. (2025) · IEEE International Conference on Software Engineering Education and Training (CSEE&T) · doi:10.1109/cseet66350.2025.00007 · Cited by 38
An Investigation of the Drivers of Novice Programmers' Intentions to Use Web Search and GenAI
Skripchuk, J., Bacher, J. T., & Price, T. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671112 · Cited by 22
How Scientists Use Large Language Models to Program
O’Brien, G. E. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713668 · Cited by 42
Investigating the Role of Cultural Values in Adopting Large Language Models for Software Engineering
Lambiase, S., Catolino, G., Palomba, F., Ferrucci, F., & Russo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3725529 · Cited by 38
Human-AI experience in integrated development environments: a systematic literature review
Sergeyuk, A., Zakharov, I., Koshchenko, E., & Izadi, M. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10793-0 · Cited by 49
Gong, X., Xu, W., & Qiao, A. (2025) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-025-00552-y · Cited by 21
Rethinking AI code generation: a one-shot correction approach based on user feedback
Le, K. T., & Andrzejak, A. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00451-y · Cited by 23
From Triumph to Uncertainty: The Journey of Software Engineering in the AI Era
Mastropaolo, A., Escobar‐Velásquez, C., & Linares‐Vásquez, M. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709360 · Cited by 30
Nikolaidis, N., Flamos, K., Gulati, K., Feitosa, D., Ampatzoglou, A., & Chatzigeorgiou, A. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering - Companion (SANER-C) · doi:10.1109/saner-c62648.2024.00018 · Cited by 23
Vibe coding: programming through conversation with artificial intelligence
Sarkar, A., & Drosos, I. (2025) · arXiv · doi:10.48550/arxiv.2506.23253 · Cited by 11
“Based on these observations, vibe coding appears to occupy a higher position on the spectrum of AI reliance in programming activities than “traditional” AI-assisted coding (e.g., as with the initial generation of GitHub Copilot as documented in Sarkar et al. (2022a) or Barke et al. (2023)).”
InfluentialStatically Contextualizing Large Language Models with Typed Holes
Blinn, A., Li, X., Kim, J., & Omar, C. (2024) · Proc. ACM Program. Lang. · doi:10.1145/3689728 · Cited by 25
Kohn, T. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70043 · Cited by 15
Qiao, H., Vermeulen, J., Fitzmaurice, G., & Matejka, J. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714103 · Cited by 11
Autonomous Agents in Software Development: A Vision Paper
Rasheed, Z., Waseem, M., Kemell, K.-K., Wang, X., Duc, A. N., Systä, K., & Abrahamsson, P. (2023) · XP Workshops · doi:10.1007/978-3-031-72781-8_2 · Cited by 37
Cipriano, B. P., Alves, P., & Denny, P. C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72315-5_6 · Cited by 14
Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories
Bouzenia, I., & Pradel, M. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00234 · Cited by 136
Developer productivity for humans, part 8: Creativity in software engineering
Inman, S., D’Angelo, S., & Vasilescu, B. (2024) · IEEE Software · doi:10.1109/ms.2023.3340831 · Cited by 9
Basha, M., & Rodríguez-Pérez, G. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107804 · Cited by 19
Khurana, A., Su, X., Wang, A. Y., & Chilana, P. K. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713431 · Cited by 28
Franc: A Lightweight Framework for High-Quality Code Generation
Siddiq, M. L., Casey, B., & Santos, J. C. S. (2024) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam63643.2024.00020 · Cited by 48
The Robots are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.00658 · Cited by 9
Poitras, E., Crane, B., & Siegel, A. (2024) · Proceedings of the 2024 on ACM Virtual Global Computing Education Conference V. 1 · doi:10.1145/3649165.3690111 · Cited by 9
Hou, K., Li, J., Liu, Y., Sun, S., Zhang, H., & Jiang, H. (2024) · Electronics · doi:10.3390/electronics13234835 · Cited by 10
Teaching IT Software Fundamentals: Strategies and Techniques for Inclusion of Large Language Models
Gumina, S., Dalton, T., & Gerdes, J. H. (2023) · Conference on Information Technology Education · doi:10.1145/3585059.3611409 · Cited by 16
NeuroSync: Intent-Aware Code-Based Problem Solving via Direct LLM Understanding Modification
Zhang, W., Shen, L., Xu, S., Wang, J., Zhao, J., Qu, H., & Yuan, L.-P. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747668 · Cited by 20
Al Haque, E., Brown, C., LaToza, T. D., & Johnson, B. (2025) · Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering · doi:10.1145/3696630.3731665 · Cited by 9
Case study: using AI-assisted code generation in mobile teams
Vasiliniuc, M.-S., & Groza, A. P. (2023) · IEEE International Conference on Intelligent Computer Communication and Processing (ICCP) · doi:10.1109/iccp60212.2023.10398656 · Cited by 18
On Mitigating Code LLM Hallucinations with API Documentation
Jain, N., Kwiatkowski, R., Ray, B., Ramanathan, M. K., & Kumar, V. (2025) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00027 · Cited by 50
Towards Effective Validation and Integration of LLM-Generated Code
Tang, N. (2024) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc60511.2024.00051 · Cited by 13
PwnPilot: Reflections on Trusting Trust in the Age of Large Language Models and AI Code Assistants
Horne, D. (2023) · Congress in Computer Science, Computer Engineering, and Applied Computing (CSCE) · doi:10.1109/csce60160.2023.00396 · Cited by 15
Sun, D., & Xu, F. (2024) · Journal of Educational Computing Research · doi:10.1177/07356331241295739 · Cited by 11
Students' Perception of ChatGPT in Software Engineering: Lessons Learned from Five Courses
Baresi, L., Lucia, A. D., Marco, A. D., Penta, M. D., Ruscio, D. D., Mariani, L., Micucci, D., Palomba, F., Rossi, M. T., & Zampetti, F. (2025) · IEEE International Conference on Software Engineering Education and Training (CSEE&T) · doi:10.1109/cseet66350.2025.00023 · Cited by 17
Helping LLMs Improve Code Generation Using Feedback from Testing and Static Analysis
Dolcetti, G., Arceri, V., Iotti, E., Maffeis, S., Cortesi, A., & Zaffanella, E. (2024) · arXiv · doi:10.48550/arxiv.2412.14841 · Cited by 8
Kharrufa, A., Alghamdi, S. S., Aziz, A., & Bull, C. (2025) · ACM Transactions on Computing Education · doi:10.1145/3779296 · Cited by 36
Helping LLMs improve code generation using feedback from testing and static analysis
Dolcetti, G., Arceri, V., Iotti, E., Maffeis, S., Cortesi, A., & Zaffanella, E. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01009-5 · Cited by 8
“Despite that, programmers frequently use LLM-powered tools (e.g., GitHub Copilot [2]) for speeding up the development process or to get a starting point for finding solutions [3], be that for simple code snippets, specific functions or entire software applications.”
Human-Machine Teaming and Team Effectiveness in AI Tools for Software Engineering
Rauf, I., Sharp, H., López, T., & Wermelinger, M. (2025) · International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1109/chase66643.2025.00017 · Cited by 18
Cognitive Apprenticeship and Artificial Intelligence Coding Assistants
Poitras, E., Crane, B. G. C., Dempsey, D., Bragg, T. A., Siegel, A. A., & Lin, M. P.-C. (2024) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-1066-3.ch013 · Cited by 8
Šavelka, J., Agarwal, A., Bogart, C., Song, Y., & Sakr, M. F. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.09325 · Cited by 7
A Transformer-Based Approach for Smart Invocation of Automatic Code Completion
de Moor, A., van Deursen, A., & Izadi, M. (2024) · Proceedings of the 1st ACM International Conference on AI-Powered Software · doi:10.1145/3664646.3664760 · Cited by 7
Fostering Responsible AI Use Through Negative Expertise: A Contextualized Autocompletion Quiz
MacNeil, S., Prather, J. E., Nabid, R. A., Gutierrez, S., Carvalho, S., Shrestha, S., Denny, P. C., Reeves, B. N., Leinonen, J., & Rossetti, R. L. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729067 · Cited by 19
Assessing the Use of GitHub Copilot on Students of Engineering of Information Systems
Cirett-Galán, F., Peralta, R. T., Navarro-Hernández, R., Ochoa, J. L., Contreras-Rivera, S., Estrada-Ríos, L. A., & Machado-Encinas, G. (2024) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194024500335 · Cited by 8
Exploring Student-AI Interactions in Vibe Coding
Geng, F., Shah, A., Li, H., Mulla, N., Swanson, S., Raj, G. S., Zingaro, D., & Porter, L. (2026) · Australasian Computing Education Conference · doi:10.1145/3786228.3786236 · Cited by 51
Ironies of Generative AI: Understanding and mitigating productivity loss in human-AI interactions
Simkute, A., Tankelevitch, L., Kewenig, V., Scott, A. E., Sellen, A., & Rintel, S. (2024) · arXiv · doi:10.48550/arxiv.2402.11364 · Cited by 7
Li, C., Treude, C., & Turel, O. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112634 · Cited by 8
Yen, R., Xie, Y., Sultanum, N., & Zhao, J. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715336.3735752 · Cited by 10
The role of firm AI capabilities in generative AI-pair coding
Bughin, J. (2024) · Journal of Decision Systems · doi:10.1080/12460125.2024.2428187 · Cited by 6
Explaining GitHub Actions Failures with Large Language Models: Challenges, Insights, and Limitations
Valenzuela-Toledo, P., Wu, C., Hernández, S., Boll, A., Machacek, R., Panichella, S., & Kehrer, T. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00037 · Cited by 11
The underlying potential of NLP for microcontroller programming education
Rocha, A., Sousa, L., Alves, M., & Sousa, A. J. (2024) · Computer Applications in Engineering Education · doi:10.1002/cae.22778 · Cited by 8
Exploring GenAI in Software Development: Insights from a Case Study in a Large Brazilian Company
Pereira, G. V., Jackson, V., Prikladnicki, R., van der Hoek, A., Fortes, L., Araújo, C., Coelho, A., Chelli, L., & Ramos, D. (2025) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00035 · Cited by 24
Development of anatomically accurate digital organ models for surgical simulation and training
Kimura, T., Takiguchi, K., Tsukita, S., Muto, M., Chiba, H., Sato, N., Kofunato, Y., Ishigame, T., Kenjo, A., Tanaka, H., et al. (2025) · PLoS ONE · doi:10.1371/journal.pone.0320816 · Cited by 9
Benario, J. G., Marroquin, J., Chan, M., Holmes, E. D. V., & Mejia, D. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701859 · Cited by 19
Luo, P., Wen, X., Jiang, Y., Parn, E. A., & Brilakis, I. (2025) · Automation in Construction · doi:10.1016/j.autcon.2025.106655 · Cited by 12
Automated Code Editing With Search-Generate-Modify
Liu, C., Cetin, P., Patodia, Y., Ray, B., Chakraborty, S., & Ding, Y. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3376387 · Cited by 18
Ashtari, N., & Chilana, P. K. (2024) · Multimodal Technologies and Interaction · doi:10.3390/mti8040035 · Cited by 5
Trust Dynamics in AI-Assisted Development: Definitions, Factors, and Implications
Sabouri, S., Eibl, P., Zhou, X., Ziyadi, M., Medvidović, N., Lindemann, L., & Chattopadhyay, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00199 · Cited by 29
Metadata
- Authors
- Barke, S., James, M. B., & Polikarpova, N.
- Affiliations
- University of California San Diego (US)
- Venue
- Proceedings of the ACM on Programming Languages
- Published
- 2023-04-06
- Volume
- 7
- Issue
- OOPSLA1
- Pages
- 85-111
- ISSN
- 2475-1421 (ISSN-L)
- OpenAlex
- W4362659486
- Cited by
- 436
- References
- 46
- Citations by year
- 2026: 147 · 2025: 131 · 2024: 119 · 2023: 39
- FWCI
- 102.07
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Proceedings of the ACM on Programming Languages (publishedVersion, cc-by)
- Funders
- National Science Foundation (2107397, 1955457)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- GitHub Copilot, acceleration mode, exploration mode, usability, human-AI collaboration, programming tasks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-27 (prompt extract-v2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.