The DOI matches its Crossref record; 105 of the 106 references it cites were found in a registry or an index; 15 of the 30 sources its claims cite were checked for support.
“It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers
Open accessAbstract
Recent developments in deep learning have resulted in code-generation models that produce source code from natural language and code-based prompts with high accuracy. This is likely to have profound effects in the classroom, where novices learning to code can now use free tools to automatically suggest solutions to programming exercises and assignments. However, little is currently known about how novices interact with these tools in practice. We present the first study that observes students at the introductory level using one such code auto-generating tool, Github Copilot, on a typical introductory programming (CS1) assignment. Through observations and interviews we explore student perceptions of the benefits and pitfalls of this technology for learning, present new observed interaction patterns, and discuss cognitive and metacognitive difficulties faced by students. We consider design implications of these findings, specifically in terms of how tools like Copilot can better support and scaffold the novice programming experience.
Overview
CoChat Summary
TL;DR
This paper reports the first observational and interview study of novice CS1 students using GitHub Copilot on a typical introductory programming assignment. It finds that novices often saw Copilot as fast and helpful, but also struggled to understand its suggestions, worried about over-reliance, and exhibited two new interaction patterns: shepherding and drifting.
Key findings
- Most students perceived Copilot as helping them write code faster and avoid syntax errors, but many also worried about not understanding generated code and becoming dependent on it; “Outsourcing” concerns appeared in 46 interview excerpts from 17 unique participants, and “Speed” in 37 excerpts from 17 unique participants.
- The study identifies two novel interaction patterns: *shepherding*, where students spent most of their time coercing Copilot toward a solution, and *drifting*, where students moved from one incorrect suggestion to the next, backtracking and deleting code without direction.
- Students showed cognitive and metacognitive difficulties: “Confused” was observed 65 times across 13 unique participants, and “Metacognitive” appeared 24 times in observations and 17 times in interviews, each across 10 unique participants.
- Some students attributed intelligence or agency to Copilot, e.g., “It like read my mind!” and “I thought it was weird that it knows what I want,” while others raised fears about future job impacts, privacy, plagiarism, and over-reliance.
Methods
- Nineteen novice programmers (5 women, 14 men, ages 18–22) enrolled in a CS1 course at a Midwestern private research university in the USA were observed one at a time in April 2022 while solving a Minesweeper-style C++ assignment in Visual Studio Code with Copilot enabled.
- Each participant had 30 minutes, followed a think-aloud protocol, and then completed a short interview about whether Copilot helped them understand the problem, whether they would use it, its advantages, and their fears or worries.
- Data from observations and interviews were analyzed together using reflexive thematic analysis following Braun and Clarke’s six phases; themes included Interactions, Cognitive, Purpose, and Speculation, with saturation treated as the point where additional data produced no new emergent themes.
Limitations
- The paper states that screens and audio were not recorded due to IRB considerations, that study conditions were somewhat lab-like rather than normal in-class assignment conditions, that only Copilot was studied, and that data collection occurred in April 2022 shortly after Copilot’s release, so novelty effects may have shaped results.
- Evident from the text: the sample is small (19 students), from a single institution and single CS1 course, with only a 30-minute exposure to Copilot, so findings may not generalize to other populations, courses, or longer-term use.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
11 Valid · 3 Not valid · 11 Not checked · 109 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Code generators like Codex have been shown to perform better than the average student on introductory-level programming problems, thus they provide an effective tool for students who might be contemplating cheating [34].”
What the sources say
- Supportedcites [34] [34] James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Australasian Computing Education Conference (Virtual Event, Australia) (ACE '22) . Association for Computing Machinery, New York, NY, USA, 10-19. https://doi.org/10.1145/3511861.3511863 - source
Section 4.1.1 (page not available) reports that Codex ranked 17th alongside 71 students on two introductory programming tests, placing it in the top quartile—above the average student. Section 5.3 also describes Codex as a way for students to obtain solutions with low risk of similarity detection, supporting the sentence’s academic-integrity context. (LLM, conf=0.92)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Our participants acknowledged that such over-reliance could hinder their own learning, a concern that has also been expressed by computing educators [32].”
What the sources say
- Supportedcites [32] [32] Neil A Ernst and Gabriele Bavota. 2022. AI-Driven Development Is Here: Should You Worry? IEEE Software 39, 2 (2022), 106-110. - source
The cited paper’s “Learning to Program” discussion (Possible Challenges with AIDE, p. 3) expresses the concern attributed to it: learners may accept AI-generated recommendations without fully understanding them. “Our participants” refers to the citing study’s participants, not to this source. (LLM, conf=0.91)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“The primary benefit that novices saw in using Copilot was that it accelerated their progress on the programming task, mirroring results with experienced programmers as reported by Barke et al. [8] and Vaithilingam et al. [80].”
What the sources say
- Supportedcites [8] [8] Shraddha Barke, Michael B. James, and Nadia Polikarpova. 2022. Grounded Copilot: How Programmers Interact with Code-Generating Models. arXiv preprint arXiv:2206.15000 (2022). https://doi.org/10.48550/ARXIV.2206.15000 - source
Barke et al. are cited for the parallel finding among experienced programmers, not for the citing paper’s finding about novices. The Introduction says Copilot helps programmers reach a known next step more quickly, and §4.1 describes its use to type out intended code faster. The Related Work section notes that their participants were mostly experienced developers. (LLM, conf=0.92)
- Not checked — no full text of the sourcecites [80] [80] Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In CHI Conference on Human Factors in Computing Systems Extended Abstracts . Association for Computing Machinery, New York, NY, USA, 1-7. - source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“On introductory level problems like the one in our study, Copilot generates correct solutions most - but not all - of the time [34].”
What the sources say
- Partly supportedcites [34] [34] James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Australasian Computing Education Conference (Virtual Event, Australia) (ACE '22) . Association for Computing Machinery, New York, NY, USA, 10-19. https://doi.org/10.1145/3511861.3511863 - sourceExperimental — accuracy not yet measured
The Introduction says Codex powers GitHub Copilot, and §4.1 reports that Codex solved 19 of 23 introductory programming questions within ten attempts. But only about half were solved on the first attempt, so the paper does not establish that a single Copilot-generated solution is correct most of the time. Page numbers are unavailable in our copy. (LLM, conf=0.87)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Still others utilized the system like a colleague who can help them when stuck, a behavior previously documented between students co-regulating their learning in study groups [69], and which matches the idea of Copilot being 'your AI pair programmer.'”
What the sources say
- Supportedcites [69] [69] James Prather, Lauren Margulieux, Jacqueline Whalley, Paul Denny, Brent N. Reeves, Brett A. Becker, Paramvir Singh, Garrett Powell, and Nigel Bosch. 2022. Getting By With Help From My Friends: Group Study in Introductory Programming Understood as Socially Shared Regulation. In Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 (Lugano and Virtual Event, Switzerland) (ICER '22) . Association for Computing Machinery, New York, NY, USA, 164-176. https://doi.org/10.1145/3501385.3543970 - source
The cited part of the sentence is the peer-helping-when-stuck behaviour documented in study groups, not the Copilot framing. Prather et al. (2022) report 'help seeking -social' as the most common co-regulation theme in their study-group data (§4.2.2 Co-regulation Themes), illustrated by student statements such as 'When we try hard but still can not figure out how to solve the problem, we tend to ask each other' and 'I work with my friends when I become stuck ... and we help each other to understand the problem better and tackle it' (§5.3). Table 3 quantifies the theme (35 responses at Lab 7, 35 at Lab 12 for co-regulation), and §5.2 discusses the shift from resources to social help-seeking within groups. Thus the source does document students treating peers as a helping colleague when stuck while co-regulating in study groups. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
56 more claims · 1 with their cited sources checked
Create a free account to see all 134 claims and to verify the 2 references still open.
- Probably not a substantial claim. 46
- It may state two separate claims as one. 38
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Our findings affirm that AI-generated outputs can boost productivity; they often require rigorous review and contextual interpretation to ensure software safety and integrity [14].”
From Redefining the Programmer: Human-AI Collaboration, LLMs, and Security in Modern Software Engineering (2025) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Moreover, the dependency on GenAI to complete academic tasks could reduce students' critical thinking and independent problem-solving skills [16,36].”
From Generative AI and Academic Integrity in Higher Education: A Systematic Review and Research Agenda (2025) · cites this paper as [36]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent research has shown promising possibilities for providing students with LLM partners in pair programming, given the right context and with the right scaffolding and support [44,108,159].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [159]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Wermelinger [191] speculated that students may spend excessive time on prompt engineering in the hope of hitting on a successful solution rather than making progress towards the solution, which was indeed observed in the study by Prather et al. [159] in the 'shepherding' interaction pattern.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [159]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“From this study, two main modalities emerged (Prather et al., 2024): in 'acceleration' mode, the programmer is aware of their next steps and utilizes the tool to speed up the coding process. In contrast, in the 'exploration' mode, the programmer is uncertain about how to proceed and employs the tool to investigate his options.”
From Understanding user mental models in AI-driven code completion tools: Insights from an elicitation study (2025) · cites this paper as Prather et al., 2024
Experimental — accuracy not yet measured
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Reflecting on reflexive thematic analysis
Braun, V., & Clarke, V. (2019) · Qualitative Research in Sport, Exercise and Health · doi:10.1080/2159676x.2019.1628806 · Cited by 19,677 · cited 1×
“Coding is fluid and involves combining and splitting codes in an iterative process as the researchers become more familiar with the data, and throughout the analysis [19].”
BackgroundKaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Hattie, J., & Timperley, H. (2007) · Review of Educational Research · doi:10.3102/003465430298487 · Cited by 12,905
Extreme Programming Explained: Embrace Change
Beck, K. L., & Andres, C. (2000) · Cited by 5,902
Conceptual and design thinking for thematic analysis.
Braun, V., & Clarke, V. (2022) · Qualitative Psychology · doi:10.1037/qup0000196 · Cited by 4,502 · cited 1×
“To analyze the data we used relexive thematic analysis [18, 20] which aims to understand pattern and meaning present in qualitative data.”
MethodLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · cited 2×
“The internal architecture of the models is opaque for most users, which has led to multiple approaches for constructing functional prompts, often called “prompt engineering” [51].”
BackgroundLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 2×
“The internal architecture of the models is opaque for most users, which has led to multiple approaches for constructing functional prompts, often called “prompt engineering” [51].”
BackgroundCognitive load theory, learning difficulty, and instructional design
Sweller, J. (1994) · Learning and Instruction · doi:10.1016/0959-4752(94)90003-5 · Cited by 3,801 · cited 1×
“We hypothesize that lengthy auto-generated code suggestions may lead to an increased cognitive load [98] among novices and call on researchers to explore this in future work.”
Background
Research works printed but not linked: 3
- Emily M. Bender. 2019. A Typology of Ethical Risks in Language Technology with an eye towards where transparent documentation can help. Future of Artificial Intelligence: Language, Ethics, Technology Workshop.
- 10.1109/SP46215.2023.00001 · doi:10.1109/sp46215.2023.00001
- Ashley D. Rittmayer and Margaret E. Beier. 2008. Overview: Self-efficacy in STEM. SWE-AWE CASEE Overviews 1, 3 (2008), 12.
Cited by
Items 101–200 of 247
Redefining the Programmer: Human-AI Collaboration, LLMs, and Security in Modern Software Engineering
Cruz, E. D. L., Le, H., Meduri, K., Nadella, G. S., & Gonaygunta, H. (2025) · Computers, Materials & Continua · doi:10.32604/cmc.2025.068137 · Cited by 9
“Explored how students interact with GitHub Copilot in educational environments [14].”
Fostering Responsible AI Use Through Negative Expertise: A Contextualized Autocompletion Quiz
MacNeil, S., Prather, J. E., Nabid, R. A., Gutierrez, S., Carvalho, S., Shrestha, S., Denny, P. C., Reeves, B. N., Leinonen, J., & Rossetti, R. L. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729067 · Cited by 19
Self-citationKharrufa, A., Alghamdi, S. S., Aziz, A., & Bull, C. (2025) · ACM Transactions on Computing Education · doi:10.1145/3779296 · Cited by 36
“These findings are confirmed by the work of Prather et al. [43] who observed and interviewed 21 novice programming students over 21 lab sessions.”
Exploring organizational implementation of generative AI: navigating benefits and safety risks
Fox, G., & Solihin, Y. (2026) · Journal of Decision System · doi:10.1080/12460125.2026.2649940 · Cited by 9
Cognitive Apprenticeship and Artificial Intelligence Coding Assistants
Poitras, E., Crane, B. G. C., Dempsey, D., Bragg, T. A., Siegel, A. A., & Lin, M. P.-C. (2024) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-1066-3.ch013 · Cited by 8
Challenges and Paths Towards AI for Software EngineeringPreprint
Gu, A., Jain, N., Li, W., Shetty, M., Shao, Y., Li, Z., Yang, D., Ellis, K., Sen, K., & Solar-Lezama, A. (2025) · Qeios · doi:10.32388/vv1661 · Cited by 49
Exploring GenAI in Software Development: Insights from a Case Study in a Large Brazilian Company
Pereira, G. V., Jackson, V., Prikladnicki, R., van der Hoek, A., Fortes, L., Araújo, C., Coelho, A., Chelli, L., & Ramos, D. (2025) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00035 · Cited by 14
“Secondly, as mentioned by some of our participants, GenAI use may impact novice developers more than more senior developers such that novices become dependent on GenAI [39].”
Passi, S., Dhanorkar, S., & Vorvoreanu, M. (2025) · Springer eBooks · doi:10.1007/978-981-95-3658-0_98 · Cited by 15
"My productivity is boosted, but …" Demystifying Users’ Perception on AI Coding Assistants
Lyu, Y., Yang, Z., Shi, J., Chang, J., Liu, Y., & Lo, D. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00024 · Cited by 19
“Some argue that AI assistants benefit experienced developers who understand how to use them effectively [54], while others suggest they are especially helpful to novices [55].”
Kim, N., & Ji, Y. G. (2026) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2026.103771 · Cited by 12
An Analysis of the Costs and Benefits of Autocomplete in IDEs
Jiang, S., & Coblenz, M. J. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660765 · Cited by 8
Potencialidades do uso de Inteligência Artificial Generativa como apoio ao Ensino de Programação
da Silva, T. L., Vidotto, K. N. S., Tarouco, L. M. R., & da Silva, P. F. (2024) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/sbie.2024.242711 · Cited by 18
Brown, N. C. C., Weill-Tessier, P., Leinonen, J., Denny, P. C., & Kölling, M. (2025) · ACM Transactions on Computing Education · doi:10.1145/3737885 · Cited by 14
Self-citationCompiler-Integrated, Conversational AI for Debugging CS1 Programs
Renzella, J., Vassar, A., Solano, L. L., & Taylor, A. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701827 · Cited by 16
Evolution of Programmers' Trust in Generative AI Programming Assistants
Shah, A., Rexin, T., Tomson, E., Griswold, W. G., Porter, L., & Raj, A. G. S. (2025) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3769994.3770029 · Cited by 12
Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows
Chen, V., Talwalkar, A., Brennan, R., & Neubig, G. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790850 · Cited by 53
“Initial analysis of this data suggests that coding agents may still lag behind humans on PR acceptance rates by a large margin, especially on complex tasks.”
Qadir, J., & Mumtaz, M. (2026) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon67543.2026.11574165 · Cited by 8
“Students accept incorrect AI code recommendations without evaluation [20]; mathematics learners show AI dependency correlating with reduced problem-solving.”
Influence of Large Language Models on Programming Assignments – A user study
Piscitelli, A., Costagliola, G., Rosa, M. D., & Fuccella, V. (2024) · International Conference on Education Technology and Computer · doi:10.1145/3702163.3702168 · Cited by 7
Rethinking Computing Students’ Help Resource Utilization through Sequentiality
Ko, S.-H., & Stephens-Martinez, K. (2025) · ACM Transactions on Computing Education · doi:10.1145/3716860 · Cited by 8
“Earlier studies around the time GenAI/LLMs tools emerged either set up GenAI tools (such as OpenAI Codex [42, 62] and GitHub Copilot [27, 67]) in controlled environments for novice programmers to interact as a learning tool, or compare the quality of help given by different LLMs to novice…”
The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B., Kimmel, B., Wright, J., & Briggs, B. (2024) · arXiv · Cited by 196
Self-citationPark, S. M., Ho, M., Lin, M. P.-C., & Ryoo, J. (2025) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon62633.2025.11016517 · Cited by 13
“Copilot emulates this dynamic by acting as a virtual partner, offering real-time code suggestions and completions as the student writes code [11].”
GenAI Voice Mode in Programming Education
Jacobs, S., & Kiesler, N. (2025) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3769994.3770001 · Cited by 7
Premature Trust: How Student Overreliance on GenAI is Seeding Tomorrow’s Security Gaps
Jeswani, S., Mittal, A., & Sewlani, S. (2025) · Conference on Information Technology Education · doi:10.1145/3769694.3771124 · Cited by 3
Shaping Programming and Data Science Education: Insights from GenAI Technical Book Trends
Bakharia, A., & Abdi, S. (2024) · IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt61570.2024.00040 · Cited by 7
“Research indicates that students often struggle to comprehend code generated by AI, which they didn’t write themselves [6].”
BackgroundAI Pair Programming Acceptance: A Value-Based Approach with AHP Analysis
Çaldağ, M. T. (2024) · International Conference on Control, Decision and Information Technologies (CoDIT) · doi:10.1109/codit62066.2024.10708135 · Cited by 5
“Intellectual Property Violation refers to licensing problems that can arise while using AI pair programming [4], [13], [14], [15].”
BackgroundMethodInfluentialLiu, Z., Xing, W., Jiao, X., Li, C., & Zhu, W. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13385-z · Cited by 8
“We used these simple prompts instead of multiple complex prompts because students may not know how to manipulate the prompts when using ChatGPT for generating answers for their assignments (Hoq et al., 2024; Prather et al., 2023).”
Gupta, R. D., Hosain, M. T., Mridha, M. F., & Ahmed, S. U. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-93864-1_5 · Cited by 10
Braga, V. (2023) · Journal of Engineering Science · doi:10.52326/jes.utm.2023.30(3).07 · Cited by 4
“Even if there exists the need to use LLMs as explainers, copilots or assistants in such complex matters like financial knowledge [22], programming [23] or medicine [24, 25].”
BackgroundExperiences and Proposals of Use of Generative AI in Advanced Software Courses
Palacios-Alonso, D., Urquiza-Fuentes, J., Velázquez-Iturbide, J. Á., & Guillén-García, J. (2024) · 2024 IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon60312.2024.10578869 · Cited by 4
“Prather et al. [16] studied the interactions and the perception of novices in an undergraduate introductory programming course who used Copilot for their first time.”
MethodHyrynsalmi, S. M., Tuape, M., & Knutas, A. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3727235 · Cited by 7
Exploring the Human-LLM Synergy in Advancing Theory-driven Qualitative Analysis
Meng, H., Yang, Y., Fu, W., Lee, J., Li, Y., & Lee, Y. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3778354 · Cited by 38
Understanding and Enhancing CS Students’ Interaction Experience with AI Coding Assistant Tools
Long, X., Tan, X. L., Zhu, Y., Jiang, J., & Zhang, L. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3785479 · Cited by 10
Qi, J., Zhao, H., Xiang, R., Zhang, Y., Zhang, X., Wang, C., Jian, L., & Ji, F. (2026) · Journal of Research on Technology in Education · doi:10.1080/15391523.2025.2601067 · Cited by 9
From Prompts to Propositions: A Logic-Based Lens on Student-LLM Interactions
Alfageeh, A., Zarkouei, S. A. K., Nam, D., Prol, D., Amoozadeh, M., Chattopadhyay, S., Prather, J. E., Denny, P. C., Leinonen, J., Hilton, M. L., et al. (2025) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3769994.3770004 · Cited by 7
“There have been several studies to understand students’ prompting behavior in solving programming tasks, e.g., [2, 6, 16, 20].”
Self-citationTholander, J., & Jönsson, M. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791847 · Cited by 5
Leveraging Generative Artificial Intelligence to Broaden Participation in Computer Science
Jayachandran, D., Maldikar, P. S., Love, T. S., & Blum, J. J. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v3i1.31262 · Cited by 3
Quille, K., Alattyanyi, C., Becker, B. A., Faherty, R., Gordon, D., Harte, M., Hensman, S., Hofmann, M., García, J. J., Kuznetsov, A., et al. (2024) · Revista de Educación a Distancia (RED) · doi:10.6018/red.603001 · Cited by 3
Self-citationProust-Androwkha, S., Psyché, V., Kyriakoglou, R., Boubaker, A., Lallé, S., Chartofylaka, L., Machado, G. M., & Mawas, N. E. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98462-4_48 · Cited by 5
Aruleba, K., Oyelere, S. S., Obaido, G., & Sanusi, I. T. (2025) · Conference on United Kingdom & Ireland Computing Education Research (UKICER) · doi:10.1145/3754508.3754517 · Cited by 6
TableTalk: Scaffolding Spreadsheet Development with a Language Agent
Liang, J. T., Kumar, A., Bajpai, Y., Gulwani, S., Le, V., Parnin, C., Radhakrishna, A., Tiwari, A., Murphy-Hill, E. R., & Soares, G. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3765286 · Cited by 12
“Generative AI, like language agents, can be effective metacognitive scaffolds for programmers [39, 57, 78, 101].”
Uncovering Behavioral Patterns in Student–LLM Conversations during Code Refactoring Tasks
de Oliveira, E. C., Keuning, H., & Jeuring, J. T. (2025) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3769994.3770000 · Cited by 9
Examining the Usage of Generative AI Models in Student Learning Activities for Software Programming
Chen, R., Jiang, S., Shen, J., Moon, A., & Wei, L. (2025) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1109/aiware69974.2025.00016 · Cited by 3
Emotional Strain and Frustration in LLM Interactions in Software Engineering
Montes, C. M., & Khojah, R. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3756681.3756951 · Cited by 8
Scientific Software in the Age of Vibe Coding
O’Brien, G. E., & Eisty, N. U. (2026) · Computing in Science & Engineering · doi:10.1109/mcse.2026.3661610 · Cited by 2
Novices' Perceptions of Web-Search and AI for Programming
Skripchuk, J., Bacher, J., Shi, Y., et al. (2024) · Technical Symposium on Computer Science Education · doi:10.1145/3626253.3635545 · Cited by 3
GPT AI in Computer Science Education: A Systematic Mapping Study
Strik, B., Menolli, A., & Brancher, J. D. (2024) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/sbie.2024.242103 · Cited by 2
An Evaluation of the Impact of Code Generation Tools on Software Development
Osório, L. F. M., Neto, P. D. A. D. S., Avelino, G. A., & Lira, W. A. L. (2025) · Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI) · doi:10.5753/sbsi.2025.246605 · Cited by 3
Concepts for Teaching Software Development in the Age of AI-Tools
Böttcher, A., Thurner, V., & Zönnchen, B. (2025) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon62633.2025.11016505 · Cited by 5
“AI-generated answers which they are not able to interpret will increase their struggling, thus even increasing the gap between students coping well and and those that are struggling even more and faster [38] (the “Matthew Effect”).”
Refuting LLM-generated Code with Reactive Task Comprehension
Hebbar, S. V., Sasmita Harini, S., & Kumar, V. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729100 · Cited by 5
Kobiella, C., Breidenstein, D., & Schmidt, A. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790757 · Cited by 7
Desolda, G., Esposito, A., Greco, F., Tucci, C., Buono, P., & Piccinno, A. (2025) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2025.103648 · Cited by 7
“From this study, two main modalities emerged (Prather et al., 2024): in “accelera-tion” mode, the programmer is aware of their next steps and utilizes the tool to speed up the coding process.”
Pang, A., Padiyath, A., Vargas, D. V., & Ericson, B. J. (2024) · ACM Virtual Global Computing Education Conference (SIGCSE Virtual) · doi:10.1145/3649165.3690099 · Cited by 6
Generative AI Tool in Learning an Introductory Programming: A Research Perspective
Sivasakthi, M., & Meenakshi, A. V. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-86296-0_10 · Cited by 1
Mohamed, S., Ismail, N., Amaya, K., Parvin, A., Oliver, M. D., & Parra, E. (2025) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/3715669.3725868 · Cited by 3
“For novices, AI tools like GitHub Copilot have been found to provide scaffolding for introductory programming students, but studies have also suggested that novices struggle to critically evaluate AI generated content [Prather et al. 2023].”
Creative Learning Using Generative AI: Empowering Problem-Solving and Programming Skills
Elanwar, R. (2025) · Springer series on cultural computing · doi:10.1007/978-3-031-86551-0_7 · Cited by 1
How Do Learners With Varying Skills Perceive Misconception Indicators and Feedback?
Evans, A. C., & Lock, D. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729073 · Cited by 1
“Additionally, learners may not know how to prompt an LLM effectively [20] and can struggle to interpret its output [20, 22].”
Ru, Q., Wang, Y., Huang, C., & Luo, H. (2025) · International Conference on Computer Science and Technologies in Education (CSTE) · doi:10.1109/cste64638.2025.11092232 · Cited by 1
“Prather (2024) mentioned that students using Copilot not only received initial guidance in the right direction but also were able to think further with its feedback [33].”
The End of Programming as We Know It: Envisioning Radical Re-Conceptualizations of Co-Coding with AI
Jönsson, M., Tholander, J., Rost, M., Sarkar, A., Fernaeus, Y., Kharrufa, A., & Ludwig, T. G. (2025) · Decennial Aarhus Conference · doi:10.1145/3737609.3747093 · Cited by 1
Artificial Intelligence as a Mediator of Scaffolding: A Scoping Review
Tenisheva, K., & Litvinova, A. (2025) · Ойкумена. Регионоведческие исследования · doi:10.63973/1998-6785/2025-3/31-39 · Cited by 3
Gardella, N., Shelton, J., Graßl, I., & Riggs, S. L. (2025) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3769994.3770015 · Cited by 2
TraceMate: Collaborating with AI in Test-Driven Programming
Wu, J., Selig, T., & Purwanto, E. (2025) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc65237.2025.00035 · Cited by 5
One Line at a Time: Scaffolding Reflective Code Evaluation through Structured AI Assistance
Qiu, V., Parker, L., Riegel, K., Giacaman, N., Denny, P. C., MacNeil, S., & Prather, J. E. (2026) · Australasian Computing Education Conference · doi:10.1145/3786228.3786243 · Cited by 4
Self-citationIntegration of a neural network assistant on programming into a digital educational platform
Кушниренко, А. Г., Leonov, A. G., Мащенко, К. А., Martynov, N. S., Пчелин, К. К., & Shlyakhov, A. V. (2026) · Informatics and Education · doi:10.32517/0234-0453-2026-41-1-23-36 · Cited by 2
Barari, G., Ortega-Moody, J., Jenab, K., Ward, T., & Siebold, K. (2026) · Applied Sciences · doi:10.3390/app16104884 · Cited by 2
Esmat, M., Amasha, N., Atta, A., et al. (2024) · Decision Making: Applications in Management and Engineering · doi:10.31181/dmame7220241459 · Cited by 1
“Previous findings indicate that many students face difficulties in planning and implementing solutions, which often leads to frustration [ 21 ] and learning setbacks.”
Ask, Code, Review: A Three-Module Framework for AI in Programming Labs
Villuri, G., Shaik, H., Eljamal, B., & Doboli, A. (2026) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3803401.3811959 · Cited by 1
Guan, Q. (2026) · Emerging Media · doi:10.1177/27523543261470125 · Cited by 4
“They can operate as a copilot (Prather et al., 2023), supporting routine analytical steps, such as coding (Gilardi et al., 2023), analyzing theme (Pham et al., 2024), summarization (Paoletti et al., 2025), and other forms of automated content analysis.”
Incremental Repair Feedback on Automated Assessment of Programming Assignments
Paiva, J. C., Leal, J. P., & Figueira, Á. (2025) · Electronics · doi:10.3390/electronics14040819 · Cited by 6
Yardım, A., Serafini, R., Jost, N., Ortloff, A.-M., Speckels, J. G., & Naiakshina, A. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791693 · Cited by 4
How do Copilot Suggestions Impact Developers' Frustration and Productivity?
Guglielmi, E., Arnoudova, V., Bavota, G., Oliveto, R., & Scalabrino, S. (2025) · arXiv · doi:10.48550/arxiv.2504.06808 · Cited by 1
“Stress plays a crucial role in cognitive performance, task efficiency, and overall well-being of developers [11], [13].”
InfluentialWu, M., & Ouyang, F. (2026) · Educational Psychology Review · doi:10.1007/s10648-026-10211-x
“Novices need scaffolds that prompt them to read, run, and test AI outputs before adopting them, whereas advanced students can work with greater autonomy (Mellado & Cubillos, 2025 ; Prather et al., 2023 ).”
The Robots Are Here: Implications of Generative AI for Computing Education
Denny, P. C. (2023) · doi:10.52843/cassyni.3j2hbh
Self-citationCan You Spot the AI? Incorporating GenAI into Technical Writing Assignments
Rajabi, P., & Kerslake, C. P. (2024) · Western Canadian Conference on Computing Education · doi:10.1145/3660650.3660673 · Cited by 2
Configuring a GPTs Chatbot to Generate Educational Contents for Programming Novice Learners
Kim, S.-K. (2024) · The Journal of Korean Association of Computer Education · doi:10.32431/kace.2024.27.4.016
Osti, G., & Roke, E. R. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825858 · Cited by 4
Design Guidelines for Human-Generative AI Interaction
Lin, L., Wang, Y., Ping, Y., Gao, J., Wang, Z., & Wang, S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-93718-7_15
Oliveira, E. C., Keuning, H., & Jeuring, J. (2025) · Annual Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3724389.3730789 · Cited by 1
“Educators are increasingly integrating LLMs into programming education [2], which has raised interest in understanding how students interact with LLMs [3].”
Rossi, V., Scantamburlo, T., & Melonio, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98459-4_12 · Cited by 2
Analysis of Students’ GenAI Prompts During a Practical SQL Test
Ramakrishnan, C., Cassidy, S., & Bower, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98465-5_23
The Effect of Generative AI on Creating Software Complements for Digital Platforms
Rochholz, D., & Beimborn, D. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-032-04512-6_3
The Impact of Generative AI on Code Expertise Models: An Exploratory Study
Cury, O., & Avelino, G. A. (2025) · Simpósio Brasileiro de Engenharia de Software (SBES) · doi:10.5753/sbes.2025.11074
Integration of AI Code-Writing Assistants in IT Higher Education
Käver, A., Leoste, J., Marmor, K., & Rodríguez-Piñero, P. T. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-032-07989-3_4 · Cited by 3
Large Language Models for Personalized and Scalable Computer Education
Fan, J. (2025) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3769994.3770045
“However, current LLMs are not adapted to learners’ prior knowledge or abilities, often producing scaffolds that are either too advanced or not pedagogically aligned [8].”
Prompts Engineering Challenges in Software Code Generation
Gutierrez, Y., Camacho, E., Pardo, C., & Villarreal, V. (2025) · IEEE Congreso Internacional en Inteligencia Ambiental, Ingeniería de Software y Salud Electrónica y Móvil (AmITIC) · doi:10.1109/amitic68284.2025.11214606
“…which are based on the use of sequential questions, feedback and educational examples to foster a progressive interaction with the model [1], [6], [20], [30], [31], [32]; (3) Accuracy, robustness and variability in code generation , focused on generating correct, adaptive and reliable code,…”
Can Large Language Models Help Students Prove Software Correctness? An Experimental Study with Dafny
Carreira, C., Silva, Á. F., Abreu, A., & Mendes, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-10444-1_13 · Cited by 2
Quiz Generation and Chat Interaction with LLMs: A Design and Implementation of an AI-Based LMS
Costagliola, G., Rosa, M. D., Fuccella, V., Piscitelli, A., & Tabari, P. (2025) · International Conference on Software, Telecommunications and Computer Networks (SoftCOM) · doi:10.23919/softcom66362.2025.11197405 · Cited by 1
“LLMs are increasingly used in education, raising concerns about students’ over-reliance and reduced engagement with problem-solving [12]– [14].”
Osório, L. F. M., Neto, P. D. A. D. S., Avelino, G. A., & Lira, W. A. L. (2025) · Simpósio Brasileiro de Qualidade de Software (SBQS) · doi:10.5753/sbqs.2025.13845
Github Copilot's Impact on Developer Productivity : A Review of Early Evidence
Adepoju, S. A. (2023) · International Journal of Scientific Research in Science and Technology · doi:10.32628/ijsrst2221192 · Cited by 3
Navigating Skill Development and Ethical Challenges in Coding Education with Generative AI
Le, H. (2025) · IGI Global eBooks · doi:10.4018/979-8-3373-2660-3.ch009
Descriptive Assessment of Student Code by LLMs: An Empirical Study
Costagliola, G., Rosa, M. D., Fuccella, V., & Piscitelli, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-13184-3_20
The AI-Competence Ceiling: Redefining Human Expertise in an AI-Augmented World
Yadav, P. S. (2026) · Future of business and finance · doi:10.1007/978-3-032-11748-9_4
Critical thinking in human-AI collaborationsPreprint
Bilen, E., & Hervé, J. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6304755 · Cited by 1
The dual-encoder detector for AI-generated go code
Liao, G., Hou, Y., Chai, L., Yue, Q., Zhang, Q., Han, W., & Ou, W. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-00626-5
“Especially in educational settings, the indiscriminate use of AI-generated code may affect students’ programming skills and reduce their academic integrity awareness (Prather et al. 2023).”
Design Implications for Student and Educator Needs in AI-Supported Programming Learning Tools
Ma, B., Xie, Y., Li, H., Li, G., Chen, L. Y. E., Shimada, A., & Konomi, S. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2730083 · Cited by 3
Liu, C., Lin, Y., Chang, J., Liu, J., Qi, B., Jiang, B., Huang, Z., & Dong, J. S. (2026) · Proceedings of the ACM on Programming Languages · doi:10.1145/3798249 · Cited by 3
A New Methodology for Computer Science Education Through the Capabilities of Artificial Intelligence
Angelova, N. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-22232-9_9
Crespi, F. G. (2026) · Computer Applications in Engineering Education · doi:10.1002/cae.70196 · Cited by 1
A Systematic Mapping Study on Developer Profile Modeling in Software Engineering
Sarı, A., Alptekin, G. I., & Tosun, A. (2026) · IEEE Access · doi:10.1109/access.2026.3696198
Performance of LLM-Generated Code
Lacher, L. L., Lewis, M. C., Ruetschle, E. M., & Sickafoose, A. A. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-22205-3_13 · Cited by 1
Liao, G., Chai, L., Hou, Y., & Yue, Q. (2026) · Procedia Computer Science · doi:10.1016/j.procs.2026.05.047
Metadata
- Authors
- Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A.
- Affiliations
- Abilene Christian University (US)
- University of Auckland (NZ)
- University College Dublin (IE)
- ORCID
- Venue
- ACM Transactions on Computer-Human Interaction
- Published
- 2023-11-29
- Volume
- 31
- Issue
- 1
- Pages
- 1-31
- ISSN
- 1073-0516 (ISSN-L)
- 1557-7325
- OpenAlex
- W4386099272
- Cited by
- 256
- References
- 91
- Citations by year
- 2026: 89 · 2025: 85 · 2024: 66 · 2023: 16
- FWCI
- 57.98
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Transactions on Computer-Human Interaction (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- GitHub Copilot, code generation, novice programmers, introductory programming, CS1, programming education, student perceptions, metacognition, cognitive load, scaffolding, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.