“It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers
Open accessThe DOI matches its Crossref record; 105 of the 106 references it cites were found in a registry or an index; 15 of the 30 sources its claims cite were checked for support.
Abstract
Recent developments in deep learning have resulted in code-generation models that produce source code from natural language and code-based prompts with high accuracy. This is likely to have profound effects in the classroom, where novices learning to code can now use free tools to automatically suggest solutions to programming exercises and assignments. However, little is currently known about how novices interact with these tools in practice. We present the first study that observes students at the introductory level using one such code auto-generating tool, Github Copilot, on a typical introductory programming (CS1) assignment. Through observations and interviews we explore student perceptions of the benefits and pitfalls of this technology for learning, present new observed interaction patterns, and discuss cognitive and metacognitive difficulties faced by students. We consider design implications of these findings, specifically in terms of how tools like Copilot can better support and scaffold the novice programming experience.
Overview
CoChat Summary
TL;DR
This paper reports the first observational and interview study of novice CS1 students using GitHub Copilot on a typical introductory programming assignment. It finds that novices often saw Copilot as fast and helpful, but also struggled to understand its suggestions, worried about over-reliance, and exhibited two new interaction patterns: shepherding and drifting.
Key findings
- Most students perceived Copilot as helping them write code faster and avoid syntax errors, but many also worried about not understanding generated code and becoming dependent on it; “Outsourcing” concerns appeared in 46 interview excerpts from 17 unique participants, and “Speed” in 37 excerpts from 17 unique participants.
- The study identifies two novel interaction patterns: *shepherding*, where students spent most of their time coercing Copilot toward a solution, and *drifting*, where students moved from one incorrect suggestion to the next, backtracking and deleting code without direction.
- Students showed cognitive and metacognitive difficulties: “Confused” was observed 65 times across 13 unique participants, and “Metacognitive” appeared 24 times in observations and 17 times in interviews, each across 10 unique participants.
- Some students attributed intelligence or agency to Copilot, e.g., “It like read my mind!” and “I thought it was weird that it knows what I want,” while others raised fears about future job impacts, privacy, plagiarism, and over-reliance.
Methods
- Nineteen novice programmers (5 women, 14 men, ages 18–22) enrolled in a CS1 course at a Midwestern private research university in the USA were observed one at a time in April 2022 while solving a Minesweeper-style C++ assignment in Visual Studio Code with Copilot enabled.
- Each participant had 30 minutes, followed a think-aloud protocol, and then completed a short interview about whether Copilot helped them understand the problem, whether they would use it, its advantages, and their fears or worries.
- Data from observations and interviews were analyzed together using reflexive thematic analysis following Braun and Clarke’s six phases; themes included Interactions, Cognitive, Purpose, and Speculation, with saturation treated as the point where additional data produced no new emergent themes.
Limitations
- The paper states that screens and audio were not recorded due to IRB considerations, that study conditions were somewhat lab-like rather than normal in-class assignment conditions, that only Copilot was studied, and that data collection occurred in April 2022 shortly after Copilot’s release, so novelty effects may have shaped results.
- Evident from the text: the sample is small (19 students), from a single institution and single CS1 course, with only a 30-minute exposure to Copilot, so findings may not generalize to other populations, courses, or longer-term use.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
11 Valid · 3 Not valid · 11 Not checked · 109 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Code generators like Codex have been shown to perform better than the average student on introductory-level programming problems, thus they provide an effective tool for students who might be contemplating cheating [34].”
What the sources say
- Supportedcites [34] [34] James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Australasian Computing Education Conference (Virtual Event, Australia) (ACE '22) . Association for Computing Machinery, New York, NY, USA, 10-19. https://doi.org/10.1145/3511861.3511863 - source
Section 4.1.1 (page not available) reports that Codex ranked 17th alongside 71 students on two introductory programming tests, placing it in the top quartile—above the average student. Section 5.3 also describes Codex as a way for students to obtain solutions with low risk of similarity detection, supporting the sentence’s academic-integrity context. (LLM, conf=0.92)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Our participants acknowledged that such over-reliance could hinder their own learning, a concern that has also been expressed by computing educators [32].”
What the sources say
- Supportedcites [32] [32] Neil A Ernst and Gabriele Bavota. 2022. AI-Driven Development Is Here: Should You Worry? IEEE Software 39, 2 (2022), 106-110. - source
The cited paper’s “Learning to Program” discussion (Possible Challenges with AIDE, p. 3) expresses the concern attributed to it: learners may accept AI-generated recommendations without fully understanding them. “Our participants” refers to the citing study’s participants, not to this source. (LLM, conf=0.91)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“The primary benefit that novices saw in using Copilot was that it accelerated their progress on the programming task, mirroring results with experienced programmers as reported by Barke et al. [8] and Vaithilingam et al. [80].”
What the sources say
- Supportedcites [8] [8] Shraddha Barke, Michael B. James, and Nadia Polikarpova. 2022. Grounded Copilot: How Programmers Interact with Code-Generating Models. arXiv preprint arXiv:2206.15000 (2022). https://doi.org/10.48550/ARXIV.2206.15000 - source
Barke et al. are cited for the parallel finding among experienced programmers, not for the citing paper’s finding about novices. The Introduction says Copilot helps programmers reach a known next step more quickly, and §4.1 describes its use to type out intended code faster. The Related Work section notes that their participants were mostly experienced developers. (LLM, conf=0.92)
- Not checked — no full text of the sourcecites [80] [80] Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In CHI Conference on Human Factors in Computing Systems Extended Abstracts . Association for Computing Machinery, New York, NY, USA, 1-7. - source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“On introductory level problems like the one in our study, Copilot generates correct solutions most - but not all - of the time [34].”
What the sources say
- Partly supportedcites [34] [34] James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Australasian Computing Education Conference (Virtual Event, Australia) (ACE '22) . Association for Computing Machinery, New York, NY, USA, 10-19. https://doi.org/10.1145/3511861.3511863 - sourceExperimental — accuracy not yet measured
The Introduction says Codex powers GitHub Copilot, and §4.1 reports that Codex solved 19 of 23 introductory programming questions within ten attempts. But only about half were solved on the first attempt, so the paper does not establish that a single Copilot-generated solution is correct most of the time. Page numbers are unavailable in our copy. (LLM, conf=0.87)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Still others utilized the system like a colleague who can help them when stuck, a behavior previously documented between students co-regulating their learning in study groups [69], and which matches the idea of Copilot being 'your AI pair programmer.'”
What the sources say
- Supportedcites [69] [69] James Prather, Lauren Margulieux, Jacqueline Whalley, Paul Denny, Brent N. Reeves, Brett A. Becker, Paramvir Singh, Garrett Powell, and Nigel Bosch. 2022. Getting By With Help From My Friends: Group Study in Introductory Programming Understood as Socially Shared Regulation. In Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 (Lugano and Virtual Event, Switzerland) (ICER '22) . Association for Computing Machinery, New York, NY, USA, 164-176. https://doi.org/10.1145/3501385.3543970 - source
The cited part of the sentence is the peer-helping-when-stuck behaviour documented in study groups, not the Copilot framing. Prather et al. (2022) report 'help seeking -social' as the most common co-regulation theme in their study-group data (§4.2.2 Co-regulation Themes), illustrated by student statements such as 'When we try hard but still can not figure out how to solve the problem, we tend to ask each other' and 'I work with my friends when I become stuck ... and we help each other to understand the problem better and tackle it' (§5.3). Table 3 quantifies the theme (35 responses at Lab 7, 35 at Lab 12 for co-regulation), and §5.2 discusses the shift from resources to social help-seeking within groups. Thus the source does document students treating peers as a helping colleague when stuck while co-regulating in study groups. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
56 more claims · 1 with their cited sources checked
Create a free account to see all 134 claims and to verify the 2 references still open.
- Probably not a substantial claim. 46
- It may state two separate claims as one. 38
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Our findings affirm that AI-generated outputs can boost productivity; they often require rigorous review and contextual interpretation to ensure software safety and integrity [14].”
From Redefining the Programmer: Human-AI Collaboration, LLMs, and Security in Modern Software Engineering (2025) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Moreover, the dependency on GenAI to complete academic tasks could reduce students' critical thinking and independent problem-solving skills [16,36].”
From Generative AI and Academic Integrity in Higher Education: A Systematic Review and Research Agenda (2025) · cites this paper as [36]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent research has shown promising possibilities for providing students with LLM partners in pair programming, given the right context and with the right scaffolding and support [44,108,159].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [159]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Wermelinger [191] speculated that students may spend excessive time on prompt engineering in the hope of hitting on a successful solution rather than making progress towards the solution, which was indeed observed in the study by Prather et al. [159] in the 'shepherding' interaction pattern.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [159]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“From this study, two main modalities emerged (Prather et al., 2024): in 'acceleration' mode, the programmer is aware of their next steps and utilizes the tool to speed up the coding process. In contrast, in the 'exploration' mode, the programmer is uncertain about how to proceed and employs the tool to investigate his options.”
From Understanding user mental models in AI-driven code completion tools: Insights from an elicitation study (2025) · cites this paper as Prather et al., 2024
Experimental — accuracy not yet measured
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Reflecting on reflexive thematic analysis
Braun, V., & Clarke, V. (2019) · Qualitative Research in Sport, Exercise and Health · doi:10.1080/2159676x.2019.1628806 · Cited by 19,677 · cited 1×
“Coding is fluid and involves combining and splitting codes in an iterative process as the researchers become more familiar with the data, and throughout the analysis [19].”
BackgroundKaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Hattie, J., & Timperley, H. (2007) · Review of Educational Research · doi:10.3102/003465430298487 · Cited by 12,905
Extreme Programming Explained: Embrace Change
Beck, K. L., & Andres, C. (2000) · Cited by 5,902
Conceptual and design thinking for thematic analysis.
Braun, V., & Clarke, V. (2022) · Qualitative Psychology · doi:10.1037/qup0000196 · Cited by 4,502 · cited 1×
“To analyze the data we used relexive thematic analysis [18, 20] which aims to understand pattern and meaning present in qualitative data.”
MethodLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · cited 2×
“The internal architecture of the models is opaque for most users, which has led to multiple approaches for constructing functional prompts, often called “prompt engineering” [51].”
BackgroundLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 2×
“The internal architecture of the models is opaque for most users, which has led to multiple approaches for constructing functional prompts, often called “prompt engineering” [51].”
BackgroundCognitive load theory, learning difficulty, and instructional design
Sweller, J. (1994) · Learning and Instruction · doi:10.1016/0959-4752(94)90003-5 · Cited by 3,801 · cited 1×
“We hypothesize that lengthy auto-generated code suggestions may lead to an increased cognitive load [98] among novices and call on researchers to explore this in future work.”
Background
Research works printed but not linked: 3
- Emily M. Bender. 2019. A Typology of Ethical Risks in Language Technology with an eye towards where transparent documentation can help. Future of Artificial Intelligence: Language, Ethics, Technology Workshop.
- 10.1109/SP46215.2023.00001 · doi:10.1109/sp46215.2023.00001
- Ashley D. Rittmayer and Margaret E. Beier. 2008. Overview: Self-efficacy in STEM. SWE-AWE CASEE Overviews 1, 3 (2008), 12.
Cited by
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396
Self-citationComputing Education in the Era of Generative AI
Denny, P., Prather, J., Becker, B. A., Finnie-Ansley, J., Hellas, A., Leinonen, J., Luxton-Reilly, A., Reeves, B. N., Santos, E. A., & Sarsa, S. (2024) · Communications of the ACM · doi:10.1145/3624720 · Cited by 316
“Novices may be intimidated by such code generation [19] or may spend too much time reading code that does not further their goals [31].”
BackgroundSelf-citationThe Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493
“We posit that these demands are exacerbated by GenAI’s non-determinism and model flexibility (not to be confused with metacognitive flexibility) in terms of (a) the wide range of explicit and implicit parameters that users can adjust, and (b) systems’ ability to work with prompts at a wide range of…”
BackgroundGenerative AI and Academic Integrity in Higher Education: A Systematic Review and Research Agenda
Bittle, K., & El-Gayar, O. (2025) · Information · doi:10.3390/info16040296 · Cited by 305
“Moreover, the dependency on GenAI to complete academic tasks could reduce students’ critical thinking and independent problem-solving skills [16,36].”
Kazemitabaar, M., Ye, R., Wang, X., Henley, A. Z., Denny, P., Craig, M., & Grossman, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642773 · Cited by 290
Self-citationLau, S., & Guo, P. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600138 · Cited by 241
“observed how students used Copilot on CS1 assignments in a lab study and then interviewed them about their first impressions [92].”
BackgroundComputer Science Curricula 2023
Kumar, A. N., Raj, R. K., Aly, S. G., Anderson, M. D., Becker, B. A., Blumenthal, R. L., Eaton, E., Epstein, S. L., Goldweber, M., Jalote, P., et al. (2024) · ACM eBooks · doi:10.1145/3664191 · Cited by 223
Self-citationNavigating the Complexity of Generative AI Adoption in Software Engineering
Russo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3652154 · Cited by 217
“For novice programmers, cognitive and metacognitive issues arise while using these tools for assignments, indicating a need for better supportive design [64].”
BackgroundExploring the Responses of Large Language Models to Beginner Programmers' Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · arXiv · doi:10.48550/arxiv.2306.05715 · Cited by 211
Self-citationCodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes
Liffiton, M., Sheese, B. E., Savelka, J., & Denny, P. (2023) · Proceedings of the 23rd Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631830 · Cited by 192
“This heavy use of the code generator raises concerns around student over-reliance which has been identified as a key challenge for educators [1, 3, 7, 28].”
BackgroundSelf-citationThe Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B. (2024) · Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3632620.3671116 · Cited by 196
Self-citationPrompt Problems: A New Programming Exercise for the Generative AI Era
Denny, P., Leinonen, J., Prather, J., Luxton-Reilly, A., Amarouche, T., Becker, B. A., & Reeves, B. N. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630909 · Cited by 187
Self-citationSavelka, J., Agarwal, A., An, M., Bogart, C., & Sakr, M. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600142 · Cited by 141
“C Y ] 1 5 Ju n 20 is how to assess learners’ skills and knowledge in the presence of ubiquitous tools (e.g., ChatGPT,1 GitHub Copilot2) that could be easily utilized to pass the assessments (at least partially).”
BackgroundInfluentialKazemitabaar, M., Hou, X., Henley, A., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Proceedings of the 23rd Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631806 · Cited by 130
The Effects of Generative AI on Computing Students’ Help-Seeking Preferences
Hou, I., Mettille, S., Man, O., Li, Z., Zastudil, C., & MacNeil, S. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636248 · Cited by 97
“Especially as many other use cases are rapidly emerging in computing education contexts [3, 9, 14, 29, 35, 41, 52, 53, 58, 59, 61].”
BackgroundDesirable Characteristics for AI Teaching Assistants in Programming Education
Denny, P. C., MacNeil, S., Šavelka, J., Porter, L., & Luxton-Reilly, A. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653574 · Cited by 153
Self-citationPrather, J., Leinonen, J., Kiesler, N., Gorson Benario, J., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., et al. (2025) · 2024 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3689187.3709614 · Cited by 96
Self-citationIronies of Generative AI: Understanding and Mitigating Productivity Loss in Human-AI Interaction
Simkute, A., Tankelevitch, L., Kewenig, V., Scott, A. E., Sellen, A., & Rintel, S. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2024.2405782 · Cited by 89
“…to effectively prompt systems, debug generated code, lose their state of flow when interrupted by long code suggestions, and get stuck in ineffective practices, such as reviewing, editing and then ultimately deleting suggestions (Barke et al., 2023; Prather et al., 2023; Sarkar et al., 2022).”
InfluentialExploring the Potential of Large Language Models to Generate Formative Programming Feedback
Kiesler, N., Lohr, D., & Keuning, H. (2023) · 2023 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10343457 · Cited by 79
“Prather et al. [18] investigated students’ use of GitHub Copilot in an introductory programming assignment through observation and interviews.”
MethodPatterns of Student Help-Seeking When Using a Large Language Model-Powered Programming Assistant
Sheese, B., Liffiton, M., Savelka, J., & Denny, P. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636249 · Cited by 74
Self-citationIncorporating Generative AI into Software Development Education
Petrovska, O., Clift, L., Moller, F., & Pearsall, R. (2024) · Proceedings of the 8th Conference on Computing Education Practice · doi:10.1145/3633053.3633057 · Cited by 75
How Beginning Programmers and Code LLMs (Mis)read Each Other
Nguyen, S., Babe, H. M., Zi, Y., Guha, A., Anderson, C. J., & Feldman, M. Q. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642706 · Cited by 66
Generative AI in Education: A Study of Educators' Awareness, Sentiments, and Influencing Factors
Ghimire, A., Pather, J., & Edwards, J. (2024) · 2024 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10892891 · Cited by 88
An Empirical Study on Usage and Perceptions of LLMs in a Software Engineering Project
Rasnayaka, S., Wang, G., Shariffdeen, R., & Iyer, G. N. (2024) · Proceedings of the 1st International Workshop on Large Language Models for Code · doi:10.1145/3643795.3648379 · Cited by 70
Cambaz, D., & Zhang, X. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630958 · Cited by 62
“Likewise, [16] hypothesized that “over-reliance on tools like Copilot could possibly worsen a novice’s metacognitive programming skills and behaviors.”
BackgroundMethodInfluentialNext-Step Hint Generation for Introductory Programming Using Large Language Models
Roest, L., Keuning, H., & Jeuring, J. T. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636259 · Cited by 123
Student-AI Interaction: A Case Study of CS1 students
Amoozadeh, M., Nam, D., Prol, D., Alfageeh, A., Prather, J. E., Hilton, M. L., Ragavan, S. S., & Alipour, M. A. (2024) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3699538.3699567 · Cited by 90
Self-citationCould ChatGPT get an engineering degree? Evaluating higher education vulnerability to AI assistants
Borges, B., Foroutan, N., Bayazit, D., Sotnikova, A., Montariol, S., Nazaretsky, T., Banaei, M., Sakhaeirad, A., Servant, P., Neshaei, S. P., et al. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2414955121 · Cited by 75
“An important dimension of these discussions emphasizes the need to revisit evaluation procedures to ensure that students acquire necessary skills and critical thinking abilities in the face of AI integration [7, 20, 38, 43].”
What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use
Ma, Q., Peng, W., Yang, C., Shen, H., Koedinger, K. R., & Wu, T. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3731756 · Cited by 111
“…prompting [70] (e.g., Prompt Maker in Figure 2) to more advanced methods like TextGrad[79],whichsearchesforoptimalwording,PromptCharm[69], whichenablesuserstoadjustdiffusionmodels’attentiontokey-wordsinprompts,andDSPy[25],whichextendsthesearchspaceto few-shotpromptsandvariouspromptingtechniques.”
Kim, Y. W., Cha, M. C., Yoon, S. H., & Lee, S. C. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2375701 · Cited by 52
“For example, non-expert end-user programmers with negligible or no programming knowledge may find it challenging to develop programs using AI-CAs (Prather et al., 2023; Sarkar et al., 2022).”
Software engineering education in the era of conversational AI: current trends and future directions
Sengul, C., Neykova, R., & Destefanis, G. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1436350 · Cited by 83
Liu, J., & Li, S. (2024) · Journal of Educational Computing Research · doi:10.1177/07356331241240460 · Cited by 62
“Prather et al. (2023) conducted structured interviews and observed introductory-level students using Github Copilot to complete introductory programming (CS1) assignments.”
Chugh, R., Turnbull, D., Morshed, A., Sabrina, F., Azad, S., Rashid, M. M., Kaisar, S., & Subramani, S. (2025) · Computer Applications in Engineering Education · doi:10.1002/cae.70002 · Cited by 57
“Part of this reluctance to embrace AI could also be attributed to an observation that AI tools are not capable of perfection when it comes to code interpretation [43].”
InfluentialHow Do Programming Students Use Generative AI?
Rahe, C., & Maalej, W. (2025) · Proceedings of the ACM on Software Engineering · doi:10.1145/3715762 · Cited by 45
“Across multiple surveys and position papers, educators have expressed concerns, particularly about over-reliance on generated content and about the ease of cheating [Becker et al. 2023; Cotton et al. 2024; Prather et al. 2023a; Zastudil et al. 2023].”
InfluentialPadiyath, A., Hou, X., Pang, A., Viramontes Vargas, D., Gu, X., Nelson-Fromm, T., Wu, Z., Guzdial, M., & Ericson, B. (2024) · Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3632620.3671098 · Cited by 35
Student Perspectives on Using a Large Language Model (LLM) for an Assignment on Professional Ethics
Grande, V., Kiesler, N., & Rodríguez, M. A. F. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653624 · Cited by 61
Combining LLM-Generated and Test-Based Feedback in a MOOC for Programming
Gabbay, H., & Cohen, A. (2024) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3657604.3662040 · Cited by 68
Scholl, A., & Kiesler, N. (2024) · 2024 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10893442 · Cited by 35
Pirzado, F. A., Ahmed, A., Mendoza-Urdiales, R. A., & Terashima‐Marín, H. (2024) · IEEE Access · doi:10.1109/access.2024.3443621 · Cited by 65
“[1], [3], [14], [22], [37], [40], [43], [44], [48], [50], [51], [52], [53], [66], [67], [72], [78], [90].”
BackgroundResultInfluentialHumble, N. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00494-x · Cited by 57
“Papers highlight the risk of students’ personal data not being safeguarded when interacting with GAI tools (Albdrani & Al-Shargabi, 2023; Gumina et al., 2023; Husain, 2024; Prather et al., 2023; Rajala, 2023; Sánchez-Gordón et al., 2023).”
InfluentialLohr, D., Keuning, H., & Kiesler, N. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.13107 · Cited by 63
Keuning, H., Alpizar-Chacon, I., Lykourentzou, I., Beehler, L., Köppe, C., de Jong, I., & Sosnovsky, S. (2024) · Proceedings of the 24th Koli Calling International Conference on Computing Education Research · doi:10.1145/3699538.3699546 · Cited by 32
Beyond Code Generation: LLM-supported Exploration of the Program Design Space
Zamfirescu-Pereira, J. D., Jun, E., Terry, M. A., Yang, Q., & Hartmann, B. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714154 · Cited by 79
Shihab, M. I. H., Hundhausen, C., Tariq, A., Haque, S., Qiao, Y., & Mulanda, B. W. (2025) · Proceedings of the 2025 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3702652.3744219 · Cited by 28
Enhancing Programming Error Messages in Real Time with Generative AI
Kimmel, B., Geisert, A. L., Yaro, L., Gipson, B., Hotchkiss, R., Osae-Asante, S. K., Vaught, H., Wininger, G., & Yamaguchi, C. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3647967 · Cited by 44
Exploring and Characterizing Large Language Models for Embedded System Development and Debugging
Englhardt, Z., Li, R. C., Nissanka, D., Zhang, Z., Narayanswamy, G., Breda, J., Liu, X., Patel, S., & Iyer, V. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650764 · Cited by 63
Choi, S., & Kim, H. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-13107-x · Cited by 50
“Such proficiencies make LLMs invaluable tools for providing personalized and contextualized support in programming education (Prather et al., 2023).”
AI chatbots in programming education: guiding success or encouraging plagiarism
Akçapınar, G., & Sidan, E. (2024) · Discover Artificial Intelligence · doi:10.1007/s44163-024-00203-7 · Cited by 83
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Caspersen, M. E., Craig, M., Keuning, H., Kiesler, N., Kohn, T., et al. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 2 · doi:10.1145/3587103.3594206 · Cited by 26
“Students using these tools can also quickly become lost while reading code they didn’t write [5] or lazily drift from code suggestion to code suggestion without understanding what they are doing [9].”
BackgroundSelf-citationNeed Help? Designing Proactive AI Assistants for Programming
Chen, V., Zhu, A., Zhao, S., Mozannar, H., Sontag, D. A., & Talwalkar, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714002 · Cited by 104
“Programming assistants surrounding code completion have been the focus of prior work, providing insight into how people use LLM-generated code completions [6, 12, 35, 41, 43, 49] and improving completions by providing explanations [53] and determining when to best show [36, 50].”
Understanding human-AI trust in education
Pitts, G., & Motamedi, S. (2025) · Telematics and Informatics Reports · doi:10.1016/j.teler.2025.100270 · Cited by 64
“In computing education, they have been explored as pair programmers capable of generating explanations and examples [17, 18] and as tools for feedback on code quality, such as identifying misleading variable names and suggesting improvements [19], potentially substituting for peers or instructors…”
van den Broek, S., Sankaran, S., de Wit, J., & de Rooij, A. (2024) · Participatory Design Conference · doi:10.1145/3661455.3669868 · Cited by 39
“While this has been explored in several ways such as using AI as a copilot programmer [33, 54] collaboratively writing or designing with AI [47, 51], its potential has not been explored in multi-stakeholder co-design contexts.”
How Instructors Incorporate Generative AI into Teaching Computing
Prather, J. E., Leinonen, J., Kiesler, N., Benario, J. G., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., et al. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649405.3659534 · Cited by 32
“However, many educators have raised concerns about adapting their curricula, student learning, inherent biases, over-reliance, and educational misconduct [2, 9, 15, 16].”
BackgroundSelf-citationNon-Expert Programmers in the Generative AI Future
Feldman, M. Q., & Anderson, C. J. (2024) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3663384.3663393 · Cited by 50
“We draw on existing studies of Code LLM use by beginning programmers in secondary school [35, 36], in CS1 courses [18, 65, 66], and after CS1 [32, 60], which have revealed barriers related to technical communication, understanding generated code, and the unpredictable nature of Code LLMs.”
MethodGenerative AI in Computer Science Education: Challenges and Opportunities
Franklin, D., Denny, P., Gonzalez-Maldonado, D. A., & Tran, M. (2025) · doi:10.1017/9781009581738 · Cited by 20
Self-citationHow much does AI impact development speed? An enterprise-based randomized controlled trial
Paradis, E., Grey, K., Madison, Q., Nam, D., Macvean, A., Meimand, V., Zhang, N., Ferrari-Church, B., & Chandra, S. (2024) · arXiv · doi:10.48550/arxiv.2410.12944 · Cited by 20
Students' Use of GitHub Copilot for Working with Large Code Bases
Shah, A., Chernova, A., Tomson, E., Porter, L., Griswold, W. G., & Soosai Raj, A. G. (2025) · Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3641554.3701800 · Cited by 19
AI-Assisted Programming Tasks Using Code Embeddings and Transformers
Kotsiantis, S. B., Verykios, V. S., & Tzagarakis, M. M. (2024) · Electronics · doi:10.3390/electronics13040767 · Cited by 38
DBox: Scaffolding Algorithmic Programming Learning through Learner-LLM Co-Decomposition
Ma, S., Wang, J., Zhang, Y., Ma, X., & Wang, A. Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713748 · Cited by 63
“Scaffolding, rooted in constructivist theory, helps bridge the gap between learners’ current abilities and desired skills by integrating new information into existing cognitive frameworks, rather than passively receiving it [10, 38, 54, 73, 81].”
Students' Perspectives on AI Code Completion: Benefits and Challenges
Takerngsaksiri, W., Warusavitarne, C., Yaacoub, C., Keng Hou, M. H., & Tantithamthavorn, C. (2024) · 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00252 · Cited by 18
“Prather et al. [10] also studied how novice programmers interact with GitHub Copilot via introductory programming assignments and interviews.”
BackgroundGenerating Programs Trivially: Student Use of Large Language Models
Prasad, S., Greenman, B., Nelson, T., & Krishnamurthi, S. (2023) · ACM Global Computing Education Conference (CompEd) · doi:10.1145/3576882.3617921 · Cited by 35
Experiences from Integrating Large Language Model Chatbots into the Classroom
Hellas, A., Leinonen, J., & Leppänen, L. (2024) · ACM Virtual Global Computing Education Conference (SIGCSE Virtual) · doi:10.1145/3649165.3690101 · Cited by 37
Self-citationKemell, K., Saarikallio, M., Nguyen‐Duc, A., & Abrahamsson, P. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107805 · Cited by 41
An Experiment on Leveraging ChatGPT for Online Teaching and Assessment of Database Students
Carr, N., Shawon, F. R., & Jamil, H. M. (2023) · IEEE International Conference on Teaching, Assessment, and Learning for Engineering · doi:10.1109/tale56641.2023.10398239 · Cited by 19
“Although scant and too early, recent research on a LLM-based AI code generator GitHub Copilot indicate learning gains [32, 39] and productivity improvement [15, 31], more formal evaluations are slowly trickling in [13, 27, 43].”
ResultGenerative AI in Programming Education: Evaluating ChatGPT’s Effect on Computational Thinking
Sivasakthi, M., & Meenakshi, A. (2025) · SN Computer Science · doi:10.1007/s42979-025-04051-9 · Cited by 16
“Though if you’re looking for one particularly tailored for programming [23], GitHub Copilot is the best artificial intelligence-based assistant available.”
Autoethnographic Insights from Neurodivergent GAI “Power Users”
Glazko, K. S., Cha, J., Lewis, A., Kosa, B., Wimer, B. L., Zheng, A., Zheng, Y., & Mankoff, J. C. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713670 · Cited by 32
Adejumo, A. A., Oyelere, S. S., Sanusi, I. T., & Suhonen, J. T. (2026) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2026.100570 · Cited by 29
Mahon, J., Namee, B. M., & Becker, B. A. (2023) · Conference on United Kingdom & Ireland Computing Education Research (UKICER) · doi:10.1145/3610969.3610982 · Cited by 23
“computing education [5], with implications for both teachers and students [30, 39].”
BackgroundSelf-citationHow Much Does AI Impact Development Speed? an Enterprise-Based Randomized Controlled Trial
Paradis, E., Grey, K., Madison, Q., Nam, D., Macvean, A., Meimand, V., Zhang, N., Ferrari-Church, B., & Chandra, S. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00060 · Cited by 20
“Researchers and educators have also developed prototype tools to assist novice programmers and students [8], [9], [10], [11].”
Mental model shifts in human-LLM interactions
Schneider, J. (2025) · Journal of Intelligent Information Systems · doi:10.1007/s10844-025-00960-6 · Cited by 30
“…to certain tasks such as design (Kumar et al., 2023; Sharma et al., 2024), code migration (Omidvar Tehrani & Anubhai, 2024), coding for novices (Prather et al., 2023), negotiation (Schneider et al., 2023) and education, i.e., assessing learning performance of students using LLMs (Kumar et…”
Not the Silver Bullet: LLM-enhanced Programming Error Messages are Ineffective in Practice
Santos, E. A., & Becker, B. A. (2024) · Conference on United Kingdom & Ireland Computing Education Research (UKICER) · doi:10.1145/3689535.3689554 · Cited by 28
Self-citationComputer Programming Education in the Age of Generative AI: Insights from Empirical ResearchPreprint
Deriba, F. G., Sanusi, I. T., Campbell, O. O., & Oyelere, S. S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4891302 · Cited by 28
How Scientists Use Large Language Models to Program
O'Brien, G. (2025) · arXiv · doi:10.48550/arxiv.2502.17348 · Cited by 14
Dawson, M. G., Deer, R., & Boguslawski, S. (2025) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2025.100724 · Cited by 30
Human-AI experience in integrated development environments: a systematic literature review
Sergeyuk, A., Zakharov, I., Koshchenko, E., & Izadi, M. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10793-0 · Cited by 50
“Studies focusing on novice programmers reveal both benefits and pronounced risks (Prather et al., 2023; Puryear and Sprint, 2022).”
Comparing Feedback from Large Language Models and Instructors: Teaching Computer Science at Scale
Nguyen, H., Stott, N., & Allan, V. H. (2024) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3657604.3664660 · Cited by 23
Mailach, A., Gorgosch, D., Siegmund, N., & Siegmund, J. (2025) · Empirical Software Engineering · doi:10.1007/s10664-024-10561-6 · Cited by 13
“Closest to our work are studies conducted by Prather and others, who observe how students interact with LLMs in a CS1 course (Prather et al. 2023), and Kazemitabaar and others, who extract interaction patterns from students using Codex in a self-paced learning environment (Kazemitabaar et al. 2023).”
How Scientists Use Large Language Models to Program
O’Brien, G. E. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713668 · Cited by 42
Bassner, P., Lenk-Ostendorf, B., Beinstingel, R., Wasner, T., & Krusche, S. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100537 · Cited by 45
Let's Ask AI About Their Programs: Exploring ChatGPT's Answers To Program Comprehension Questions
Lehtinen, T., Koutcheme, C., & Hellas, A. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1145/3639474.3640058 · Cited by 32
An Investigation of the Drivers of Novice Programmers' Intentions to Use Web Search and GenAI
Skripchuk, J., Bacher, J. T., & Price, T. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671112 · Cited by 22
Stone, I. (2024) · Conference on United Kingdom & Ireland Computing Education Research (UKICER) · doi:10.1145/3689535.3689553 · Cited by 22
“GenAI can accelerate programming progress, enabling learners to work at a higher level of abstraction [33].”
InfluentialInsights from the Frontline: GenAI Utilization Among Software Engineering Students
Choudhuri, R., Ramakrishnan, A., Chatterjee, A., Trinkenreich, B., Steinmacher, I., Gerosa, M. A., & Sarma, A. (2025) · IEEE International Conference on Software Engineering Education and Training (CSEE&T) · doi:10.1109/cseet66350.2025.00007 · Cited by 38
“User studies with genAI : Research on genAI-based programming tools [32, 33, 34, 35, 36, 37, 38, 39] has shown their usefulness and limitations.”
Zhu, M., & Zhang, K. (2025) · Technology, Knowledge and Learning · doi:10.1007/s10758-025-09859-1 · Cited by 13
“Prather et al. (2023) found that utilizing GitHub Copilot can boost students’ cognitive coding, metacognition, positive emotions, and learning efficiency.”
A Bibliometric Exposition and Review on Leveraging LLMs for Programming Education
Amos, J. P., Amodu, O. A., Mahmood, R. A. R., AbdulQudus, A. B., Zakaria, A. F., Iyanda, A. R., Bukar, U. A., & Hanapi, Z. M. (2025) · IEEE Access · doi:10.1109/access.2025.3554627 · Cited by 24
“The study in [19] examines how novice programmers interact with AI-based automated code-generated tools like GitHub Copilot while completing typical CS1 (introductory programming) assignments.”
The Trees in the Forest: Characterizing Computing Students' Individual Help-Seeking Approaches
Ko, S.-H., & Stephens-Martinez, K. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671099 · Cited by 13
Can AI teach me employability? A multi-national study in three countries
Aditya, D., Silvestri, K., & Otermans, P. C. J. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1461158 · Cited by 24
Kohn, T. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70043 · Cited by 15
Ethical Issues in AI-Generated Texts: A Systematic Review and Analysis
Zhang, Y., Zhou, T., Qiao, H., & Li, T. X. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2530071 · Cited by 24
“The positive reactions mainly come from AI ’ s ability to generate fast required content, which improves human productivity (Kuhail et al., 2024; Prather et al., 2023), but the negative perceptions are diverse.”
InfluentialAnagnostopoulos, C. (2023) · F1000Research · doi:10.12688/f1000research.141958.1 · Cited by 19
Solving Proof Block Problems Using Large Language Models
Poulsen, S., Sarsa, S., Prather, J. E., Leinonen, J., Becker, B. A., Hellas, A., Denny, P. C., & Reeves, B. N. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630928 · Cited by 21
Self-citationBasha, M., & Rodríguez-Pérez, G. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107804 · Cited by 19
Outcomes, Perceptions, and Interaction Strategies of Novice Programmers Studying with ChatGPT
Penney, J., Acharya, P., Hilbert, P., Parekh, P., Sarma, A., Steinmacher, I., & Gerosa, M. A. (2025) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3719160.3736625 · Cited by 17
CodingGenie: A Proactive LLM-Powered Programming Assistant
Zhao, S., Zhu, A., Mozannar, H., Sontag, D. A., Talwalkar, A., & Chen, V. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3728603 · Cited by 32
Pirzado, F. A., Ahmed, A., Alipour, M. A., & Terashima‐Marín, H. (2025) · Journal of Computers in Education · doi:10.1007/s40692-025-00375-w · Cited by 23
“Additionally, students with less programming experience find it more challenging to interact with AI-CAs, raising concerns about their implications while remaining optimistic about the potential integration of AI-CAs into programming education in the future (Prather et al., 2023).”
The Robots are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.00658 · Cited by 9
Self-citationAugmenting Virtual Labs with Artificial Intelligence for Hybrid Learning
Murali, R., Ravi, N., & Surendran, A. (2024) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon60312.2024.10578649 · Cited by 21
Exploring Student-AI Interactions in Vibe Coding
Geng, F., Shah, A., Li, H., Mulla, N., Swanson, S., Raj, G. S., Zingaro, D., & Porter, L. (2026) · Australasian Computing Education Conference · doi:10.1145/3786228.3786236 · Cited by 51
A Transformer-Based Approach for Smart Invocation of Automatic Code Completion
de Moor, A., van Deursen, A., & Izadi, M. (2024) · Proceedings of the 1st ACM International Conference on AI-Powered Software · doi:10.1145/3664646.3664760 · Cited by 7
“Since its inception, several user studies and surveys [2, 24, 31, 33, 34, 42, 44, 49] have been performed on GitHub Copilot 4 .”
BackgroundInfluential
Metadata
- Authors
- Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A.
- Affiliations
- Abilene Christian University (US)
- University of Auckland (NZ)
- University College Dublin (IE)
- ORCID
- Venue
- ACM Transactions on Computer-Human Interaction
- Published
- 2023-11-29
- Volume
- 31
- Issue
- 1
- Pages
- 1-31
- ISSN
- 1073-0516 (ISSN-L)
- 1557-7325
- OpenAlex
- W4386099272
- Cited by
- 256
- References
- 91
- Citations by year
- 2026: 89 · 2025: 85 · 2024: 66 · 2023: 16
- FWCI
- 57.98
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Transactions on Computer-Human Interaction (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- GitHub Copilot, code generation, novice programmers, introductory programming, CS1, programming education, student perceptions, metacognition, cognitive load, scaffolding, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.