The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Open accessThe DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.
Abstract
Novice programmers often struggle through programming problem solving due to a lack of metacognitive awareness and strategies. Previous research has shown that novices can encounter multiple metacognitive difficulties while programming, such as forming incorrect conceptual models of the problem or having a false sense of progress after testing their solution. Novices are typically unaware of how these difficulties are hindering their progress. Meanwhile, many novices are now programming with generative AI (GenAI), which can provide complete solutions to most introductory programming problems, code suggestions, hints for next steps when stuck, and explain cryptic error messages. Its impact on novice metacognition has only started to be explored. Here we replicate a previous study that examined novice programming problem solving behavior and extend it by incorporating GenAI tools. Through 21 lab sessions consisting of participant observation, interview, and eye tracking, we explore how novices are coding with GenAI tools. Although 20 of 21 students completed the assigned programming problem, our findings show an unfortunate divide in the use of GenAI tools between students who did and did not struggle. Some students who did not struggle were able to use GenAI to accelerate, creating code they already intended to make, and were able to ignore unhelpful or incorrect inline code suggestions. But for students who struggled, our findings indicate that previously known metacognitive difficulties persist, and that GenAI unfortunately can compound them and even introduce new metacognitive difficulties. Furthermore, struggling students often expressed cognitive dissonance about their problem solving ability, thought they performed better than they did, and finished with an illusion of competence. Based on our observations from both groups, we propose ways to scaffold the novice GenAI experience and make suggestions for future work.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
Interrater reliability: The kappa statistic
McHugh, M. L. (2012) · Biochemia Medica · doi:10.11613/bm.2012.031 · Cited by 19,877
Protocol Analysis: Verbal Reports as Data
Ericsson, K. A., & Simon, H. A. (1993) · doi:10.7551/mitpress/5657.001.0001 · Cited by 3,537
Effects of beliefs about the nature of knowledge on comprehension.
Schommer, M. (1990) · Journal of Educational Psychology · doi:10.1037//0022-0663.82.3.498 · Cited by 2,214 · cited 1×
“Metacognitive awareness, along with self-regulation, are two essential components of cognitive control [60], which involves students recognizing and implementing strategies to guide their problem solving and learning.”
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Impact of AI assistance on student agency
Darvishi, A., Khosravi, H., Sadiq, S., Gašević, D., & Siemens, G. (2024) · Computers & Education · doi:10.1016/j.compedu.2023.104967 · Cited by 548
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497
Self-citationThe Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493 · cited 4×
“Therefore, it is imperative that we scaffold the novice programmer experience of GenAI tools from the very beginning by lowering the metacognitive demand on the user [62].”
InfluentialThe Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463
Self-citationGrounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436 · cited 3×
“For professionals, Copilot use can be categorized into “acceleration” where they utilize it for completing a task faster and “exploration” where they explore potential approaches to solve the task [4].”
InfluentialBecker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569759 · Cited by 401 · cited 3×
“However, since GenAI entered the mainstream, researchers have been concerned about the possibility of student over-reliance [6].”
Self-citationMotivated Strategies for Learning Questionnaire
Pintrich, P. R., & De Groot, E. V. (2012) · PsycTESTS Dataset · doi:10.1037/t09161-000 · Cited by 399
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396 · cited 5×
“ChatGPT is a general purpose tool that students are increasingly using for their work [25, 33, 50, 64].”
InfluentialSelf-citationComputing Education in the Era of Generative AI
Denny, P., Prather, J., Becker, B. A., Finnie-Ansley, J., Hellas, A., Leinonen, J., Luxton-Reilly, A., Reeves, B. N., Santos, E. A., & Sarsa, S. (2024) · Communications of the ACM · doi:10.1145/3624720 · Cited by 316
Self-citationKazemitabaar, M., Ye, R., Wang, X., Henley, A. Z., Denny, P., Craig, M., & Grossman, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642773 · Cited by 290 · cited 2×
“Many computing education tools that utilize generative AI and LLMs have recently emerged such as CodeHelp [32], CodeAid [27], and Promptly [13].”
The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023) · arXiv · doi:10.48550/arxiv.2302.06590 · Cited by 287
Self-efficacy and mental models in learning to program
Ramalingam, V., LaBelle, D., & Wiedenbeck, S. (2004) · Proceedings of the 9th annual SIGCSE conference on Innovation and technology in computer science education · doi:10.1145/1007996.1008042 · Cited by 280
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256
Self-citationLau, S., & Guo, P. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600138 · Cited by 241 · cited 2×
“This has raised concerns from educators about potential student over-reliance [6, 29, 61, 72].”
Using Large Language Models to Enhance Programming Error Messages
Leinonen, J., Hellas, A., Sarsa, S., Reeves, B., Denny, P., Prather, J., & Becker, B. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569770 · Cited by 235
Self-citationProgramming, Problem Solving, and Self-Awareness: Effects of Explicit Guidance
Loksa, D., Ko, A. J., Jernigan, W., Oleson, A., Mendez, C. J., & Burnett, M. M. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858252 · Cited by 223
Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · arXiv · doi:10.48550/arxiv.2306.05715 · Cited by 211
Self-citationComparing Code Explanations Created by Students and Large Language Models
Leinonen, J., Denny, P., MacNeil, S., Sarsa, S., Bernstein, S., Kim, J., Tran, A., & Hellas, A. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588785 · Cited by 210
Self-citationMacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569785 · Cited by 195
Self-citationMacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J. (2022) · arXiv · doi:10.48550/arxiv.2211.02265 · Cited by 195
Self-citationCodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes
Liffiton, M., Sheese, B. E., Savelka, J., & Denny, P. (2023) · Proceedings of the 23rd Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631830 · Cited by 192 · cited 2×
“Many computing education tools that utilize generative AI and LLMs have recently emerged such as CodeHelp [32], CodeAid [27], and Promptly [13].”
A theory of instruction for introductory programming skills
Xie, B., Loksa, D., Nelson, G. L., Davidson, M. J., Dong, D., Kwik, H., Tan, A. H., Hwa, L., Li, M., & Ko, A. J. (2019) · Computer Science Education · doi:10.1080/08993408.2019.1565235 · Cited by 191 · cited 1×
“The theory of programming instruction by Xie et al. [69] states that student learning should start at the code reading and tracing level and then move down into code writing.”
Prompt Problems: A New Programming Exercise for the Generative AI Era
Denny, P., Leinonen, J., Prather, J., Luxton-Reilly, A., Amarouche, T., Becker, B. A., & Reeves, B. N. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630909 · Cited by 187 · cited 2×
“Many computing education tools that utilize generative AI and LLMs have recently emerged such as CodeHelp [32], CodeAid [27], and Promptly [13].”
Self-citationMy AI Wants to Know if This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises
Finnie-Ansley, J., Denny, P., Luxton-Reilly, A., Santos, E. A., Prather, J., & Becker, B. A. (2023) · Proceedings of the 25th Australasian Computing Education Conference · doi:10.1145/3576123.3576134 · Cited by 186 · cited 1×
“LLMs can solve most introductory programming exercises in both CS1 [18] and CS2 courses [19].”
Self-citationTeaching CS50 with AI: Leveraging Generative Artificial Intelligence in Computer Science Education
Liu, R., Zenke, C., Liu, C., Holmes, A., Thornton, P., & Malan, D. J. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630938 · Cited by 185
A Survey on the Usage of Eye-Tracking in Computer Programming
Obaidellah, U., Al Haek, M., & Cheng, P. C.-H. (2019) · ACM Computing Surveys · doi:10.1145/3145904 · Cited by 162 · cited 1×
“Eye tracking has been utilized in programming research for decades [7, 10, 11] and is mostly used to measure code comprehension and debugging [42], novice gaze patterns [8] and automatic skill level detection [1].”
Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600139 · Cited by 145 · cited 1×
“Due to good performance on these tasks, there have been calls to integrate LLMs into the classroom [9, 72] (albeit with guardrails [23]).”
Self-citationEye tracking in computing education
Busjahn, T., Schulte, C., Sharif, B., Simon, Begel, A., Hansen, M., Bednarik, R., Orlov, P. A., Ihantola, P., Shchekotova, G., et al. (2014) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2632320.2632344 · Cited by 144 · cited 2×
“Eye tracking has been utilized in programming research for decades [7, 10, 11] and is mostly used to measure code comprehension and debugging [42], novice gaze patterns [8] and automatic skill level detection [1].”
The Role of Self-Regulation in Programming Problem Solving Process and Success
Loksa, D., & Ko, A. J. (2016) · Proceedings of the 2016 ACM Conference on International Computing Education Research · doi:10.1145/2960310.2960334 · Cited by 143 · cited 1×
“We take this caution seriously and note that even before the advent of GenAI, researchers found that students struggled to move through the programming problem solving process due to a lack of metacognitive awareness [34, 35, 51, 52].”
The Impact of Large Language Models on Programming Education and Student Learning Outcomes
Jošt, G., Taneski, V., & Karakatič, S. (2024) · Applied Sciences · doi:10.3390/app14104115 · Cited by 143
Savelka, J., Agarwal, A., An, M., Bogart, C., & Sakr, M. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600142 · Cited by 141
Metacognition and Self-Regulation in Programming Education: Theories and Exemplars of Use
Loksa, D., Margulieux, L., Becker, B. A., Craig, M., Denny, P., Pettit, R., & Prather, J. (2022) · ACM Transactions on Computing Education · doi:10.1145/3487050 · Cited by 140
Self-citationGenerative AI in Computing Education: Perspectives of Students and Instructors
Zastudil, C., Rogalska, M., Kapp, C., Vaughn, J., & MacNeil, S. (2023) · 2023 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10343467 · Cited by 136 · cited 3×
“This has raised concerns from educators about potential student over-reliance [6, 29, 61, 72].”
Self-citationHandbook of usability testing: How to plan, design, and conduct effective tests
Shaw, D. (1996) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199603)47:3<258::aid-asi18>3.0.co;2-# · Cited by 134
What Do We Think We Think We Are Doing?: Metacognition and Self-Regulation in Programming
Prather, J., Becker, B. A., Craig, M., Denny, P., Loksa, D., & Margulieux, L. (2020) · Proceedings of the 2020 ACM Conference on International Computing Education Research · doi:10.1145/3372782.3406263 · Cited by 131
Self-citationMetacognitive Difficulties Faced by Novice Programmers in Automated Assessment Tools
Prather, J., Pettit, R., McMurry, K., Peters, A., Homer, J., & Cohen, M. (2018) · Proceedings of the 2018 ACM Conference on International Computing Education Research · doi:10.1145/3230977.3230981 · Cited by 129 · cited 10×
“Unlike our prior study [52], which we conducted before the advent of GenAI, the course in which this study took place utilized GenAI from the beginning.”
InfluentialSelf-citationMeasuring GitHub Copilot's Impact on Productivity
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2024) · Communications of the ACM · doi:10.1145/3633453 · Cited by 124 · cited 2×
“Third, a study in Communications of the ACM in March, 2024 by Ziegler et al. investigated how GitHub Copilot is impacting programmer productivity [73].”
Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641936 · Cited by 114
Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2022) · arXiv · doi:10.48550/arxiv.2210.14306 · Cited by 114 · cited 1×
“Mozannar et al. recently showed a possible way to achieve this via reflective retrospective labeling of coding sessions [41].”
(2019) · doi:10.1145/3287324.3287374 · Cited by 104
Self-citationThe Effects of Generative AI on Computing Students’ Help-Seeking Preferences
Hou, I., Mettille, S., Man, O., Li, Z., Zastudil, C., & MacNeil, S. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636248 · Cited by 97 · cited 4×
“ChatGPT is a general purpose tool that students are increasingly using for their work [25, 33, 50, 64].”
InfluentialSelf-citationCS1-LLM: Integrating LLMs into CS1 Instruction
Vadaparty, A., Zingaro, D., Smith IV, D. H., Padala, M., Alvarado, C., Gorson Benario, J., & Porter, L. (2024) · Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3649217.3653584 · Cited by 77 · cited 4×
“Second, recent work by Vadaparty et al. [64] describes the usage of GenAI tools from the very beginning in a large introductory programming course.”
InfluentialOn Novices' Interaction with Compiler Error Messages: A Human Factors Approach
Prather, J., Pettit, R., McMurry, K. H., Peters, A., Homer, J., Simone, N., & Cohen, M. (2017) · Proceedings of the 2017 ACM Conference on International Computing Education Research · doi:10.1145/3105726.3106169 · Cited by 72 · cited 1×
“Next, the participant was given a warm-up task because following a think-aloud protocol can be difficult to do while also doing cognitively demanding tasks, as suggested by [53, 63, 67].”
Self-citationIvie: Lightweight Anchored Explanations of Just-Generated Code
Yan, L., Hwang, A., Wu, Z., & Head, A. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642239 · Cited by 67
An Eye For Design: Gaze Visualizations for Remote Collaborative Work
D'Angelo, S., & Gergle, D. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173923 · Cited by 62 · cited 2×
“Eye tracking has been utilized in programming research for decades [7, 10, 11] and is mostly used to measure code comprehension and debugging [42], novice gaze patterns [8] and automatic skill level detection [1].”
Cambaz, D., & Zhang, X. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630958 · Cited by 62
Margulieux, L. E., Prather, J., Reeves, B. N., Becker, B. A., Cetin Uzun, G., Loksa, D., Leinonen, J., & Denny, P. (2024) · Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3649217.3653621 · Cited by 59 · cited 3×
“Another key component for moving forward in the era of GenAI is to finally invest in explicitly teaching novice programmers metacognitive behaviors and skills [38, 62, 68].”
Self-citationInstructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study
Sheard, J., Denny, P., Hellas, A., Leinonen, J., Malmi, L., & Simon (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630880 · Cited by 54 · cited 2×
“This has raised concerns from educators about potential student over-reliance [6, 29, 61, 72].”
Self-citationA Large Scale RCT on Effective Error Messages in CS1
Wang, S., Mitchell, J. C., & Piech, C. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630764 · Cited by 53
A qualitative think-aloud study of novice programmers' code writing strategies
Whalley, J., & Kasto, N. (2014) · Proceedings of the 2014 conference on Innovation & technology in computer science education - ITiCSE '14 · doi:10.1145/2591708.2591762 · Cited by 52 · cited 1×
“Next, the participant was given a warm-up task because following a think-aloud protocol can be difficult to do while also doing cognitively demanding tasks, as suggested by [53, 63, 67].”
Hou, I., Man, O., Mettille, S., Gutierrez, S., Angelikas, K., & MacNeil, S. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636247 · Cited by 47 · cited 1×
“They can correctly answer multiple-choice questions related to introductory programming [59] and solve Parsons problems based only on an image of the problem [24].”
Self-citationAn Analysis of Using Many Small Programs in CS1
Allen, J. M., Vahid, F., Edgcomb, A., Downey, K., & Miller, K. (2019) · Proceedings of the 50th ACM Technical Symposium on Computer Science Education · doi:10.1145/3287324.3287466 · Cited by 36 · cited 1×
“It may also be the kind of phenomenon that is not readily apparent to instructors due to students possibly having a perception of learning while courses evaluate programming skills in traditional ways, such as through the popular “many small programs” approach [2, 3].”
Always Provide Context: The Effects of Code Context on Programming Error Message Enhancement
Santos, E. A., Prasad, P., & Becker, B. A. (2023) · ACM Global Computing Education Conference (CompEd) · doi:10.1145/3576882.3617909 · Cited by 36
Self-citationBecker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2022) · arXiv · doi:10.48550/arxiv.2212.01020 · Cited by 36
Self-citationOleson, A., Xie, B., Salac, J., Everson, J., Kivuva, F. M., & Ko, A. J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543967 · Cited by 35
Metacodenition: Scaffolding the Problem-Solving Process for Novice Programmers
Pechorina, Y., Anderson, K., & Denny, P. (2023) · Proceedings of the 25th Australasian Computing Education Conference · doi:10.1145/3576123.3576130 · Cited by 33 · cited 1×
“Other interventions suggested by Prather et al. [51] and Pechorina et al. [44] provide additional fruitful ground for interventions designed to scaffold novice metacognition.”
A qualitative think aloud study of the early neo-piagetian stages of reasoning in novice programmers
Teague, D., Corney, M., Ahadi, A., & Lister, R. (2013) · QUT ePrints (Queensland University of Technology) · doi:10.5555/2667199.2667209 · Cited by 31
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23 · cited 3×
“For professionals, Copilot use can be categorized into “acceleration” where they utilize it for completing a task faster and “exploration” where they explore potential approaches to solve the task [4].”
InfluentialFrom Novice to Expert: Analysis of Token Level Effects in a Longitudinal Eye Tracking Study
Al Madi, N., Peterson, C. S., Sharif, B., & Maletic, J. I. (2021) · 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc52881.2021.00025 · Cited by 20 · cited 1×
“Eye tracking has been utilized in programming research for decades [7, 10, 11] and is mostly used to measure code comprehension and debugging [42], novice gaze patterns [8] and automatic skill level detection [1].”
iTrace-Toolkit: A Pipeline for Analyzing Eye-Tracking Data of Software Engineering Studies
Behler, J. A. C., Weston, P., Guarnera, H. M., Sharif, B., & Maletic, J. I. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion58688.2023.00022 · Cited by 18 · cited 1×
“The professor (one of the researchers) encouraged students to install VSCode with GitHub Copilot on the first day of class and coded live in front of students in that environment.”
Developing Novice Programmers’ Self-Regulation Skills with Code Replays
Xie, B., Lim, J. O., Pham, P. K. D., Li, M., & Ko, A. J. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600127 · Cited by 18 · cited 3×
“Another key component for moving forward in the era of GenAI is to finally invest in explicitly teaching novice programmers metacognitive behaviors and skills [38, 62, 68].”
Generative AI in Education: A Study of Educators' Awareness, Sentiments, and Influencing FactorsPreprint
Ghimire, A., Prather, J. E., & Edwards, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.15586 · Cited by 18
Self-citationMany Small Programs in CS1: Usage Analysis from Multiple Universities
Allen, J., Downey, K., Miller, K., Edgcomb, A., & Vahid, F. (2020) · 2019 ASEE Annual Conference & Exposition Proceedings · doi:10.18260/1-2--33084 · Cited by 7 · cited 1×
“It may also be the kind of phenomenon that is not readily apparent to instructors due to students possibly having a perception of learning while courses evaluate programming skills in traditional ways, such as through the popular “many small programs” approach [2, 3].”
Iris: a tool for designing contextually relevant gaze visualizations
D'Angelo, S., Brewer, J., & Gergle, D. (2019) · Eye Tracking Research & Application · doi:10.1145/3317958.3318228 · Cited by 6 · cited 1×
“Eye tracking has been utilized in programming research for decades [7, 10, 11] and is mostly used to measure code comprehension and debugging [42], novice gaze patterns [8] and automatic skill level detection [1].”
Elias, S., Cromarty, E., & Wilson-Jones, L. (2022) · DigitalCommons@Fayetteville State University (Fayetteville State University) · Cited by 1
"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2023) · arXiv
Self-citationDesirable Characteristics for AI Teaching Assistants in Programming Education
Denny, P., MacNeil, S., Savelka, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.14178
Self-citation
Research works printed but not linked: 6
- [5] Brett A. Becker, Michelle Craig, Paul Denny, Hieke Keuning, Natalie Kiesler, Juho Leinonen, Andrew Luxton-Reilly, Lauri Malmi, James Prather, and Keith Quille. 2023. Generative AI in Introductory Programming. https://csed.acm.org/large-language-models-in-introductory-programming Google Scholar
- [21] David Freedman, Robert Pisani, and Roger Purves. 2007. Statistics (international student edition). Pisani, R. Purves, 4th edn. WW Norton & Company, New York (2007). Google Scholar
- [40] Marvin Lee Minsky. 1994. Negative Expertise. International Journal of Expert Systems Research and Applications 7 (1994), 13–18. Issue 1. Digital Library Google Scholar
- [46] Bruno Pereira Cipriano and Pedro Alves. 2024. " ChatGPT Is Here to Help, Not to Replace Anybody"–An Evaluation of Students’ Opinions On Integrating ChatGPT In CS Courses. arXiv e-prints (2024), arXiv–2404. Google Scholar
- [48] Leo Porter and Daniel Zingaro. 2023. Learn AI-Assisted Python Programming with GitHub Copilot and ChatGPT. Manning, Shelter Island, NY, USA. Google Scholar
- [70] Yuankai Xue, Hanlin Chen, Gina R Bai, Robert Tairas, and Yu Huang. 2004. Does ChatGPT Help With Introductory Programming? An Experiment of Students Using ChatGPT in CS1. In Proceedings of the 46th ACM/IEEE International Conference on Software Engineering (Lisbon, Portugal) (ICSE-SEET 2024). Association for Computing Machinery, New York, NY, USA. Google Scholar
Cited by
Lee, H.-P. (., Sarkar, A., Tankelevitch, L., Drosos, I., Rintel, S., Banks, R., & Wilson, N. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713778 · Cited by 490
Prather, J., Leinonen, J., Kiesler, N., Gorson Benario, J., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., et al. (2025) · 2024 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3689187.3709614 · Cited by 96
“Similarly, in an observational study, researchers found that although a larger percentage of students are able to complete programming tasks with the aid of GenAI tools, students faced new metacognitive challenges [113].”
Self-citationTang, Q., Deng, W., Huang, Y., Wang, S., & Zhang, H. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70027 · Cited by 55
Student-AI Interaction: A Case Study of CS1 students
Amoozadeh, M., Nam, D., Prol, D., Alfageeh, A., Prather, J. E., Hilton, M. L., Ragavan, S. S., & Alipour, M. A. (2024) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3699538.3699567 · Cited by 90
Self-citationLepp, M., & Kaimre, J. (2025) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2025.100642 · Cited by 92
Metadata
- Authors
- Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B.
- Affiliations
- Abilene Christian University (US)
- Aalto University (FI)
- Temple University (US)
- University College Dublin (IE)
- ORCID
- Venue
- Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1
- Publisher
- ACM
- Published
- 2024-08-12
- Pages
- 469-486
- OpenAlex
- W4401379837
- Cited by
- 196
- References
- 59
- Citations by year
- 2026: 86 · 2025: 103 · 2024: 7
- FWCI
- 85.68
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on International Computing Education Research (ICER) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- Aaltodoc (Aalto University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- generative AI, novice programmers, metacognition, programming problem solving, conceptual model, illusion of competence, cognitive dissonance, code completion, scaffolding, introductory programming, eye tracking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A second model labelled each claim on 2026-10-27 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.