The DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index.
The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Open accessAbstract
Novice programmers often struggle through programming problem solving due to a lack of metacognitive awareness and strategies. Previous research has shown that novices can encounter multiple metacognitive difficulties while programming, such as forming incorrect conceptual models of the problem or having a false sense of progress after testing their solution. Novices are typically unaware of how these difficulties are hindering their progress. Meanwhile, many novices are now programming with generative AI (GenAI), which can provide complete solutions to most introductory programming problems, code suggestions, hints for next steps when stuck, and explain cryptic error messages. Its impact on novice metacognition has only started to be explored. Here we replicate a previous study that examined novice programming problem solving behavior and extend it by incorporating GenAI tools. Through 21 lab sessions consisting of participant observation, interview, and eye tracking, we explore how novices are coding with GenAI tools. Although 20 of 21 students completed the assigned programming problem, our findings show an unfortunate divide in the use of GenAI tools between students who did and did not struggle. Some students who did not struggle were able to use GenAI to accelerate, creating code they already intended to make, and were able to ignore unhelpful or incorrect inline code suggestions. But for students who struggled, our findings indicate that previously known metacognitive difficulties persist, and that GenAI unfortunately can compound them and even introduce new metacognitive difficulties. Furthermore, struggling students often expressed cognitive dissonance about their problem solving ability, thought they performed better than they did, and finished with an illusion of competence. Based on our observations from both groups, we propose ways to scaffold the novice GenAI experience and make suggestions for future work.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
Interrater reliability: The kappa statistic
McHugh, M. L. (2012) · Biochemia Medica · doi:10.11613/bm.2012.031 · Cited by 19,877
Protocol Analysis: Verbal Reports as Data
Ericsson, K. A., & Simon, H. A. (1993) · doi:10.7551/mitpress/5657.001.0001 · Cited by 3,537
Effects of beliefs about the nature of knowledge on comprehension.
Schommer, M. (1990) · Journal of Educational Psychology · doi:10.1037//0022-0663.82.3.498 · Cited by 2,214 · cited 1×
“Metacognitive awareness, along with self-regulation, are two essential components of cognitive control [60], which involves students recognizing and implementing strategies to guide their problem solving and learning.”
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Impact of AI assistance on student agency
Darvishi, A., Khosravi, H., Sadiq, S., Gašević, D., & Siemens, G. (2024) · Computers & Education · doi:10.1016/j.compedu.2023.104967 · Cited by 548
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497
Self-citationThe Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493 · cited 4×
“Therefore, it is imperative that we scaffold the novice programmer experience of GenAI tools from the very beginning by lowering the metacognitive demand on the user [62].”
InfluentialThe Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463
Self-citationGrounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436 · cited 3×
“For professionals, Copilot use can be categorized into “acceleration” where they utilize it for completing a task faster and “exploration” where they explore potential approaches to solve the task [4].”
InfluentialBecker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569759 · Cited by 401 · cited 3×
“However, since GenAI entered the mainstream, researchers have been concerned about the possibility of student over-reliance [6].”
Self-citation
Research works printed but not linked: 6
- Brett A. Becker Michelle Craig Paul Denny Hieke Keuning Natalie Kiesler Juho Leinonen Andrew Luxton-Reilly Lauri Malmi James Prather and Keith Quille. 2023. Generative AI in Introductory Programming. https://csed.acm.org/large-language-models-in-introductory-programming
- David Freedman, Robert Pisani, and Roger Purves. 2007. Statistics (international student edition). Pisani, R. Purves, 4th edn. WW Norton & Company, New York (2007).
- Marvin Lee Minsky. 1994. Negative Expertise. International Journal of Expert Systems Research and Applications 7 (1994), 13–18. Issue 1.
- Bruno Pereira Cipriano and Pedro Alves. 2024. " ChatGPT Is Here to Help, Not to Replace Anybody"–An Evaluation of Students’ Opinions On Integrating ChatGPT In CS Courses. arXiv e-prints (2024), arXiv–2404.
- Leo Porter and Daniel Zingaro. 2023. Learn AI-Assisted Python Programming with GitHub Copilot and ChatGPT. Manning, Shelter Island, NY, USA.
- Yuankai Xue, Hanlin Chen, Gina R Bai, Robert Tairas, and Yu Huang. 2004. Does ChatGPT Help With Introductory Programming? An Experiment of Students Using ChatGPT in CS1. In Proceedings of the 46th ACM/IEEE International Conference on Software Engineering (Lisbon, Portugal) (ICSE-SEET 2024). Association for Computing Machinery, New York, NY, USA.
Cited by
Lee, H.-P. (., Sarkar, A., Tankelevitch, L., Drosos, I., Rintel, S., Banks, R., & Wilson, N. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713778 · Cited by 490
Prather, J. E., Leinonen, J., Kiesler, N., Benario, J. G., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., et al. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3689187.3709614 · Cited by 173
“Similarly, in an observational study, researchers found that although a larger percentage of students are able to complete programming tasks with the aid of GenAI tools, students faced new metacognitive challenges [113].”
Self-citationTang, Q., Deng, W., Huang, Y., Wang, S., & Zhang, H. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70027 · Cited by 55
Student-AI Interaction: A Case Study of CS1 students
Amoozadeh, M., Nam, D., Prol, D., Alfageeh, A., Prather, J. E., Hilton, M. L., Ragavan, S. S., & Alipour, M. A. (2024) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3699538.3699567 · Cited by 70
Self-citationLepp, M., & Kaimre, J. (2025) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2025.100642 · Cited by 77
Metadata
- Authors
- Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B.
- Affiliations
- Abilene Christian University (US)
- Aalto University (FI)
- Temple University (US)
- University College Dublin (IE)
- ORCID
- Venue
- Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1
- Publisher
- ACM
- Published
- 2024-08-12
- Pages
- 469-486
- OpenAlex
- W4401379837
- Cited by
- 196
- References
- 59
- Citations by year
- 2026: 86 · 2025: 103 · 2024: 7
- FWCI
- 85.68
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on International Computing Education Research (ICER) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- Aaltodoc (Aalto University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- generative AI, novice programmers, metacognition, programming problem solving, conceptual model, illusion of competence, cognitive dissonance, code completion, scaffolding, introductory programming, eye tracking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A second model labelled each claim on 2026-10-27 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers. https://app.cochat.ai/papers/10.1145/3632620.3671116
- The paper
- Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B. (2024). The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers. Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1. https://doi.org/10.1145/3632620.3671116
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.