Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index.

The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers

Open access
Prather, James, Reeves, Brent N, Leinonen, Juho, MacNeil, Stephen, Randrianasolo, Arisoa S, Becker, Brett A., Kimmel, Bailey, Wright, Jared, Briggs, Ben·2024·Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1

Abstract

Novice programmers often struggle through programming problem solving due to a lack of metacognitive awareness and strategies. Previous research has shown that novices can encounter multiple metacognitive difficulties while programming, such as forming incorrect conceptual models of the problem or having a false sense of progress after testing their solution. Novices are typically unaware of how these difficulties are hindering their progress. Meanwhile, many novices are now programming with generative AI (GenAI), which can provide complete solutions to most introductory programming problems, code suggestions, hints for next steps when stuck, and explain cryptic error messages. Its impact on novice metacognition has only started to be explored. Here we replicate a previous study that examined novice programming problem solving behavior and extend it by incorporating GenAI tools. Through 21 lab sessions consisting of participant observation, interview, and eye tracking, we explore how novices are coding with GenAI tools. Although 20 of 21 students completed the assigned programming problem, our findings show an unfortunate divide in the use of GenAI tools between students who did and did not struggle. Some students who did not struggle were able to use GenAI to accelerate, creating code they already intended to make, and were able to ignore unhelpful or incorrect inline code suggestions. But for students who struggled, our findings indicate that previously known metacognitive difficulties persist, and that GenAI unfortunately can compound them and even introduce new metacognitive difficulties. Furthermore, struggling students often expressed cognitive dissonance about their problem solving ability, thought they performed better than they did, and finished with an illusion of competence. Based on our observations from both groups, we propose ways to scaffold the novice GenAI experience and make suggestions for future work.

Overview

Integrity signals

Registry

Crossref · 10.1145/3632620.3671116

Cites

70 of 70 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

The text we found for this paper looks like another document, so its claims and self-check weren’t run. The registry and citation checks still apply.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The text we found for this paper looks like another document, so the self-check wasn’t run.

Citations

Cites

Show all 59 references

Research works printed but not linked: 6

  1. Brett A. Becker Michelle Craig Paul Denny Hieke Keuning Natalie Kiesler Juho Leinonen Andrew Luxton-Reilly Lauri Malmi James Prather and Keith Quille. 2023. Generative AI in Introductory Programming. https://csed.acm.org/large-language-models-in-introductory-programming
  2. David Freedman, Robert Pisani, and Roger Purves. 2007. Statistics (international student edition). Pisani, R. Purves, 4th edn. WW Norton & Company, New York (2007).
  3. Marvin Lee Minsky. 1994. Negative Expertise. International Journal of Expert Systems Research and Applications 7 (1994), 13–18. Issue 1.
  4. Bruno Pereira Cipriano and Pedro Alves. 2024. " ChatGPT Is Here to Help, Not to Replace Anybody"–An Evaluation of Students’ Opinions On Integrating ChatGPT In CS Courses. arXiv e-prints (2024), arXiv–2404.
  5. Leo Porter and Daniel Zingaro. 2023. Learn AI-Assisted Python Programming with GitHub Copilot and ChatGPT. Manning, Shelter Island, NY, USA.
  6. Yuankai Xue, Hanlin Chen, Gina R Bai, Robert Tairas, and Yu Huang. 2004. Does ChatGPT Help With Introductory Programming? An Experiment of Students Using ChatGPT in CS1. In Proceedings of the 46th ACM/IEEE International Conference on Software Engineering (Lisbon, Portugal) (ICSE-SEET 2024). Association for Computing Machinery, New York, NY, USA.

Cited by

See all 196

Metadata

Authors
Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B.
Affiliations
  • Abilene Christian University (US)
  • Aalto University (FI)
  • Temple University (US)
  • University College Dublin (IE)
Venue
Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1
Publisher
ACM
Published
2024-08-12
Pages
469-486
OpenAlex
W4401379837
Cited by
196
References
59
Citations by year
2026: 86 · 2025: 103 · 2024: 7
FWCI
85.68
Type
conference paper
Language
en
Open access
Locations
  • ACM Conference on International Computing Education Research (ICER) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Aaltodoc (Aalto University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
Keywords
generative AI, novice programmers, metacognition, programming problem solving, conceptual model, illusion of competence, cognitive dissonance, code completion, scaffolding, introductory programming, eye tracking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A second model labelled each claim on 2026-10-27 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers. https://app.cochat.ai/papers/10.1145/3632620.3671116
The paper
Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B. (2024). The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers. Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1. https://doi.org/10.1145/3632620.3671116

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.