Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index; 15 of the 46 sources its claims cite were checked for support.

How Do Programming Students Use Generative AI?

Open access
Rahe, Christian, Maalej, Walid · 2025 · Proceedings of the ACM on Software Engineering

Abstract

Exploratory experiment with 37 programming students. Majority prompted ChatGPT to generate full solutions. Students got trapped in a vicious cycle of submitting wrong generated code and asking the bot for a fix. Regular AI users more likely to prompt for full solutions.

Overview

CoChat Summary

TL;DR

This paper combines a GPT task-solving evaluation of an introductory programming course with a monitored experiment in which 37 students could use ChatGPT while solving a code-comprehension exercise. It finds that GPT can solve most course exercises, but students often used the chatbot to request full code solutions, fell into ineffective error-fix cycles, and showed over-reliance rather than asking for explanations.

Key findings

  • In the task-solving evaluation, both GPT models produced syntactically correct code in all responses, but GPT-3.5 had more logical errors and failed exercise requirements more than twice as often as GPT-4; the authors conclude a student could not pass the course by relying solely on GenAI because of in-person reviews and secondary assessments.
  • In the student experiment, 42 participated and 37 were analyzed; 62.2% completed the entire exercise correctly and 78.4% solved at least one problem. 37.8% did not use the chatbot at all, while among chatbot users the prompt ratio was 62% Codegen to 38% Support, and 73.9% submitted a Solve prompt at some point.
  • Manual analysis of code changes with chatbot interactions found 54.3% of submissions semantically identical to generated code (Copy All), 13.6% reused part of the generated solution (Idea), 11.1% copied a language construct (Syntax), 2.5% were based on generated explanation, and 18.5% were unrelated to chatbot interactions.
  • Consecutive submissions were significantly less similar when chatbot interactions occurred between them: 64.1% vs 82.2% without chatbot interactions (t = 5.76, p < 0.0001); in 62% of cases with generations, the modified submission was more similar to generated code than to the previous submission. GPT-3.5 generated correct solutions for only 8.8% of 80 P1 solutions, but 76.9% of 13 P2 solutions.

Methods

  • Task-solving evaluation: all course assignments were converted into prompts for GPT-3.5 (`gpt-3.5-turbo-1106`) and GPT-4 (`gpt-4-1106-preview`) with temperature 0.0; generated solutions were inserted into project files and evaluated against tutor criteria including syntax, logic, requirements, style, learnability, and intent.
  • Student experiment: an in-person controlled study in the SE1 course at University of Hamburg, November–December 2023; participants solved a two-problem Moodle/CodeRunner exercise with optional access to a ChatGPT-like interface using `gpt-3.5-turbo-1106`, while code changes, chatbot conversations, copy/paste events, and submissions were recorded.
  • Analysis: quantitative metrics and manual labeling of prompts and responses; code similarity used Ratcliff/Obershelp with a >90% threshold for close match; sequence pattern mining used NOSEP and NETLAP to identify interaction patterns.

Limitations

  • Stated: the referenced LLMs may be updated and yield different outputs; prompt phrasing and the entirely German material may affect model performance; participant self-selection may bias the pool toward better-performing students; the sample came from one university course in one year; the experiment setting and recording may have influenced chatbot use and self-reported GenAI use; only one fairly complex exercise was tested.
  • Stated: the authors recommend future studies test students’ understanding after completion and use exit surveys or think-aloud observations, because the study did not capture much information on students’ thoughts and impressions during the experiment.
  • Evident from the text: the analyzed student sample is small (37 participants), and the findings are tied to a single course, task, and chatbot configuration, limiting generalizability beyond that setting.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3715762

Cites

59 of 59 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

10 Valid · 6 Not valid · 20 Not checked · 171 Own results and interpretations

109 more claims

Create a free account to see all 207 claims and every source verdict.

See all claims — free account
  • Borderline: one of the tests for a substantial claim is unclear. 1
  • Probably not a substantial claim. 49
  • It may state two separate claims as one. 63
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

1 of 1

Citations

Cites

Show all 43 references

Research works printed but not linked: 3

  1. David J Barnes and Michael Kölling. 2009. Java lernen mit BlueJ: Eine Einführung in die objektorientierte Programmierung. Pearson Deutschland GmbH.
  2. Marlo Häring and Walid Maalej. 2019. A Socio-Technical Framework for Face-to-Face Teaching in Large Software Development Courses.. In Software Engineering (Workshops). 3–6.
  3. John W. Ratcliff and DM Metzener. 1988. Gestalt: an introduction to the Ratcliff/Obershelp pattern matching algorithm. Dr. Dobbs Journal 7 (1988), 46.

Cited by

See all 45

Metadata

Authors
Rahe, C., & Maalej, W.
Affiliations
Universität Hamburg (DE)
Venue
Proceedings of the ACM on Software Engineering
Published
2025-06-19
Volume
2
Issue
FSE
Pages
978-1000
ISSN
2994-970X (ISSN-L)
OpenAlex
W4406604115
Cited by
45
References
43
Citations by year
2026: 33 · 2025: 12
FWCI
39.98
Type
journal article
Language
en
Open access
Locations
  • Proceedings of the ACM on software engineering. (publishedVersion, cc-by-nc)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Social Sciences › Education › Artificial Intelligence in Education
Keywords
generative AI, ChatGPT, programming education, student use of AI, overreliance on AI, critical thinking, program comprehension, code reasoning, student learning behavior

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: How Do Programming Students Use Generative AI? https://app.cochat.ai/papers/10.1145/3715762
The paper
Rahe, C., & Maalej, W. (2025). How Do Programming Students Use Generative AI? Proceedings of the ACM on Software Engineering. https://doi.org/10.1145/3715762

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.