The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index; 15 of the 46 sources its claims cite were checked for support.
How Do Programming Students Use Generative AI?
Open accessAbstract
Exploratory experiment with 37 programming students. Majority prompted ChatGPT to generate full solutions. Students got trapped in a vicious cycle of submitting wrong generated code and asking the bot for a fix. Regular AI users more likely to prompt for full solutions.
Overview
CoChat Summary
TL;DR
This paper combines a GPT task-solving evaluation of an introductory programming course with a monitored experiment in which 37 students could use ChatGPT while solving a code-comprehension exercise. It finds that GPT can solve most course exercises, but students often used the chatbot to request full code solutions, fell into ineffective error-fix cycles, and showed over-reliance rather than asking for explanations.
Key findings
- In the task-solving evaluation, both GPT models produced syntactically correct code in all responses, but GPT-3.5 had more logical errors and failed exercise requirements more than twice as often as GPT-4; the authors conclude a student could not pass the course by relying solely on GenAI because of in-person reviews and secondary assessments.
- In the student experiment, 42 participated and 37 were analyzed; 62.2% completed the entire exercise correctly and 78.4% solved at least one problem. 37.8% did not use the chatbot at all, while among chatbot users the prompt ratio was 62% Codegen to 38% Support, and 73.9% submitted a Solve prompt at some point.
- Manual analysis of code changes with chatbot interactions found 54.3% of submissions semantically identical to generated code (Copy All), 13.6% reused part of the generated solution (Idea), 11.1% copied a language construct (Syntax), 2.5% were based on generated explanation, and 18.5% were unrelated to chatbot interactions.
- Consecutive submissions were significantly less similar when chatbot interactions occurred between them: 64.1% vs 82.2% without chatbot interactions (t = 5.76, p < 0.0001); in 62% of cases with generations, the modified submission was more similar to generated code than to the previous submission. GPT-3.5 generated correct solutions for only 8.8% of 80 P1 solutions, but 76.9% of 13 P2 solutions.
Methods
- Task-solving evaluation: all course assignments were converted into prompts for GPT-3.5 (`gpt-3.5-turbo-1106`) and GPT-4 (`gpt-4-1106-preview`) with temperature 0.0; generated solutions were inserted into project files and evaluated against tutor criteria including syntax, logic, requirements, style, learnability, and intent.
- Student experiment: an in-person controlled study in the SE1 course at University of Hamburg, November–December 2023; participants solved a two-problem Moodle/CodeRunner exercise with optional access to a ChatGPT-like interface using `gpt-3.5-turbo-1106`, while code changes, chatbot conversations, copy/paste events, and submissions were recorded.
- Analysis: quantitative metrics and manual labeling of prompts and responses; code similarity used Ratcliff/Obershelp with a >90% threshold for close match; sequence pattern mining used NOSEP and NETLAP to identify interaction patterns.
Limitations
- Stated: the referenced LLMs may be updated and yield different outputs; prompt phrasing and the entirely German material may affect model performance; participant self-selection may bias the pool toward better-performing students; the sample came from one university course in one year; the experiment setting and recording may have influenced chatbot use and self-reported GenAI use; only one fairly complex exercise was tested.
- Stated: the authors recommend future studies test students’ understanding after completion and use exit surveys or think-aloud observations, because the study did not capture much information on students’ thoughts and impressions during the experiment.
- Evident from the text: the analyzed student sample is small (37 participants), and the findings are tied to a single course, task, and chatbot configuration, limiting generalizability beyond that setting.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 6 Not valid · 20 Not checked · 171 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“This supports prior findings by Mailach et al. [2025] and Kruse et al. [2024] that some students do not understand what information precisely needs to be shared with the chatbot.”
What the sources say
- Not checked — no full text of the sourcecites Mailach et al. [2025] source
- Supportedcites Kruse et al. [2024] source
In C. Code Comprehension (page unavailable), the paper reports that less experienced students initially omitted relevant documentation keywords from their prompts and received less useful explanations. The Abstract (page unavailable) also says students wanted more support in formulating prompts. Together, these support the claim in the context of prompting a chatbot-like tool for code documentation. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Using LLMs for code generation shifts the developer's focus from ideation [Wei et al. 2024] to parsing and debugging unfamiliar code.”
What the sources say
- Partly supportedcites Wei et al. 2024 sourceExperimental — accuracy not yet measured
The UI Code Generation section describes LLM-generated HTML, and the Conclusion identifies UI design as a process involving ideation. But neither passage says that code generation shifts developers’ focus from ideation to parsing and debugging unfamiliar code; the UI Code Adjustment section discusses fixing generated HTML through additional prompts instead. The cited paper supports the code-generation and ideation context, not the claimed shift. (LLM, conf=0.82)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“However, as the results are fairly consistent with similar surveys around the time [Kazemitabaar et al. 2023; Prather et al. 2023a; Zastudil et al. 2023] and we were commonly able to observe behavior that could be considered unethical in an academic context [Prather et al. 2023a], we do not believe this to be a major limitation.”
What the sources say
- Could not verifycites Kazemitabaar et al. 2023 sourceExperimental — accuracy not yet measured
The citation is attached to the citing authors’ assertion that their results are consistent with similar surveys, not directly to their conclusion about social desirability bias. Sections 4.3 and 5.4 describe a post-study survey of learners’ perceptions of a Codex code generator, while Section 5.1.3 reports measured generator use. I found no passage addressing self-reported ChatGPT usage, its consistency with the citing survey, or whether social desirability bias is a major limitation. (LLM, conf=0.75)
- Not matchedcites Prather et al. 2023a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites Zastudil et al. 2023 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Our findings suggest that once learners decide turn to a ChatGPT-like assistant for help, especially if they already have prior experience with generative AI, they predominantly seek code solutions rather than support in creatively working through the problem [Wei et al. 2024].”
What the sources say
- Could not verifycites Wei et al. 2024 sourceExperimental — accuracy not yet measured
The UI Code Generation section describes generating HTML, while Boosting Creativity with AI and the design-process discussion concern designers and practitioner teams. Our copy does not address learners’ help-seeking choices, whether prior generative-AI experience affects them, or whether learners predominantly seek code solutions rather than creative support. Searches for learners, prior experience, and code-solution preferences found no passage on those points; the cited passages have no page numbers. (LLM, conf=0.75)
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All participants had access to the chatbot interface. Still, 37.8% did not use it at all during the experiment.”
- Also stated in: § 3.2 Chatbot Use (RQ2) · plain
“Still, 37.8% did not use it at all during the experiment.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3.2 Chatbot Use (RQ2) · plain
109 more claims
Create a free account to see all 207 claims and every source verdict.
- Borderline: one of the tests for a substantial claim is unclear. 1
- Probably not a substantial claim. 49
- It may state two separate claims as one. 63
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“In total, 177 prompts were sent to the chatbot and 172 responses were received. The remaining 5 prompts went unanswered due to network interruptions. On average, each participant submitted 4.57 prompts (M = 2, 𝜎 = 5.96).”
Reported 4.57 · recomputed 4.783784
Citations
Cites
The Content Analysis Guidebook
Neuendorf, K. A. (2017) · doi:10.4135/9781071802878 · Cited by 9,286 · cited 1×
“The labels were initially created through a deductive approach [Maalej and Robillard 2013; Neuendorf 2017], and then consolidated after reviewing the actual chat logs.”
ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · doi:10.35542/osf.io/5er8f · Cited by 7,552
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326 · cited 2×
“…highlight the risk of learners relying too heavily on the model, noting that “the model simplifies the acquisition of answers or information, which can amplify laziness and counteract the learners’ interest to [...] come to their own conclusions and solutions” [Kasneci et al. 2023, p. 5].”
InfluentialChatting and cheating: Ensuring academic integrity in the era of ChatGPT
Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2024) · Innovations in Education and Teaching International · doi:10.1080/14703297.2023.2190148 · Cited by 3,087
Chatting and Cheating. Ensuring academic integrity in the era of ChatGPT
Cotton, D., Cotton, P., & Shipway, J. R. (2023) · doi:10.35542/osf.io/mrz8h · Cited by 3,079
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 1×
“…as ChatGPT [OpenAI 2022] and GitHub Copilot [GitHub 2022], students in programming courses now have access to code authoring tools capable of solving coding exercises [Berrezueta-Guzman and Krusche 2023; Chen et al. 2021; Savelka et al. 2023] and exam questions entirely [Finnie-Ansley et al. 2023].”
The role of trust in automation reliance
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., & Beck, H. P. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00038-7 · Cited by 1,276
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660 · cited 1×
“Prior work has shown that this is a challenge for professional developers [Sarkar et al. 2022; Vaithilingam et al. 2022].”
How is ChatGPT's behavior changing over time?
Chen, L., Zaharia, M., & Zou, J. (2023) · arXiv · doi:10.48550/arxiv.2307.09009 · Cited by 507 · cited 1×
“5 and GPT-4 over time [Chen et al. 2023], though it is not clear whether these results are based on architectural changes, continuous training, or simply the non-deterministic nature of the LLMs.”
How Is ChatGPT’s Behavior Changing Over Time?
Chen, L., Zaharia, M., & Zou, J. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.5317da47 · Cited by 507
Research works printed but not linked: 3
- David J Barnes and Michael Kölling. 2009. Java lernen mit BlueJ: Eine Einführung in die objektorientierte Programmierung. Pearson Deutschland GmbH.
- Marlo Häring and Walid Maalej. 2019. A Socio-Technical Framework for Face-to-Face Teaching in Large Software Development Courses.. In Software Engineering (Workshops). 3–6.
- John W. Ratcliff and DM Metzener. 1988. Gestalt: an introduction to the Ratcliff/Obershelp pattern matching algorithm. Dr. Dobbs Journal 7 (1988), 46.
Cited by
Reihanian, I., Hou, Y., & Sun, Q. (2025) · AI · doi:10.3390/ai7010006 · Cited by 15
Stoyanova, D., Stoyanova-Petrova, S. V., Shotarova, S., Lyubomirov, S. Y., & Mileva, N. S. (2025) · Education Sciences · doi:10.3390/educsci16010019 · Cited by 13
``I Would Have Written My Code Differently': Beginners Struggle to Understand LLM-Generated Code
Zi, Y., Li, L., Guha, A., Anderson, C. J., & Feldman, M. Q. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3731663 · Cited by 33
Examining the Usage of Generative AI Models in Student Learning Activities for Software Programming
Chen, R., Jiang, S., Shen, J., Moon, A., & Wei, L. (2025) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1109/aiware69974.2025.00016 · Cited by 3
Rizvi, S. W. F., Yankouskaya, A., Alshakhsi, S. A. A., Al‐Thani, D., & Ali, R. (2025) · International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm67465.2025.11391082 · Cited by 2
“In programming context, an experimental study showed concerns of students who regularly use AI chatbot, highlighting potential overreliance and reduced engagement in problem-solving tasks [18].”
Metadata
- Authors
- Rahe, C., & Maalej, W.
- Affiliations
- Universität Hamburg (DE)
- Venue
- Proceedings of the ACM on Software Engineering
- Published
- 2025-06-19
- Volume
- 2
- Issue
- FSE
- Pages
- 978-1000
- ISSN
- 2994-970X (ISSN-L)
- OpenAlex
- W4406604115
- Cited by
- 45
- References
- 43
- Citations by year
- 2026: 33 · 2025: 12
- FWCI
- 39.98
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the ACM on software engineering. (publishedVersion, cc-by-nc)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- generative AI, ChatGPT, programming education, student use of AI, overreliance on AI, critical thinking, program comprehension, code reasoning, student learning behavior
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: How Do Programming Students Use Generative AI? https://app.cochat.ai/papers/10.1145/3715762
- The paper
- Rahe, C., & Maalej, W. (2025). How Do Programming Students Use Generative AI? Proceedings of the ACM on Software Engineering. https://doi.org/10.1145/3715762
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.