The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index; 15 of the 46 sources its claims cite were checked for support.
How Do Programming Students Use Generative AI?
Open accessAbstract
Exploratory experiment with 37 programming students. Majority prompted ChatGPT to generate full solutions. Students got trapped in a vicious cycle of submitting wrong generated code and asking the bot for a fix. Regular AI users more likely to prompt for full solutions.
Overview
CoChat Summary
TL;DR
This paper combines a GPT task-solving evaluation of an introductory programming course with a monitored experiment in which 37 students could use ChatGPT while solving a code-comprehension exercise. It finds that GPT can solve most course exercises, but students often used the chatbot to request full code solutions, fell into ineffective error-fix cycles, and showed over-reliance rather than asking for explanations.
Key findings
- In the task-solving evaluation, both GPT models produced syntactically correct code in all responses, but GPT-3.5 had more logical errors and failed exercise requirements more than twice as often as GPT-4; the authors conclude a student could not pass the course by relying solely on GenAI because of in-person reviews and secondary assessments.
- In the student experiment, 42 participated and 37 were analyzed; 62.2% completed the entire exercise correctly and 78.4% solved at least one problem. 37.8% did not use the chatbot at all, while among chatbot users the prompt ratio was 62% Codegen to 38% Support, and 73.9% submitted a Solve prompt at some point.
- Manual analysis of code changes with chatbot interactions found 54.3% of submissions semantically identical to generated code (Copy All), 13.6% reused part of the generated solution (Idea), 11.1% copied a language construct (Syntax), 2.5% were based on generated explanation, and 18.5% were unrelated to chatbot interactions.
- Consecutive submissions were significantly less similar when chatbot interactions occurred between them: 64.1% vs 82.2% without chatbot interactions (t = 5.76, p < 0.0001); in 62% of cases with generations, the modified submission was more similar to generated code than to the previous submission. GPT-3.5 generated correct solutions for only 8.8% of 80 P1 solutions, but 76.9% of 13 P2 solutions.
Methods
- Task-solving evaluation: all course assignments were converted into prompts for GPT-3.5 (`gpt-3.5-turbo-1106`) and GPT-4 (`gpt-4-1106-preview`) with temperature 0.0; generated solutions were inserted into project files and evaluated against tutor criteria including syntax, logic, requirements, style, learnability, and intent.
- Student experiment: an in-person controlled study in the SE1 course at University of Hamburg, November–December 2023; participants solved a two-problem Moodle/CodeRunner exercise with optional access to a ChatGPT-like interface using `gpt-3.5-turbo-1106`, while code changes, chatbot conversations, copy/paste events, and submissions were recorded.
- Analysis: quantitative metrics and manual labeling of prompts and responses; code similarity used Ratcliff/Obershelp with a >90% threshold for close match; sequence pattern mining used NOSEP and NETLAP to identify interaction patterns.
Limitations
- Stated: the referenced LLMs may be updated and yield different outputs; prompt phrasing and the entirely German material may affect model performance; participant self-selection may bias the pool toward better-performing students; the sample came from one university course in one year; the experiment setting and recording may have influenced chatbot use and self-reported GenAI use; only one fairly complex exercise was tested.
- Stated: the authors recommend future studies test students’ understanding after completion and use exit surveys or think-aloud observations, because the study did not capture much information on students’ thoughts and impressions during the experiment.
- Evident from the text: the analyzed student sample is small (37 participants), and the findings are tied to a single course, task, and chatbot configuration, limiting generalizability beyond that setting.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 6 Not valid · 20 Not checked · 171 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“This supports prior findings by Mailach et al. [2025] and Kruse et al. [2024] that some students do not understand what information precisely needs to be shared with the chatbot.”
What the sources say
- Not checked — no full text of the sourcecites Mailach et al. [2025] source
- Supportedcites Kruse et al. [2024] source
In C. Code Comprehension (page unavailable), the paper reports that less experienced students initially omitted relevant documentation keywords from their prompts and received less useful explanations. The Abstract (page unavailable) also says students wanted more support in formulating prompts. Together, these support the claim in the context of prompting a chatbot-like tool for code documentation. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Using LLMs for code generation shifts the developer's focus from ideation [Wei et al. 2024] to parsing and debugging unfamiliar code.”
What the sources say
- Partly supportedcites Wei et al. 2024 sourceExperimental — accuracy not yet measured
The UI Code Generation section describes LLM-generated HTML, and the Conclusion identifies UI design as a process involving ideation. But neither passage says that code generation shifts developers’ focus from ideation to parsing and debugging unfamiliar code; the UI Code Adjustment section discusses fixing generated HTML through additional prompts instead. The cited paper supports the code-generation and ideation context, not the claimed shift. (LLM, conf=0.82)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“However, as the results are fairly consistent with similar surveys around the time [Kazemitabaar et al. 2023; Prather et al. 2023a; Zastudil et al. 2023] and we were commonly able to observe behavior that could be considered unethical in an academic context [Prather et al. 2023a], we do not believe this to be a major limitation.”
What the sources say
- Could not verifycites Kazemitabaar et al. 2023 sourceExperimental — accuracy not yet measured
The citation is attached to the citing authors’ assertion that their results are consistent with similar surveys, not directly to their conclusion about social desirability bias. Sections 4.3 and 5.4 describe a post-study survey of learners’ perceptions of a Codex code generator, while Section 5.1.3 reports measured generator use. I found no passage addressing self-reported ChatGPT usage, its consistency with the citing survey, or whether social desirability bias is a major limitation. (LLM, conf=0.75)
- Not matchedcites Prather et al. 2023a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites Zastudil et al. 2023 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Our findings suggest that once learners decide turn to a ChatGPT-like assistant for help, especially if they already have prior experience with generative AI, they predominantly seek code solutions rather than support in creatively working through the problem [Wei et al. 2024].”
What the sources say
- Could not verifycites Wei et al. 2024 sourceExperimental — accuracy not yet measured
The UI Code Generation section describes generating HTML, while Boosting Creativity with AI and the design-process discussion concern designers and practitioner teams. Our copy does not address learners’ help-seeking choices, whether prior generative-AI experience affects them, or whether learners predominantly seek code solutions rather than creative support. Searches for learners, prior experience, and code-solution preferences found no passage on those points; the cited passages have no page numbers. (LLM, conf=0.75)
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All participants had access to the chatbot interface. Still, 37.8% did not use it at all during the experiment.”
- Also stated in: § 3.2 Chatbot Use (RQ2) · plain
“Still, 37.8% did not use it at all during the experiment.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3.2 Chatbot Use (RQ2) · plain
109 more claims
Create a free account to see all 207 claims and every source verdict.
- Borderline: one of the tests for a substantial claim is unclear. 1
- Probably not a substantial claim. 49
- It may state two separate claims as one. 63
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“In total, 177 prompts were sent to the chatbot and 172 responses were received. The remaining 5 prompts went unanswered due to network interruptions. On average, each participant submitted 4.57 prompts (M = 2, 𝜎 = 5.96).”
Reported 4.57 · recomputed 4.783784
Citations
Cites
The Content Analysis Guidebook
Neuendorf, K. A. (2017) · doi:10.4135/9781071802878 · Cited by 9,286 · cited 1×
“The labels were initially created through a deductive approach [Maalej and Robillard 2013; Neuendorf 2017], and then consolidated after reviewing the actual chat logs.”
ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · doi:10.35542/osf.io/5er8f · Cited by 7,552
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326 · cited 2×
“…highlight the risk of learners relying too heavily on the model, noting that “the model simplifies the acquisition of answers or information, which can amplify laziness and counteract the learners’ interest to [...] come to their own conclusions and solutions” [Kasneci et al. 2023, p. 5].”
InfluentialChatting and cheating: Ensuring academic integrity in the era of ChatGPT
Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2024) · Innovations in Education and Teaching International · doi:10.1080/14703297.2023.2190148 · Cited by 3,087
Chatting and Cheating. Ensuring academic integrity in the era of ChatGPT
Cotton, D., Cotton, P., & Shipway, J. R. (2023) · doi:10.35542/osf.io/mrz8h · Cited by 3,079
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 1×
“…as ChatGPT [OpenAI 2022] and GitHub Copilot [GitHub 2022], students in programming courses now have access to code authoring tools capable of solving coding exercises [Berrezueta-Guzman and Krusche 2023; Chen et al. 2021; Savelka et al. 2023] and exam questions entirely [Finnie-Ansley et al. 2023].”
The role of trust in automation reliance
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., & Beck, H. P. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00038-7 · Cited by 1,276
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660 · cited 1×
“Prior work has shown that this is a challenge for professional developers [Sarkar et al. 2022; Vaithilingam et al. 2022].”
How is ChatGPT's behavior changing over time?
Chen, L., Zaharia, M., & Zou, J. (2023) · arXiv · doi:10.48550/arxiv.2307.09009 · Cited by 507 · cited 1×
“5 and GPT-4 over time [Chen et al. 2023], though it is not clear whether these results are based on architectural changes, continuous training, or simply the non-deterministic nature of the LLMs.”
How Is ChatGPT’s Behavior Changing Over Time?
Chen, L., Zaharia, M., & Zou, J. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.5317da47 · Cited by 507
Lee, H.-P. (., Sarkar, A., Tankelevitch, L., Drosos, I., Rintel, S., Banks, R., & Wilson, N. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713778 · Cited by 490 · cited 1×
“Further, GenAI tools can increase developers’ perceived productivity [Ziegler et al. 2024] and level of understanding [Prather et al. 2024] without delivering actual improvements, potentially leading to a false sense of confidence [Lee et al. 2025] among novices.”
The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463
GitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417
GitHub Copilot AI pair programmer: Asset or Liability?
Dakhel, A. M., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., Ming, Z., & Jiang (2022) · arXiv · doi:10.48550/arxiv.2206.15331 · Cited by 417
Becker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569759 · Cited by 401 · cited 1×
“Across multiple surveys and position papers, educators have expressed concerns, particularly about over-reliance on generated content and about the ease of cheating [Becker et al. 2023; Cotton et al. 2024; Prather et al. 2023a; Zastudil et al. 2023].”
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396 · cited 8×
“Across multiple surveys and position papers, educators have expressed concerns, particularly about over-reliance on generated content and about the ease of cheating [Becker et al. 2023; Cotton et al. 2024; Prather et al. 2023a; Zastudil et al. 2023].”
InfluentialStudying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580919 · Cited by 366 · cited 7×
“Furthermore, reliance on code generation can negatively impact code authoring skills [Jošt et al. 2024; Kazemitabaar et al. 2023].”
InfluentialDenny, P., Kumar, V., & Giacaman, N. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569823 · Cited by 287 · cited 1×
“The phrasing of the prompt and problem statement can significantly impact whether the LLM will solve it correctly [Denny et al. 2023].”
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256 · cited 8×
“Across multiple surveys and position papers, educators have expressed concerns, particularly about over-reliance on generated content and about the ease of cheating [Becker et al. 2023; Cotton et al. 2024; Prather et al. 2023a; Zastudil et al. 2023].”
InfluentialLau, S., & Guo, P. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600138 · Cited by 241 · cited 3×
“Two major approaches appear to emerge from this development: either a) embracing GenAI as a tool not only for teaching but also as a core software development tool, or b) preventing and restricting its use [Lau and Guo 2023].”
InfluentialLai, V. D., Ngo, N. T., Veyseh, A. P. B., Man, H., Dernoncourt, F., Bui, T., & Nguyen, T. H. (2023) · arXiv · doi:10.48550/arxiv.2304.05613 · Cited by 233 · cited 1×
“Furthermore, the material used is entirely in German, which may affect the performance of the language models [Lai et al. 2023].”
Lai, V., Ngo, N., Pouran Ben Veyseh, A., Man, H., Dernoncourt, F., Bui, T., & Nguyen, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.878 · Cited by 233
Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · arXiv · doi:10.48550/arxiv.2306.05715 · Cited by 211
Comparing Code Explanations Created by Students and Large Language Models
Leinonen, J., Denny, P., MacNeil, S., Sarsa, S., Bernstein, S., Kim, J., Tran, A., & Hellas, A. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588785 · Cited by 210 · cited 3×
“Leinonen et al. [2023] found that automatically generated code explanations by GPT-3 were rated as both a more accurate and easier to understand than explanations created by students.”
Generating Diverse Code Explanations using the GPT-3 Large Language Model
MacNeil, S., Tran, A., Mogil, D., Bernstein, S., Ross, E., & Huang, Z. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 2 · doi:10.1145/3501709.3544280 · Cited by 200
The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B. (2024) · Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3632620.3671116 · Cited by 196 · cited 3×
“Further, GenAI tools can increase developers’ perceived productivity [Ziegler et al. 2024] and level of understanding [Prather et al. 2024] without delivering actual improvements, potentially leading to a false sense of confidence [Lee et al. 2025] among novices.”
The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B., Kimmel, B., Wright, J., & Briggs, B. (2024) · arXiv · Cited by 196 · cited 3×
“Further, GenAI tools can increase developers’ perceived productivity [Ziegler et al. 2024] and level of understanding [Prather et al. 2024] without delivering actual improvements, potentially leading to a false sense of confidence [Lee et al. 2025] among novices.”
MacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569785 · Cited by 195 · cited 1×
“In a subsequent work, MacNeil et al. [2023] found that the explanations were mostly rated useful by students, both for their own understanding and as a general learning tool [MacNeil et al. 2023].”
MacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J. (2022) · arXiv · doi:10.48550/arxiv.2211.02265 · Cited by 195
My AI Wants to Know if This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises
Finnie-Ansley, J., Denny, P., Luxton-Reilly, A., Santos, E. A., Prather, J., & Becker, B. A. (2023) · Proceedings of the 25th Australasian Computing Education Conference · doi:10.1145/3576123.3576134 · Cited by 186 · cited 1×
“…as ChatGPT [OpenAI 2022] and GitHub Copilot [GitHub 2022], students in programming courses now have access to code authoring tools capable of solving coding exercises [Berrezueta-Guzman and Krusche 2023; Chen et al. 2021; Savelka et al. 2023] and exam questions entirely [Finnie-Ansley et al. 2023].”
Patterns of Knowledge in API Reference Documentation
Maalej, W., & Robillard, M. P. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2013.12 · Cited by 183 · cited 1×
“The labels were initially created through a deductive approach [Maalej and Robillard 2013; Neuendorf 2017], and then consolidated after reviewing the actual chat logs.”
Self-citationOn the Comprehension of Program Comprehension
Maalej, W., Tiarks, R., Roehm, T., & Koschke, R. (2014) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/2622669 · Cited by 180 · cited 1×
“Students had to comprehend and reason about the code [Maalej et al. 2014] to arrive at the solution or to formulate a relevant prompt for getting effective assistance from the bot.”
Self-citationKabir, S., Udo-Imeh, D. N., Kou, B., & Zhang, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642596 · Cited by 160 · cited 2×
“However, such tools also tend to confidently present incorrect information [Cao et al. 2021; Kabir et al. 2024; Tian et al. 2024] or generate subtly incorrect code [Dakhel et al. 2023] which may be difficult to detect for beginners [Popovici 2023; Stanik et al. 2018].”
Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600139 · Cited by 145 · cited 3×
“Prior work found ChatGPT to struggle with identifying issues in code, especially for logic errors [Hellas et al. 2023; Tian et al. 2024].”
The Impact of Large Language Models on Programming Education and Student Learning Outcomes
Jošt, G., Taneski, V., & Karakatič, S. (2024) · Applied Sciences · doi:10.3390/app14104115 · Cited by 143 · cited 2×
“Furthermore, reliance on code generation can negatively impact code authoring skills [Jošt et al. 2024; Kazemitabaar et al. 2023].”
Lobb, R., & Harlow, J. (2016) · ACM Inroads · doi:10.1145/2810041 · Cited by 142 · cited 1×
“The assessments also include code authoring questions using the Moodle CodeRunner [Lobb and Harlow 2016] plugin.”
Savelka, J., Agarwal, A., An, M., Bogart, C., & Sakr, M. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600142 · Cited by 141 · cited 1×
“…as ChatGPT [OpenAI 2022] and GitHub Copilot [GitHub 2022], students in programming courses now have access to code authoring tools capable of solving coding exercises [Berrezueta-Guzman and Krusche 2023; Chen et al. 2021; Savelka et al. 2023] and exam questions entirely [Finnie-Ansley et al. 2023].”
Generative AI in Computing Education: Perspectives of Students and Instructors
Zastudil, C., Rogalska, M., Kapp, C., Vaughn, J., & MacNeil, S. (2023) · 2023 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10343467 · Cited by 136 · cited 6×
“These concerns have been echoed by a smaller interview study by Zastudil et al. [2023], where all instructors and a majority of the students reported anticipating an increase in academic dishonesty due to GenAI tools.”
InfluentialMeasuring GitHub Copilot's Impact on Productivity
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2024) · Communications of the ACM · doi:10.1145/3633453 · Cited by 124 · cited 1×
“Further, GenAI tools can increase developers’ perceived productivity [Ziegler et al. 2024] and level of understanding [Prather et al. 2024] without delivering actual improvements, potentially leading to a false sense of confidence [Lee et al. 2025] among novices.”
NOSEP: Nonoverlapping Sequence Pattern Mining With Gap Constraints
Wu, Y., Tong, Y., Zhu, X., et al. (2017) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2017.2750691 · Cited by 99 · cited 2×
“For this analysis, we used the NOSEP algorithm by Wu et al. [2017b], as we were looking for consecutive, non-overlapping subsequences that may occur multiple times in one sequence.”
Does ChatGPT Help With Introductory Programming?An Experiment of Students Using ChatGPT in CS1
Xue, Y., Chen, H., Bai, G. R., Tairas, R., & Huang, Y. (2024) · Proceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training · doi:10.1145/3639474.3640076 · Cited by 96 · cited 1×
“Initial studies have shown conflicting results regarding the impact of GenAI on learners, with some reporting neutral or slightly positive changes [Kazemitabaar et al. 2023; Vadaparty et al. 2024; Xue et al. 2024], while others even found adverse effects [Jošt et al. 2024; Prather et al. 2024].”
CS1-LLM: Integrating LLMs into CS1 Instruction
Vadaparty, A., Zingaro, D., Smith IV, D. H., Padala, M., Alvarado, C., Gorson Benario, J., & Porter, L. (2024) · Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3649217.3653584 · Cited by 77 · cited 1×
“Initial studies have shown conflicting results regarding the impact of GenAI on learners, with some reporting neutral or slightly positive changes [Kazemitabaar et al. 2023; Vadaparty et al. 2024; Xue et al. 2024], while others even found adverse effects [Jošt et al. 2024; Prather et al. 2024].”
ChatGPT in the Classroom. Exploring Its Potential and Limitations in a Functional Programming Course
Popovici, M.-D. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2023.2269006 · Cited by 70 · cited 10×
“Furthermore, reliance on code generation can negatively impact code authoring skills [Jošt et al. 2024; Kazemitabaar et al. 2023].”
InfluentialDevGPT: Studying Developer-ChatGPT Conversations
Xiao, T., Treude, C., Hata, H., & Matsumoto, K. (2024) · Proceedings of the 21st International Conference on Mining Software Repositories · doi:10.1145/3643991.3648400 · Cited by 56
What is it like to program with artificial intelligence?
Sarkar, A., Gordon, A. D., Negreanu, C., Poelitz, C., Ragavan, S. S., & Zorn, B. (2022) · arXiv · doi:10.48550/arxiv.2208.06213 · Cited by 47 · cited 1×
“Prior work has shown that this is a challenge for professional developers [Sarkar et al. 2022; Vaithilingam et al. 2022].”
Strict pattern matching under non-overlapping condition
Wu, Y., Shen, C., Jiang, H., et al. (2016) · Science China Information Sciences · doi:10.1007/s11432-015-0935-3 · Cited by 37
A Simple NLP-Based Approach to Support Onboarding and Retention in Open Source Communities
Stanik, C., Montgomery, L., Martens, D., Fucci, D., & Maalej, W. (2018) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2018.00027 · Cited by 36
Self-citationBecker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2022) · arXiv · doi:10.48550/arxiv.2212.01020 · Cited by 36
Recommendations to Create Programming Exercises to Overcome ChatGPT
Berrezueta-Guzman, J., & Krusche, S. (2023) · 2023 IEEE 35th International Conference on Software Engineering Education and Training (CSEE&T) · doi:10.1109/cseet58097.2023.00031 · Cited by 27
Can ChatGPT Support Developers? An Empirical Evaluation of Large Language Models for Code Generation
Jin, K., Wang, C.-Y., Pham, H. V., & Hemmati, H. (2024) · Proceedings of the 21st International Conference on Mining Software Repositories · doi:10.1145/3643991.3645074 · Cited by 24
Mailach, A., Gorgosch, D., Siegmund, N., & Siegmund, J. (2025) · Empirical Software Engineering · doi:10.1007/s10664-024-10561-6 · Cited by 13 · cited 3×
“This supports prior findings by Mailach et al. [2025] and Kruse et al. [2024] that some students do not understand what information precisely needs to be shared with the chatbot.”
Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases
Cao, B., Lin, H., Han, X., Sun, L., Yan, L., Liao, M., Xue, T., & Xu, J. (2021) · arXiv · doi:10.48550/arxiv.2106.09231 · Cited by 10 · cited 1×
“However, such tools also tend to confidently present incorrect information [Cao et al. 2021; Kabir et al. 2024; Tian et al. 2024] or generate subtly incorrect code [Dakhel et al. 2023] which may be difficult to detect for beginners [Popovici 2023; Stanik et al. 2018].”
Can Developers Prompt? A Controlled Experiment for Code Documentation Generation
Kruse, H.-A., Puhlfürß, T., & Maalej, W. (2024) · 2024 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme58944.2024.00058 · Cited by 8 · cited 2×
“Exit surveys or think-aloud observations can be used to gain more insight into student sentiment while using GenAI tools for programming [Kruse et al. 2024; Prather et al. 2024].”
Self-citationPham, Y. D., Bouraffa, A., Hillen, M., & Maalej, W. (2021) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re51729.2021.00018 · Cited by 6 · cited 1×
“Alternatively, different chatbot behaviors could encourage students to think through the problem themselves or “nudge” them towards finding the solution [Pham et al. 2021], potentially even beyond the model’s inherent reasoning capabilities.”
Self-citationDevGPT: Studying Developer Conversations with ChatGPT
Xiao, T., Treude, C., Hata, H., & Matsumoto, K. (2023) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.8242141 · Cited by 4
DebugBench: Evaluating Debugging Capability of Large Language Models
Tian, R., Ye, Y., Qin, Y., Cong, X., Lin, Y., Pan, Y., Wu, Y., Hui, H., Liu, W., Liu, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.04621 · Cited by 2
Wei, J., Courbis, A.-L., Lambolais, T., Dray, G., & Maalej, W. (2024) · arXiv · doi:10.48550/arxiv.2406.13631 · Cited by 1
Self-citation"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2023) · arXiv · cited 8×
“Across multiple surveys and position papers, educators have expressed concerns, particularly about over-reliance on generated content and about the ease of cheating [Becker et al. 2023; Cotton et al. 2024; Prather et al. 2023a; Zastudil et al. 2023].”
Influential
Research works printed but not linked: 3
- David J Barnes and Michael Kölling. 2009. Java lernen mit BlueJ: Eine Einführung in die objektorientierte Programmierung. Pearson Deutschland GmbH.
- Marlo Häring and Walid Maalej. 2019. A Socio-Technical Framework for Face-to-Face Teaching in Large Software Development Courses.. In Software Engineering (Workshops). 3–6.
- John W. Ratcliff and DM Metzener. 1988. Gestalt: an introduction to the Ratcliff/Obershelp pattern matching algorithm. Dr. Dobbs Journal 7 (1988), 46.
Cited by
Reihanian, I., Hou, Y., & Sun, Q. (2025) · AI · doi:10.3390/ai7010006 · Cited by 15
Stoyanova, D., Stoyanova-Petrova, S. V., Shotarova, S., Lyubomirov, S. Y., & Mileva, N. S. (2025) · Education Sciences · doi:10.3390/educsci16010019 · Cited by 13
``I Would Have Written My Code Differently': Beginners Struggle to Understand LLM-Generated Code
Zi, Y., Li, L., Guha, A., Anderson, C. J., & Feldman, M. Q. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3731663 · Cited by 33
Examining the Usage of Generative AI Models in Student Learning Activities for Software Programming
Chen, R., Jiang, S., Shen, J., Moon, A., & Wei, L. (2025) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1109/aiware69974.2025.00016 · Cited by 3
Rizvi, S. W. F., Yankouskaya, A., Alshakhsi, S. A. A., Al‐Thani, D., & Ali, R. (2025) · International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm67465.2025.11391082 · Cited by 2
“In programming context, an experimental study showed concerns of students who regularly use AI chatbot, highlighting potential overreliance and reduced engagement in problem-solving tasks [18].”
Metadata
- Authors
- Rahe, C., & Maalej, W.
- Affiliations
- Universität Hamburg (DE)
- Venue
- Proceedings of the ACM on Software Engineering
- Published
- 2025-06-19
- Volume
- 2
- Issue
- FSE
- Pages
- 978-1000
- ISSN
- 2994-970X (ISSN-L)
- OpenAlex
- W4406604115
- Cited by
- 45
- References
- 43
- Citations by year
- 2026: 33 · 2025: 12
- FWCI
- 39.98
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the ACM on software engineering. (publishedVersion, cc-by-nc)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- generative AI, ChatGPT, programming education, student use of AI, overreliance on AI, critical thinking, program comprehension, code reasoning, student learning behavior
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.