The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
How Novice Programmers Use and Experience ChatGPT when Solving Programming Exercises in an Introductory Course
Open accessAbstract
This research paper contributes to the computing education research community's understanding of Generative AI (GenAI) in the context of introductory programming, and specifically, how students utilize related tools, such as ChatGPT. An increased understanding of students' use is mandatory for educators and higher education institutions, as GenAI is here to stay, and its performance is likely to improve rapidly in the near future. Learning about students' use patterns is not only crucial to support their learning, but to develop adequate forms of instruction and assessment. With the rapid advancement of AI, its broad availability, and ubiquitous presence in educational environments, elaborating how AI can enhance learning experiences, especially in courses such as introductory programming is important. To date, most studies have focused on the educator's perspective on GenAI, its performance, characteristics, and limitations. However, the student perspective, and how they actually use GenAI tools in course contexts, has not been subject to a great number of studies. Therefore, this study is guided by the following research questions: (1) What do students report on their use pattern of ChatGPT in the context of introductory programming exercises? and (2) How do students perceive ChatGPT in the context of introductory programming exercises? To address these questions, computing students at a large German university were asked to solve programming tasks with the assistance of ChatGPT as part of their introductory programming course. Students (n=298) provided information regarding the use of ChatGPT, and their evaluation of the tool via an online survey. This research provides a comprehensive evaluation of ChatGPT-3.5's application by novice programmers in a higher education context. The findings reveal that while students widely adopt GenAI, their use varies significantly, ranging from acceptance of generated solutions to dynamic, and critical engagement. Therefore, this work has implications for educators designing guardrails or forms of instructions on the use of GenAI tools in the classroom.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Students use GenAI for a variety of tasks, including content generation, problem-solving, debugging, conceptual understanding, and exam preparation.
“Students use GenAI for a variety of tasks, including content generation, problem-solving, debugging, conceptual understanding, and exam preparation [25, 308].”
From Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education (2025) · cites this paper as [308]
The Results section, under “Students’ reports on their use patterns of ChatGPT (RQ1)” (page not supplied), reports code generation, debugging, problem understanding, and conceptual understanding among students’ uses. The study concerns introductory programming exercises; our copy does not report students using GenAI for exam preparation. Its mention of exams concerns tool performance or course grades, not that use. (LLM, conf=0.89)
Experimental — accuracy not yet measuredStudents are aware of problems associated with the use of GenAI, such as hallucination, overconfidence, and over-reliance.
“Students are also aware of problems associated with the use of GenAI, such as hallucination, overconfidence, and over-reliance [308].”
From Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education (2025) · cites this paper as [308]
CS students are concerned about privacy when using GenAI tools.
“CS students are concerned about privacy when using GenAI tools [308, 324].”
From Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education (2025) · cites this paper as [308]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ChatGPT User Experience: Implications for Education
Zhai, X. (2023) · doi:10.2139/ssrn.4312418 · Cited by 935 · cited 1×
“As GenAI tools challenge the integrity of academic work, educational institutions and faculty need to take action on several levels [3], [33].”
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436 · cited 3×
“Barke et al. [44] further conclude that over-reliance on Copilot can hinder task completion, and that multiple suggestions can lead to a high cognitive load.”
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396
Self-citationTeaching CS50 with AI: Leveraging Generative Artificial Intelligence in Computer Science Education
Liu, R., Zenke, C., Liu, C., Holmes, A., Thornton, P., & Malan, D. J. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630938 · Cited by 185
Generative AI in Computing Education: Perspectives of Students and Instructors
Zastudil, C., Rogalska, M., Kapp, C., Vaughn, J., & MacNeil, S. (2023) · 2023 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10343467 · Cited by 136
Does ChatGPT Help With Introductory Programming?An Experiment of Students Using ChatGPT in CS1
Xue, Y., Chen, H., Bai, G. R., Tairas, R., & Huang, Y. (2024) · Proceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training · doi:10.1145/3639474.3640076 · Cited by 96
Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Code
Savelka, J., Agarwal, A., Bogart, C., & Sakr, M. (2023) · arXiv · doi:10.48550/arxiv.2303.08033 · Cited by 56
Towards Giving Timely Formative Feedback and Hints to Novice Programmers
Jeuring, J., Keuning, H., Marwan, S., Bouvier, D., Izu, C., Kiesler, N., Lehtinen, T., Lohr, D., Peterson, A., & Sarsa, S. (2022) · Proceedings of the 2022 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3571785.3574124 · Cited by 40
Self-citationGrounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23 · cited 3×
“Barke et al. [44] further conclude that over-reliance on Copilot can hinder task completion, and that multiple suggestions can lead to a high cognitive load.”
Modeling Programming Competency: A Qualitative Analysis
Kiesler, N. (2024) · doi:10.1007/978-3-031-47148-3 · Cited by 7 · cited 1×
“Even though it is challenging to predict which programming competencies [4] students will require in the future and how to adapt curricula respectively [5], [6], computing educators need to gain an understanding of how students currently apply GenAI tools.”
Self-citation"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2023) · arXiv · cited 2×
“In a study by Prather et al. [40], students’ use of GitHub Copilot as part of an introductory programming assignment was evaluated through observations and interviews.”
Flagged first, then by citations
Research works printed but not linked: 42
- Amani, 2023, Generative AI Perceptions: A Survey to Measure the Perceptions of Faculty, Staff, and Students on Generative AI Tools in Academia
- Raman, 2023, University students as early adopters of ChatGPT: Innovation Diffusion Study
- Becker, 2023, Generative AI in Introductory Programming
- Kiesler, 2023, Beyond the Textbook: Rethinking Students’ Competencies in the LLM Era, Generative AI: Implications for Teaching and Learning, Uppsala, Sweden
- Geng, 2023, Can ChatGPT Pass An Introductory Level Functional Language Programming Course?
- Kiesler, 2023, Large Language Models in Introductory Programming Education: ChatGPT’s Performance and Implications for Assessments
- 10.5220/0011996900003470 · DOI
- 10.1145/3545945.3569770 · DOI · Paper page
- 10.1145/3501385.3543957 · DOI · Paper page
- 10.1145/3545945.3569785 · DOI · Paper page
- 10.1145/3587102.3588785 · DOI · Paper page
- Bengtsson, 2023, Assessment Accuracy of a Large Language Model on Programming Assignments
- 10.1109/FIE58773.2023.10343457 · DOI
- 10.3991/ijep.v13i8.45621 · DOI
- 10.1145/3649217.3653594 · DOI
- 10.1145/3636243.3636259 · DOI
- 10.1109/IEEECONF56852.2023.10104622 · DOI
- 10.1145/3626252.3630799 · DOI
- 10.1145/3626252.3630803 · DOI
- 10.1145/3649405.3659534 · DOI
- 10.1145/3626252.3630842 · DOI · Paper page
- 10.1145/3626252.3630784 · DOI · Paper page
- 10.1145/3649217.3653624 · DOI
- 10.1145/3626253.3635369 · DOI
- 10.1145/3576123.3576134 · DOI · Paper page
- Parlante, 2024, CodingBat
- 10.1016/j.iotcps.2023.06.002 · DOI
- Phung, 2023, Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models
- Zhang, 2022, Repairing Bugs in Python Assignments Using Large Language Models, arXiv preprint
- Narciss, 2006, Informatives Tutorielles Feedback: Entwicklungs- und Evaluationsprinzipien auf der Basis instruktionspsychologischer Erkenntnisse, Munster: Waxmann Verlag
- 10.1145/3626252.3630822 · DOI
- 10.1145/3613904.3642773 · DOI · Paper page
- 10.1145/3526113.3545659 · DOI · Paper page
- Denny, 2023, Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code Generators
- 10.59287/ijanser.562 · DOI
- 2023, Prompt Engineering
- Mayring, 2000, Qualitative content analysis, Forum: qualitative social research, 1
- Ballstaedt, 1981, Texte verstehen, Texte gestalten. Munchen: Urban u. Schwarzenberg
- Mandl, 1981, Zur Psychologie der Textverarbeitung: Ansatze, Befunde, Probleme
- Scholl, 2024, Data: Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT
- Roethlisberger, 1939, Management and the Worker
- Scholl, 2024, Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT
Cited by
Sorted by citations
Shihab, M. I. H., Hundhausen, C., Tariq, A., Haque, S., Qiao, Y., & Mulanda, B. W. (2025) · Proceedings of the 2025 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3702652.3744219 · Cited by 28
“They discovered two novel patterns: novice programmers often (a) type code that matches Copilot’s suggestions, but end up not accepting those suggestions ( shepherding ); and (b) accept incorrect Copilot-generated code, leading to debugging rabbit holes that stray further from the correct solutions…”
Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education
Mak, J., Nakatumba-Nabende, J., Clear, T., Clear, A., Abluwi, I., Andrei, O., Angeli, L., MacNeil, S., Oyelere, S. S., Hale Rattigan, M., et al. (2025) · Proceedings of the 2025 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3760545.3783975 · Cited by 12
Metadata
- Authors
- Scholl, A., & Kiesler, N.
- Affiliations
- Computer Science Nuremberg Tech,Nuremberg,Germany
- Venue
- 2024 IEEE Frontiers in Education Conference (FIE)
- Publisher
- IEEE
- Published
- 2024-10-13
- Pages
- 1-9
- ISSN
- 0190-5848 (ISSN-L)
- OpenAlex
- W4407949461
- Cited by
- 35
- References
- 13
- Citations by year
- 2026: 10 · 2025: 24 · 2024: 1
- FWCI
- 15.35
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Frontiers in Education Conference (FIE) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- ChatGPT, generative AI in education, introductory programming, novice programmers, student perceptions of AI, programming exercises, AI-assisted learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
