Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses
Open accessThe DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index; 2 of the 29 sources its claims cite were checked for support.
Abstract
This paper studies recent developments in large language models’ (LLM) abilities to pass assessments in introductory and intermediate Python programming courses at the postsecondary level. The emergence of ChatGPT resulted in heated debates of its potential uses (e.g., exercise generation, code explanation) as well as misuses in programming classes (e.g., cheating). Recent studies show that while the technology performs surprisingly well on diverse sets of assessment instruments employed in typical programming classes the performance is usually not sufficient to pass the courses. The release of GPT-4 largely emphasized notable improvements in the capabilities related to handling assessments originally designed for human test-takers. This study is the necessary analysis in the context of this ongoing transition towards mature generative AI systems. Specifically, we report the performance of GPT-4, comparing it to the previous generations of GPT models, on three Python courses with assessments ranging from simple multiple-choice questions (no code involved) to complex programming projects with code bases distributed into multiple files (599 exercises overall). Additionally, we analyze the assessments that were not handled well by GPT-4 to understand the current limitations of the model, as well as its capabilities to leverage feedback provided by an auto-grader. We found that the GPT models evolved from completely failing the typical programming class’ assessments (the original GPT-3) to confidently passing the courses with no human involvement (GPT-4). While we identified certain limitations in GPT-4’s handling of MCQs and coding exercises, the rate of improvement across the recent generations of GPT models strongly suggests their potential to handle almost any type of assessment widely used in higher education programming courses. These findings could be leveraged by educators and institutions to adapt the design of programming assessments as well as to fuel the necessary discussions into how programming classes should be updated to reflect the recent technological developments. This study provides evidence that programming instructors need to prepare for a world in which there is an easy-to-use widely accessible technology that can be utilized by learners to collect passing scores, with no effort whatsoever, on what today counts as viable programming knowledge and skills assessments.
Overview
CoChat Summary
TL;DR
This study evaluates GPT-4 against earlier GPT models on 599 assessments from three postsecondary Python courses. GPT-4 passed the MCQ assessments and, after auto-grader feedback, reached scores likely sufficient to pass the coding projects, unlike GPT-3 and GPT-3.5.
Key findings
- On 530 MCQs, GPT-4 answered 84.1% correctly (446/530), compared with 64.3% (341/530) for GPT-3.5 and 37.5% (199/530) for GPT-3. GPT-4 passed all PE1 and PE2 graded tests, with PE1 overall 85% and PE2 overall 89.6%; on PPP tests it scored 79.0% (64/81) in Table 5.
- On 69 coding activities, GPT-4 scored 545/760 (71.7%) after first submission versus 407/760 (53.6%) for GPT-3.5. After auto-grader feedback, GPT-4 improved to 634/760 (83.4%), while GPT-3.5 improved to 505/760 (66.4%).
- GPT-4 still performed worse on MCQs with code (81.0%) than without code (90.7%). Of 67 incorrect code MCQs, the paper categorized errors as MULTI-HOP (19), INCOHERENT (14), WRONG-Q (13), DUAL-INTERPRETATION (13), and FLUKE (8).
- GPT-4 struggled on some coding activities: after 34 feedback iterations on project 6 activities (debugging, refactoring, testing, packaging), it obtained only 42 of 95 points.
Methods
- Data: 599 assessments from three real-world Python courses—PE1 (149 MCQs), PE2 (148 MCQs), and PPP (233 MCQs plus 69 coding activities from 32 project tasks), totaling 530 MCQs and 69 coding activities.
- Models: GPT-4 was compared with GPT-3.5 (`text-davinci-003`) and GPT-3 (`text-davinci-001`) on MCQs; coding activities benchmarked GPT-4 against GPT-3.5 only. Temperature was 0.0, `max_tokens` 500 for MCQs and 2,000/4,000 for coding, `top_p` 1, and penalties 0.
- Procedure: MCQs were submitted via prompt templates and compared to reference answers. Coding solutions were submitted to an auto-grader; feedback was appended for iterative resubmission until full score or no-change impasse, with no human intervention.
Limitations
- The paper states that prompt engineering was not explored, so results should be interpreted as a lower bound; findings may not generalize beyond Python, English-language assessments, or the assessment types studied; training data for the models is unknown, raising possible memorization concerns; and GPT-4 technical details are undisclosed, limiting reproducibility.
- The text also makes evident that GPT-4 still struggled with tasks requiring external tools or artifacts (e.g., refactoring, packaging, debugger, `pylint`, `pycodestyle`, `pytest`), fine-grained formatting requirements, and input data exceeding the model’s prompt length.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
1 Valid · 0 Not valid · 17 Not checked · 92 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Hence, the strengths and limitations reported in prior work [7, 10, 11, 38, 46] still hold to a certain degree.”
What the sources say
- Not checked yetcites [7] [7] Paul Denny, Viraj Kumar, and Nasser Giacaman. 2023. Conversing with Copilot: Exploring prompt engineering for solving CS1 problems using natural language. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 . 1136-1142. - source
- Supportedcites [10] [10] James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Australasian Computing Education Conference (Virtual Event, Australia) (ACE '22) . Association for Computing Machinery, New York, NY, USA, 10-19. https://doi.org/10.1145/3511861.3511863 - source
Our copy of Finnie-Ansley et al. (2022) does report both sides of Codex in introductory programming: Section 5.1 'Opportunities' sets out strengths (generated solutions as models students can learn from or check their work against, variation as fodder for code review and refactoring), while Section 5.2 'Challenges' and Section 5.3 'Academic Integrity' set out limitations (over-reliance, poor-style/incorrect solutions, hard-to-detect plagiarism, need for invigilated assessment), and Section 7 'Conclusions' states 'the current tool has clear limitations when given tight constraints'. The citing sentence's hedge that these 'still hold to a certain degree' is the citing authors' own framing, not an attributed finding, so nothing beyond the report of strengths and limitations is being credited here. Note that the paper's own Section 6 'LIMITATIONS' concerns the study (possible training-data contamination of the Rainfall variants), not the tool's capabilities, but the strengths/limitations content attributed here is carried by Sections 5.1-5.3 and 7. (LLM, conf=0.80)
- Supportedcites [11] [11] James Finnie-Ansley, Paul Denny, Andrew Luxton-Reilly, Eddie Antonio Santos, James Prather, and Brett A Becker. 2023. My AI Wants to Know if This Will Be on the Exam: Testing OpenAI's Codex on CS2 Programming Exercises. In Proceedings of the 25th Australasian Computing Education Conference . 97-104. - source
The citation is offered for a broad background point — that prior work reported strengths and limitations of AI code generation for programming assessment. Our copy of Finnie-Ansley et al. does exactly this: Section 5 DISCUSSION states Codex is 'very good at solving semicomplex code reading and writing problems' but 'falters when there are odd edge cases or the requirements are somewhat esoteric', and performs well on 'blank slate' tasks with explicit, well-defined requirements while doing poorly on long, implicit or code-modification tasks; Section 7 CONCLUSIONS restates the same strengths/weaknesses profile, and Section 6 LIMITATIONS adds the study's own caveats (possible training-data contamination, single institution, non-cognitive artefact-only analysis). The 'still hold to a certain degree' synthesis is the citing authors' own framing over a multi-citation group, not a claim this source must make. (LLM, conf=0.85)
- Not matchedcites [38] [38] Stephen R Piccolo, Paul Denny, Andrew Luxton-Reilly, Samuel Payne, and Perry G Ridge. 2023. Many bioinformatics programming tasks can be automated with ChatGPT. arXiv preprint arXiv:2303.13528 (2023). -
We searched by its title and found no matching paper.
- Not checked yetcites [46] [46] Jaromir Savelka, Arav Agarwal, Christopher Bogart, Yifan Song, and Majd Sakr. 2023. Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses?. In Proceedings of the 28th Annual ACM Conference on Innovation and Technology in Computer Science Education . - source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This indeed corrected some, but not all, of the errors.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“through the better performing GPT-3.5 (65.4%)”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“from the weakest GPT-3 model (30.9% on the tests)”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT's explanations in some cases exposed problems in the original questions.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
65 more claims
Create a free account to see all 110 claims and to verify the 24 references still open.
- It may state two separate claims as one. 13
- The quote only partly states this claim. 2
- Probably not a substantial claim. 28
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Unsurprisingly, GPT-4 outperformed GPT-3.5 and Copilot, which corroborates previous work where GPT-4 has outperformed other LLMs for various tasks [21,151,175,184].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [175]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This has been inspired by many other instances where LLMs have already demonstrated remarkable effectiveness in generating novel language artifacts. These include course readings [22], code explanations [23, 28, 29, 41], model solutions [9, 35, 42], feedback [34], or responses to help requests [18, 26, 46, 47].”
From A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education (2024) · cites this paper as [42]
In the citing paper
“Some looked into other types of exercises such as multiple-choice questions [175][176][177]189], textbook questions [103], exam questions [79], computational thinking tasks [39], and textual reports [48].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [175]
In the citing paper
“Some recent work has found that GPT-4 outperforms earlier models in tasks such as visual programming [98,184], Socratic questioning of novices [21], solving multiple-choice questions and programming exercises [175], and that performance can be close to human tutors for some tasks [151] while not for others [21].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [175]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 5.1 (RQ1) To what degree can GPT-4 generate correct answers to MCQs? · p. 8
“successfully handled 446 questions (84.1%).”
§ 5.2 (RQ2) Does GPT-4 struggle with programming MCQs containing code? · p. 10
“| Overall | 199/530 | 347/530 | 448/530 | | | (37.5%) | (65.5%) | (84.5%) |”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 5.1 (RQ1) To what degree can GPT-4 generate correct answers to MCQs? · p. 7
“GPT-3.5 correctly answered 341 MCQs (64.3%).”
§ 5.2 (RQ2) Does GPT-4 struggle with programming MCQs containing code? · p. 10
“| Overall | 199/530 | 347/530 | 448/530 | | | (37.5%) | (65.5%) | (84.5%) |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3 DATA · p. 4
“There are 148 questions in PE2 out of which 83 have code snippets.”
§ 3 DATA · p. 5
“| PE2 | 4 | 0 | 7 | 31 | 10 | 0 | 0 | 21 | 27 | 0 | 52 | 148 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5.3 (RQ3) To what degree can GPT-4 produce solutions to complex coding tasks? · p. 10
“the performance of GPT-3.5 on MCQ tests (65.4%) and projects (54.6%)”
§ 5.3 (RQ3) To what degree can GPT-4 produce solutions to complex coding tasks? · p. 10
“the GPT-3.5 model obtained 407 points from the available 760 (53.6%) after a single submission to each activity”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5.1 (RQ1) To what degree can GPT-4 generate correct answers to MCQs? · p. 8
“to the best performing GPT-4 model (77.8%).”
§ 5.2 (RQ2) Does GPT-4 struggle with programming MCQs containing code? · p. 9
“| Course Total | 61/152 | 91/152 | 120/152 | 25/81 | 53/81 | 64/81 | | | (40.1%) | (59.9%) | (78.9%) | (30.9%) | (65.4%) | (79.0%) |”
7 more findings
Create a free account to see every finding of this check.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“The original GPT model [40] is a 12-layer decoder-only transformer [47] with masked self-attention heads.”
MethodExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 4×
“The third generation of GPT models [4] uses almost the same architecture as GPT-2.”
BackgroundMethodInfluentialKaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · cited 3×
“apply InstructGPT [36] and Codex to OpenBookQA [31], StoryCloze [33], and RACE-m [23] data sets which focus on multi-hop reasoning, recall, and reading comprehension, reporting 77.”
MethodInfluentialKung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · cited 1×
“Others have used GPT models to solve various MCQ-based exams, including the United States Medical Licensing Examination (USMLE), with accuracy around 50% [13, 22, 27], the Multistate Bar Examination (MBE) [3, 18], and the American Institute of Certified Public Accountants’ (AICPA) Regulation (REG) exam [2].”
MethodLearn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringPreprint
Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., & Kalyan, A. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2209.09513 · Cited by 2,884 · cited 2×
“0 worked well for our setting, which falls inline with the findings and precedence of existing work regarding multiple-choice questions [2, 27, 28].”
BackgroundResult
Research works printed but not linked: 5
- Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:abs/2107.03374 (2021). https://doi.org/10.48550/ARXIV.2107.03374 · DOI · Paper page
- Amelia Gilson Conrad W. Safranek Tao Huang Vimig Socrates Lim Sze Chi Roderick A. Taylor and David Chartash. 2022. How Well Does ChatGPT Do When Taking the Medical Licensing Exams? The Implications of Large Language Models for Medical Education and Knowledge Assessment. In medRxiv. https://doi.org/10.1101/2022.12.23.22283901. · DOI · Paper page
- [15] Sajed Jalil, Suzzana Rafi, Thomas D. LaToza, Kevin Moran, and Wing Lam. 2023. ChatGPT and Software Testing Education: Promises & Perils. https://doi.org/10. 48550/arXiv.2302.03287 arXiv:2302.03287 [cs.SE] - · arXiv
- [20] Donald Ervin Knuth. 1984. Literate programming. The computer journal 27, 2 (1984), 97-111. -
- [24] Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas. 2023. Comparing Code Explanations Created by Students and Large Language Models. https://doi.org/10.48550/arXiv.2304. 03938 arXiv:2304.03938 [cs.CY] - · arXiv · doi:10.48550/arxiv.2304
Cited by
The promise and challenges of generative AI in education
Giannakos, M., Azevedo, R., Brusilovsky, P., Cukurova, M., Dimitriadis, Y., Hernandez-Leo, D., Järvelä, S., Mavrikis, M., & Rienties, B. (2025) · Behaviour & Information Technology · doi:10.1080/0144929x.2024.2394886 · Cited by 543
“For tasks such as code explanation generation, success and correctness rates might be higher, and newer models tend to perform better than older models (i.e. Codex vs. GPT 3.5 vs. GPT 4) on most types of tasks (Savelka et al. 2023).”
InfluentialThe Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396
“Indeed, recent results with GPT-4 suggest that it can solve most exercises in introductory programming courses [164], which is also supported by our benchmarking work presented in Section 8.”
BackgroundResultInfluentialKazemitabaar, M., Ye, R., Wang, X., Henley, A. Z., Denny, P., Craig, M., & Grossman, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642773 · Cited by 290
“In terms of generating learning resources, early work by Sarsa et al. [53] analyzed the novelty, plausibility, and readiness of 120 programming exercises generated by OpenAI Codex and proposed the potential of using such models to come up with coding assignments.”
MethodCodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes
Liffiton, M., Sheese, B. E., Savelka, J., & Denny, P. (2023) · Proceedings of the 23rd Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631830 · Cited by 192
The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B. (2024) · Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3632620.3671116 · Cited by 196
A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education
Doughty, J., Wan, Z., Bompelli, A., Qayum, J., Wang, T., Zhang, J., Zheng, Y., Doyle, A., Sridhar, P., Agarwal, A., et al. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636256 · Cited by 132
Self-citationTowards an understanding of large language models in software engineering tasks
Zheng, Z., Ning, K., Zhong, Q., Chen, J., Chen, W., Guo, L., Wang, W., & Wang, Y. (2025) · Empirical Software Engineering · doi:10.1007/s10664-024-10602-0 · Cited by 116
“Moreover, a considerable portion of the research evaluates the performance of LLMs in education and programming courses (Leinonen et al. 2023a; Hellas et al. 2023; Sarsa et al. 2022; Savelka et al. 2023), with positive feedback.”
Can ChatGPT Pass High School Exams on English Language Comprehension?
de Winter, J. C. F. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00372-z · Cited by 115
“…encompass aiding the development of writing skills, facilitating comprehension through step-by-step explanations, speeding up information delivery via summarization of texts, and enhancing engagement through person - alized feedback (Kasneci et al., 2023; Rudolph et al., 2023; Šlapeta, 2023).”
BackgroundThe Effects of Generative AI on Computing Students’ Help-Seeking Preferences
Hou, I., Mettille, S., Man, O., Li, Z., Zastudil, C., & MacNeil, S. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636248 · Cited by 97
“…of generative AI including generating code explanations [34, 39, 42], enhancing error messages [35], identifying bugs [38], solving multiple-choice programming problems [59, 61], solving Parsons problems [22], and creating instructional materials and programming assignments [14, 58, 65].”
BackgroundPrather, J., Leinonen, J., Kiesler, N., Gorson Benario, J., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., et al. (2025) · 2024 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3689187.3709614 · Cited by 96
“Beginning in early 2022, computing education researchers showed that these models had incredible accuracy solving programming problems and exam questions in multiple courses and contexts [6, 40, 41, 48, 68, 119, 125, 126].”
Desirable Characteristics for AI Teaching Assistants in Programming Education
Denny, P. C., MacNeil, S., Šavelka, J., Porter, L., & Luxton-Reilly, A. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653574 · Cited by 152
“There is a growing body of work focusing on the capabilities of LLMs for solving programming exercises [5, 7, 9, 30], as well as answering [29] and generating programming-related MCQs [35].”
BackgroundMethodLarge Language Models in Computer Science Education: A Systematic Literature Review
Raihan, M. N., Siddiq, M. L., Santos, J. C. S., & Zampieri, M. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701863 · Cited by 222
Patterns of Student Help-Seeking When Using a Large Language Model-Powered Programming Assistant
Sheese, B., Liffiton, M., Savelka, J., & Denny, P. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636249 · Cited by 74
“Forexample,astudybyPardos andBhandarithatcomparedtheefficacyofhintsgeneratedby ChatGPTandbyhumantutorsinthesubjectofalgebrafoundthat significantlearninggainswereonlyobservedforstudentswho wereprovidedhuman-createdhints[22].”
MethodTransforming education with generative AI (GAI): key insights and future prospects
Lang, Q., Wang, M., Yin, M., Liang, S., & Song, W. (2025) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2025.3537618 · Cited by 76
“Savelka et al. [45] conducted an in-depth exploration of the performance and limitations of LLMs in multiple-choice questions and complex programming projects within higher education Python courses.”
How to Teach Programming in the AI Era? Using LLMs as a Teachable Agent for Debugging
Ma, Q., Shen, H., Koedinger, K., & Wu, S. T. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-64302-6_19 · Cited by 63
“Despite their prevalence, LLMs often produce unpredictable mistakes [Ganguli et al. , 2022], e.g., GPT-4 can still make mistakes 17% of the time in coding tasks for introductory and intermediate programming courses [Savelka et al. , 2023].”
Wang, P., Jing, Y., & Shen, S. (2025) · The Internet and Higher Education · doi:10.1016/j.iheduc.2025.100996 · Cited by 118
Álvarez Ariza, J., Benitez Restrepo, M., & Hernández Hernández, C. (2025) · IEEE Access · doi:10.1109/access.2025.3541424 · Cited by 51
Chen, J., Lu, X., Du, Y., Rejtig, M., Bagley, R., Horn, M., & Wilensky, U. J. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642377 · Cited by 110
GAIDE: A Framework for Using Generative AI to Assist in Course Content Development
Dickey, E., & Bejarano, A. (2024) · 2024 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10893132 · Cited by 48
Koutcheme, C., Dainese, N., Sarsa, S., Hellas, A., Leinonen, J., & Denny, P. (2024) · Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3649217.3653612 · Cited by 47
“Following prior work [6, 36], we generate a single feedback using greedy decoding (i.e., selecting the token with the highest probability as the next element in the sequence).”
MethodDenny, P. C., Smith, D. H., Fowler, M., Prather, J. E., Becker, B. A., & Leinonen, J. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653587 · Cited by 86
ChatGPT in Education: Empowering Educators through Methods for Recognition and Assessment
de Winter, J. C. F., Dodou, D., & Stienen, A. H. A. (2023) · Informatics · doi:10.3390/informatics10040087 · Cited by 40
Garcia, M. B. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13452-5 · Cited by 67
“Savelka et al. (2023) also asserted the importance of cultivating an educational environment where the emphasis is placed on learning and personal development rather than solely on achieving the correct answers.”
Chen, B., Lewis, C. M., West, M., & Zilles, C. (2024) · Proceedings of the Eleventh ACM Conference on Learning @ Scale · doi:10.1145/3657604.3662046 · Cited by 39
“This is particularly relevant to an introductory programming course, because ChatGPT has been shown to perform well solving introductory programming problems [15, 34].”
Ma, T. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100336 · Cited by 78
AI-Enhanced Auto-Correction of Programming Exercises: How Effective is GPT-3.5?
Azaiz, I., Deckarm, O., & Strickroth, S. (2023) · International Journal of Engineering Pedagogy (iJEP) · doi:10.3991/ijep.v13i8.45621 · Cited by 61
“5 have emerged, demonstrating their ability to solve programming assignments and pass exams [8,9,10,11].”
BackgroundHou, I., Man, O., Hamilton, K., Muthusekaran, S., Johnykutty, J., Zadeh, L., & MacNeil, S. (2025) · Proceedings of the 30th ACM Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3724363.3729024 · Cited by 37
“However, problems are also being identified, such as inequitable access to these tools [20, 59], negative impacts on students’ metacognition [45], and threats to assessment [16, 19, 27, 48].”
Computing education using generative artificial intelligence tools: A systematic literature review
Agbo, F. J., Olivia, C., Oguibe, G. F., Sanusi, I. T., & Sani, G. (2025) · Computers and Education Open · doi:10.1016/j.caeo.2025.100266 · Cited by 76
How Do Programming Students Use Generative AI?
Rahe, C., & Maalej, W. (2025) · Proceedings of the ACM on Software Engineering · doi:10.1145/3715762 · Cited by 45
“…as ChatGPT [OpenAI 2022] and GitHub Copilot [GitHub 2022], students in programming courses now have access to code authoring tools capable of solving coding exercises [Berrezueta-Guzman and Krusche 2023; Chen et al. 2021; Savelka et al. 2023] and exam questions entirely [Finnie-Ansley et al. 2023].”
Padiyath, A., Hou, X., Pang, A., Viramontes Vargas, D., Gu, X., Nelson-Fromm, T., Wu, Z., Guzdial, M., & Ericson, B. (2024) · Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3632620.3671098 · Cited by 35
“Note that prior work has found homework instructions in a similar style can be used to generate code from LLMs rather quickly [77].”
Kazemitabaar, M., Huang, O., Suh, S., Henley, A. Z., & Grossman, T. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712104 · Cited by 92
“…Ko et al.: uncertainty about next steps, selecting appropriate programming constructs, combining constructs, using code construct correctly, understanding code failures, and inspecting program behavior [66], with similar mistakes and misconceptions identified by other research [1, 65, 67, 76, 79].”
Pirzado, F. A., Ahmed, A., Mendoza-Urdiales, R. A., & Terashima‐Marín, H. (2024) · IEEE Access · doi:10.1109/access.2024.3443621 · Cited by 65
“They may hold conversations, answer inquiries, explain things, and even create original material on various topics and genres [6], [26], [38], [46], [58], [59], [62], [64], [83], [87], [88], [95].”
BackgroundDickey, E., Bejarano, A., & Garg, C. (2024) · SN Computer Science · doi:10.1007/s42979-024-03074-y · Cited by 33
Humble, N. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00494-x · Cited by 58
“…that GAI can be used by students to pass exams with little to no effort or knowledge in the topic at hand (Karnalim et al., 2023; Li et al., 2023; Savelka et al., 2023a; Xie et al., 2023; Zeng et al., 2024) or seek unauthorized assistance during examinations (Budhiraja et al., 2024; Cambaz &…”
InfluentialDecoding Logic Errors: A Comparative Study on Bug Detection by Students and Large Language Models
MacNeil, S., Denny, P. C., Tran, A., Leinonen, J., Bernstein, S., Hellas, A., Sarsa, S., & Kim, J. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636245 · Cited by 57
“While we acknowledge the potential impact of further prompt optimization on mitigating false positives in the correct code condition, it’s essential to note the dynamic nature of these models, characterized by continuous changes in verbosity and performance [44, 45, 53].”
BackgroundHou, I., Man, O., Mettille, S., Gutierrez, S., Angelikas, K., & MacNeil, S. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636247 · Cited by 47
“…generative models are at solving programming problems [51, 68], automatically repairing bugs in code [22, 34, 36], and passing multiple choice quizzes [56, 57, 59], researchers and practitioners have become concerned about the implications regarding assessment and academic integrity [3, 40, 72].”
BackgroundResultEnhancing Programming Error Messages in Real Time with Generative AI
Kimmel, B., Geisert, A. L., Yaro, L., Gipson, B., Hotchkiss, R., Osae-Asante, S. K., Vaught, H., Wininger, G., & Yamaguchi, C. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3647967 · Cited by 44
“Recent work with GPT-4 has shown that current models outperform their predecessors by getting nearly every question correct, placing this model near the top spot when compared to humans [19, 26].”
Literature Review on the Integration of Generative AI in Programming Education
Nathaniel, J., Oyelere, S. S., Suhonen, J. T., & Tedre, M. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00524-3 · Cited by 70
“GenAI have shown to have the potential to enhance program - ming through code auto-completion (Phung et al., 2023) and generation (Savelka et al., 2023a), serving as programming assistance (Savelka et al., 2023b), and encourag - ing personalized learning, and fostering greater student engagement.”
Abbasnejad, B., Soltani, S., Taghizadeh, F., & Ali, Z. (2025) · Interactive Technology and Smart Education · doi:10.1108/itse-12-2024-0314 · Cited by 45
How Instructors Incorporate Generative AI into Teaching Computing
Prather, J. E., Leinonen, J., Kiesler, N., Benario, J. G., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., et al. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649405.3659534 · Cited by 32
“Early work on GenAI in computing education, starting in 2022, showed amazing accuracy in solving programming problems [5, 6, 8, 18].”
BackgroundPrather, J. E., Reeves, B. N., Denny, P. C., Leinonen, J., MacNeil, S., Luxton-Reilly, A., Orvalho, J. G., Alipour, M. A., Alfageeh, A., Amarouche, T., et al. (2025) · Australasian Computing Education Conference · doi:10.1145/3716640.3716649 · Cited by 44
“LLMs can also solve multiple-choice questions related to programming [56].”
DBox: Scaffolding Algorithmic Programming Learning through Learner-LLM Co-Decomposition
Ma, S., Wang, J., Zhang, Y., Ma, X., & Wang, A. Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713748 · Cited by 65
“Based on GPT-4’s performance on coding tasks [20, 65], we selected 25 easy-level LeetCode problems covering various algorithms and data structures problems (e.g., dynamic programming, sorting, greedy algorithms).”
Generative AI in Computer Science Education: Challenges and Opportunities
Franklin, D., Denny, P., Gonzalez-Maldonado, D. A., & Tran, M. (2025) · doi:10.1017/9781009581738 · Cited by 20
Prather, J., Reeves, B. N., Denny, P., Leinonen, J., MacNeil, S., Luxton-Reilly, A., Orvalho, J., Alipour, A., Alfageeh, A., Amarouche, T., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.12800 · Cited by 20
“LLMs can also solve multiple-choice questions related to programming [56].”
Ali, M., Rao, P. V., Mai, Y., & Xie, B. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671097 · Cited by 33
Awal, M. R., & Asaduzzaman (2024) · Higher Education Skills and Work-based Learning · doi:10.1108/heswbl-09-2023-0249 · Cited by 23
A comparative study of AI ‐generated and human‐crafted learning objectives in computing education
Doyle, A., Sridhar, P., Agarwal, A., Šavelka, J., & Sakr, M. F. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.13092 · Cited by 30
Self-citationChang, C. I., Choi, W. C., & Choi, I. C. (2024) · International Conference on Big Data and Education (ICBDE) · doi:10.1145/3704289.3704301 · Cited by 47
“The tool’s ability to quickly generate ], [6], [7], [10], [11], [12], [13], [14], [17], [20], [22], [24], [25], [26], [ [15], [23] alternative solutions and make minor adjustments based on additional input demonstrated its potential for providing rapid, accurate feedback.”
InfluentialExamining Student and Teacher Perspectives on Undisclosed Use of Generative AI in Academic Work
Adnin, R., Pandkar, A., Yao, B., Wang, D., & Das, M. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713393 · Cited by 16
Feng, T. H., Denny, P., Wuensche, B., Luxton-Reilly, A., & Hooper, S. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636263 · Cited by 16
Analyzing LLM Usage in an Advanced Computing Class in India
Arora, U., Garg, A., Gupta, A., Jain, S., Mehta, R. L., Oberoi, R., Prachi, Raina, A., Saini, M., Sharma, S., et al. (2025) · Australasian Computing Education Conference · doi:10.1145/3716640.3716657 · Cited by 52
“Code Generation using LLMs: Most of the existing studies have evaluated the effectiveness of LLMs for generating code in introductory programming exercises [7, 8, 12, 14, 23, 25, 29, 30, 33].”
Let's Ask AI About Their Programs: Exploring ChatGPT's Answers To Program Comprehension Questions
Lehtinen, T., Koutcheme, C., & Hellas, A. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1145/3639474.3640058 · Cited by 32
Comparing and assessing four AI chatbots’ competence in economics
Hultberg, P. T., Calonge, D. S., Kamalov, F., & Smail, L. (2024) · PLoS ONE · doi:10.1371/journal.pone.0297804 · Cited by 26
“5, and GPT-4, in programming courses was carried out by Savelka et al. (2023) who found a significant rate of improvement which strongly suggests their potential to handle almost any type of assessment widely used in higher education programming courses [18].”
BackgroundComputer Programming Education in the Age of Generative AI: Insights from Empirical ResearchPreprint
Deriba, F. G., Sanusi, I. T., Campbell, O. O., & Oyelere, S. S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4891302 · Cited by 28
Cipriano, B. P., & Alves, P. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1145/3639474.3640052 · Cited by 35
Comparing Feedback from Large Language Models and Instructors: Teaching Computer Science at Scale
Nguyen, H., Stott, N., & Allan, V. H. (2024) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3657604.3664660 · Cited by 23
“LLMs have shown potential to complete and explain code [11, 12, 18], detect errors [1], and provide suggestions to improve code [8, 17].”
An Investigation of the Drivers of Novice Programmers' Intentions to Use Web Search and GenAI
Skripchuk, J., Bacher, J. T., & Price, T. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671112 · Cited by 22
“GenAI is becoming increasingly competent at solving programming problems common to introductory CS, ranging from short programming questions, to larger projects [18, 42, 47], but GenAI still struggles with tasks such as finding subtle issues in assignments and giving out of scope explanations [45].”
Chang, C. I., Choi, W. C., & Choi, I. C. (2024) · Proceedings of the 2024 7th Artificial Intelligence and Cloud Computing Conference · doi:10.1145/3719384.3719439 · Cited by 11
“Savelka et al. [12] highlighted the genuine concern that learners might rely excessively on ChatGPT models for completing course assignments, thus compromising their ability to develop critical thinking skills.”
Šavelka, J., Agarwal, A., Bogart, C., & Sakr, M. F. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-53656-4_8 · Cited by 21
Self-citationClassifying Sentiments on Social Media Texts: A GPT-4 Preliminary Study
Maceda, L. L., Llovido, J. L., Artiaga, M. B., & Abisado, M. (2023) · International Conference on Natural Language Processing and Information Retrieval (NLPIR) · doi:10.1145/3639233.3639353 · Cited by 22
“These inaccuracies and inconsistencies have been also observed in the study of [27].”
ResultClos, J., & Chen, Y. Y. (2024) · International Symposium on Trustworthy Autonomous Systems (TAS) · doi:10.1145/3686038.3686063 · Cited by 17
“It cannot be detected [5] and can pass exams [19, 20]; therefore, we need to figure out how it should be used to make our lives easier rather than harder.”
A Bibliometric Exposition and Review on Leveraging LLMs for Programming Education
Amos, J. P., Amodu, O. A., Mahmood, R. A. R., AbdulQudus, A. B., Zakaria, A. F., Iyanda, A. R., Bukar, U. A., & Hanapi, Z. M. (2025) · IEEE Access · doi:10.1109/access.2025.3554627 · Cited by 24
“Most of these papers have been covered in the previous section on introductory programming (Section V-A) and computing and computer science and programming education (Section V-B) except for [33] which is discussed in this section.”
Solving Proof Block Problems Using Large Language Models
Poulsen, S., Sarsa, S., Prather, J. E., Leinonen, J., Becker, B. A., Hellas, A., Denny, P. C., & Reeves, B. N. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630928 · Cited by 21
“Similarly, the performance of GPT-4 in passing various programming assessments is better compared to earlier models [42].”
ResultGu, P., Wu, J., Cheng, Z., Xia, Y., Cheng, M., & Dong, Y. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13367-1 · Cited by 24
“Instead, "student-centered" and constructivist learning are preferred (Peng et al., 2017; Savelka et al., 2023).”
Kohn, T. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70043 · Cited by 15
Cipriano, B. P., Alves, P., & Denny, P. C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72315-5_6 · Cited by 14
Feng, H., Denny, P. C., Wuensche, B. C., Luxton-Reilly, A., & Whalley, J. L. (2024) · doi:10.1145/3680533.3697064 · Cited by 15
“In CS (Computer Science), LLMs achieved high performance in CS1 [Denny et al. 2023; Finnie-Ansley et al. 2022], CS2 [Finnie-Ansley et al. 2023], and programming-related MCQs (multiple-choice questions) [Savelka et al. 2023], often surpassing average student performance.”
Designing LLM-Resistant Programming Assignments: Insights and Strategies for CS Educators
McDanel, B., & Novak, E. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701872 · Cited by 18
Cognitive Computing with Large Language Models for Student Assessment Feedback
Abbas, N., & Atwell, E. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9050112 · Cited by 16
“Complementing this, Savelka et al. [16] conducted a comprehensive evaluation of GPT-4 across several assessment tasks in programming courses, revealing significant improvements over previous GPT versions in handling multiple-choice and coding assessments.”
Large Language Models (LLMs) in Programming Learning: The Current Research State and Agenda
Fu, Q., Zhao, Y., Jia, Z., & Zheng, Y. (2025) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2025.3622043 · Cited by 19
Memarian, B., & Doleck, T. (2025) · SN Computer Science · doi:10.1007/s42979-025-04569-y · Cited by 21
“Savelka et al. [105] explore the performance of GPT-4, comparing it to the previous generations of GPT models, on three Python courses with assessments ranging from simple multiple-choice questions to complex programming projects.”
InfluentialThe Robots are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.00658 · Cited by 9
“Indeed, recent results with GPT-4 suggest that it can solve most exercises in introductory programming courses [164], which is also supported by our benchmarking work presented in Section 8.”
BackgroundResultInfluentialLi, N., Broner, S., Kim, Y., Mizuo, K., Sauder, E., To, C., Wang, A., Gila, O., & Shindler, M. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701946 · Cited by 15
“Shortly thereafter, Savelka et al. [38] demonstrated that the use of GPT-4 led to the models being able to pass, although some limitations still held.”
Conversational Alignment With Artificial Intelligence in Context
Sterken, R., & Kirkpatrick, J. R. (2024) · Philosophical Perspectives · doi:10.1111/phpe.12205 · Cited by 16
Denny, P. C., Kumar, V., MacNeil, S., Prather, J. E., & Leinonen, J. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729093 · Cited by 14
WIP: Large Language Model-Enhanced Smart Tutor for Undergraduate Circuit Analysis
Chen, L., Xie, H., Rohde, J. A., & Zhang, Y. (2025) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie63693.2025.11328403 · Cited by 14
“However, most LLM-based educational tools have been developed in fields where LLMs perform well, such as programming [1] and medical education [3], while comparatively fewer efforts have focused on the broader field of engineering education.”
Simon, B. F., Hartvég, Á., Dénes-Fazakas, L., Eigner, G., & Szilágyi, L. S. (2024) · Information · doi:10.3390/info15060297 · Cited by 13
Generative AI in CS Education: Literature Review through a SWOT Lens
Roberts, J., & Mohamed, A. (2024) · Western Canadian Conference on Computing Education · doi:10.1145/3660650.3660657 · Cited by 14
“This issue has motivated a lot of literature on remedies and strategies to support academic integrity [12, 19, 23, 26–29].”
BackgroundProbeable Problems for Beginner-level Programming-with-AI Contests
Pawagi, M., & Kumar, V. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671108 · Cited by 11
“For instance, Aniche et al. [2] note that developers reasoning about the adequacy of their test cases tend to focus on their source code, their test code, and its code coverage while often failing to consider the documented specification.”
SENSAI: Large Language Models as Applied Cybersecurity Tutors
Nelson, C., Doupé, A., & Shoshitaishvili, Y. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701801 · Cited by 28
“Already, the current state-of-the-art, GPT-4, is possibly capable of passing some assessments, such as multiple choice question tests and coding exercises, in introductory and intermediate programming courses [29].”
Gutierrez, S., Hou, I., Lee, J., Angelikas, K., Man, O., Mettille, S., Prather, J. E., Denny, P. C., & MacNeil, S. (2025) · Australasian Computing Education Conference · doi:10.1145/3716640.3716654 · Cited by 15
“However, the capabilities of generative AI extend beyond mere coding support; students can also leverage these technologies to answer multiple-choice questions [58, 60] and solve text-based programming problems [25, 33, 55].”
Fostering Responsible AI Use Through Negative Expertise: A Contextualized Autocompletion Quiz
MacNeil, S., Prather, J. E., Nabid, R. A., Gutierrez, S., Carvalho, S., Shrestha, S., Denny, P. C., Reeves, B. N., Leinonen, J., & Rossetti, R. L. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729067 · Cited by 19
Pre‐trained language models: What do they know?
Guimar�ães, N., Campos, R., & Jorge, A. (2023) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1518 · Cited by 17
“An important highlight from the authors in this and a similar paper (Savelka, Agarwal, Bogart, & Sakr, 2023) is the lack of ability of GPT-3.”
BackgroundEvaluating GPT for use in K-12 Block Based CS Instruction Using a Transpiler and Prompt Engineering
Gonzalez-Maldonado, D., Liu, J., & Franklin, D. M. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701910 · Cited by 9
Kannam, S., Yang, Y., Dharm, A., & Lin, K. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701806 · Cited by 16
Chang, C. I., Choi, W. C., Choi, I. C., & Lei, H. (2024) · International Conference on Education and E-Learning (ICEEL) · doi:10.1145/3719487.3719519 · Cited by 17
“[12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [31], [32], [33], [34], [35] Others 0 0 1 4.2% [30] Table 3: Distribution of AIGC Tools in Programming Education Research”
InfluentialThe Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B., Kimmel, B., Wright, J., & Briggs, B. (2024) · arXiv · Cited by 196
Generating Effective Distractors for Introductory Programming Challenges: LLMs vs Humans
Hassany, M., Brusilovsky, P. L., Šavelka, J., Narayanan, A. B. L., Akhüseyinoğlu, K., Agarwal, A., & Hendrawan, R. A. (2025) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3706468.3706529 · Cited by 11
Self-citationStudents' Attitudes Towards Cheating Before and After ChatGPT
Kann, V. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729108 · Cited by 9
Haque, S., & Hundhausen, C. D. (2025) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3702652.3744231 · Cited by 14
“Research on introductory programming has focused on GenAI’s performance in solving basic problems and course assignments [11, 17–19] while other studies have assessed these models in completing exams and practical coding tasks in computer science (CS) courses [29, 33, 34, 47, 50, 61].”
Comparative Performance of GPT-4, RAG-Augmented GPT-4, and Students in MOOCs
Miladi, F., Psyché, V., & Lemire, D. (2024) · BBGI · doi:10.1007/978-3-031-65996-6_7 · Cited by 5
“Instructors should adopt a more sustainable assessment approach that priori-tizes the cultivation of higher-order cognitive abilities, including critical thinking [18], problem-solving, and creativity [19].”
BackgroundSyahidi, A. A., Kiyokawa, K., & Nuchitprasitchai, S. (2025) · IEEE Symposium on Computers & Informatics (ISCI) · doi:10.1109/isci65687.2025.11166787 · Cited by 7
“…built upon Generative Pretrained Transformer 4 (GPT-4), a state-of-the-art Large Language Models (LLMs) that significantly outperforms its predecessors—including GPT-3 and traditional rule-based systems—in contextual understanding, response accuracy, and handling complex queries [11], [12], [13].”
Vemula, S. (2024) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10893561 · Cited by 7
“In a study involving undergraduate students, the integration of an LLM-based assistant led to a measurable improvement in students’ test scores and debugging skills within a semester-long Python programming course [15], [16].”
AI Generated Code Plagiarism Detection in Computer Science Courses: A Literature Mapping
Simmons, A., de Holanda, M. T., Chamon, C., & Silva, D. D. (2024) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10893117 · Cited by 11
Munisamy, M., Osman, S. Z. M., & Sanmugam, M. (2024) · International Journal of Modern Education · doi:10.35631/ijmoe.620027 · Cited by 5
Towards the Integration of Large Language Models in an Object-Oriented Programming Course
Cipriano, B. P. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649405.3659473 · Cited by 7
Integrating AI into Design Ideation: Assessing ChatGPT’s Role in Human-Centered Design EducationPreprint
Papachristos, E., İnal, Y., Vicient, C., Johansen, E. A., & Hermansen, M. (2024) · doi:10.36227/techrxiv.171656320.09963657/v1 · Cited by 7
Chen, P., & Alias, S. B. (2024) · International Symposium on Artificial Intelligence for Education (ISAIE) · doi:10.1145/3700297.3700342 · Cited by 6
“In language teaching and assessment, LLMs have improved text generation capabilities, though they still require careful prompting and adjustment to achieve optimal results, and have not yet sur-passed the state-of-the-art in automated grading and error correction [13].”
Can ChatGPT Solve Undergraduate Exams from Warehousing Studies? An Investigation
Franke, S., Pott, C., Rutinowski, J., Pauly, M., Reining, C., & Kirchheim, A. (2025) · Computers · doi:10.3390/computers14020052 · Cited by 4
Quille, K., Alattyanyi, C., Becker, B. A., Faherty, R., Gordon, D., Harte, M., Hensman, S., Hofmann, M., García, J. J., Kuznetsov, A., et al. (2024) · Revista de Educación a Distancia (RED) · doi:10.6018/red.603001 · Cited by 5
“…GPT-4 on complicated coding projects including multiple-file code bases, noting ”the rate of improvement across the recent generations of GPT models strongly suggests their potential to handle almost any type of assessment widely used in higher education programming courses” (Savelka et al., 2023).”
Background
Metadata
- Authors
- Savelka, J., Agarwal, A., An, M., Bogart, C., & Sakr, M.
- Affiliations
- Carnegie Mellon University, USA
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 2023 ACM Conference on International Computing Education Research V.1
- Publisher
- ACM
- Published
- 2023-08-07
- Pages
- 78-92
- OpenAlex
- W4381587445
- Cited by
- 141
- References
- 48
- Citations by year
- 2026: 21 · 2025: 54 · 2024: 57 · 2023: 9
- FWCI
- 42.59
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on International Computing Education Research (ICER) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- large language models, GPT-4, programming education, Python programming course, automated assessment, automated grading, multiple-choice questions, programming exercises, generative AI in higher education, assessment design, GPT-3
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.