Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses
Open accessThe DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index; 2 of the 29 sources its claims cite were checked for support.
Abstract
This paper studies recent developments in large language models’ (LLM) abilities to pass assessments in introductory and intermediate Python programming courses at the postsecondary level. The emergence of ChatGPT resulted in heated debates of its potential uses (e.g., exercise generation, code explanation) as well as misuses in programming classes (e.g., cheating). Recent studies show that while the technology performs surprisingly well on diverse sets of assessment instruments employed in typical programming classes the performance is usually not sufficient to pass the courses. The release of GPT-4 largely emphasized notable improvements in the capabilities related to handling assessments originally designed for human test-takers. This study is the necessary analysis in the context of this ongoing transition towards mature generative AI systems. Specifically, we report the performance of GPT-4, comparing it to the previous generations of GPT models, on three Python courses with assessments ranging from simple multiple-choice questions (no code involved) to complex programming projects with code bases distributed into multiple files (599 exercises overall). Additionally, we analyze the assessments that were not handled well by GPT-4 to understand the current limitations of the model, as well as its capabilities to leverage feedback provided by an auto-grader. We found that the GPT models evolved from completely failing the typical programming class’ assessments (the original GPT-3) to confidently passing the courses with no human involvement (GPT-4). While we identified certain limitations in GPT-4’s handling of MCQs and coding exercises, the rate of improvement across the recent generations of GPT models strongly suggests their potential to handle almost any type of assessment widely used in higher education programming courses. These findings could be leveraged by educators and institutions to adapt the design of programming assessments as well as to fuel the necessary discussions into how programming classes should be updated to reflect the recent technological developments. This study provides evidence that programming instructors need to prepare for a world in which there is an easy-to-use widely accessible technology that can be utilized by learners to collect passing scores, with no effort whatsoever, on what today counts as viable programming knowledge and skills assessments.
Overview
CoChat Summary
TL;DR
This study evaluates GPT-4 against earlier GPT models on 599 assessments from three postsecondary Python courses. GPT-4 passed the MCQ assessments and, after auto-grader feedback, reached scores likely sufficient to pass the coding projects, unlike GPT-3 and GPT-3.5.
Key findings
- On 530 MCQs, GPT-4 answered 84.1% correctly (446/530), compared with 64.3% (341/530) for GPT-3.5 and 37.5% (199/530) for GPT-3. GPT-4 passed all PE1 and PE2 graded tests, with PE1 overall 85% and PE2 overall 89.6%; on PPP tests it scored 79.0% (64/81) in Table 5.
- On 69 coding activities, GPT-4 scored 545/760 (71.7%) after first submission versus 407/760 (53.6%) for GPT-3.5. After auto-grader feedback, GPT-4 improved to 634/760 (83.4%), while GPT-3.5 improved to 505/760 (66.4%).
- GPT-4 still performed worse on MCQs with code (81.0%) than without code (90.7%). Of 67 incorrect code MCQs, the paper categorized errors as MULTI-HOP (19), INCOHERENT (14), WRONG-Q (13), DUAL-INTERPRETATION (13), and FLUKE (8).
- GPT-4 struggled on some coding activities: after 34 feedback iterations on project 6 activities (debugging, refactoring, testing, packaging), it obtained only 42 of 95 points.
Methods
- Data: 599 assessments from three real-world Python courses—PE1 (149 MCQs), PE2 (148 MCQs), and PPP (233 MCQs plus 69 coding activities from 32 project tasks), totaling 530 MCQs and 69 coding activities.
- Models: GPT-4 was compared with GPT-3.5 (`text-davinci-003`) and GPT-3 (`text-davinci-001`) on MCQs; coding activities benchmarked GPT-4 against GPT-3.5 only. Temperature was 0.0, `max_tokens` 500 for MCQs and 2,000/4,000 for coding, `top_p` 1, and penalties 0.
- Procedure: MCQs were submitted via prompt templates and compared to reference answers. Coding solutions were submitted to an auto-grader; feedback was appended for iterative resubmission until full score or no-change impasse, with no human intervention.
Limitations
- The paper states that prompt engineering was not explored, so results should be interpreted as a lower bound; findings may not generalize beyond Python, English-language assessments, or the assessment types studied; training data for the models is unknown, raising possible memorization concerns; and GPT-4 technical details are undisclosed, limiting reproducibility.
- The text also makes evident that GPT-4 still struggled with tasks requiring external tools or artifacts (e.g., refactoring, packaging, debugger, `pylint`, `pycodestyle`, `pytest`), fine-grained formatting requirements, and input data exceeding the model’s prompt length.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
1 Valid · 0 Not valid · 17 Not checked · 92 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Hence, the strengths and limitations reported in prior work [7, 10, 11, 38, 46] still hold to a certain degree.”
What the sources say
- Not checked yetcites [7] [7] Paul Denny, Viraj Kumar, and Nasser Giacaman. 2023. Conversing with Copilot: Exploring prompt engineering for solving CS1 problems using natural language. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 . 1136-1142. - source
- Supportedcites [10] [10] James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Australasian Computing Education Conference (Virtual Event, Australia) (ACE '22) . Association for Computing Machinery, New York, NY, USA, 10-19. https://doi.org/10.1145/3511861.3511863 - source
Our copy of Finnie-Ansley et al. (2022) does report both sides of Codex in introductory programming: Section 5.1 'Opportunities' sets out strengths (generated solutions as models students can learn from or check their work against, variation as fodder for code review and refactoring), while Section 5.2 'Challenges' and Section 5.3 'Academic Integrity' set out limitations (over-reliance, poor-style/incorrect solutions, hard-to-detect plagiarism, need for invigilated assessment), and Section 7 'Conclusions' states 'the current tool has clear limitations when given tight constraints'. The citing sentence's hedge that these 'still hold to a certain degree' is the citing authors' own framing, not an attributed finding, so nothing beyond the report of strengths and limitations is being credited here. Note that the paper's own Section 6 'LIMITATIONS' concerns the study (possible training-data contamination of the Rainfall variants), not the tool's capabilities, but the strengths/limitations content attributed here is carried by Sections 5.1-5.3 and 7. (LLM, conf=0.80)
- Supportedcites [11] [11] James Finnie-Ansley, Paul Denny, Andrew Luxton-Reilly, Eddie Antonio Santos, James Prather, and Brett A Becker. 2023. My AI Wants to Know if This Will Be on the Exam: Testing OpenAI's Codex on CS2 Programming Exercises. In Proceedings of the 25th Australasian Computing Education Conference . 97-104. - source
The citation is offered for a broad background point — that prior work reported strengths and limitations of AI code generation for programming assessment. Our copy of Finnie-Ansley et al. does exactly this: Section 5 DISCUSSION states Codex is 'very good at solving semicomplex code reading and writing problems' but 'falters when there are odd edge cases or the requirements are somewhat esoteric', and performs well on 'blank slate' tasks with explicit, well-defined requirements while doing poorly on long, implicit or code-modification tasks; Section 7 CONCLUSIONS restates the same strengths/weaknesses profile, and Section 6 LIMITATIONS adds the study's own caveats (possible training-data contamination, single institution, non-cognitive artefact-only analysis). The 'still hold to a certain degree' synthesis is the citing authors' own framing over a multi-citation group, not a claim this source must make. (LLM, conf=0.85)
- Not matchedcites [38] [38] Stephen R Piccolo, Paul Denny, Andrew Luxton-Reilly, Samuel Payne, and Perry G Ridge. 2023. Many bioinformatics programming tasks can be automated with ChatGPT. arXiv preprint arXiv:2303.13528 (2023). -
We searched by its title and found no matching paper.
- Not checked yetcites [46] [46] Jaromir Savelka, Arav Agarwal, Christopher Bogart, Yifan Song, and Majd Sakr. 2023. Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses?. In Proceedings of the 28th Annual ACM Conference on Innovation and Technology in Computer Science Education . - source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This indeed corrected some, but not all, of the errors.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“through the better performing GPT-3.5 (65.4%)”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“from the weakest GPT-3 model (30.9% on the tests)”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT's explanations in some cases exposed problems in the original questions.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
65 more claims
Create a free account to see all 110 claims and to verify the 24 references still open.
- It may state two separate claims as one. 13
- The quote only partly states this claim. 2
- Probably not a substantial claim. 28
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Unsurprisingly, GPT-4 outperformed GPT-3.5 and Copilot, which corroborates previous work where GPT-4 has outperformed other LLMs for various tasks [21,151,175,184].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [175]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This has been inspired by many other instances where LLMs have already demonstrated remarkable effectiveness in generating novel language artifacts. These include course readings [22], code explanations [23, 28, 29, 41], model solutions [9, 35, 42], feedback [34], or responses to help requests [18, 26, 46, 47].”
From A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education (2024) · cites this paper as [42]
In the citing paper
“Some looked into other types of exercises such as multiple-choice questions [175][176][177]189], textbook questions [103], exam questions [79], computational thinking tasks [39], and textual reports [48].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [175]
In the citing paper
“Some recent work has found that GPT-4 outperforms earlier models in tasks such as visual programming [98,184], Socratic questioning of novices [21], solving multiple-choice questions and programming exercises [175], and that performance can be close to human tutors for some tasks [151] while not for others [21].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [175]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 5.1 (RQ1) To what degree can GPT-4 generate correct answers to MCQs? · p. 8
“successfully handled 446 questions (84.1%).”
§ 5.2 (RQ2) Does GPT-4 struggle with programming MCQs containing code? · p. 10
“| Overall | 199/530 | 347/530 | 448/530 | | | (37.5%) | (65.5%) | (84.5%) |”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 5.1 (RQ1) To what degree can GPT-4 generate correct answers to MCQs? · p. 7
“GPT-3.5 correctly answered 341 MCQs (64.3%).”
§ 5.2 (RQ2) Does GPT-4 struggle with programming MCQs containing code? · p. 10
“| Overall | 199/530 | 347/530 | 448/530 | | | (37.5%) | (65.5%) | (84.5%) |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3 DATA · p. 4
“There are 148 questions in PE2 out of which 83 have code snippets.”
§ 3 DATA · p. 5
“| PE2 | 4 | 0 | 7 | 31 | 10 | 0 | 0 | 21 | 27 | 0 | 52 | 148 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5.3 (RQ3) To what degree can GPT-4 produce solutions to complex coding tasks? · p. 10
“the performance of GPT-3.5 on MCQ tests (65.4%) and projects (54.6%)”
§ 5.3 (RQ3) To what degree can GPT-4 produce solutions to complex coding tasks? · p. 10
“the GPT-3.5 model obtained 407 points from the available 760 (53.6%) after a single submission to each activity”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5.1 (RQ1) To what degree can GPT-4 generate correct answers to MCQs? · p. 8
“to the best performing GPT-4 model (77.8%).”
§ 5.2 (RQ2) Does GPT-4 struggle with programming MCQs containing code? · p. 9
“| Course Total | 61/152 | 91/152 | 120/152 | 25/81 | 53/81 | 64/81 | | | (40.1%) | (59.9%) | (78.9%) | (30.9%) | (65.4%) | (79.0%) |”
7 more findings
Create a free account to see every finding of this check.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“The original GPT model [40] is a 12-layer decoder-only transformer [47] with masked self-attention heads.”
MethodExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 4×
“The third generation of GPT models [4] uses almost the same architecture as GPT-2.”
BackgroundMethodInfluentialKaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · cited 3×
“apply InstructGPT [36] and Codex to OpenBookQA [31], StoryCloze [33], and RACE-m [23] data sets which focus on multi-hop reasoning, recall, and reading comprehension, reporting 77.”
MethodInfluentialKung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · cited 1×
“Others have used GPT models to solve various MCQ-based exams, including the United States Medical Licensing Examination (USMLE), with accuracy around 50% [13, 22, 27], the Multistate Bar Examination (MBE) [3, 18], and the American Institute of Certified Public Accountants’ (AICPA) Regulation (REG) exam [2].”
MethodLearn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringPreprint
Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., & Kalyan, A. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2209.09513 · Cited by 2,884 · cited 2×
“0 worked well for our setting, which falls inline with the findings and precedence of existing work regarding multiple-choice questions [2, 27, 28].”
BackgroundResult
Research works printed but not linked: 5
- Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:abs/2107.03374 (2021). https://doi.org/10.48550/ARXIV.2107.03374 · DOI · Paper page
- Amelia Gilson Conrad W. Safranek Tao Huang Vimig Socrates Lim Sze Chi Roderick A. Taylor and David Chartash. 2022. How Well Does ChatGPT Do When Taking the Medical Licensing Exams? The Implications of Large Language Models for Medical Education and Knowledge Assessment. In medRxiv. https://doi.org/10.1101/2022.12.23.22283901. · DOI · Paper page
- [15] Sajed Jalil, Suzzana Rafi, Thomas D. LaToza, Kevin Moran, and Wing Lam. 2023. ChatGPT and Software Testing Education: Promises & Perils. https://doi.org/10. 48550/arXiv.2302.03287 arXiv:2302.03287 [cs.SE] - · arXiv
- [20] Donald Ervin Knuth. 1984. Literate programming. The computer journal 27, 2 (1984), 97-111. -
- [24] Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas. 2023. Comparing Code Explanations Created by Students and Large Language Models. https://doi.org/10.48550/arXiv.2304. 03938 arXiv:2304.03938 [cs.CY] - · arXiv · doi:10.48550/arxiv.2304
Cited by
Items 101–143 of 143
A Code Analysis Tool to Help Students in the Age of Generative AI
Dosaru, D.-F., Simion, D.-M., Ignat, A.-H., Negreanu, L., & Olteanu, A.-C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72312-4_31 · Cited by 4
Leveling up Learning: Serious Games for Computing Education - Long-Term Opportunities and Risks
Wuensche, B. C., Hooper, S., Whalley, J. L., Straand, I. J., Denny, P. C., Crow, T. V. H., Lange-Nawka, D. C., Luxton-Reilly, A., & Thompson, S. E. R. (2025) · Australasian Computing Education Conference · doi:10.1145/3716640.3716655 · Cited by 5
The Impact of Chatbots on Students' Reflective Thinking in Introductory Programming Course
Rachmat, A., Watterson, C. A., & Lundqvist, K. Ø. (2025) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon62633.2025.11016417 · Cited by 9
“Moreover, students require additional training to acquire the understanding and skill to utilize chat-bots for learning benefits, especially in programming courses due to the capability to solve programming assignments [14].”
Manakina, O., & Müldner, K. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98414-3_19 · Cited by 7
Beyond the ban: A theoretical framework for integrating Generative AI in assessment
Chapman, B. L., & Dalgarno, B. (2025) · Policy Futures in Education · doi:10.1177/14782103251411725 · Cited by 6
“…these concerns, particularly as reports emerge of GenAI con fi dently passing standardised exams (Chau et al., 2024; Hargreaves, 2023; Raftery, 2023; Savelka et al., 2023), drafting peer-reviewed scienti fi c articles (Kacena et al., 2024), and impersonating humans on social media platforms (Hajli…”
From Code Generation to Conceptual Learning: Student Use of LLMs in a Web Programming Course
Isa, H.-Y., Weston, M., Wellyanto, M. R., Karna, I., Talton, J. O., & Kumar, R. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3793207 · Cited by 4
From Priming Modalities to Educational AI Chatbot Engagement: A Study of 67 Learners
Xuan, T., Salminen, J. O., Kaate, I., Ahmed, F., Amin, D., Patil, R., Jung, S., Azem, J. Y., & Jansen, B. J. J. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2654818 · Cited by 2
“Chatbots are also employed in the programming field to generate code and provide detailed code explanations (Savelka et al., 2023).”
Análise dos efeitos do idioma na geração automática de respostas por aplicações de LLM
Delbone, F. R., Wiese, I., & Silva, M. A. G. (2024) · doi:10.5753/educomp_estendido.2024.238811 · Cited by 3
Ramesh, A., Agarwal, A., Doughty, J. A., Ramaneti, K., Šavelka, J., & Sakr, M. F. (2024) · ACM Virtual Global Computing Education Conference (SIGCSE Virtual) · doi:10.1145/3649165.3690123 · Cited by 7
“These include course readings [17], learning objectives [34], code explanations [18, 21], model solutions [33], feedback [28], or responses to help requests [14, 19].”
Self-citationKasinidou, M., Kleanthous, S., & Otterbacher, J. (2024) · ACM Virtual Global Computing Education Conference (SIGCSE Virtual) · doi:10.1145/3649165.3690113 · Cited by 5
Does GPT Really Get It? A Hierarchical Scale to Quantify Human and AI’s Understanding of Algorithms
Reid, M., & Vempala, S. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i2.32140 · Cited by 3
“This is in line with other findings showing the competence of GPT in code generation (Vaithilingam, Zhang, and Glassman 2022; Savelka et al. 2023).”
ResultÅkerfeldt, A., Mannila, L., Kjällander, S., & Heintz, F. (2025) · ACM Transactions on Computing Education · doi:10.1145/3767723 · Cited by 2
“Some countries have introduced CS as a subject of its own (e.g., Computing in England [19]), while others have decided to integrate it with other subjects, by for instance making programming an interdisciplinary element throughout the curriculum (e.g., through digital competence in Sweden [73] and…”
Design Implications for Student and Educator Needs in AI-Supported Programming Learning Tools
Ma, B., Xie, Y., Li, H., Li, G., Chen, L. Y. E., Shimada, A., & Konomi, S. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2730083 · Cited by 3
AI-Generated Slides: Are They Good? Can Students Tell?
Leinonen, J., Zhang, L., & Hellas, A. (2026) · Western Canadian Conference on Computing Education · doi:10.1145/3828792.3828803 · Cited by 2
“More recently, Denny et al. [4] explored using natural language prompts to generate CS1 solutions, while Savelka et al. [25] documented GPT-4’s improved performance on higher education programming assessments, underscoring rapid capability growth.”
Could ChatGPT Be Used for Reviewing Learnersourced Exercises?
Pirttinen, N., & Leinonen, J. (2023) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631845 · Cited by 1
“In computing education, LLMs have been found to solve introductory programming problems better than the average student [4, 22], to generate code explanations [21] that students find useful [13] and that are better than explanations created by their peers [10], and there are preliminary results…”
BackgroundEnhancing CS1 Education through Experiential Learning with Robotics Projects
Borela, R., Liding, Z. C., & McDaniel, M. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701810 · Cited by 4
Kılınç, S. (2025) · Future education and learning spaces · doi:10.1007/978-981-96-9251-4_2 · Cited by 3
Beyond Verbal Self-Explanations: Student Annotations of a Code-Tracing Example Produced by ChatGPT
Gandhi, A. (2024) · Carleton University's Institutional Repository (MacOdrum Library, Carleton University) · doi:10.22215/etd/2024-16309
"I'm not sure, but...": Expert Practices that Enable Effective Code Comprehension in Data Science
Lum, C., Xu, G., & Lau, S. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701933 · Cited by 1
“This need is heightened by the emergence of large language models (LLMs), which, in the context of data science programming, can often generate code that runs without raising an error but is slightly incorrect and requires careful debugging [29].”
Generating Distractors for Code Completion Problems: Can LLM Assist Instructors?
Hassany, M., Akhüseyinoğlu, K., Narayanan, A. B. L., Agarwal, A., Šavelka, J., & Brusilovsky, P. L. (2025) · Proceedings of the ... International Florida Artificial Intelligence Research Society Conference · doi:10.32473/flairs.38.1.138995
Self-citationTHE ROLE OF GENERATIVE ARTIFICIAL INTELLIGENCE IN PROGRAMMER TRAINING: OPPORTUNITIES AND CHALLENGES
Hadzhikoleva, S., Rachovski, T., Hadzhikolev, E. N., & Ivanov, I. (2025) · Environment Technology Resources Proceedings of the International Scientific and Practical Conference · doi:10.17770/etr2025vol2.8608
“Savelka and colleagues investigate the ability of large language models (LLMs) to pass assessments in postgraduate-level Python programming courses [9].”
Alrobai, A., & Cristea, A. I. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-99261-2_31 · Cited by 1
Analysis of Students’ GenAI Prompts During a Practical SQL Test
Ramakrishnan, C., Cassidy, S., & Bower, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98465-5_23
How Effectively Can Off-the-Shelf Generative AI Tools Solve Circuit Design Problems?
Azhaan, M., Afeef, M., & Kumar, V. (2025) · Lecture notes in educational technology · doi:10.1007/978-981-95-1734-3_21
Skodová, J., & Hynninen, T. (2025) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3769994.3770059 · Cited by 1
“Given that LLMs can pass most introductory programming code problems [6, 21] or act as teaching assistants [9], they can be used to simulate students to produce data for pilot studies when such data does not yet exist or collecting it is difficult due to privacy reasons.”
ChatGPT-4: A fresh lens on holistic education’s reimagination
Baskara, F. R. (2025) · AIP conference proceedings · doi:10.1063/5.0261979
Laaksonen, A. (2026) · Undergraduate topics in computer science · doi:10.1007/978-3-032-07453-9_11
Markova, A., Kasam, A., Hackel, B., Langlois, M., & Lau, S. (2026) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3770762.3772518 · Cited by 1
Bamfo, K. A., Hall-Holt, O. A., Ola, O., Owusu, D. A., & Yeluripati, G. R. (2026) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3770762.3772659 · Cited by 3
Unraveling Ambiguities: Analyzing Student Approaches to Solving Probeable Problems
Kumar, V., Denny, P. C., Leinonen, J., Prather, J. E., & MacNeil, S. (2026) · Australasian Computing Education Conference · doi:10.1145/3786228.3786255 · Cited by 1
Feng, H., Hooper, S., Denny, P. C., Wünsche, B. C., & Luxton-Reilly, A. (2026) · Australasian Computing Education Conference · doi:10.1145/3786228.3786238 · Cited by 1
“GenAI’s ability to solve a comprehensive range of problems [8, 10, 36, 44] has created opportunities to improve Computing Education [2, 34], such as: instructor-facing tools for generating exercise questions, model solutions, and code explanations [9, 20, 41]; and student-facing tools for…”
Yadav, J., Arora, S., & Kashyap, N. (2026) · AIP conference proceedings · doi:10.1063/5.0317818
QVAS: A Multi-Dimensional Framework for Scoring Exam Question Vulnerability to Large Language Models
Ratul, Q. E. A., Zaman, E., & Das, S. (2026) · Intermountain Engineering, Technology and Computing (IETC) · doi:10.1109/ietc69527.2026.11568581
“In CS, Savelka et al. [36] reported 84.1% GPT-4 accuracy on 530 programming MCQs, and Bordt and von Luxburg [37] showed GPT-4 matched the average student on a CS algorithms exam.”
InfluentialAn LLM-Based Guided Programming Assistance System for Code Quality Feedback and Formative Assessment
Liu, G. (2026) · Applied Sciences · doi:10.3390/app16136455
Wyk, R. J. V., & Chaipah, K. (2026) · International Joint Conference on Computer Science and Software Engineering · doi:10.1109/jcsse68839.2026.11597033
Xuan, T. T. T., Salminen, J. O., Tillotson, J. S., Akhtar, W., Bui, T. V., Aldous, K. K., Medina, J., Jung, S., & Jansen, B. J. (2026) · Journal of Educational Technology Systems · doi:10.1177/00472395261467419
“Some students possess little to no exposure to AI technologies (Alsobeh & Woodward, 2023; Savelka et al., 2023).”
Álvarez-Jiménez, M., Pomagualli, A. G. C., Gamboa, D., & Oscullo, D. S. P. (2026) · Computers · doi:10.3390/computers15070466 · Cited by 2
“Several studies report cases in which GenAI completes exams or assignments with little evidence of authentic learning, undermining academic integrity and, in some instances, leading to self-sabotaging behaviors when students rely on inaccurate outputs [59–61].”
Khan, T. I. (2026) · Research Square · doi:10.21203/rs.3.rs-10513554/v1
Scaffolding Autocomplete: Improving Guidance for Learners using Generative Code Suggestions
Prather, J. E., MacNeil, S., Luxton-Reilly, A., Margulieux, L. E., Reeves, B. N., Denny, P. C., Leinonen, J., Homer, J., Nabid, R. A., & Rossetti, R. L. (2026) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3765964.3811654
Tran, K., Harper, C., & Price, T. (2026) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3765964.3811653 · Cited by 2
Evaluating LLM-Generated Contextualized Algorithm Design Problems
Goodwin, E., Braught, K., Liu, J., Newar, D. K. P., Gertner, Y., Poulsen, S., & Franklin, D. M. (2026) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3765964.3811658
Wang, X., Hou, Z., Lang, S., & Yu, Z. (2026) · IEEE Conference on Cybernetics and Intelligent Systems · doi:10.1109/cis-ram71614.2026.11668538
“Large language models have demonstrated performance on programming tasks that approaches, and in some cases surpasses, that of novice human programmers [5-6] .”
Explanation First Prompting for Reviewing AI Programming Answers in Undergraduate Courses
Raiter, O. К. (2026) · International Conference on Advanced Computer Information Technologies (ACIT) · doi:10.1109/acit69831.2026.11700007
Metadata
- Authors
- Savelka, J., Agarwal, A., An, M., Bogart, C., & Sakr, M.
- Affiliations
- Carnegie Mellon University, USA
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 2023 ACM Conference on International Computing Education Research V.1
- Publisher
- ACM
- Published
- 2023-08-07
- Pages
- 78-92
- OpenAlex
- W4381587445
- Cited by
- 141
- References
- 48
- Citations by year
- 2026: 21 · 2025: 54 · 2024: 57 · 2023: 9
- FWCI
- 42.59
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on International Computing Education Research (ICER) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- large language models, GPT-4, programming education, Python programming course, automated assessment, automated grading, multiple-choice questions, programming exercises, generative AI in higher education, assessment design, GPT-3
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.