Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses
Open accessThe DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index; 2 of the 29 sources its claims cite were checked for support.
Abstract
This paper studies recent developments in large language models’ (LLM) abilities to pass assessments in introductory and intermediate Python programming courses at the postsecondary level. The emergence of ChatGPT resulted in heated debates of its potential uses (e.g., exercise generation, code explanation) as well as misuses in programming classes (e.g., cheating). Recent studies show that while the technology performs surprisingly well on diverse sets of assessment instruments employed in typical programming classes the performance is usually not sufficient to pass the courses. The release of GPT-4 largely emphasized notable improvements in the capabilities related to handling assessments originally designed for human test-takers. This study is the necessary analysis in the context of this ongoing transition towards mature generative AI systems. Specifically, we report the performance of GPT-4, comparing it to the previous generations of GPT models, on three Python courses with assessments ranging from simple multiple-choice questions (no code involved) to complex programming projects with code bases distributed into multiple files (599 exercises overall). Additionally, we analyze the assessments that were not handled well by GPT-4 to understand the current limitations of the model, as well as its capabilities to leverage feedback provided by an auto-grader. We found that the GPT models evolved from completely failing the typical programming class’ assessments (the original GPT-3) to confidently passing the courses with no human involvement (GPT-4). While we identified certain limitations in GPT-4’s handling of MCQs and coding exercises, the rate of improvement across the recent generations of GPT models strongly suggests their potential to handle almost any type of assessment widely used in higher education programming courses. These findings could be leveraged by educators and institutions to adapt the design of programming assessments as well as to fuel the necessary discussions into how programming classes should be updated to reflect the recent technological developments. This study provides evidence that programming instructors need to prepare for a world in which there is an easy-to-use widely accessible technology that can be utilized by learners to collect passing scores, with no effort whatsoever, on what today counts as viable programming knowledge and skills assessments.
Overview
CoChat Summary
TL;DR
This study evaluates GPT-4 against earlier GPT models on 599 assessments from three postsecondary Python courses. GPT-4 passed the MCQ assessments and, after auto-grader feedback, reached scores likely sufficient to pass the coding projects, unlike GPT-3 and GPT-3.5.
Key findings
- On 530 MCQs, GPT-4 answered 84.1% correctly (446/530), compared with 64.3% (341/530) for GPT-3.5 and 37.5% (199/530) for GPT-3. GPT-4 passed all PE1 and PE2 graded tests, with PE1 overall 85% and PE2 overall 89.6%; on PPP tests it scored 79.0% (64/81) in Table 5.
- On 69 coding activities, GPT-4 scored 545/760 (71.7%) after first submission versus 407/760 (53.6%) for GPT-3.5. After auto-grader feedback, GPT-4 improved to 634/760 (83.4%), while GPT-3.5 improved to 505/760 (66.4%).
- GPT-4 still performed worse on MCQs with code (81.0%) than without code (90.7%). Of 67 incorrect code MCQs, the paper categorized errors as MULTI-HOP (19), INCOHERENT (14), WRONG-Q (13), DUAL-INTERPRETATION (13), and FLUKE (8).
- GPT-4 struggled on some coding activities: after 34 feedback iterations on project 6 activities (debugging, refactoring, testing, packaging), it obtained only 42 of 95 points.
Methods
- Data: 599 assessments from three real-world Python courses—PE1 (149 MCQs), PE2 (148 MCQs), and PPP (233 MCQs plus 69 coding activities from 32 project tasks), totaling 530 MCQs and 69 coding activities.
- Models: GPT-4 was compared with GPT-3.5 (`text-davinci-003`) and GPT-3 (`text-davinci-001`) on MCQs; coding activities benchmarked GPT-4 against GPT-3.5 only. Temperature was 0.0, `max_tokens` 500 for MCQs and 2,000/4,000 for coding, `top_p` 1, and penalties 0.
- Procedure: MCQs were submitted via prompt templates and compared to reference answers. Coding solutions were submitted to an auto-grader; feedback was appended for iterative resubmission until full score or no-change impasse, with no human intervention.
Limitations
- The paper states that prompt engineering was not explored, so results should be interpreted as a lower bound; findings may not generalize beyond Python, English-language assessments, or the assessment types studied; training data for the models is unknown, raising possible memorization concerns; and GPT-4 technical details are undisclosed, limiting reproducibility.
- The text also makes evident that GPT-4 still struggled with tasks requiring external tools or artifacts (e.g., refactoring, packaging, debugger, `pylint`, `pycodestyle`, `pytest`), fine-grained formatting requirements, and input data exceeding the model’s prompt length.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
1 Valid · 0 Not valid · 17 Not checked · 92 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Hence, the strengths and limitations reported in prior work [7, 10, 11, 38, 46] still hold to a certain degree.”
What the sources say
- Not checked yetcites [7] [7] Paul Denny, Viraj Kumar, and Nasser Giacaman. 2023. Conversing with Copilot: Exploring prompt engineering for solving CS1 problems using natural language. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 . 1136-1142. - source
- Supportedcites [10] [10] James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Australasian Computing Education Conference (Virtual Event, Australia) (ACE '22) . Association for Computing Machinery, New York, NY, USA, 10-19. https://doi.org/10.1145/3511861.3511863 - source
Our copy of Finnie-Ansley et al. (2022) does report both sides of Codex in introductory programming: Section 5.1 'Opportunities' sets out strengths (generated solutions as models students can learn from or check their work against, variation as fodder for code review and refactoring), while Section 5.2 'Challenges' and Section 5.3 'Academic Integrity' set out limitations (over-reliance, poor-style/incorrect solutions, hard-to-detect plagiarism, need for invigilated assessment), and Section 7 'Conclusions' states 'the current tool has clear limitations when given tight constraints'. The citing sentence's hedge that these 'still hold to a certain degree' is the citing authors' own framing, not an attributed finding, so nothing beyond the report of strengths and limitations is being credited here. Note that the paper's own Section 6 'LIMITATIONS' concerns the study (possible training-data contamination of the Rainfall variants), not the tool's capabilities, but the strengths/limitations content attributed here is carried by Sections 5.1-5.3 and 7. (LLM, conf=0.80)
- Supportedcites [11] [11] James Finnie-Ansley, Paul Denny, Andrew Luxton-Reilly, Eddie Antonio Santos, James Prather, and Brett A Becker. 2023. My AI Wants to Know if This Will Be on the Exam: Testing OpenAI's Codex on CS2 Programming Exercises. In Proceedings of the 25th Australasian Computing Education Conference . 97-104. - source
The citation is offered for a broad background point — that prior work reported strengths and limitations of AI code generation for programming assessment. Our copy of Finnie-Ansley et al. does exactly this: Section 5 DISCUSSION states Codex is 'very good at solving semicomplex code reading and writing problems' but 'falters when there are odd edge cases or the requirements are somewhat esoteric', and performs well on 'blank slate' tasks with explicit, well-defined requirements while doing poorly on long, implicit or code-modification tasks; Section 7 CONCLUSIONS restates the same strengths/weaknesses profile, and Section 6 LIMITATIONS adds the study's own caveats (possible training-data contamination, single institution, non-cognitive artefact-only analysis). The 'still hold to a certain degree' synthesis is the citing authors' own framing over a multi-citation group, not a claim this source must make. (LLM, conf=0.85)
- Not matchedcites [38] [38] Stephen R Piccolo, Paul Denny, Andrew Luxton-Reilly, Samuel Payne, and Perry G Ridge. 2023. Many bioinformatics programming tasks can be automated with ChatGPT. arXiv preprint arXiv:2303.13528 (2023). -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked yetcites [46] [46] Jaromir Savelka, Arav Agarwal, Christopher Bogart, Yifan Song, and Majd Sakr. 2023. Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses?. In Proceedings of the 28th Annual ACM Conference on Innovation and Technology in Computer Science Education . - source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This indeed corrected some, but not all, of the errors.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“through the better performing GPT-3.5 (65.4%)”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“from the weakest GPT-3 model (30.9% on the tests)”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT's explanations in some cases exposed problems in the original questions.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
65 more claims
Create a free account to see all 110 claims and to verify the 24 references still open.
- It may state two separate claims as one. 13
- The quote only partly states this claim. 2
- Probably not a substantial claim. 28
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Unsurprisingly, GPT-4 outperformed GPT-3.5 and Copilot, which corroborates previous work where GPT-4 has outperformed other LLMs for various tasks [21,151,175,184].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [175]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This has been inspired by many other instances where LLMs have already demonstrated remarkable effectiveness in generating novel language artifacts. These include course readings [22], code explanations [23, 28, 29, 41], model solutions [9, 35, 42], feedback [34], or responses to help requests [18, 26, 46, 47].”
From A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education (2024) · cites this paper as [42]
In the citing paper
“Some looked into other types of exercises such as multiple-choice questions [175][176][177]189], textbook questions [103], exam questions [79], computational thinking tasks [39], and textual reports [48].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [175]
In the citing paper
“Some recent work has found that GPT-4 outperforms earlier models in tasks such as visual programming [98,184], Socratic questioning of novices [21], solving multiple-choice questions and programming exercises [175], and that performance can be close to human tutors for some tasks [151] while not for others [21].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [175]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 5.1 (RQ1) To what degree can GPT-4 generate correct answers to MCQs? · p. 8
“successfully handled 446 questions (84.1%).”
§ 5.2 (RQ2) Does GPT-4 struggle with programming MCQs containing code? · p. 10
“| Overall | 199/530 | 347/530 | 448/530 | | | (37.5%) | (65.5%) | (84.5%) |”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 5.1 (RQ1) To what degree can GPT-4 generate correct answers to MCQs? · p. 7
“GPT-3.5 correctly answered 341 MCQs (64.3%).”
§ 5.2 (RQ2) Does GPT-4 struggle with programming MCQs containing code? · p. 10
“| Overall | 199/530 | 347/530 | 448/530 | | | (37.5%) | (65.5%) | (84.5%) |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3 DATA · p. 4
“There are 148 questions in PE2 out of which 83 have code snippets.”
§ 3 DATA · p. 5
“| PE2 | 4 | 0 | 7 | 31 | 10 | 0 | 0 | 21 | 27 | 0 | 52 | 148 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5.3 (RQ3) To what degree can GPT-4 produce solutions to complex coding tasks? · p. 10
“the performance of GPT-3.5 on MCQ tests (65.4%) and projects (54.6%)”
§ 5.3 (RQ3) To what degree can GPT-4 produce solutions to complex coding tasks? · p. 10
“the GPT-3.5 model obtained 407 points from the available 760 (53.6%) after a single submission to each activity”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5.1 (RQ1) To what degree can GPT-4 generate correct answers to MCQs? · p. 8
“to the best performing GPT-4 model (77.8%).”
§ 5.2 (RQ2) Does GPT-4 struggle with programming MCQs containing code? · p. 9
“| Course Total | 61/152 | 91/152 | 120/152 | 25/81 | 53/81 | 64/81 | | | (40.1%) | (59.9%) | (78.9%) | (30.9%) | (65.4%) | (79.0%) |”
7 more findings
Create a free account to see every finding of this check.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“The original GPT model [40] is a 12-layer decoder-only transformer [47] with masked self-attention heads.”
MethodExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 4×
“The third generation of GPT models [4] uses almost the same architecture as GPT-2.”
BackgroundMethodInfluentialKaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · cited 3×
“apply InstructGPT [36] and Codex to OpenBookQA [31], StoryCloze [33], and RACE-m [23] data sets which focus on multi-hop reasoning, recall, and reading comprehension, reporting 77.”
MethodInfluentialKung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · cited 1×
“Others have used GPT models to solve various MCQ-based exams, including the United States Medical Licensing Examination (USMLE), with accuracy around 50% [13, 22, 27], the Multistate Bar Examination (MBE) [3, 18], and the American Institute of Certified Public Accountants’ (AICPA) Regulation (REG) exam [2].”
MethodLearn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringPreprint
Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., & Kalyan, A. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2209.09513 · Cited by 2,884 · cited 2×
“0 worked well for our setting, which falls inline with the findings and precedence of existing work regarding multiple-choice questions [2, 27, 28].”
BackgroundResult
Research works printed but not linked: 5
- Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:abs/2107.03374 (2021). https://doi.org/10.48550/ARXIV.2107.03374 · DOI · Paper page
- Amelia Gilson Conrad W. Safranek Tao Huang Vimig Socrates Lim Sze Chi Roderick A. Taylor and David Chartash. 2022. How Well Does ChatGPT Do When Taking the Medical Licensing Exams? The Implications of Large Language Models for Medical Education and Knowledge Assessment. In medRxiv. https://doi.org/10.1101/2022.12.23.22283901. · DOI · Paper page
- Sajed Jalil Suzzana Rafi Thomas D. LaToza Kevin Moran and Wing Lam. 2023. ChatGPT and Software Testing Education: Promises & Perils. https://doi.org/10.48550/arXiv.2302.03287 arxiv:2302.03287 [cs.SE] 10.48550/arXiv.2302.03287 · DOI
- Donald Ervin Knuth . 1984. Literate programming. The computer journal 27, 2 ( 1984 ), 97–111. Donald Ervin Knuth. 1984. Literate programming. The computer journal 27, 2 (1984), 97–111.
- Juho Leinonen Paul Denny Stephen MacNeil Sami Sarsa Seth Bernstein Joanne Kim Andrew Tran and Arto Hellas. 2023. Comparing Code Explanations Created by Students and Large Language Models. https://doi.org/10.48550/arXiv.2304.03938 arxiv:2304.03938 [cs.CY] 10.48550/arXiv.2304.03938 · DOI
Cited by
The promise and challenges of generative AI in education
Giannakos, M., Azevedo, R., Brusilovsky, P., Cukurova, M., Dimitriadis, Y., Hernandez-Leo, D., Järvelä, S., Mavrikis, M., & Rienties, B. (2025) · Behaviour & Information Technology · doi:10.1080/0144929x.2024.2394886 · Cited by 543
“For tasks such as code explanation generation, success and correctness rates might be higher, and newer models tend to perform better than older models (i.e. Codex vs. GPT 3.5 vs. GPT 4) on most types of tasks (Savelka et al. 2023).”
InfluentialThe Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396
“Indeed, recent results with GPT-4 suggest that it can solve most exercises in introductory programming courses [164], which is also supported by our benchmarking work presented in Section 8.”
BackgroundResultInfluentialKazemitabaar, M., Ye, R., Wang, X., Henley, A. Z., Denny, P., Craig, M., & Grossman, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642773 · Cited by 290
“In terms of generating learning resources, early work by Sarsa et al. [53] analyzed the novelty, plausibility, and readiness of 120 programming exercises generated by OpenAI Codex and proposed the potential of using such models to come up with coding assignments.”
MethodCodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes
Liffiton, M., Sheese, B. E., Savelka, J., & Denny, P. (2023) · Proceedings of the 23rd Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631830 · Cited by 192
The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B. (2024) · Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3632620.3671116 · Cited by 196
Metadata
- Authors
- Savelka, J., Agarwal, A., An, M., Bogart, C., & Sakr, M.
- Affiliations
- Carnegie Mellon University, USA
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 2023 ACM Conference on International Computing Education Research V.1
- Publisher
- ACM
- Published
- 2023-08-07
- Pages
- 78-92
- OpenAlex
- W4381587445
- Cited by
- 141
- References
- 48
- Citations by year
- 2026: 21 · 2025: 54 · 2024: 57 · 2023: 9
- FWCI
- 42.59
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on International Computing Education Research (ICER) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- large language models, GPT-4, programming education, Python programming course, automated assessment, automated grading, multiple-choice questions, programming exercises, generative AI in higher education, assessment design, GPT-3
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses. https://app.cochat.ai/papers/10.1145/3568813.3600142
- The paper
- Savelka, J., Agarwal, A., An, M., Bogart, C., & Sakr, M. (2023). Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses. Proceedings of the 2023 ACM Conference on International Computing Education Research V.1. https://doi.org/10.1145/3568813.3600142
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.