CoChat

Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses

Open access
Savelka, Jaromir, Agarwal, Arav, An, Marshall, Bogart, Chris, Sakr, Majd · 2023 · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1

The DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index; 2 of the 29 sources its claims cite were checked for support.

Abstract

This paper studies recent developments in large language models’ (LLM) abilities to pass assessments in introductory and intermediate Python programming courses at the postsecondary level. The emergence of ChatGPT resulted in heated debates of its potential uses (e.g., exercise generation, code explanation) as well as misuses in programming classes (e.g., cheating). Recent studies show that while the technology performs surprisingly well on diverse sets of assessment instruments employed in typical programming classes the performance is usually not sufficient to pass the courses. The release of GPT-4 largely emphasized notable improvements in the capabilities related to handling assessments originally designed for human test-takers. This study is the necessary analysis in the context of this ongoing transition towards mature generative AI systems. Specifically, we report the performance of GPT-4, comparing it to the previous generations of GPT models, on three Python courses with assessments ranging from simple multiple-choice questions (no code involved) to complex programming projects with code bases distributed into multiple files (599 exercises overall). Additionally, we analyze the assessments that were not handled well by GPT-4 to understand the current limitations of the model, as well as its capabilities to leverage feedback provided by an auto-grader. We found that the GPT models evolved from completely failing the typical programming class’ assessments (the original GPT-3) to confidently passing the courses with no human involvement (GPT-4). While we identified certain limitations in GPT-4’s handling of MCQs and coding exercises, the rate of improvement across the recent generations of GPT models strongly suggests their potential to handle almost any type of assessment widely used in higher education programming courses. These findings could be leveraged by educators and institutions to adapt the design of programming assessments as well as to fuel the necessary discussions into how programming classes should be updated to reflect the recent technological developments. This study provides evidence that programming instructors need to prepare for a world in which there is an easy-to-use widely accessible technology that can be utilized by learners to collect passing scores, with no effort whatsoever, on what today counts as viable programming knowledge and skills assessments.

Overview

CoChat Summary

TL;DR

This study evaluates GPT-4 against earlier GPT models on 599 assessments from three postsecondary Python courses. GPT-4 passed the MCQ assessments and, after auto-grader feedback, reached scores likely sufficient to pass the coding projects, unlike GPT-3 and GPT-3.5.

Key findings

  • On 530 MCQs, GPT-4 answered 84.1% correctly (446/530), compared with 64.3% (341/530) for GPT-3.5 and 37.5% (199/530) for GPT-3. GPT-4 passed all PE1 and PE2 graded tests, with PE1 overall 85% and PE2 overall 89.6%; on PPP tests it scored 79.0% (64/81) in Table 5.
  • On 69 coding activities, GPT-4 scored 545/760 (71.7%) after first submission versus 407/760 (53.6%) for GPT-3.5. After auto-grader feedback, GPT-4 improved to 634/760 (83.4%), while GPT-3.5 improved to 505/760 (66.4%).
  • GPT-4 still performed worse on MCQs with code (81.0%) than without code (90.7%). Of 67 incorrect code MCQs, the paper categorized errors as MULTI-HOP (19), INCOHERENT (14), WRONG-Q (13), DUAL-INTERPRETATION (13), and FLUKE (8).
  • GPT-4 struggled on some coding activities: after 34 feedback iterations on project 6 activities (debugging, refactoring, testing, packaging), it obtained only 42 of 95 points.

Methods

  • Data: 599 assessments from three real-world Python courses—PE1 (149 MCQs), PE2 (148 MCQs), and PPP (233 MCQs plus 69 coding activities from 32 project tasks), totaling 530 MCQs and 69 coding activities.
  • Models: GPT-4 was compared with GPT-3.5 (`text-davinci-003`) and GPT-3 (`text-davinci-001`) on MCQs; coding activities benchmarked GPT-4 against GPT-3.5 only. Temperature was 0.0, `max_tokens` 500 for MCQs and 2,000/4,000 for coding, `top_p` 1, and penalties 0.
  • Procedure: MCQs were submitted via prompt templates and compared to reference answers. Coding solutions were submitted to an auto-grader; feedback was appended for iterative resubmission until full score or no-change impasse, with no human intervention.

Limitations

  • The paper states that prompt engineering was not explored, so results should be interpreted as a lower bound; findings may not generalize beyond Python, English-language assessments, or the assessment types studied; training data for the models is unknown, raising possible memorization concerns; and GPT-4 technical details are undisclosed, limiting reproducibility.
  • The text also makes evident that GPT-4 still struggled with tasks requiring external tools or artifacts (e.g., refactoring, packaging, debugger, `pylint`, `pycodestyle`, `pytest`), fine-grained formatting requirements, and input data exceeding the model’s prompt length.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3568813.3600142

Cites

64 of 64 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

1 Valid · 0 Not valid · 17 Not checked · 92 Own results and interpretations

65 more claims

Create a free account to see all 110 claims and to verify the 24 references still open.

See all claims — free account
  • It may state two separate claims as one. 13
  • The quote only partly states this claim. 2
  • Probably not a substantial claim. 28

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 12 · the rest aren’t loaded

7 more findings

Create a free account to see every finding of this check.

See all findings — free account

Citations

Cites

Show all 64 references

Research works printed but not linked: 5

  1. Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:abs/2107.03374 (2021). https://doi.org/10.48550/ARXIV.2107.03374 · DOI · Paper page
  2. Amelia Gilson Conrad W. Safranek Tao Huang Vimig Socrates Lim Sze Chi Roderick A. Taylor and David Chartash. 2022. How Well Does ChatGPT Do When Taking the Medical Licensing Exams? The Implications of Large Language Models for Medical Education and Knowledge Assessment. In medRxiv. https://doi.org/10.1101/2022.12.23.22283901. · DOI · Paper page
  3. Sajed Jalil Suzzana Rafi Thomas D. LaToza Kevin Moran and Wing Lam. 2023. ChatGPT and Software Testing Education: Promises & Perils. https://doi.org/10.48550/arXiv.2302.03287 arxiv:2302.03287 [cs.SE] 10.48550/arXiv.2302.03287 · DOI
  4. Donald Ervin Knuth . 1984. Literate programming. The computer journal 27, 2 ( 1984 ), 97–111. Donald Ervin Knuth. 1984. Literate programming. The computer journal 27, 2 (1984), 97–111.
  5. Juho Leinonen Paul Denny Stephen MacNeil Sami Sarsa Seth Bernstein Joanne Kim Andrew Tran and Arto Hellas. 2023. Comparing Code Explanations Created by Students and Large Language Models. https://doi.org/10.48550/arXiv.2304.03938 arxiv:2304.03938 [cs.CY] 10.48550/arXiv.2304.03938 · DOI

Cited by

See all 143

Metadata

Authors
Savelka, J., Agarwal, A., An, M., Bogart, C., & Sakr, M.
Affiliations
  • Carnegie Mellon University, USA
  • Carnegie Mellon University (US)
Venue
Proceedings of the 2023 ACM Conference on International Computing Education Research V.1
Publisher
ACM
Published
2023-08-07
Pages
78-92
OpenAlex
W4381587445
Cited by
141
References
48
Citations by year
2026: 21 · 2025: 54 · 2024: 57 · 2023: 9
FWCI
42.59
Type
conference paper
Language
en
Open access
Locations
  • ACM Conference on International Computing Education Research (ICER) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Social Sciences › Education › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Keywords
large language models, GPT-4, programming education, Python programming course, automated assessment, automated grading, multiple-choice questions, programming exercises, generative AI in higher education, assessment design, GPT-3

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses. https://app.cochat.ai/papers/10.1145/3568813.3600142
The paper
Savelka, J., Agarwal, A., An, M., Bogart, C., & Sakr, M. (2023). Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses. Proceedings of the 2023 ACM Conference on International Computing Education Research V.1. https://doi.org/10.1145/3568813.3600142

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.