Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses
Open accessThe DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index; 2 of the 29 sources its claims cite were checked for support.
Abstract
This paper studies recent developments in large language models’ (LLM) abilities to pass assessments in introductory and intermediate Python programming courses at the postsecondary level. The emergence of ChatGPT resulted in heated debates of its potential uses (e.g., exercise generation, code explanation) as well as misuses in programming classes (e.g., cheating). Recent studies show that while the technology performs surprisingly well on diverse sets of assessment instruments employed in typical programming classes the performance is usually not sufficient to pass the courses. The release of GPT-4 largely emphasized notable improvements in the capabilities related to handling assessments originally designed for human test-takers. This study is the necessary analysis in the context of this ongoing transition towards mature generative AI systems. Specifically, we report the performance of GPT-4, comparing it to the previous generations of GPT models, on three Python courses with assessments ranging from simple multiple-choice questions (no code involved) to complex programming projects with code bases distributed into multiple files (599 exercises overall). Additionally, we analyze the assessments that were not handled well by GPT-4 to understand the current limitations of the model, as well as its capabilities to leverage feedback provided by an auto-grader. We found that the GPT models evolved from completely failing the typical programming class’ assessments (the original GPT-3) to confidently passing the courses with no human involvement (GPT-4). While we identified certain limitations in GPT-4’s handling of MCQs and coding exercises, the rate of improvement across the recent generations of GPT models strongly suggests their potential to handle almost any type of assessment widely used in higher education programming courses. These findings could be leveraged by educators and institutions to adapt the design of programming assessments as well as to fuel the necessary discussions into how programming classes should be updated to reflect the recent technological developments. This study provides evidence that programming instructors need to prepare for a world in which there is an easy-to-use widely accessible technology that can be utilized by learners to collect passing scores, with no effort whatsoever, on what today counts as viable programming knowledge and skills assessments.
Overview
CoChat Summary
TL;DR
This study evaluates GPT-4 against earlier GPT models on 599 assessments from three postsecondary Python courses. GPT-4 passed the MCQ assessments and, after auto-grader feedback, reached scores likely sufficient to pass the coding projects, unlike GPT-3 and GPT-3.5.
Key findings
- On 530 MCQs, GPT-4 answered 84.1% correctly (446/530), compared with 64.3% (341/530) for GPT-3.5 and 37.5% (199/530) for GPT-3. GPT-4 passed all PE1 and PE2 graded tests, with PE1 overall 85% and PE2 overall 89.6%; on PPP tests it scored 79.0% (64/81) in Table 5.
- On 69 coding activities, GPT-4 scored 545/760 (71.7%) after first submission versus 407/760 (53.6%) for GPT-3.5. After auto-grader feedback, GPT-4 improved to 634/760 (83.4%), while GPT-3.5 improved to 505/760 (66.4%).
- GPT-4 still performed worse on MCQs with code (81.0%) than without code (90.7%). Of 67 incorrect code MCQs, the paper categorized errors as MULTI-HOP (19), INCOHERENT (14), WRONG-Q (13), DUAL-INTERPRETATION (13), and FLUKE (8).
- GPT-4 struggled on some coding activities: after 34 feedback iterations on project 6 activities (debugging, refactoring, testing, packaging), it obtained only 42 of 95 points.
Methods
- Data: 599 assessments from three real-world Python courses—PE1 (149 MCQs), PE2 (148 MCQs), and PPP (233 MCQs plus 69 coding activities from 32 project tasks), totaling 530 MCQs and 69 coding activities.
- Models: GPT-4 was compared with GPT-3.5 (`text-davinci-003`) and GPT-3 (`text-davinci-001`) on MCQs; coding activities benchmarked GPT-4 against GPT-3.5 only. Temperature was 0.0, `max_tokens` 500 for MCQs and 2,000/4,000 for coding, `top_p` 1, and penalties 0.
- Procedure: MCQs were submitted via prompt templates and compared to reference answers. Coding solutions were submitted to an auto-grader; feedback was appended for iterative resubmission until full score or no-change impasse, with no human intervention.
Limitations
- The paper states that prompt engineering was not explored, so results should be interpreted as a lower bound; findings may not generalize beyond Python, English-language assessments, or the assessment types studied; training data for the models is unknown, raising possible memorization concerns; and GPT-4 technical details are undisclosed, limiting reproducibility.
- The text also makes evident that GPT-4 still struggled with tasks requiring external tools or artifacts (e.g., refactoring, packaging, debugger, `pylint`, `pycodestyle`, `pytest`), fine-grained formatting requirements, and input data exceeding the model’s prompt length.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
1 Valid · 0 Not valid · 17 Not checked · 92 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Hence, the strengths and limitations reported in prior work [7, 10, 11, 38, 46] still hold to a certain degree.”
What the sources say
- Not checked yetcites [7] [7] Paul Denny, Viraj Kumar, and Nasser Giacaman. 2023. Conversing with Copilot: Exploring prompt engineering for solving CS1 problems using natural language. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 . 1136-1142. - source
- Supportedcites [10] [10] James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Australasian Computing Education Conference (Virtual Event, Australia) (ACE '22) . Association for Computing Machinery, New York, NY, USA, 10-19. https://doi.org/10.1145/3511861.3511863 - source
Our copy of Finnie-Ansley et al. (2022) does report both sides of Codex in introductory programming: Section 5.1 'Opportunities' sets out strengths (generated solutions as models students can learn from or check their work against, variation as fodder for code review and refactoring), while Section 5.2 'Challenges' and Section 5.3 'Academic Integrity' set out limitations (over-reliance, poor-style/incorrect solutions, hard-to-detect plagiarism, need for invigilated assessment), and Section 7 'Conclusions' states 'the current tool has clear limitations when given tight constraints'. The citing sentence's hedge that these 'still hold to a certain degree' is the citing authors' own framing, not an attributed finding, so nothing beyond the report of strengths and limitations is being credited here. Note that the paper's own Section 6 'LIMITATIONS' concerns the study (possible training-data contamination of the Rainfall variants), not the tool's capabilities, but the strengths/limitations content attributed here is carried by Sections 5.1-5.3 and 7. (LLM, conf=0.80)
- Supportedcites [11] [11] James Finnie-Ansley, Paul Denny, Andrew Luxton-Reilly, Eddie Antonio Santos, James Prather, and Brett A Becker. 2023. My AI Wants to Know if This Will Be on the Exam: Testing OpenAI's Codex on CS2 Programming Exercises. In Proceedings of the 25th Australasian Computing Education Conference . 97-104. - source
The citation is offered for a broad background point — that prior work reported strengths and limitations of AI code generation for programming assessment. Our copy of Finnie-Ansley et al. does exactly this: Section 5 DISCUSSION states Codex is 'very good at solving semicomplex code reading and writing problems' but 'falters when there are odd edge cases or the requirements are somewhat esoteric', and performs well on 'blank slate' tasks with explicit, well-defined requirements while doing poorly on long, implicit or code-modification tasks; Section 7 CONCLUSIONS restates the same strengths/weaknesses profile, and Section 6 LIMITATIONS adds the study's own caveats (possible training-data contamination, single institution, non-cognitive artefact-only analysis). The 'still hold to a certain degree' synthesis is the citing authors' own framing over a multi-citation group, not a claim this source must make. (LLM, conf=0.85)
- Not matchedcites [38] [38] Stephen R Piccolo, Paul Denny, Andrew Luxton-Reilly, Samuel Payne, and Perry G Ridge. 2023. Many bioinformatics programming tasks can be automated with ChatGPT. arXiv preprint arXiv:2303.13528 (2023). -
We searched by its title and found no matching paper.
- Not checked yetcites [46] [46] Jaromir Savelka, Arav Agarwal, Christopher Bogart, Yifan Song, and Majd Sakr. 2023. Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses?. In Proceedings of the 28th Annual ACM Conference on Innovation and Technology in Computer Science Education . - source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This indeed corrected some, but not all, of the errors.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“through the better performing GPT-3.5 (65.4%)”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“from the weakest GPT-3 model (30.9% on the tests)”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT's explanations in some cases exposed problems in the original questions.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
65 more claims
Create a free account to see all 110 claims and to verify the 24 references still open.
- It may state two separate claims as one. 13
- The quote only partly states this claim. 2
- Probably not a substantial claim. 28
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Unsurprisingly, GPT-4 outperformed GPT-3.5 and Copilot, which corroborates previous work where GPT-4 has outperformed other LLMs for various tasks [21,151,175,184].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [175]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This has been inspired by many other instances where LLMs have already demonstrated remarkable effectiveness in generating novel language artifacts. These include course readings [22], code explanations [23, 28, 29, 41], model solutions [9, 35, 42], feedback [34], or responses to help requests [18, 26, 46, 47].”
From A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education (2024) · cites this paper as [42]
In the citing paper
“Some looked into other types of exercises such as multiple-choice questions [175][176][177]189], textbook questions [103], exam questions [79], computational thinking tasks [39], and textual reports [48].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [175]
In the citing paper
“Some recent work has found that GPT-4 outperforms earlier models in tasks such as visual programming [98,184], Socratic questioning of novices [21], solving multiple-choice questions and programming exercises [175], and that performance can be close to human tutors for some tasks [151] while not for others [21].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [175]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 5.1 (RQ1) To what degree can GPT-4 generate correct answers to MCQs? · p. 8
“successfully handled 446 questions (84.1%).”
§ 5.2 (RQ2) Does GPT-4 struggle with programming MCQs containing code? · p. 10
“| Overall | 199/530 | 347/530 | 448/530 | | | (37.5%) | (65.5%) | (84.5%) |”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 5.1 (RQ1) To what degree can GPT-4 generate correct answers to MCQs? · p. 7
“GPT-3.5 correctly answered 341 MCQs (64.3%).”
§ 5.2 (RQ2) Does GPT-4 struggle with programming MCQs containing code? · p. 10
“| Overall | 199/530 | 347/530 | 448/530 | | | (37.5%) | (65.5%) | (84.5%) |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3 DATA · p. 4
“There are 148 questions in PE2 out of which 83 have code snippets.”
§ 3 DATA · p. 5
“| PE2 | 4 | 0 | 7 | 31 | 10 | 0 | 0 | 21 | 27 | 0 | 52 | 148 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5.3 (RQ3) To what degree can GPT-4 produce solutions to complex coding tasks? · p. 10
“the performance of GPT-3.5 on MCQ tests (65.4%) and projects (54.6%)”
§ 5.3 (RQ3) To what degree can GPT-4 produce solutions to complex coding tasks? · p. 10
“the GPT-3.5 model obtained 407 points from the available 760 (53.6%) after a single submission to each activity”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5.1 (RQ1) To what degree can GPT-4 generate correct answers to MCQs? · p. 8
“to the best performing GPT-4 model (77.8%).”
§ 5.2 (RQ2) Does GPT-4 struggle with programming MCQs containing code? · p. 9
“| Course Total | 61/152 | 91/152 | 120/152 | 25/81 | 53/81 | 64/81 | | | (40.1%) | (59.9%) | (78.9%) | (30.9%) | (65.4%) | (79.0%) |”
7 more findings
Create a free account to see every finding of this check.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“The original GPT model [40] is a 12-layer decoder-only transformer [47] with masked self-attention heads.”
MethodExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 4×
“The third generation of GPT models [4] uses almost the same architecture as GPT-2.”
BackgroundMethodInfluentialKaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · cited 3×
“apply InstructGPT [36] and Codex to OpenBookQA [31], StoryCloze [33], and RACE-m [23] data sets which focus on multi-hop reasoning, recall, and reading comprehension, reporting 77.”
MethodInfluentialKung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · cited 1×
“Others have used GPT models to solve various MCQ-based exams, including the United States Medical Licensing Examination (USMLE), with accuracy around 50% [13, 22, 27], the Multistate Bar Examination (MBE) [3, 18], and the American Institute of Certified Public Accountants’ (AICPA) Regulation (REG) exam [2].”
MethodLearn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringPreprint
Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., & Kalyan, A. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2209.09513 · Cited by 2,884 · cited 2×
“0 worked well for our setting, which falls inline with the findings and precedence of existing work regarding multiple-choice questions [2, 27, 28].”
BackgroundResultGilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078
Project-based learning: A review of the literature
Kokotsaki, D., Menzies, V., & Wiggins, A. (2016) · Improving Schools · doi:10.1177/1365480216659733 · Cited by 1,760 · cited 1×
“In comparison to PE1 and PE2, PPP mostly employs the projectbased education model [21].”
MethodSparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · cited 1×
“GPT models have been shown to handle a number of programming languages [5, 8].”
BackgroundEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 1×
“This is to be expected, as the major breakthrough in OpenAI GPT models’ capabilities in handling computer code was Codex (code-davinci-002) [6] which is the predecessor of text-davinci-003 (i.”
BackgroundMaking Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.295 · Cited by 1,380 · cited 1×
“Numerous studies have explored prompts which include a number of examples to demonstrate what the desired output should be [4, 12, 50].”
BackgroundRACE: Large-scale ReAding Comprehension Dataset From Examinations
Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1082 · Cited by 1,044 · cited 2×
“apply InstructGPT [36] and Codex to OpenBookQA [31], StoryCloze [33], and RACE-m [23] data sets which focus on multi-hop reasoning, recall, and reading comprehension, reporting 77.”
MethodInfluentialCompetition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · cited 1×
“3% ranking amongst 5,000 participants [26].”
Competition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043 · cited 1×
“3% ranking amongst 5,000 participants [26].”
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Reynolds, L., & McDonell, K. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451760 · Cited by 952 · cited 1×
“However, the current research literature remains inconclusive as to the efficacy of adding examples to natural language prompts, with multiple studies suggesting that the order and number of examples can dramatically influence the performance of LLMs across various tasks [42, 50].”
BackgroundCan a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
Mihaylov, T., Clark, P., Khot, T., & Sabharwal, A. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1260 · Cited by 649 · cited 2×
“apply InstructGPT [36] and Codex to OpenBookQA [31], StoryCloze [33], and RACE-m [23] data sets which focus on multi-hop reasoning, recall, and reading comprehension, reporting 77.”
MethodInfluentialA Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories
Mostafazadeh, N., Chambers, N., He, X., Parikh, D., Batra, D., Vanderwende, L., Kohli, P., & Allen, J. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1098 · Cited by 598 · cited 2×
“apply InstructGPT [36] and Codex to OpenBookQA [31], StoryCloze [33], and RACE-m [23] data sets which focus on multi-hop reasoning, recall, and reading comprehension, reporting 77.”
MethodInfluentialCalibrate Before Use: Improving Few-Shot Performance of Language Models
Zhao, T. Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021) · arXiv · doi:10.48550/arxiv.2102.09690 · Cited by 513 · cited 1×
“In general, finding the best prompt for a specific task is challenging, with prompts that are semantically similar sometimes providing large differences in performance [51].”
BackgroundAutomatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497 · cited 1×
“used code-davinci-001 to generate 240 introductory programming exercises, along with tests, sample solutions and explanations, reporting that over 75% of the generated exercises were novel and suitable for use in a university setting [44].”
MethodThe Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.10625 · Cited by 380
Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580919 · Cited by 366 · cited 1×
“studied learners using OpenAI’s Codex during traditional code creation tasks, and demonstrated the use of Codex did not harm their performance, with experienced students performing significantly better [19].”
MethodLearn to Explain: Multimodal Reasoning Via Thought Chains for Science Question Answering
Lu, P., Mishra, S. R., Xia, T., Qiu, L., Chang, K.-W., Zhu, S. C., Tafjord, O., Clark, P., & Kalyan, A. (2022) · neural information processing systems · doi:10.52202/068431-0182 · Cited by 362
Can large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2024) · Patterns · doi:10.1016/j.patter.2024.100943 · Cited by 343 · cited 2×
“Others have used GPT models to solve various MCQ-based exams, including the United States Medical Licensing Examination (USMLE), with accuracy around 50% [13, 22, 27], the Multistate Bar Examination (MBE) [3, 18], and the American Institute of Certified Public Accountants’ (AICPA) Regulation (REG) exam [2].”
MethodResultCan large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2022) · arXiv · doi:10.48550/arxiv.2207.08143 · Cited by 343 · cited 2×
“Others have used GPT models to solve various MCQ-based exams, including the United States Medical Licensing Examination (USMLE), with accuracy around 50% [13, 22, 27], the Multistate Bar Examination (MBE) [3, 18], and the American Institute of Certified Public Accountants’ (AICPA) Regulation (REG) exam [2].”
MethodResultIs ChatGPT A Good Translator? Yes With GPT-4 As The Engine
Jiao, W., Wang, W., Huang, J.-T., Wang, X., Shi, S., & Tu, Z. (2023) · arXiv · doi:10.48550/arxiv.2301.08745 · Cited by 317 · cited 1×
“consider the performance of GPT-4 on academic translation tasks, demonstrating that the ChatGPT service achieves significantly better performance compared to existing commercial translation products [16].”
BackgroundAn empirical evaluation of GitHub copilot's code suggestions
Nguyen, N., & Nadi, S. (2022) · Proceedings of the 19th International Conference on Mining Software Repositories · doi:10.1145/3524842.3528470 · Cited by 313 · cited 1×
“Nguyen and Nadi reported Copilot’s effectiveness on LeetCode10 problems, achieving 42% accuracy [34].”
BackgroundMeasuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291 · cited 2×
“Finally, they noted that GPT does poorly on tests of legal and moral reasoning [14].”
BackgroundMethodDenny, P., Kumar, V., & Giacaman, N. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569823 · Cited by 287 · cited 7×
“the strengths and limitations reported in prior work [7, 10, 11, 38, 46] still hold to a certain degree.”
BackgroundMethodResultInfluentialChatGPT and Software Testing Education: Promises & Perils
Jalil, S., Rafi, S., LaToza, T. D., Moran, K., & Lam, W. (2023) · 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw58534.2023.00078 · Cited by 270
Do Users Write More Insecure Code with AI Assistants?
Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2023) · Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3576915.3623157 · Cited by 259 · cited 7×
“Perhaps, a more immediate question that likely occupies minds of many instructors is how to assess learners’ skills and knowledge in the presence of ubiquitous tools (e.g., ChatGPT, 1 GitHub Copilot 2 ) that could be easily utilized to pass the assessments (at least partially).”
BackgroundInfluentialPrather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256 · cited 10×
“C Y ] 1 5 Ju n 20 is how to assess learners’ skills and knowledge in the presence of ubiquitous tools (e.g., ChatGPT,1 GitHub Copilot2) that could be easily utilized to pass the assessments (at least partially).”
BackgroundInfluentialUsing Large Language Models to Enhance Programming Error Messages
Leinonen, J., Hellas, A., Sarsa, S., Reeves, B., Denny, P., Prather, J., & Becker, B. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569770 · Cited by 235 · cited 1×
“used Codex to generate more readable error messages for learners to use for project-level debugging, suggesting that model-created explanations can serve as effective scaffolding for students learning to program [25].”
MethodComparing Code Explanations Created by Students and Large Language Models
Leinonen, J., Denny, P., MacNeil, S., Sarsa, S., Bernstein, S., Kim, J., Tran, A., & Hellas, A. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588785 · Cited by 210 · cited 1×
“compared learner-authored code explanations with those generated by GPT-3, showing that students perceived GPT-3 generated explanations as more readable of the two [24].”
ResultGPT Takes the Bar ExamPreprint
Bommarito, M., & Katz, D. (2022) · Social Science Research Network · doi:10.48550/arxiv.2212.14402 · Cited by 195
MacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569785 · Cited by 195 · cited 1×
“integrated LLM-generated code explanations into an interactive e-book and compared several different explanation types, such as line-by-line or summarization-oriented explanations, reporting students using line-by-line explanations the most despite them being considered the least useful according to the students [29].”
MacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J. (2022) · arXiv · doi:10.48550/arxiv.2211.02265 · Cited by 195 · cited 1×
“integrated LLM-generated code explanations into an interactive e-book and compared several different explanation types, such as line-by-line or summarization-oriented explanations, reporting students using line-by-line explanations the most despite them being considered the least useful according to the students [29].”
My AI Wants to Know if This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises
Finnie-Ansley, J., Denny, P., Luxton-Reilly, A., Santos, E. A., Prather, J., & Becker, B. A. (2023) · Proceedings of the 25th Australasian Computing Education Conference · doi:10.1145/3576123.3576134 · Cited by 186 · cited 6×
“[7, 10, 11, 38, 46] (RQ4) Can GPT-4 successfully utilize feedback to fix solutions of coding tasks?”
BackgroundMethodResultInfluentialBommarito, M. J., & Katz, D. M. (2022) · doi:10.2139/ssrn.4314839 · Cited by 181
Gilson, A., Safranek, C., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2022) · doi:10.1101/2022.12.23.22283901 · Cited by 172 · cited 1×
“Others have used GPT models to solve various MCQ-based exams, including the United States Medical Licensing Examination (USMLE), with accuracy around 50% [13, 22, 27], the Multistate Bar Examination (MBE) [3, 18], and the American Institute of Certified Public Accountants’ (AICPA) Regulation (REG) exam [2].”
MethodAugmented Language Models: a Survey
Mialon, G., Dessì, R., Lomeli, M., Nalmpantis, C., Pasunuru, R., Raileanu, R., Rozière, B., Schick, T., Dwivedi-Yu, J., Celikyilmaz, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.07842 · Cited by 160 · cited 1×
“Note that it would be possible to equip LLMs, including GPT models, with the ability to manipulate such external tools [30].”
BackgroundMaking Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2020) · arXiv · doi:10.48550/arxiv.2012.15723 · Cited by 124 · cited 1×
“Numerous studies have explored prompts which include a number of examples to demonstrate what the desired output should be [4, 12, 50].”
BackgroundSavelka, J., Agarwal, A., Bogart, C., Song, Y., & Sakr, M. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588792 · Cited by 109 · cited 8×
“evaluated the capability of text-davinci-003, to pass a diverse set of assessment instruments, including MCQs, in the realistic context of full-fledged programming courses [46].”
BackgroundMethodResultInfluentialSelf-citationPrompting Large Language Model for Machine Translation: A Case Study
Zhang, B., Haddow, B., & Birch, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.07069 · Cited by 69 · cited 2×
“However, the current research literature remains inconclusive as to the efficacy of adding examples to natural language prompts, with multiple studies suggesting that the order and number of examples can dramatically influence the performance of LLMs across various tasks [42, 50].”
BackgroundPiccolo, S. R., Denny, P., Luxton-Reilly, A., Payne, S. H., & Ridge, P. G. (2023) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011511 · Cited by 69 · cited 4×
“Hence, the findings reported in [46] appear to be challenged by the more capable GPT-4 model as well as the findings reported in [7, 10, 11, 38].”
BackgroundResultInfluentialMany bioinformatics programming tasks can be automated with ChatGPT
Piccolo, S. R., Denny, P., Luxton-Reilly, A., Payne, S., & Ridge, P. G. (2023) · arXiv · doi:10.48550/arxiv.2303.13528 · Cited by 69
Gpt as Knowledge Worker: A Zero-Shot Evaluation of (AI)CPA Capabilities
Bommarito, J., Bommarito, M. J., Katz, J., & Katz, D. M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4322372 · Cited by 68 · cited 2×
“Others have used GPT models to solve various MCQ-based exams, including the United States Medical Licensing Examination (USMLE), with accuracy around 50% [13, 22, 27], the Multistate Bar Examination (MBE) [3, 18], and the American Institute of Certified Public Accountants’ (AICPA) Regulation (REG) exam [2].”
MethodResultLarge Language Models (GPT) Struggle to Answer Multiple-Choice Questions About Code
Savelka, J., Agarwal, A., Bogart, C., & Sakr, M. (2023) · Proceedings of the 15th International Conference on Computer Supported Education · doi:10.5220/0011996900003470 · Cited by 56
Self-citationLarge Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Code
Savelka, J., Agarwal, A., Bogart, C., & Sakr, M. (2023) · arXiv · doi:10.48550/arxiv.2303.08033 · Cited by 56 · cited 8×
“They also observed that an important limitation of the GPT models was their apparent struggle with activities that required multi-hop reasoning, and that there appeared to be a difference in success rate between MCQs that contained a code snippet and those that did not [45, 46].”
BackgroundMethodResultInfluentialSelf-citationHow Important Are Good Method Names in Neural Code Generation? A Model Robustness Perspective
Yang, G., Zhou, Y., Yang, W., Yue, T., Chen, X., & Chen, T. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3630010 · Cited by 45 · cited 1×
“For example, work from multiple studies suggest that models that interpret code depend heavily on function names and input variables [32, 49].”
BackgroundLeveraging Large Language Models for Multiple Choice Question Answering
Robinson, J. A., Rytting, C. M., & Wingate, D. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.12353 · Cited by 37
Becker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2022) · arXiv · doi:10.48550/arxiv.2212.01020 · Cited by 36 · cited 1×
“The study highlights that the risk of learners becoming overly reliant on GPT models when completing programming course assignments and assessments is a genuine concern that must be taken seriously which is consistent with [1].”
BackgroundCodex Hacks HackerRank: Memorization Issues and a Framework for Code Synthesis EvaluationPreprint
Karmakar, A., Prenner, J. A., D’Ambros, M., & Robbes, R. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.02684 · Cited by 31 · cited 1×
“reported 96% pass rate for Codex on a data set of 115 programming problems from HackerRank9 [17].”
BackgroundSolving Linear Algebra by Program SynthesisPreprint
Drori, I., & Verma, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.08171 · Cited by 18 · cited 2×
“Drori and Verma used Codex to write Python programs to solve 60 computational linear algebra MCQs, reporting 100% accuracy [9].”
BackgroundMethodDestefanis, G., Bartolucci, S., & Ortu, M. (2023) · arXiv · doi:10.48550/arxiv.2305.09402 · Cited by 16 · cited 1×
“GPT models have been shown to handle a number of programming languages [5, 8].”
BackgroundHow Important are Good Method Names in Neural Code Generation? A Model Robustness PerspectivePreprint
Yang, G., Zhou, Y., Yang, W., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.15844 · Cited by 5 · cited 1×
“For example, work from multiple studies suggest that models that interpret code depend heavily on function names and input variables [32, 49].”
BackgroundExplainable AI for Pre-Trained Code Models: What Do They Learn? When They Do Not Work?Preprint
Mohammadkhani, A. H., Tantithamthavorn, C. K., & Hemmati, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.12821 · Cited by 4 · cited 1×
“For example, work from multiple studies suggest that models that interpret code depend heavily on function names and input variables [32, 49].”
BackgroundDo Users Write More Insecure Code with AI Assistants?
Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2022) · arXiv · cited 7×
“Perhaps, a more immediate question that likely occupies minds of many instructors is how to assess learners’ skills and knowledge in the presence of ubiquitous tools (e.g., ChatGPT, 1 GitHub Copilot 2 ) that could be easily utilized to pass the assessments (at least partially).”
BackgroundInfluential"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2023) · arXiv · cited 10×
“C Y ] 1 5 Ju n 20 is how to assess learners’ skills and knowledge in the presence of ubiquitous tools (e.g., ChatGPT,1 GitHub Copilot2) that could be easily utilized to pass the assessments (at least partially).”
BackgroundInfluentialOpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Research works printed but not linked: 5
- Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:abs/2107.03374 (2021). https://doi.org/10.48550/ARXIV.2107.03374 · DOI · Paper page
- Amelia Gilson Conrad W. Safranek Tao Huang Vimig Socrates Lim Sze Chi Roderick A. Taylor and David Chartash. 2022. How Well Does ChatGPT Do When Taking the Medical Licensing Exams? The Implications of Large Language Models for Medical Education and Knowledge Assessment. In medRxiv. https://doi.org/10.1101/2022.12.23.22283901. · DOI · Paper page
- [15] Sajed Jalil, Suzzana Rafi, Thomas D. LaToza, Kevin Moran, and Wing Lam. 2023. ChatGPT and Software Testing Education: Promises & Perils. https://doi.org/10. 48550/arXiv.2302.03287 arXiv:2302.03287 [cs.SE] - · arXiv
- [20] Donald Ervin Knuth. 1984. Literate programming. The computer journal 27, 2 (1984), 97-111. -
- [24] Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas. 2023. Comparing Code Explanations Created by Students and Large Language Models. https://doi.org/10.48550/arXiv.2304. 03938 arXiv:2304.03938 [cs.CY] - · arXiv · doi:10.48550/arxiv.2304
Cited by
The promise and challenges of generative AI in education
Giannakos, M., Azevedo, R., Brusilovsky, P., Cukurova, M., Dimitriadis, Y., Hernandez-Leo, D., Järvelä, S., Mavrikis, M., & Rienties, B. (2025) · Behaviour & Information Technology · doi:10.1080/0144929x.2024.2394886 · Cited by 543
“For tasks such as code explanation generation, success and correctness rates might be higher, and newer models tend to perform better than older models (i.e. Codex vs. GPT 3.5 vs. GPT 4) on most types of tasks (Savelka et al. 2023).”
InfluentialThe Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396
“Indeed, recent results with GPT-4 suggest that it can solve most exercises in introductory programming courses [164], which is also supported by our benchmarking work presented in Section 8.”
BackgroundResultInfluentialKazemitabaar, M., Ye, R., Wang, X., Henley, A. Z., Denny, P., Craig, M., & Grossman, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642773 · Cited by 290
“In terms of generating learning resources, early work by Sarsa et al. [53] analyzed the novelty, plausibility, and readiness of 120 programming exercises generated by OpenAI Codex and proposed the potential of using such models to come up with coding assignments.”
MethodCodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes
Liffiton, M., Sheese, B. E., Savelka, J., & Denny, P. (2023) · Proceedings of the 23rd Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631830 · Cited by 192
The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B. (2024) · Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3632620.3671116 · Cited by 196
Metadata
- Authors
- Savelka, J., Agarwal, A., An, M., Bogart, C., & Sakr, M.
- Affiliations
- Carnegie Mellon University, USA
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 2023 ACM Conference on International Computing Education Research V.1
- Publisher
- ACM
- Published
- 2023-08-07
- Pages
- 78-92
- OpenAlex
- W4381587445
- Cited by
- 141
- References
- 48
- Citations by year
- 2026: 21 · 2025: 54 · 2024: 57 · 2023: 9
- FWCI
- 42.59
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on International Computing Education Research (ICER) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- large language models, GPT-4, programming education, Python programming course, automated assessment, automated grading, multiple-choice questions, programming exercises, generative AI in higher education, assessment design, GPT-3
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.