Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors
The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index; none of the 8 sources its claims cite is checked for support yet.
Abstract
Generative AI and large language models hold great promise in enhancing computing education by powering next-generation educational technologies. State-of-the-art models like OpenAI’s ChatGPT [8] and GPT-4 [9] could enhance programming education in various roles, e.g., by acting as a personalized digital tutor for a student, a digital assistant for an educator, and a digital peer for collaborative learning [1, 2, 7]. In our work, we seek to comprehensively evaluate and benchmark state-of-the-art large language models for various scenarios in programming education.
Overview
CoChat Summary
## TL;DR
The paper benchmarks ChatGPT (GPT-3.5), GPT-4, and human tutors on six introductory Python programming-education scenarios using 25 instances per scenario. GPT-4 consistently improves over ChatGPT and sometimes approaches human performance, but it still lags on grading feedback and task synthesis.
## Key findings
- For program repair, GPT-4 achieved 88.0% Correct, compared with 68.0% for ChatGPT and 100.0% for Tutor; GPT-4 used more edits (36.6 EditTokens) than Tutor (19.0).
- For hint generation, GPT-4 reached 66.0% Overall, versus 18.0% for ChatGPT and 92.0% for Tutor; the paper notes the gap was worst on PALINDROME.
- For grading feedback, GPT-4 scored 16.0% Match, compared with 10.0% for ChatGPT and 52.0% for Tutor; the paper says the largest gap was on D-CorrectEdge.
- For pair programming, GPT-4 achieved 64.0% Overall and 92.0% Correct, versus 38.0% Overall for ChatGPT and 82.0% Overall for Tutor; GPT-4 tended to make more edits and not keep the partial-program context as well as Tutor.
- For contextualized explanation, GPT-4 reached 84.0% Overall and 88.0% Correct, close to Tutor at 92.0% Overall and 92.0% Correct.
- For task synthesis, GPT-4 scored 22.0% Overall, above ChatGPT’s 10.0% but well below Tutor’s 74.0%; the paper highlights the gap on SimilarBugs.
## Methods
- Evaluated three methods: ChatGPT (GPT-3.5 via web), GPT-4 via web, and Tutor (2 human experts).
- Used five introductory Python problems and 25 buggy programs total, drawn from publicly accessible geeksforgeeks.org submissions; the scenarios were evaluated with 25 instances each.
- Compared methods across six scenarios: program repair, hint generation, grading feedback, pair programming, contextualized explanation, and task synthesis, using expert-based annotations and some automated scripts/test suites depending on the scenario.
## Limitations
- The paper states it used only two human experts as tutors and evaluators, and says the study should be scaled up.
- It focused only on introductory Python and English, and the authors say future work should test other languages, domains, and multilingual settings.
- It used expert-based assessments only and did not involve students; this is stated by the authors.
- Evident from the text: the benchmark is limited to five problems / 25 buggy programs, so generalization beyond these specific tasks is not established.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 1 Not valid · 4 Not checked · 78 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These results also highlight that GPT-4 still struggles with more challenging scenarios, e.g., grading feedback and task synthesis, where the performance of GPT-4 is quite low compared to that of human tutors.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These results also highlight settings where GPT-4 still struggles, providing exciting future directions on developing techniques to improve the performance of these models.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results show that GPT-4 drastically outperforms ChatGPT (based on GPT-3.5) and comes close to human tutors' performance for several scenarios.”
- Also stated in: § 1 Introduction · plain
“Our results show that GPT-4 drastically outperforms ChatGPT (based on GPT-3.5) and comes close to human tutors' performance for several scenarios.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 3 assumptions not shown here.
- Also stated in: § 1 Introduction · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results show that GPT-4 drastically outperforms ChatGPT (based on GPT-3.5) and comes close to human tutors' performance for several scenarios.”
- Also stated in: § 1 Introduction · plain
“Our results show that GPT-4 drastically outperforms ChatGPT (based on GPT-3.5) and comes close to human tutors' performance for several scenarios.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 3 assumptions not shown here.
- Also stated in: § 1 Introduction · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT was able to solve four out of five problems; it consistently failed on FIBONACCI across multiple queries.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
38 more claims
Create a free account to see all 83 claims and to verify the 8 references still open.
- Probably not a substantial claim. 35
- The quote only partly states this claim. 3
- It may state two separate claims as one. 12
Citations
Cites
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · cited 1×
“State-of-the-art models like OpenAI’s ChatGPT [2] and GPT-4 [3] could enhance programming education in various roles, e.g., by acting as a personalized digital tutor for a student, as a digital assistant for an educator, or as a digital peer for collaborative learning [4–6].”
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Lim, W. M., Gunasekara, A., Pallant, J. L., Pallant, J. I., & Pechenkina, E. (2023) · The International Journal of Management Education · doi:10.1016/j.ijme.2023.100790 · Cited by 1,261 · cited 1×
“State-of-the-art models like OpenAI’s ChatGPT [2] and GPT-4 [3] could enhance programming education in various roles, e.g., by acting as a personalized digital tutor for a student, as a digital assistant for an educator, or as a digital peer for collaborative learning [4–6].”
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497 · cited 2×
“This section is dedicated to the programming education scenario of contextualized explanation [9, 21].”
BackgroundThe Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463 · cited 1×
“Recent works have studied several LLMs for different scenarios relevant to programming education [7– 11].”
BackgroundUsing Large Language Models to Enhance Programming Error Messages
Leinonen, J., Hellas, A., Sarsa, S., Reeves, B., Denny, P., Prather, J., & Becker, B. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569770 · Cited by 235 · cited 1×
“This section is dedicated to the programming education scenario of hint generation [10, 11].”
BackgroundReading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2022) · arXiv · doi:10.48550/arxiv.2210.14306 · Cited by 114
Repairing Bugs in Python Assignments Using Large Language ModelsPreprint
Zhang, J.-L., Cambronero, J., Gulwani, S., Le, V., Piskac, R., Soares, G., & Verbruggen, G. (2022) · arXiv.org · doi:10.48550/arxiv.2209.14876 · Cited by 68 · cited 1×
“This section is dedicated to the programming education scenario of program repair [8, 11, 14].”
BackgroundSelf-citationIs AI the better programming partner? Human-Human Pair Programming vs. Human-AI pAIr ProgrammingPreprint
Ma, Q., Wu, S., & Koedinger, K. (2023) · LLM@AIED · doi:10.48550/arxiv.2306.05153 · Cited by 61
MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education
Shen, J. T., Yamashita, M., Prihar, E., Heffernan, N., Wu, X., Graff, B., & Lee, D. (2021) · arXiv · doi:10.48550/arxiv.2106.07340 · Cited by 47 · cited 1×
“This section is dedicated to the programming education scenario of grading feedback [15, 16].”
Background
Research works printed but not linked: 1
- David Baidoo-Anu and Leticia Owusu Ansah. 2023. Education in the Era of Generative Artificial Intelligence (AI): Understanding the Potential Benefits of ChatGPT in Promoting Teaching and Learning. Available at SSRN 4337484 (2023).
Cited by
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
Large Language Models in Computer Science Education: A Systematic Literature Review
Raihan, M. N., Siddiq, M. L., Santos, J. C. S., & Zampieri, M. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701863 · Cited by 219
AI Literacy in K-12 and Higher Education in the Wake of Generative AI: An Integrative Review
Gu, X., & Ericson, B. J. (2025) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3702652.3744217 · Cited by 163
“The vast increase in the ability of LLMs to perform a wide range of tasks, coupled with lowered barriers of access given their natural language interface, prompted researchers to rethink what future students could do with the help of AI [25, 36, 49, 91, 93, 94].”
Alsaiari, O. A. S., Baghaei, N., Lahza, H. F., Lodge, J. M., Bodén, M., & Khosravi, H. (2025) · Computers & Education · doi:10.1016/j.compedu.2025.105363 · Cited by 102
AI Literacy in K-12 and Higher Education in the Wake of Generative AI: An Integrative Review
Gu, X., & Ericson, B. J. (2025) · arXiv · doi:10.48550/arxiv.2503.00079 · Cited by 73
“The vast increase in the ability of LLMs to perform a wide range of tasks, coupled with lowered barriers of access given their natural language interface, prompted researchers to rethink what future students could do with the help of AI [25, 36, 49, 91, 93, 94].”
Metadata
- Authors
- Phung, T., Pădurean, V.-A., Cambronero, J., Gulwani, S., Kohn, T., Majumdar, R., Singla, A., & Soares, G.
- Affiliations
- Max Planck Institute for Software Systems (DE)
- MPI-INF, Germany
- Microsoft (United States) (US)
- TU Wien (AT)
- ORCID
- Venue
- Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 2
- Publisher
- ACM
- Published
- 2023-08-07
- Pages
- 41-42
- OpenAlex
- W4386719008
- Semantic Scholar
- 449ab79be2d6086b88832cfc9c5d502524c8524f
- Cited by
- 112
- References
- 4
- Citations by year
- 2026: 26 · 2025: 51 · 2024: 30 · 2023: 5
- FWCI
- 33.87
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Conference on International Computing Education Research (ICER) (publishedVersion)
- Funders
- European Research Council (ERC, TOPS, 101039090)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- generative AI, programming education, large language models, ChatGPT, GPT-4, human tutors, benchmarking, AI tutors, personalized tutoring, cooperative learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors. https://app.cochat.ai/papers/10.1145/3568812.3603476
- The paper
- Phung, T., Pădurean, V.-A., Cambronero, J., Gulwani, S., Kohn, T., Majumdar, R., Singla, A., & Soares, G. (2023). Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors. Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 2. https://doi.org/10.1145/3568812.3603476
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.