Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors
The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index; none of the 8 sources its claims cite is checked for support yet.
Abstract
Generative AI and large language models hold great promise in enhancing computing education by powering next-generation educational technologies. State-of-the-art models like OpenAI’s ChatGPT [8] and GPT-4 [9] could enhance programming education in various roles, e.g., by acting as a personalized digital tutor for a student, a digital assistant for an educator, and a digital peer for collaborative learning [1, 2, 7]. In our work, we seek to comprehensively evaluate and benchmark state-of-the-art large language models for various scenarios in programming education.
Overview
CoChat Summary
## TL;DR
The paper benchmarks ChatGPT (GPT-3.5), GPT-4, and human tutors on six introductory Python programming-education scenarios using 25 instances per scenario. GPT-4 consistently improves over ChatGPT and sometimes approaches human performance, but it still lags on grading feedback and task synthesis.
## Key findings
- For program repair, GPT-4 achieved 88.0% Correct, compared with 68.0% for ChatGPT and 100.0% for Tutor; GPT-4 used more edits (36.6 EditTokens) than Tutor (19.0).
- For hint generation, GPT-4 reached 66.0% Overall, versus 18.0% for ChatGPT and 92.0% for Tutor; the paper notes the gap was worst on PALINDROME.
- For grading feedback, GPT-4 scored 16.0% Match, compared with 10.0% for ChatGPT and 52.0% for Tutor; the paper says the largest gap was on D-CorrectEdge.
- For pair programming, GPT-4 achieved 64.0% Overall and 92.0% Correct, versus 38.0% Overall for ChatGPT and 82.0% Overall for Tutor; GPT-4 tended to make more edits and not keep the partial-program context as well as Tutor.
- For contextualized explanation, GPT-4 reached 84.0% Overall and 88.0% Correct, close to Tutor at 92.0% Overall and 92.0% Correct.
- For task synthesis, GPT-4 scored 22.0% Overall, above ChatGPT’s 10.0% but well below Tutor’s 74.0%; the paper highlights the gap on SimilarBugs.
## Methods
- Evaluated three methods: ChatGPT (GPT-3.5 via web), GPT-4 via web, and Tutor (2 human experts).
- Used five introductory Python problems and 25 buggy programs total, drawn from publicly accessible geeksforgeeks.org submissions; the scenarios were evaluated with 25 instances each.
- Compared methods across six scenarios: program repair, hint generation, grading feedback, pair programming, contextualized explanation, and task synthesis, using expert-based annotations and some automated scripts/test suites depending on the scenario.
## Limitations
- The paper states it used only two human experts as tutors and evaluators, and says the study should be scaled up.
- It focused only on introductory Python and English, and the authors say future work should test other languages, domains, and multilingual settings.
- It used expert-based assessments only and did not involve students; this is stated by the authors.
- Evident from the text: the benchmark is limited to five problems / 25 buggy programs, so generalization beyond these specific tasks is not established.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 1 Not valid · 4 Not checked · 78 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These results also highlight that GPT-4 still struggles with more challenging scenarios, e.g., grading feedback and task synthesis, where the performance of GPT-4 is quite low compared to that of human tutors.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These results also highlight settings where GPT-4 still struggles, providing exciting future directions on developing techniques to improve the performance of these models.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results show that GPT-4 drastically outperforms ChatGPT (based on GPT-3.5) and comes close to human tutors' performance for several scenarios.”
- Also stated in: § 1 Introduction · plain
“Our results show that GPT-4 drastically outperforms ChatGPT (based on GPT-3.5) and comes close to human tutors' performance for several scenarios.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 3 assumptions not shown here.
- Also stated in: § 1 Introduction · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results show that GPT-4 drastically outperforms ChatGPT (based on GPT-3.5) and comes close to human tutors' performance for several scenarios.”
- Also stated in: § 1 Introduction · plain
“Our results show that GPT-4 drastically outperforms ChatGPT (based on GPT-3.5) and comes close to human tutors' performance for several scenarios.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 3 assumptions not shown here.
- Also stated in: § 1 Introduction · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT was able to solve four out of five problems; it consistently failed on FIBONACCI across multiple queries.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
38 more claims
Create a free account to see all 83 claims and to verify the 8 references still open.
- Probably not a substantial claim. 35
- The quote only partly states this claim. 3
- It may state two separate claims as one. 12
Citations
Cites
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · cited 1×
“State-of-the-art models like OpenAI’s ChatGPT [2] and GPT-4 [3] could enhance programming education in various roles, e.g., by acting as a personalized digital tutor for a student, as a digital assistant for an educator, or as a digital peer for collaborative learning [4–6].”
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Lim, W. M., Gunasekara, A., Pallant, J. L., Pallant, J. I., & Pechenkina, E. (2023) · The International Journal of Management Education · doi:10.1016/j.ijme.2023.100790 · Cited by 1,261 · cited 1×
“State-of-the-art models like OpenAI’s ChatGPT [2] and GPT-4 [3] could enhance programming education in various roles, e.g., by acting as a personalized digital tutor for a student, as a digital assistant for an educator, or as a digital peer for collaborative learning [4–6].”
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497 · cited 2×
“This section is dedicated to the programming education scenario of contextualized explanation [9, 21].”
BackgroundThe Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463 · cited 1×
“Recent works have studied several LLMs for different scenarios relevant to programming education [7– 11].”
BackgroundUsing Large Language Models to Enhance Programming Error Messages
Leinonen, J., Hellas, A., Sarsa, S., Reeves, B., Denny, P., Prather, J., & Becker, B. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569770 · Cited by 235 · cited 1×
“This section is dedicated to the programming education scenario of hint generation [10, 11].”
BackgroundReading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2022) · arXiv · doi:10.48550/arxiv.2210.14306 · Cited by 114
Is AI the better programming partner? Human-Human Pair Programming vs. Human-AI pAIr ProgrammingPreprint
Ma, Q., Wu, T., & Koedinger, K. R. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.05153 · Cited by 92
Repairing Bugs in Python Assignments Using Large Language ModelsPreprint
Zhang, J.-L., Cambronero, J., Gulwani, S., Le, V., Piskac, R., Soares, G., & Verbruggen, G. (2022) · arXiv.org · doi:10.48550/arxiv.2209.14876 · Cited by 68 · cited 1×
“This section is dedicated to the programming education scenario of program repair [8, 11, 14].”
BackgroundSelf-citationMathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education
Shen, J. T., Yamashita, M., Prihar, E., Heffernan, N., Wu, X., Graff, B., & Lee, D. (2021) · arXiv · doi:10.48550/arxiv.2106.07340 · Cited by 47 · cited 1×
“This section is dedicated to the programming education scenario of grading feedback [15, 16].”
Background
Research works printed but not linked: 4
- [5] David Baidoo-Anu and Leticia Owusu Ansah. Education in the Era of Generative Artificial Intelligence (AI): Understanding the Potential Benefits of ChatGPT in Promoting Teaching and Learning. Available at SSRN 4337484 , 2023. -
- [14] Jooyong Yi, Umair Z. Ahmed, Amey Karkare, Shin Hwei Tan, and Abhik Roychoudhury. A Feasibility Study of Using Automated Program Repair for Introductory Programming Assignments. In ESEC/FSE , 2017. -
- [19] Saki Imai. Is Github Copilot a Substitute for Human Pair-programming? An Empirical Study. In ICSE Companion Proceedings , 2022. -
- [22] Umair Z. Ahmed, Maria Christakis, Aleksandr Efremov, Nigel Fernandez, Ahana Ghosh, Abhik Roychoudhury, and Adish Singla. Synthesizing Tasks for Block-based Programming. In NeurIPS , 2020. -
Cited by
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
Large Language Models in Computer Science Education: A Systematic Literature Review
Raihan, M. N., Siddiq, M. L., Santos, J. C. S., & Zampieri, M. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701863 · Cited by 222
AI Literacy in K-12 and Higher Education in the Wake of Generative AI: An Integrative Review
Gu, X., & Ericson, B. J. (2025) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3702652.3744217 · Cited by 166
“The vast increase in the ability of LLMs to perform a wide range of tasks, coupled with lowered barriers of access given their natural language interface, prompted researchers to rethink what future students could do with the help of AI [25, 36, 49, 91, 93, 94].”
Alsaiari, O. A. S., Baghaei, N., Lahza, H. F., Lodge, J. M., Bodén, M., & Khosravi, H. (2025) · Computers & Education · doi:10.1016/j.compedu.2025.105363 · Cited by 103
AI Literacy in K-12 and Higher Education in the Wake of Generative AI: An Integrative Review
Gu, X., & Ericson, B. J. (2025) · arXiv · doi:10.48550/arxiv.2503.00079 · Cited by 73
“The vast increase in the ability of LLMs to perform a wide range of tasks, coupled with lowered barriers of access given their natural language interface, prompted researchers to rethink what future students could do with the help of AI [25, 36, 49, 91, 93, 94].”
Phung, T., Pădurean, V.-A., Singh, A., Brooks, C., Cambronero, J., Gulwani, S., Singla, A., & Soares, G. (2024) · Proceedings of the 14th Learning Analytics and Knowledge Conference · doi:10.1145/3636555.3636846 · Cited by 63
“5 with the more advanced model, GPT-4, as the simulated “student” model, there is actually a reduction in precision.”
BackgroundMethodInfluentialSelf-citationAutoTutor meets Large Language Models: A Language Model Tutor with Rich Pedagogy and Guardrails
Chowdhury, S. P., Zouhar, V., & Sachan, M. (2024) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3657604.3662041 · Cited by 130
Tang, Q., Deng, W., Huang, Y., Wang, S., & Zhang, H. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70027 · Cited by 55
Generative AI in Education: Technical Foundations, Applications, and Challenges
Rashid, S. F., Duong‐Trung, N., & Pinkwart, N. (2024) · Artificial intelligence · doi:10.5772/intechopen.1005402 · Cited by 71
Ma, B., Chen, L., & Konomi, S. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-64315-6_9 · Cited by 73
Gaitantzi, A., & Kazanidis, I. (2025) · Applied Sciences · doi:10.3390/app15073960 · Cited by 65
Tailor, P. D., Dalvin, L. A., Chen, J. J.-W., Iezzi, R., Olsen, T. W., Scruggs, B. A., Barkmeier, A. J., Bakri, S. J., Ryan, E. H., Tang, P. H., et al. (2024) · Ophthalmology Science · doi:10.1016/j.xops.2024.100485 · Cited by 46
Literature Review on the Integration of Generative AI in Programming Education
Nathaniel, J., Oyelere, S. S., Suhonen, J. T., & Tedre, M. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00524-3 · Cited by 70
“GenAI have shown to have the potential to enhance program - ming through code auto-completion (Phung et al., 2023) and generation (Savelka et al., 2023a), serving as programming assistance (Savelka et al., 2023b), and encourag - ing personalized learning, and fostering greater student engagement.”
InfluentialRubric Is All You Need: Improving LLM-Based Code Evaluation With Question-Specific Rubrics
Pathak, A., Gandhi, R., Uttam, V. G., Ramamoorthy, A., Ghosh, P., Jindal, A. R., Verma, S., Mittal, A., Ased, A., Khatri, C., et al. (2025) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3702652.3744220 · Cited by 96
Generating In-Context, Personalized Feedback for Intelligent Tutors with Large Language Models
Reddig, J. M., Arora, A., & MacLellan, C. J. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00505-6 · Cited by 62
Evaluating the Application of Large Language Models to Generate Feedback in Programming Education
Jacobs, S., & Jaschke, S. (2024) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon60312.2024.10578838 · Cited by 62
“If the generated student response does not pass the task-specific unit tests, this information can be used again to improve the feedback [20].”
BackgroundPirzado, F. A., Ahmed, A., Mendoza-Urdiales, R. A., & Terashima‐Marín, H. (2024) · IEEE Access · doi:10.1109/access.2024.3443621 · Cited by 65
“They may hold conversations, answer inquiries, explain things, and even create original material on various topics and genres [6], [26], [38], [46], [58], [59], [62], [64], [83], [87], [88], [95].”
BackgroundCastillo-Segura, P., Alario‐Hoyos, C., Kloos, C. D., & Fernández-Panadero, C. (2023) · World Engineering Education Forum - Global Engineering Deans Council (WEEF-GEDC) · doi:10.1109/weef-gedc59520.2023.10344098 · Cited by 50
“…and then, in March 2023, to release GPT4 to be used from ChatGPT at a reduced cost of $20 per month meant that many people around the world could test and share different scenarios and use cases, which helped to increase the popularity of this tool in particular and of generative AI in general [5].”
BackgroundRubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubrics
Pathak, A., Gandhi, R., Uttam, V., Ramamoorthy, A., Ghosh, P., Jindal, A. R., Verma, S., Mittal, A., Ased, A., Khatri, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.23989 · Cited by 30
“These general rubrics often fail to capture the nuances of specific programming problems, leading to misaligned evaluations, still struggling with grading feed-back compared to human tutors [35].”
InfluentialNormative Requirements Operationalization with Large Language Models
Feng, N., Marsso, L., Yaman, S. G., Standen, I., Baatartogtokh, Y., Ayad, R., de Mello, V. O., Townsend, B., Bartels, H., Cavalcanti, A., et al. (2024) · IEEE International Requirements Engineering Conference · doi:10.1109/re59067.2024.00022 · Cited by 23
“They have been shown effective in answering questions regarding common sense [46], [7], [13] and domain-specific knowledge such as programming [32], [41], healthcare [10], [27], and law [40], [48].”
BackgroundCámara, J., Troya, J., Montes-Torres, J., & Jaime, F. J. (2024) · IEEE Software · doi:10.1109/ms.2024.3385309 · Cited by 31
Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programming
Alkafaween, U., Al-Bluwi, I., & Denny, P. C. (2024) · Journal of Computer Assisted Learning · doi:10.1111/jcal.13100 · Cited by 35
Advanced large language models and visualization tools for data analytics learning
Valverde-Rebaza, J., González, A., Navarro-Hinojosa, O., & Noguez, J. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1418006 · Cited by 26
Lee, H.-Y., Chen, P.-H., Lin, C.-J., Huang, Y.-M., & Wu, T.-T. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13733-z · Cited by 21
“The integration of LLMs, such as ChatGPT, in programming education has gained significant attention in recent years (Phung et al., 2023 ; Yilmaz & Karaoglan Yilmaz, 2023 ).”
BugSpotter: Automated Generation of Code Debugging Exercises
Pădurean, V.-A., Denny, P. C., & Singla, A. K. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701974 · Cited by 35
Self-citationEnhancing Engineering Education Through LLM-Driven Adaptive Quiz Generation: A RAG-Based Approach
Gopi, S., Sreekanth, D., & Dehbozorgi, N. (2024) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10893146 · Cited by 30
Lyu, W., Wang, Y., Sun, Y., & Zhang, Y. (2025) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3698205.3729544 · Cited by 36
“GenAI demonstrated strong capabilities in coding tasks [15, 28], prompting researchers to investigate its role in CS education [12, 41, 42].”
Penney, J., Pimentel, J. F., Steinmacher, I., & Gerosa, M. A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-54975-5_12 · Cited by 29
“Phung et al. [50] quantify the skills of existing LLM’s, displaying not only where the state-of-the-art is at but also discussing where it is headed.”
BackgroundInfluentialEvaluation of Generative AI Models in Python Code Generation: A Comparative Study
Palla, D., & Slabý, A. (2025) · IEEE Access · doi:10.1109/access.2025.3560244 · Cited by 32
“From the analysis, it is clear that Anthropic’s latest models, such as Claude 3.5 Sonnet, have now surpassed previous leaders like GPT-4, particularly in accuracy and reliability, as shown in [55].”
InfluentialComputer Programming Education in the Age of Generative AI: Insights from Empirical ResearchPreprint
Deriba, F. G., Sanusi, I. T., Campbell, O. O., & Oyelere, S. S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4891302 · Cited by 28
Li, H., Huang, Q.-R., Wen, L.-P., Chen, W., & Xu, Z.-Z. (2025) · SAGE Open · doi:10.1177/21582440251377986 · Cited by 23
“Its applications extend across multiple domains, from text generation to coding, demonstrating significant potential for enhancing programming education (Phung et al., 2023).”
Tang, B., Liang, J., Hu, W., & Luo, H. (2025) · Systems · doi:10.3390/systems13070555 · Cited by 21
Enhancing Programming Learning with LLMs: Prompt Engineering and Flipped Interaction
Cowan, B., Watanobe, Y., & Shirafuji, A. (2023) · Asia Service Sciences and Software Engineering Conference (ASSE) · doi:10.1145/3634814.3634816 · Cited by 18
FhGenie: A Custom, Confidentiality-Preserving Chat AI for Corporate and Scientific Use
Weber, I. M., Linka, H., Mertens, D., Muryshkin, T., Opgenoorth, H., & Langer, S. (2024) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c63560.2024.00011 · Cited by 17
Outcomes, Perceptions, and Interaction Strategies of Novice Programmers Studying with ChatGPT
Penney, J., Acharya, P., Hilbert, P., Parekh, P., Sarma, A., Steinmacher, I., & Gerosa, M. A. (2025) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3719160.3736625 · Cited by 17
Plan More, Debug Less: Applying Metacognitive Theory to AI-Assisted Programming Education
Phung, T., Choi, H., Wu, M., Singla, A. K., & Brooks, C. A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98414-3_1 · Cited by 23
Self-citationLarge Language Models (LLMs) in Programming Learning: The Current Research State and Agenda
Fu, Q., Zhao, Y., Jia, Z., & Zheng, Y. (2025) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2025.3622043 · Cited by 19
Frank, L., Herth, F., Stuwe, P., Klaiber, M., Gerschner, F., & Theissler, A. (2024) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon60312.2024.10578933 · Cited by 11
“Phung et al. [4] themselves point out the limitation that they restrict their work to Python programming and that it would be interesting to conduct a similar study for other programming languages.”
BackgroundMethodResultInfluentialGenç, O. (2025) · Engineering Construction & Architectural Management · doi:10.1108/ecam-07-2024-0890 · Cited by 16
GPT-4 shows comparable performance to human examiners in ranking open-text answers
Zubaer, A. A., Granitzer, M., Geschwind, S., Lambsdorff, J. G., & Voss, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-21572-8 · Cited by 13
“When it comes to using GPT as a teacher, recent studies suggest a positive effect of using generative AI to assist teaching activities 20–22 .”
Sun, D., Zheng, Y., Xu, J., & Yang, Z. (2026) · Journal of Computer Assisted Learning · doi:10.1002/jcal.70210 · Cited by 17
Song, E. S., & Lee, S. (2024) · International Journal of Dental Hygiene · doi:10.1111/idh.12848 · Cited by 14
The Robots are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.00658 · Cited by 9
Self-citationEnnion, M., & McLellan, R. (2025) · doi:10.21428/8c225f6e.540b41b5 · Cited by 17
Hidayatullah, E., Untari, R., & Fifardin, F. (2024) · UNION Jurnal Ilmiah Pendidikan Matematika · doi:10.30738/union.v12i2.17548 · Cited by 15
“Contrary to the study by Phung et al., (2023) which only conducted assessments without involving students.”
ResultCross-Language Code Development with Generative AI: A Source-to-Source Translation Perspective
Rai, L., Khatiwada, S., Deng, C., & Liu, F. (2024) · IEEE International Conference on Electronic Information and Communication Technology (ICEICT) · doi:10.1109/iceict61637.2024.10671366 · Cited by 10
A Systematic Review of Gen-AI Applications in Education: Rewards, Challenges and Future Prospects
Wanyonyi, E. N., & Murithi, M. (2025) · Pan-African journal of education and social sciences · doi:10.56893/pajes2025v06i01.01 · Cited by 8
Understanding Programming Students' Help-Seeking Preferences in the Era of Generative AI
Penney, J., Acharya, P., Hilbert, P., Parekh, P., Sarma, A., Steinmacher, I., & Gerosa, M. A. (2025) · ACM Global Computing Education Conference (CompEd) · doi:10.1145/3736181.3747165 · Cited by 11
Gambo, O. O., Onifade, O. J., Olanrewaju-Smart, W., Abuya, O. J., Salawu, R. E., Massenon, R., & Gambo, I. (2026) · Discover Education · doi:10.1007/s44217-026-01142-4 · Cited by 14
“5 and GPT-4 used in this study, serves as a transformative tool in programming education by acting as a personalized tutor, educator assistant [75].”
Developing a GPT Chatbot Model for Students Programming Education
Modran, H. A., Ursuțiu, D., Samoilă, C., & Gherman-Dolhăscu, E.-C. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-61905-2_8 · Cited by 10
Synthesizing High-Quality Programming Tasks with LLM-Based Expert and Student Agents
Nguyen, M. H., Pădurean, V.-A., Gotovos, A., Tschiatschek, S., & Singla, A. K. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98414-3_6 · Cited by 8
Self-citationStudent Engagement with GenAI's Tutoring Feedback: A Mixed Methods Study
Jacobs, S., Haas, J., & Kiesler, N. (2025) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3769994.3770034 · Cited by 9
“As a result, there is growing research on how to enhance the quality of the feedback messages with techniques like Retrieval Augmented Generation [15, 29], multiple prompts [3, 19], chain-of-thought prompting [58], fine-tuning [30, 50], or validation of feedback with a smaller model [38].”
El-Khalili, N. H., Al-Nashashibi, M. Y., & Al-E’mari, S. (2024) · International Conference on Cyber Resilience (ICCR) · doi:10.1109/iccr61006.2024.10532824 · Cited by 6
“Furthermore, work done in [8] shows that GPT-4 performs close to human tutors’ in scenarios that teach programming to students.”
BackgroundPropagating Large Language Models Programming Feedback
Koutcheme, C., & Hellas, A. (2024) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3657604.3664665 · Cited by 11
A Report on the Use of ChatGPT in Software Engineering and Systems Analysis Courses
Albonico, M., & Varela, P. J. (2023) · Brazilian Symposium on Software Engineering · doi:10.1145/3613372.3614189 · Cited by 7
From Assistance to Misconduct: Unpacking the Complex Role of Generative AI in Student Learning
Axelsson, A., Wallgren, D. T., Verma, U., Cajander, Å., Daniels, M., Eckerdal, A., & McDermott, R. (2024) · 2024 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10893133 · Cited by 5
“These technologies can tailor educational content to diverse learning approaches and speeds, potentially overcoming the constraints of conventional one-size-fits-all teaching methods [2].”
Open, Small, Rigmarole – Evaluating Llama 3.2 3B’s Feedback for Programming Exercises
Azaiz, I., Kiesler, N., Strickroth, S., & Zhang, A. (2025) · International Journal of Engineering Pedagogy (iJEP) · doi:10.3991/ijep.v15i5.55359 · Cited by 11
Xu, H., Liu, H., Wu, Y., Kang, X., Chen, X., & Liu, Y. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112731 · Cited by 6
Yılmaz, F. G. K., Olpak, Y. Z., & Yılmaz, R. (2026) · Journal of Research on Technology in Education · doi:10.1080/15391523.2026.2683793 · Cited by 6
An AI-driven Approach for Tailoring Java Programming Instructions
Thiyagarajan, G., & Vijayalakshmi, V. (2025) · International Conference on Intelligent Data Communication Technologies and Internet of Things (IDCIoT) · doi:10.1109/idciot64235.2025.10914687 · Cited by 5
Integrating AI into Design Ideation: Assessing ChatGPT’s Role in Human-Centered Design EducationPreprint
Papachristos, E., İnal, Y., Vicient, C., Johansen, E. A., & Hermansen, M. (2024) · doi:10.36227/techrxiv.171656320.09963657/v1 · Cited by 7
Meng, X., & Yang, Y. (2025) · Research Square · doi:10.21203/rs.3.rs-6107039/v1 · Cited by 5
Exploring the role of automated feedback in programming education: a systematic literature review
Jung, Y., Lee, Y., Bae, J., Kim, D., Choi, H., Kang, M., & Lee, U. (2026) · Interactive Learning Environments · doi:10.1080/10494820.2026.2701430 · Cited by 4
ElSayary, A. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-7332-3.ch015 · Cited by 3
Gandhi, A., & Müldner, K. (2025) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3702652.3744228 · Cited by 4
“One often cited ‘promise’ of ChatGPT for education is its potential to serve as a personalized tutor (see, e.g., [18, 20]).”
Ding, S., Smith, J. B., & Magerko, B. S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98459-4_36 · Cited by 8
Reeping, D. P., & Shah, A. (2024) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--47047 · Cited by 8
Mahir, A., Shohel, M. M. C., & Sall, W. (2024) · Advances in higher education and professional development book series · doi:10.4018/979-8-3693-1499-9.ch012 · Cited by 3
Instruction Tuning with LLMs for Programming Exercise Generation
Zeng, G., Xue, Q., & Lu, X. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-7707-5_31 · Cited by 5
Phung, T., Choi, H., Wu, M., Brooks, C., Gulwani, S., & Singla, A. K. (2026) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3770762.3772612 · Cited by 8
“The growing capabilities of AI have created new opportunities to enhance computing education [3, 6, 26, 28], particularly in providing immediate and tailored feedback to students [8, 16, 25].”
Self-citation(2024) · International Research Journal of Modernization in Engineering Technology and Science · doi:10.56726/irjmets64283 · Cited by 3
Customizing ChatGPT to Help Computer Science Principles Students Learn Through Conversation
Frazier, M., Damevski, K., & Pollock, L. (2024) · Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3649217.3653570 · Cited by 2
da Silva, D. C. A., Mello, C. E., & García, A. C. B. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-87330-0_15 · Cited by 3
Mekterović, I., & Brkić, L. (2025) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro65660.2025.11131962 · Cited by 2
da Silva, E. P., & dos Reis, J. C. (2025) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/sbie.2025.12723 · Cited by 2
“However, these rubrics are generally developed by specialists beforehand [Phung et al. 2023, Duong and Meng 2024].”
Lin, Y.-T. L. Y.-T., Chang, C.-C., & Lin, Y.-C. (2026) · International Journal of Information and Education Technology · doi:10.18178/ijiet.2026.16.3.2531 · Cited by 3
“[28] reports that GPT-4 outperforms GPT-3.”
Менлибай, Ж. Ғ., & Керімбаев, Н. Н. (2026) · Bulletin of the Karaganda University Pedagogy series · doi:10.31489/2026ped1/150-164 · Cited by 2
Daryanto, T., Ding, X., Ping, K., Wilhelm, L. A., Chen, Y., Brown, C., & Rho, E. H. R. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791773 · Cited by 9
“These advancements have led to its adoption in various domains, including programming education [86].”
Превір, М., & Kyrylenko, O. I. (2025) · Problems of Modern Textbook · doi:10.32405/2411-1309-2025-35-94-106 · Cited by 2
Hasan, A. A., Saha, S., & Imran, M. M. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3786580.3786946 · Cited by 4
SpeedGen: Enhancing Code Efficiency through Large Language Model-Based Performance Optimization
Purschke, N., Kirchner, S., & Knoll, A. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00045 · Cited by 4
Hints Help Finding and Fixing Bugs Differently in Python and Text-Based Program Representations
Rawal, R., Pădurean, V.-A., Apel, S., Singla, A. K., & Toneva, M. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00192 · Cited by 4
“The hint conditions are carefully chosen to target different aspects of overall bug understanding, being grounded in providing support to students in programming education [12], [14], [15].”
Self-citationKılınç, S. (2025) · Future education and learning spaces · doi:10.1007/978-981-96-9251-4_2 · Cited by 3
Task Synthesis for Elementary Visual Programming in XLogoOnline Environment
Wen, C., Ghosh, A., Staub, J., & Singla, A. K. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-64312-5_37 · Cited by 2
Self-citationPortela, A. K. E., & Vieira, J. K. M. (2025) · Anais do XXXIII Workshop sobre Educação em Computação (WEI 2025) · doi:10.5753/wei.2025.8316 · Cited by 1
Fredericks, E. M., Bobeldyk, D., & Moore, J. M. (2025) · Genetic and evolutionary computation · doi:10.1007/978-981-96-0077-9_16 · Cited by 2
Åkerfeldt, A., Mannila, L., Kjällander, S., & Heintz, F. (2025) · ACM Transactions on Computing Education · doi:10.1145/3767723 · Cited by 2
“Some countries have introduced CS as a subject of its own (e.g., Computing in England [19]), while others have decided to integrate it with other subjects, by for instance making programming an interdisciplinary element throughout the curriculum (e.g., through digital competence in Sweden [73] and…”
Enhancing Educational Content Through AI-Assisted Transformation of Video Transcriptions
Zabala, I. D., Ribalta, C. T., & Mataix, J. B. (2025) · IEEE Digital Education and MOOCS Conference (DEMOcon) · doi:10.1109/democon65705.2025.11282786 · Cited by 2
“In [8] a systematic comparison of ChatGPT and GPT-4 performance on programming education is performed, finding significant improvements in GPT-4's ability to provide educational support that was closer to the one of human tutors.”
The Web Dev Challenge: Designing GenAI-Supported Gamified Learning in a University Classroom
Zolezzi, D., Adorni, G., Torre, I., & Vercelli, G. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-032-05070-0_17 · Cited by 2
How Large Language Models Shape Programming Skill Development Beyond Task Completion
Orehovački, T. (2026) · AI · doi:10.3390/ai7070252 · Cited by 1
“…and active engagement with underlying concepts. programming-support settings shows strong performance in debugging and explanation-generation tasks [27], while broader work in computer science education argues that generative AI challenges assessment models that rely primarily on final code…”
Tives, H. A., Marioti, M. B., & Jaques, P. A. (2025) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/sbie.2025.12361 · Cited by 2
“GPT-4, for instance, tends to make extensive changes to code, not always respecting the original context provided by the student [Phung et al. 2023].”
Examining Student-ChatGPT Interactions in Programming Education
Ma, B., Chen, L., & Konomi, S. (2025) · Cognition and exploratory learning in the digital age · doi:10.1007/978-3-032-05817-1_5 · Cited by 5
高格格 (2026) · Advances in Education · doi:10.12677/ae.2026.163465 · Cited by 1
Liu, J., Zhang, Y., Zhang, X., Jing, L., & Gui, M. (2026) · International Conference on Digital Education and Information Technology (DEIT) · doi:10.1145/3802607.3802640 · Cited by 1
“This improvement likely occurred because the tool offered immediate, targeted, and comprehensible process-oriented prompts, thereby reducing frustration, and strengthening their efficacy [ 20 ] .”
Digital Enhancement: Applying Generative AI in Students Decision Making Process
Cabral, C. A. A., & Pereira, R. (2025) · Studies in systems, decision and control · doi:10.1007/978-3-031-78412-5_7 · Cited by 2
Intelligent Mathematical Tutor Based on ChatGPT or DeepSeek
Rothkrantz, C., Fitrianie, S., & Rothkrantz, L. J. M. (2025) · International Conference on Computer Systems and Technologies (CompSysTech) · doi:10.1109/compsystech65493.2025.11136856 · Cited by 1
Chat GPT Wrote it: What HCI Educators Can Learn from their Students?
Sikorski, M. (2024) · Proceedings of the International Conference on Information Systems Development · doi:10.62036/isd.2024.25
Simulated Interactive Debugging
Noller, Y., Chandra, E., Srinidhi, H. C., Choo, K. T. W., Jégourel, C., Kurniawan, O., & POSKITT, C. M. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00338 · Cited by 2
“Moreover, existing works [20], [21] benchmarked various models regarding their capabilities to support programming education.”
Future Directions and Innovations in Digital Educational Escape Room Games
(2025) · IGI Global eBooks · doi:10.4018/979-8-3693-4219-0.ch008
ChatGPT for Coding: User Insights and Challenges in Program Generation
Talib, M. W. A., Bettayeb, A. M., Altayasinah, A. Z. S., & Dakalbab, F. M. (2025) · International Wireless Communications and Mobile Computing Conference (IWCMC) · doi:10.1109/iwcmc65282.2025.11059508
Metadata
- Authors
- Phung, T., Pădurean, V.-A., Cambronero, J., Gulwani, S., Kohn, T., Majumdar, R., Singla, A., & Soares, G.
- Affiliations
- Max Planck Institute for Software Systems (DE)
- MPI-INF, Germany
- Microsoft (United States) (US)
- TU Wien (AT)
- ORCID
- Venue
- Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 2
- Publisher
- ACM
- Published
- 2023-08-07
- Pages
- 41-42
- OpenAlex
- W4386719008
- Semantic Scholar
- 449ab79be2d6086b88832cfc9c5d502524c8524f
- Cited by
- 112
- References
- 4
- Citations by year
- 2026: 26 · 2025: 51 · 2024: 30 · 2023: 5
- FWCI
- 33.87
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Conference on International Computing Education Research (ICER) (publishedVersion)
- Funders
- European Research Council (ERC, TOPS, 101039090)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- generative AI, programming education, large language models, ChatGPT, GPT-4, human tutors, benchmarking, AI tutors, personalized tutoring, cooperative learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.