My AI Wants to Know if This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
The introduction of OpenAI Codex sparked a surge of interest in the impact of generative AI models on computing education practices. Codex is also the underlying model for GitHub Copilot, a plugin which makes AI-generated code accessible to students through auto-completion in popular code editors. Research in this area, particularly on the educational implications, is nascent and has focused almost exclusively on introductory programming (or CS1) questions. Very recent work has shown that Codex performs considerably better on typical CS1 exam questions than most students. It is not clear, however, what Codex’s limits are with regard to more complex programming assignments and exams. In this paper, we present results detailing how Codex performs on more advanced CS2 (data structures and algorithms) exam questions taken from past exams. We compare these results to those of students who took the same exams under normal conditions, demonstrating that Codex outscores most students. We consider the implications of such tools for the future of undergraduate computing education.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, this approach may have become obsolete given how easily most LLMs can now solve introductory computing problems with simple prompts [85,86,167,175].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [86]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Teachers who shift away from using many small problems to create larger and more authentic problems in an attempt to reduce reliance of generative AI will lose access to the quick and easy assessment methods such as automated grading associated with many introductory programming courses [86].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [86]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the consistent pattern is that ChatGPT and other LLMs are capable of producing content sufficient to pass undergraduate level assessment in diverse curriculum areas.”
From Bob or Bot: Exploring ChatGPT's Answers to University Computer Science Assessment (2024) · cites this paper as [20]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, the strengths and limitations reported in prior work [7, 10, 11, 38, 46] still hold to a certain degree.”
From Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses (2023) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“The impressive documented performance of large language models in solving typical CS1 and CS2 problems suggests that some rethinking of assessments is essential [86].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [86]
9 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043
Competition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497 · cited 2×
“explored the potential for Codex to generate novel programming exercises and explanations of code [23].”
BackgroundSelf-citationIntroductory programming: a systematic literature review
Luxton-Reilly, A., Simon, Albluwi, I., Becker, B. A., Giannakos, M., Kumar, A. N., Ott, L., Paterson, J., Scott, M. J., Sheard, J., et al. (2018) · Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3293881.3295779 · Cited by 485
Self-citationThe Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463
Self-citationGitHub Copilot AI pair programmer: Asset or Liability?
Dakhel, A. M., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., Ming, Z., & Jiang (2022) · arXiv · doi:10.48550/arxiv.2206.15331 · Cited by 417 · cited 2×
“explored Copilot’s ability to generate code for various introductory algorithms, as well as five introductory Python programming problems [6].”
BackgroundGitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417
An empirical evaluation of GitHub copilot's code suggestions
Nguyen, N., & Nadi, S. (2022) · Proceedings of the 19th International Conference on Mining Software Repositories · doi:10.1145/3524842.3528470 · Cited by 313 · cited 1×
“Nguyen and Nadi evaluated the performance of Copilot on 33 LeetCode questions, with varying difficulty, observing correctness rates of between 27% and 57% across four languages [21].”
BackgroundA theory of instruction for introductory programming skills
Xie, B., Loksa, D., Nelson, G. L., Davidson, M. J., Dong, D., Kwik, H., Tan, A. H., Hwa, L., Li, M., & Ko, A. J. (2019) · Computer Science Education · doi:10.1080/08993408.2019.1565235 · Cited by 191
Research works printed but not linked: 3
- Brett Becker. 2017. Artificial Intelligence in Education: What Is It, Where Is It Now, Where Is It Going. Ireland’s Yearbook of Education 2018 (2017), 42–46.
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, 2021. Evaluating Large Language Models Trained on Code. https://arxiv.org/abs/2107.03374. (2021). · arXiv
- ACM Joint Task Force on Computing Curriculaand IEEE Computer Society. 2013. Computer Science Curricula 2013: Curriculum Guidelines for Undergraduate Degree Programs in Computer Science. ACM, NY, NY, USA.
Cited by
The promise and challenges of generative AI in education
Giannakos, M., Azevedo, R., Brusilovsky, P., Cukurova, M., Dimitriadis, Y., Hernandez-Leo, D., Järvelä, S., Mavrikis, M., & Rienties, B. (2025) · Behaviour & Information Technology · doi:10.1080/0144929x.2024.2394886 · Cited by 543
Becker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569759 · Cited by 401
Self-citationThe Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396
Self-citationComputing Education in the Era of Generative AI
Denny, P., Prather, J., Becker, B. A., Finnie-Ansley, J., Hellas, A., Leinonen, J., Luxton-Reilly, A., Reeves, B. N., Santos, E. A., & Sarsa, S. (2024) · Communications of the ACM · doi:10.1145/3624720 · Cited by 316
“Often, all that is required of a student is to accept an auto-generated suggestion by an IDE plugin [10, 11].”
BackgroundSelf-citationKazemitabaar, M., Ye, R., Wang, X., Henley, A. Z., Denny, P., Craig, M., & Grossman, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642773 · Cited by 290
Self-citationDenny, P., Kumar, V., & Giacaman, N. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569823 · Cited by 287
Self-citationChatGPT and Software Testing Education: Promises & Perils
Jalil, S., Rafi, S., LaToza, T. D., Moran, K., & Lam, W. (2023) · 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw58534.2023.00078 · Cited by 270
“One study found that Codex already performs better than most students on the code writing questions found in typical introductory programming exams [20] as well as more advanced exams on data structures and algorithms [21].”
BackgroundPrather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256
“A later study conducted by the same group found that Codex was similarly performant in Data Structures and Algorithms (CS2) exams [43].”
ResultSelf-citationLau, S., & Guo, P. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600138 · Cited by 241
“We sought out instructors who taught programming-based university CS1 or CS2 courses since those are many students’ entry points into computing.”
BackgroundMethodInfluentialLa nueva realidad de la educación ante los avances de la inteligencia artificial generativa
García Peñalvo, F. J., Llorens-Largo, F., & Vidal, J. (2023) · RIED-Revista Iberoamericana de Educación a Distancia · doi:10.5944/ried.27.1.37716 · Cited by 238
Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · arXiv · doi:10.48550/arxiv.2306.05715 · Cited by 211
The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B. (2024) · Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3632620.3671116 · Cited by 196
“LLMs can solve most introductory programming exercises in both CS1 [18] and CS2 courses [19].”
Self-citationPrompt Problems: A New Programming Exercise for the Generative AI Era
Denny, P., Leinonen, J., Prather, J., Luxton-Reilly, A., Amarouche, T., Becker, B. A., & Reeves, B. N. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630909 · Cited by 187
Self-citationAI Literacy in K-12 and Higher Education in the Wake of Generative AI: An Integrative Review
Gu, X., & Ericson, B. J. (2025) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3702652.3744217 · Cited by 166
Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600139 · Cited by 145
Savelka, J., Agarwal, A., An, M., Bogart, C., & Sakr, M. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600142 · Cited by 141
“[7, 10, 11, 38, 46] (RQ4) Can GPT-4 successfully utilize feedback to fix solutions of coding tasks?”
BackgroundMethodResultInfluentialKazemitabaar, M., Hou, X., Henley, A., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Proceedings of the 23rd Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631806 · Cited by 130
Next-Step Hint Generation for Introductory Programming Using Large Language Models
Roest, L., Keuning, H., & Jeuring, J. T. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636259 · Cited by 125
Denny, P. C., Khosravi, H., Hellas, A., Leinonen, J., & Sarsa, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.10509 · Cited by 119
Self-citationFeedback-Generation for Programming Exercises With GPT-4
Azaiz, I., Kiesler, N., & Strickroth, S. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653594 · Cited by 116
Savelka, J., Agarwal, A., Bogart, C., Song, Y., & Sakr, M. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588792 · Cited by 109
“Finnie-Ansley et al. also studied Codex’ performance on CS2 tasks [14].”
BackgroundEvaluating Contextually Personalized Programming Exercises Created with Generative AI
Logacheva, E., Hellas, A., Prather, J. E., Sarsa, S., & Leinonen, J. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671103 · Cited by 105
Self-citationDetecting ChatGPT-Generated Code Submissions in a CS1 Course Using Machine Learning Models
Hoq, M., Shi, Y., Leinonen, J., Babalola, D., Lynch, C. F., Price, T., & Akram, B. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630826 · Cited by 99
Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code GeneratorsPreprint
Denny, P. C., Leinonen, J., Prather, J. E., Luxton-Reilly, A., Amarouche, T., Becker, B. A., & Reeves, B. N. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.16364 · Cited by 97
Self-citationPrather, J., Leinonen, J., Kiesler, N., Gorson Benario, J., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., et al. (2025) · 2024 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3689187.3709614 · Cited by 96
Self-citationEvaluating LLM-generated Worked Examples in an Introductory Programming Course
Jury, B., Lorusso, A., Leinonen, J., Denny, P., & Luxton-Reilly, A. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636252 · Cited by 95
Self-citationKazemitabaar, M., Huang, O., Suh, S., Henley, A. Z., & Grossman, T. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712104 · Cited by 92
Lepp, M., & Kaimre, J. (2025) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2025.100642 · Cited by 92
Generative AI in Education: A Study of Educators' Awareness, Sentiments, and Influencing Factors
Ghimire, A., Pather, J., & Edwards, J. (2024) · 2024 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10892891 · Cited by 88
Denny, P. C., Smith, D. H., Fowler, M., Prather, J. E., Becker, B. A., & Leinonen, J. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653587 · Cited by 86
Self-citationSoftware engineering education in the era of conversational AI: current trends and future directions
Sengul, C., Neykova, R., & Destefanis, G. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1436350 · Cited by 84
Exploring the Potential of Large Language Models to Generate Formative Programming Feedback
Kiesler, N., Lohr, D., & Keuning, H. (2023) · 2023 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10343457 · Cited by 79
Computing education using generative artificial intelligence tools: A systematic literature review
Agbo, F. J., Olivia, C., Oguibe, G. F., Sanusi, I. T., & Sani, G. (2025) · Computers and Education Open · doi:10.1016/j.caeo.2025.100266 · Cited by 76
Incorporating Generative AI into Software Development Education
Petrovska, O., Clift, L., Moller, F., & Pearsall, R. (2024) · Proceedings of the 8th Conference on Computing Education Practice · doi:10.1145/3633053.3633057 · Cited by 75
“Finnie-Ansley et al. [6] found that OpenAI’s Codex solves the majority of CS2 programming exercises in Python.”
BackgroundAI Literacy in K-12 and Higher Education in the Wake of Generative AI: An Integrative Review
Gu, X., & Ericson, B. J. (2025) · arXiv · doi:10.48550/arxiv.2503.00079 · Cited by 73
How Far Are We? The Triumphs and Trials of Generative AI in Learning Software Engineering
Choudhuri, R., Liu, D., Steinmacher, I., Gerosa, M., & Sarma, A. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639201 · Cited by 72
ChatGPT in the Classroom. Exploring Its Potential and Limitations in a Functional Programming Course
Popovici, M.-D. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2023.2269006 · Cited by 70
Literature Review on the Integration of Generative AI in Programming Education
Nathaniel, J., Oyelere, S. S., Suhonen, J. T., & Tedre, M. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00524-3 · Cited by 70
Many bioinformatics programming tasks can be automated with ChatGPT
Piccolo, S. R., Denny, P., Luxton-Reilly, A., Payne, S., & Ridge, P. G. (2023) · arXiv · doi:10.48550/arxiv.2303.13528 · Cited by 69
Self-citationCombining LLM-Generated and Test-Based Feedback in a MOOC for Programming
Gabbay, H., & Cohen, A. (2024) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3657604.3662040 · Cited by 69
Piccolo, S. R., Denny, P., Luxton-Reilly, A., Payne, S. H., & Ridge, P. G. (2023) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011511 · Cited by 69
Self-citationPirzado, F. A., Ahmed, A., Mendoza-Urdiales, R. A., & Terashima‐Marín, H. (2024) · IEEE Access · doi:10.1109/access.2024.3443621 · Cited by 65
DBox: Scaffolding Algorithmic Programming Learning through Learner-LLM Co-Decomposition
Ma, S., Wang, J., Zhang, Y., Ma, X., & Wang, A. Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713748 · Cited by 65
Gaitantzi, A., & Kazanidis, I. (2025) · Applied Sciences · doi:10.3390/app15073960 · Cited by 65
Liu, J., & Li, S. (2024) · Journal of Educational Computing Research · doi:10.1177/07356331241240460 · Cited by 62
Student Perspectives on Using a Large Language Model (LLM) for an Assignment on Professional Ethics
Grande, V., Kiesler, N., & Rodríguez, M. A. F. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653624 · Cited by 61
AI-Enhanced Auto-Correction of Programming Exercises: How Effective is GPT-3.5?
Azaiz, I., Deckarm, O., & Strickroth, S. (2023) · International Journal of Engineering Pedagogy (iJEP) · doi:10.3991/ijep.v13i8.45621 · Cited by 61
Can Students without Prior Knowledge Use ChatGPT to Answer Test Questions? An Empirical Study
Shoufan, A. (2023) · ACM Transactions on Computing Education · doi:10.1145/3628162 · Cited by 59
“In another study [11], the same authors reported similar findings for programming tasks in advanced computing courses.”
ResultMargulieux, L. E., Prather, J., Reeves, B. N., Becker, B. A., Cetin Uzun, G., Loksa, D., Leinonen, J., & Denny, P. (2024) · Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3649217.3653621 · Cited by 59
Self-citationZamfirescu-Pereira, J. D., Qi, L., Hartmann, B., DeNero, J., & Norouzi, N. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701864 · Cited by 58
Nathaniel, J., Oyelere, S. S., Suhonen, J. T., & Tedre, M. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100460 · Cited by 58
Joshi, I., Budhiraja, R., Dev, H., Kadia, J., Ataullah, M. O., Mitra, S., Akolekar, H. D., & Kumar, D. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630803 · Cited by 58
Humble, N. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00494-x · Cited by 58
Chugh, R., Turnbull, D., Morshed, A., Sabrina, F., Azad, S., Rashid, M. M., Kaisar, S., & Subramani, S. (2025) · Computer Applications in Engineering Education · doi:10.1002/cae.70002 · Cited by 57
Reeves, B., Sarsa, S., Prather, J., Denny, P., Becker, B. A., Hellas, A., Kimmel, B., Powell, G., & Leinonen, J. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588805 · Cited by 56
“This work has mainly focused on LLM performance in solving code writing exercises [4, 14, 15] and on using LLMs for creating or improving educational resources [6, 27, 29–31 Early work by Finnie-Ansley et al. [14] explored the performance of the OpenAI Codex model on typical introductory (CS1)…”
BackgroundMethodSelf-citationAkolekar, H. D., Jhamnani, P., Kumar, V., Tailor, V., Pote, A., Meena, A., Kumar, K., Challa, J. S., & Kumar, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-93871-z · Cited by 56
Integrating AI Tutors in a Programming Course
Ma, I., Krone-Martins, A., & Lopes, C. V. (2024) · ACM Virtual Global Computing Education Conference (SIGCSE Virtual) · doi:10.1145/3649165.3690094 · Cited by 55
Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study
Sheard, J., Denny, P., Hellas, A., Leinonen, J., Malmi, L., & Simon (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630880 · Cited by 54
“Similar performance was found in a follow-up study that used Codex to solve more complex problems in data structures and algorithms (CS2) [12].”
ResultSelf-citationBudhiraja, R., Joshi, I., Challa, J. S., Akolekar, H. D., & Kumar, D. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636257 · Cited by 52
Bob or Bot: Exploring ChatGPT's Answers to University Computer Science Assessment
Richards, M., Waugh, K., Slaymaker, M., Petre, M., Woodthorpe, J., & Gooch, D. (2024) · ACM Transactions on Computing Education · doi:10.1145/3633287 · Cited by 52
Analyzing LLM Usage in an Advanced Computing Class in India
Arora, U., Garg, A., Gupta, A., Jain, S., Mehta, R. L., Oberoi, R., Prachi, Raina, A., Saini, M., Sharma, S., et al. (2025) · Australasian Computing Education Conference · doi:10.1145/3716640.3716657 · Cited by 52
Melker, S., Gabrils, E., Villavicencio, V., Faraon, M., & Rönkkö, K. (2025) · International Journal of Technology and Design Education · doi:10.1007/s10798-025-09964-3 · Cited by 48
Chang, C. I., Choi, W. C., & Choi, I. C. (2024) · International Conference on Big Data and Education (ICBDE) · doi:10.1145/3704289.3704301 · Cited by 47
Pan, Z., Xie, Z., Liu, T., & Xia, T. (2024) · Systems · doi:10.3390/systems12050176 · Cited by 46
How Do Programming Students Use Generative AI?
Rahe, C., & Maalej, W. (2025) · Proceedings of the ACM on Software Engineering · doi:10.1145/3715762 · Cited by 45
“…as ChatGPT [OpenAI 2022] and GitHub Copilot [GitHub 2022], students in programming courses now have access to code authoring tools capable of solving coding exercises [Berrezueta-Guzman and Krusche 2023; Chen et al. 2021; Savelka et al. 2023] and exam questions entirely [Finnie-Ansley et al. 2023].”
Enhancing Programming Error Messages in Real Time with Generative AI
Kimmel, B., Geisert, A. L., Yaro, L., Gipson, B., Hotchkiss, R., Osae-Asante, S. K., Vaught, H., Wininger, G., & Yamaguchi, C. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3647967 · Cited by 44
Prather, J. E., Reeves, B. N., Denny, P. C., Leinonen, J., MacNeil, S., Luxton-Reilly, A., Orvalho, J. G., Alipour, M. A., Alfageeh, A., Amarouche, T., et al. (2025) · Australasian Computing Education Conference · doi:10.1145/3716640.3716649 · Cited by 44
Self-citationTowards Giving Timely Formative Feedback and Hints to Novice Programmers
Jeuring, J., Keuning, H., Marwan, S., Bouvier, D., Izu, C., Kiesler, N., Lehtinen, T., Lohr, D., Peterson, A., & Sarsa, S. (2022) · Proceedings of the 2022 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3571785.3574124 · Cited by 40
Chen, Y., Xiao, S., Song, Y., Li, Z., Sun, L., & Chen, L. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2475991 · Cited by 39
Joshi, I., Budhiraja, R., Tanna, P. D., Jain, L., Deshpande, M., Srivastava, A., Rallapalli, S., Akolekar, H. D., Challa, J. S., & Kumar, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.10694 · Cited by 38
Bernstein, S., Rahman, A., Sharifi, N., Terbish, A., & MacNeil, S. (2025) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3769994.3770036 · Cited by 38
Generating Programs Trivially: Student Use of Large Language Models
Prasad, S., Greenman, B., Nelson, T., & Krishnamurthi, S. (2023) · ACM Global Computing Education Conference (CompEd) · doi:10.1145/3576882.3617921 · Cited by 37
Using GPT-4 to Provide Tiered, Formative Code Feedback
Nguyen, H., & Allan, V. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630960 · Cited by 37
Smith, D. H., & Zilles, C. B. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653582 · Cited by 37
Always Provide Context: The Effects of Code Context on Programming Error Message Enhancement
Santos, E. A., Prasad, P., & Becker, B. A. (2023) · ACM Global Computing Education Conference (CompEd) · doi:10.1145/3576882.3617909 · Cited by 36
Self-citationScholl, A., & Kiesler, N. (2024) · 2024 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10893442 · Cited by 35
“The performance of GenAI tools can be impressive, especially in introductory programming exercises and respective exams [29], [30].”
Lauren, P., & Watta, P. B. (2023) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10342970 · Cited by 35
Cipriano, B. P., & Alves, P. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1145/3639474.3640052 · Cited by 35
Ali, M., Rao, P. V., Mai, Y., & Xie, B. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671097 · Cited by 33
Can ChatGPT Play the Role of a Teaching Assistant in an Introductory Programming Course?Preprint
Anishka, Mehta, A., Gupta, N., Balachandran, A., Kumar, D., & Jalote, P. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.07343 · Cited by 33
How Instructors Incorporate Generative AI into Teaching Computing
Prather, J. E., Leinonen, J., Kiesler, N., Benario, J. G., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., et al. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649405.3659534 · Cited by 32
Self-citationLet's Ask AI About Their Programs: Exploring ChatGPT's Answers To Program Comprehension Questions
Lehtinen, T., Koutcheme, C., & Hellas, A. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1145/3639474.3640058 · Cited by 32
Ma, B., Li, H., Li, G., Chen, L., Tang, C., Xie, Y., Gu, C., Shimada, A., & Konomi, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.04144 · Cited by 30
Computer Programming Education in the Age of Generative AI: Insights from Empirical ResearchPreprint
Deriba, F. G., Sanusi, I. T., Campbell, O. O., & Oyelere, S. S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4891302 · Cited by 28
A Global Survey of Introductory Programming Courses
Mason, R., Simon, N. P., Becker, B. A., Crick, T., & Davenport, J. H. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630761 · Cited by 28
Self-citationNot the Silver Bullet: LLM-enhanced Programming Error Messages are Ineffective in Practice
Santos, E. A., & Becker, B. A. (2024) · Conference on United Kingdom & Ireland Computing Education Research (UKICER) · doi:10.1145/3689535.3689554 · Cited by 27
Self-citationWuisang, M. C., Kurniawan, M., Santosa, K. A. W., Gunawan, A. A. S., & Saputra, K. E. (2023) · International Seminar on Application for Technology of Information and Communication (iSemantic) · doi:10.1109/isemantic59612.2023.10295323 · Cited by 27
Interactions with Prompt Problems: A New Way to Teach Programming with Large Language ModelsPreprint
Prather, J. E., Denny, P. C., Leinonen, J., Smith, D. H., Reeves, B. N., MacNeil, S., Becker, B. A., Luxton-Reilly, A., Amarouche, T., & Kimmel, B. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.10759 · Cited by 27
Self-citationAgarwal, V., Garg, M. K., Dharmavaram, S., & Kumar, D. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.01687 · Cited by 27
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Caspersen, M. E., Craig, M., Keuning, H., Kiesler, N., Kohn, T., et al. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 2 · doi:10.1145/3587103.3594206 · Cited by 26
Self-citationKorpimies, K., Laaksonen, A., & Luukkainen, M. (2024) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3699538.3699541 · Cited by 25
Pirzado, F. A., Ahmed, A., Alipour, M. A., & Terashima‐Marín, H. (2025) · Journal of Computers in Education · doi:10.1007/s40692-025-00375-w · Cited by 23
Mahon, J., Namee, B. M., & Becker, B. A. (2023) · Conference on United Kingdom & Ireland Computing Education Research (UKICER) · doi:10.1145/3610969.3610982 · Cited by 23
Self-citationSantos, P., Cantalejo, K. U., & Moreno, V. (2024) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro60963.2024.10569779 · Cited by 22
ExGen: Ready-To-Use Exercise Generation in Introductory Programming Courses
Nguyen Binh Duong, T. A., Nguyen, H. L., & Gottipati, S. (2023) · International Conference on Computers in Education · doi:10.58459/icce.2023.953 · Cited by 22
Solving Proof Block Problems Using Large Language Models
Poulsen, S., Sarsa, S., Prather, J. E., Leinonen, J., Becker, B. A., Hellas, A., Denny, P. C., & Reeves, B. N. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630928 · Cited by 21
Self-citationPickering, M. A., Williams, H., Gan, A., He, W., Park, H., Vélez, F., Littman, M. L., & Ur, B. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713271 · Cited by 21
Šavelka, J., Agarwal, A., Bogart, C., & Sakr, M. F. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-53656-4_8 · Cited by 21
Supervised Fine-Tuning LLMs to Behave as Pedagogical Agents in Programming EducationPreprint
Ross, E., Kansal, Y., Renzella, J., Vassar, A., & Taylor, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.20527 · Cited by 21
Prather, J., Reeves, B. N., Denny, P., Leinonen, J., MacNeil, S., Luxton-Reilly, A., Orvalho, J., Alipour, A., Alfageeh, A., Amarouche, T., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.12800 · Cited by 20
Self-citation
Metadata
- Authors
- Finnie-Ansley, J., Denny, P., Luxton-Reilly, A., Santos, E. A., Prather, J., & Becker, B. A.
- Affiliations
- University of Auckland (NZ)
- University College Dublin (IE)
- Abilene Christian University (US)
- ORCID
- Venue
- Proceedings of the 25th Australasian Computing Education Conference
- Publisher
- ACM
- Published
- 2023-01-30
- Pages
- 97-104
- OpenAlex
- W4317502110
- Cited by
- 186
- References
- 21
- Citations by year
- 2026: 34 · 2025: 59 · 2024: 59 · 2023: 34
- FWCI
- 55.98
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Australasian Computing Education Conference (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- OpenAI Codex, generative AI in education, data structures and algorithms, AI-generated code, GitHub Copilot, student performance comparison, undergraduate computer science education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.