My AI Wants to Know if This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
The introduction of OpenAI Codex sparked a surge of interest in the impact of generative AI models on computing education practices. Codex is also the underlying model for GitHub Copilot, a plugin which makes AI-generated code accessible to students through auto-completion in popular code editors. Research in this area, particularly on the educational implications, is nascent and has focused almost exclusively on introductory programming (or CS1) questions. Very recent work has shown that Codex performs considerably better on typical CS1 exam questions than most students. It is not clear, however, what Codex’s limits are with regard to more complex programming assignments and exams. In this paper, we present results detailing how Codex performs on more advanced CS2 (data structures and algorithms) exam questions taken from past exams. We compare these results to those of students who took the same exams under normal conditions, demonstrating that Codex outscores most students. We consider the implications of such tools for the future of undergraduate computing education.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, this approach may have become obsolete given how easily most LLMs can now solve introductory computing problems with simple prompts [85,86,167,175].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [86]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Teachers who shift away from using many small problems to create larger and more authentic problems in an attempt to reduce reliance of generative AI will lose access to the quick and easy assessment methods such as automated grading associated with many introductory programming courses [86].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [86]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the consistent pattern is that ChatGPT and other LLMs are capable of producing content sufficient to pass undergraduate level assessment in diverse curriculum areas.”
From Bob or Bot: Exploring ChatGPT's Answers to University Computer Science Assessment (2024) · cites this paper as [20]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, the strengths and limitations reported in prior work [7, 10, 11, 38, 46] still hold to a certain degree.”
From Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses (2023) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“The impressive documented performance of large language models in solving typical CS1 and CS2 problems suggests that some rethinking of assessments is essential [86].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [86]
9 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043
Competition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497 · cited 2×
“explored the potential for Codex to generate novel programming exercises and explanations of code [23].”
BackgroundSelf-citationIntroductory programming: a systematic literature review
Luxton-Reilly, A., Simon, Albluwi, I., Becker, B. A., Giannakos, M., Kumar, A. N., Ott, L., Paterson, J., Scott, M. J., Sheard, J., et al. (2018) · Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3293881.3295779 · Cited by 485
Self-citationThe Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463
Self-citationGitHub Copilot AI pair programmer: Asset or Liability?
Dakhel, A. M., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., Ming, Z., & Jiang (2022) · arXiv · doi:10.48550/arxiv.2206.15331 · Cited by 417 · cited 2×
“explored Copilot’s ability to generate code for various introductory algorithms, as well as five introductory Python programming problems [6].”
BackgroundGitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417
An empirical evaluation of GitHub copilot's code suggestions
Nguyen, N., & Nadi, S. (2022) · Proceedings of the 19th International Conference on Mining Software Repositories · doi:10.1145/3524842.3528470 · Cited by 313 · cited 1×
“Nguyen and Nadi evaluated the performance of Copilot on 33 LeetCode questions, with varying difficulty, observing correctness rates of between 27% and 57% across four languages [21].”
BackgroundA theory of instruction for introductory programming skills
Xie, B., Loksa, D., Nelson, G. L., Davidson, M. J., Dong, D., Kwik, H., Tan, A. H., Hwa, L., Li, M., & Ko, A. J. (2019) · Computer Science Education · doi:10.1080/08993408.2019.1565235 · Cited by 191
Research works printed but not linked: 3
- Brett Becker. 2017. Artificial Intelligence in Education: What Is It, Where Is It Now, Where Is It Going. Ireland’s Yearbook of Education 2018 (2017), 42–46.
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, 2021. Evaluating Large Language Models Trained on Code. https://arxiv.org/abs/2107.03374. (2021). · arXiv
- ACM Joint Task Force on Computing Curriculaand IEEE Computer Society. 2013. Computer Science Curricula 2013: Curriculum Guidelines for Undergraduate Degree Programs in Computer Science. ACM, NY, NY, USA.
Cited by
Items 201–279 of 279
Hey, ChatGPT, Look at My Work: Using Conversational AI in Requirements Engineering Education
Badihi, S., Tegegn, M., Riddell, E., Czarnecki, K., & Rubin, J. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773112 · Cited by 1
Characterization and Effects of CS2 Learning with GenAI, Visualization, and Human Support
Yong, Q., Nacenta, M. A., & Estey, A. (2026) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3765964.3811663 · Cited by 1
AI-Assisted Help-Seeking Trajectories in Programming Education from an SRL-Informed PerspectivePreprint
Ma, B., Li, H., Li, G., Chen, L., Shimada, A., & Konomi, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.22809 · Cited by 1
Pan, D. P., Luo, L., Zhu, J., Gao, Z., Tong, X., & Hui, P. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791410 · Cited by 1
MACEDON : Supporting Programmers with Real-Time Multi-Dimensional Code Evaluation and Optimization
Liu, X., You, Y., Qiu, X., Ma, T., & Zhao, J. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747655 · Cited by 1
Generative artificial intelligence for education: The present and the future
Reynaud, T. V. (2024) · Repositório do ISCTE-IUL · Cited by 1
Bergh, C. H., Tag, B., Vassar, A., & Renzella, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.21194 · Cited by 1
Do Not Copy/Paste: Soft Barriers for Copying in AI-Assisted Programming
Olatunji, I. E., Djiré, A. E., Klein, J., & Bissyandé, T. F. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834554 · Cited by 1
Ohjelmointitehtävien ratkaisu ChatGPT:llä ja sen vaikutukset ohjelmoinnin opetukseen
Rytilahti, J. (2024) · UTUPub (University of Turku) · Cited by 1
Suomalainen, E. (2024) · LUTPub (LUT University) · Cited by 1
Paiva, J. C. C. (2025) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) · doi:10.34626/jwhx-jn48 · Cited by 1
Scaffolding Autocomplete: Improving Guidance for Learners using Generative Code Suggestions
Prather, J. E., MacNeil, S., Luxton-Reilly, A., Margulieux, L. E., Reeves, B. N., Denny, P. C., Leinonen, J., Homer, J., Nabid, R. A., & Rossetti, R. L. (2026) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3765964.3811654
Self-citationMODEL OF AUTOMATED TEST EVALUATION USING GENERATIVE AI TOOLS
Aritonovic, G. (2026) · SCIENCE International Journal · doi:10.35120/sciencej0502227a
Gardella, N., Prather, J. E., Leinonen, J., Denny, P. C., Pettit, R., & Riggs, S. L. (2026) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3765964.3811664
Self-citationPrompts Engineering Challenges in Software Code Generation
Gutierrez, Y., Camacho, E., Pardo, C., & Villarreal, V. (2025) · IEEE Congreso Internacional en Inteligencia Ambiental, Ingeniería de Software y Salud Electrónica y Móvil (AmITIC) · doi:10.1109/amitic68284.2025.11214606
109PwnPilot: could an adversary be pair programming with our most trusted software engineers?
Horne, D. (2025) · De Gruyter eBooks · doi:10.1515/9783111436548-006
Explanation First Prompting for Reviewing AI Programming Answers in Undergraduate Courses
Raiter, O. К. (2026) · International Conference on Advanced Computer Information Technologies (ACIT) · doi:10.1109/acit69831.2026.11700007
Decoding AI Impact: Longitudinal Analysis of Code Submissions in Programming MOOCs
Leppänen, L., Leinonen, J., Hellas, A., Kaila, E., & Mannila, L. (2025) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie63693.2025.11328633
Comparative Analysis of Generative AI Performance in University Programming Courses
Suomalainen, E., Vanhala, E., & Ikonen, J. (2026) · Lecture notes in business information processing · doi:10.1007/978-3-032-14518-5_16
Student use of accurate and inaccurate chatbot content: an empirical study
Lee, J. S. Y. (2025) · International Conference on Computing and Artificial Intelligence (ICCAI) · doi:10.1109/iccai66501.2025.00065
Thị, H. T. (2025) · International Journal of Social Science and Human Research · doi:10.47191/ijsshr/v8-i7-44
Analysis of Students’ GenAI Prompts During a Practical SQL Test
Ramakrishnan, C., Cassidy, S., & Bower, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98465-5_23
The AI-Competence Ceiling: Redefining Human Expertise in an AI-Augmented World
Yadav, P. S. (2026) · Future of business and finance · doi:10.1007/978-3-032-11748-9_4
PAV: Project-Led, AI-Permissive, and Verified Assessment for GenAI-Ready IT Education
Li, B., Zhao, L., Ye, W., Pradhan, S., & Miao, Y. (2026) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse72781.2026.00045
Aplicação de IA generativa no processo de tomada de decisão dos estudantes
Cabral, C. A. A. (2024) · Repositório do ISCTE-IUL
Kituku, B., Araka, E., & Maina, E. M. (2025) · International Conference on Information and Communication Technology for Development for Africa (ICT4DA) · doi:10.1109/ict4da67218.2025.11282535
Describing Functionality in Natural Language May Improve Decomposition Behaviors
Burns, M., Edwards, W., & Edwards, J. (2026) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3770762.3772513
Sonawane, A. V., Garad, A. P., Puri, H. S. G., & Karande, A. P. (2026) · International Conference on Innovation & Growth in Next-Gen Intelligent Technology & Engineering (IGNITE) · doi:10.1109/ignite69592.2026.11706531
Exploring Continuance Intention in the Use of AI Coding Assistants
Wangsadjaja, A., Chandra, C. S., & Gui, A. (2025) · International Conference on Intelligent & Innovative Practices in Engineering & Management (IIPEM) · doi:10.1109/iipem65914.2025.11548454
Summative Assessment of Program Comprehension in CS1 with Questions About Learners' Code
Caniço, A. B., & Santos, A. L. (2026) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/oasics.icpec.2026.1
Programmers' Views on IDE Compilation Mechanisms
Karvelas, I., Dillane, J., & Becker, B. A. (2023) · ACM Global Computing Education Conference (CompEd) · doi:10.1145/3576882.3617915
Self-citationChatGPT for Coding: User Insights and Challenges in Program Generation
Talib, M. W. A., Bettayeb, A. M., Altayasinah, A. Z. S., & Dakalbab, F. M. (2025) · International Wireless Communications and Mobile Computing Conference (IWCMC) · doi:10.1109/iwcmc65282.2025.11059508
Towards Introductory Programming with ChatGPT-Assisted MTL-Based Program Visualization
Nyamawe, A. S. (2026) · Journal of Computer Languages · doi:10.1016/j.cola.2026.101425
Liang, H., & Brown, C. (2026) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3803437.3805792
Evaluating LLM-Generated Contextualized Algorithm Design Problems
Goodwin, E., Braught, K., Liu, J., Newar, D. K. P., Gertner, Y., Poulsen, S., & Franklin, D. M. (2026) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3765964.3811658
Wang, W., & Bakharia, A. (2026) · IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt68022.2026.00104
Yazılım Mühendisliği Eğitiminde Üretken Yapay Zekânın Bibliyometrik Analiz Yöntemi ile İncelenmesi
Baz, F. Ç., & Keleş, Ş. (2026) · Osmaniye Korkut Ata Üniversitesi Fen Bilimleri Enstitüsü Dergisi · doi:10.47495/okufbed.1766554
How Effectively Can Off-the-Shelf Generative AI Tools Solve Circuit Design Problems?
Azhaan, M., Afeef, M., & Kumar, V. (2025) · Lecture notes in educational technology · doi:10.1007/978-981-95-1734-3_21
Alsaid, D. (2026) · Annual Symposium on Computer-Human Interaction in Play · doi:10.1145/3800965.3834672
Ye, R., Fan, J., Bernuy, A. Z., Karnalim, O., Denny, P. C., Leinonen, J., & Liut, M. (2026) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3856208.3856226
Self-citationHou, X., Weng, Y., Yeh, C. H., Lee, D. L., Zheng, L., Li, F., Wang, W., & Liu, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.22993
Tabarsi, B. T., Yasir, T., Reichert, H., Tian, X., Gadireddy, S., DiMarco, C., Briceno, D., Battestilli, L., & Barnes, T. M. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7463479
Li, Z., Wu, J., Liu, Q., Sun, Q., Li, J., & Rong, W. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11729842
GENERATIVE AI REDUCES PERIPHERAL BUT NOT MENTAL COGNITIVE LOAD IN PROGRAMMING EDUCATION
Аркабаев, Н. К., Ойчуева, Р., & Ajibekova, A. (2026) · Jurnal Ilmiah Ilmu Terapan Universitas Jambi|JIITUJ| · doi:10.22437/jiituj.v10i4.58071
Lepp, M., & Kaimre, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.16318
Nathaniel, J. (2026) · UEF eRepo (University of Eastern Finland)
Sundell, K., & Öberg, E. (2023) · Uppsala University Publications (Uppsala University)
Investigating Conversational Agents to Support Secondary School Students Learning CSPPreprint
Frazier, M., Damevski, K., & Pollock, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.16213
Investigating conversational agents to support secondary school computer science exploratory search
Frazier, M. (2024) · Library, Museums and Press - UDSpace (University of Delaware) · doi:10.58088/1gk5-9t60
Cipriano, B. M. P. (2025) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT)
Scaling CS1 Support with Compiler-Integrated Conversational AIPreprint
Renzella, J., Vassar, A., Solano, L. L., & Taylor, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.02378
da Silva, T. L. (2026) · Lume (Universidade Federal do Rio Grande do Sul)
Understanding Student Use of Large Language Models Across Computer Science SubfieldsPreprint
Nizamani, S. B., Lee, Y., Chandrashekar, N. D., Ellis, M., & Ramakrishnan, N. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.01158
Yalwa, A. S., Othman, M. S., Yusuf, L. M., & Almarshadi, M. S. (2026) · Cogent Education · doi:10.1080/2331186x.2026.2745324
Sriprasert, A., & Chavanich, S. (2026) · EAU Heritage Journal Science and Technology (online)
Beyond Copy-Paste: Detecting and Understanding Students' Use of Unauthorized Aid when Monitored
Price, T., Titus, K., Jiao, S., & Tran, K. (2026) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3856208.3856225
HUMAN-CENTERED AI DEVELOPMENT FOR AUGMENTED COGNITION
Doliashvili, M. (2024) · ScholarSpace (University of Hawaii at Manoa)
Who Is Doing the Thinking? Characterising Human Agency in First-Year Students' AI InteractionsPreprint
Shakiba, A., & Chaffey, T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.04783
Proficiency and Usage of AI in an Introductory Object-Oriented Programming Course
Kaimre, J. (2024) · DSpace repository (University of Tartu)
KI in der Programmierausbildung
Klein, J. T. (2025) · Repository KITopen (Karlsruhe Institute of Technology) · doi:10.5445/ir/1000181842
Wang, Y., Yang, C., Hou, S., Xu, X., Cui, B., 蒋严冰, & Yang, T. (2026) · Data Science and Engineering · doi:10.1007/s41019-026-00371-9
Evaluating LLMs for Answering Student Questions in Introductory Programming CoursesPreprint
Mullem, T. V., Mesuere, B., & Dawyndt, P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.28295
Code and Beyond: A Review on the Impact of AI on Modern Software Development
Lohana, A., Ghulamani, S., Khowaja, K., Talpur, K. R., & Shah, A. (2024) · Pakistan Journal of Engineering Technology & Science · doi:10.22555/pjets.v12i2.1210
Segers, M. (. (2023) · Research Repository (Delft University of Technology)
Hey, ChatGPT, look at my work : using conversational AI in requirements engineering education
Tegegn, M. (2025) · cIRcle (University of British Columbia) · doi:10.14288/1.0450984
Nga, H. T. T., & Thuy, N. T. P. (2026) · Saudi Journal of Engineering and Technology · doi:10.36348/sjet.2026.v11i07.002
Is Solving Better Than Evaluating GenAI Solutions?Preprint
Dickey, E., Mertzanidis, M., & Psomas, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.27586
Gupta, R. D., Hosain, M. T., Mridha, M. F., & Ahmed, S. U. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.06552
Sengul, C., Neykova, R., & Destefanis, G. (2024) · doi:10.3389/frai.2024.1436350.s001
Pecuchova, J., & Drlík, M. (2026) · Procedia Computer Science · doi:10.1016/j.procs.2026.09.162
АЛЄКСЄЄВА, Г., HORBATIYK, L., & Сергійович, О. О. (2025) · Scientific papers of Berdiansk State Pedagogical University Series Pedagogical sciences · doi:10.32782/2412-9208-2025-3-392-402
UXMind: An Intelligent Agent for AI-Assisted Interface Evaluation in UX Education
Pontes, D. P. N. (2026) · The Paris Conference on Education, official conference proceedings · doi:10.22492/issn.2758-0962.2026.48
Imashev, A., Abisheva, Z., Tusseyeva, I., Kozhirbayev, Z., Alizadeh, T., & Matkarimov, B. T. (2026) · Research Square · doi:10.21203/rs.3.rs-10677896/v1
Inteligencia artificial: retos y oportunidades en los cursos de programación para ingeniería
Valencia, J., Cano-Quintero, J. B., & Velilla, E. (2025) · Encuentro Internacional de Educación en Ingeniería · doi:10.26507/paper.4395
Teaching Algorithmic Thinking in a Liberal STEM Curriculum: An Experience Report
Venkatasubramanian, V., Chandra, P., Vibhavari, R., & Satish, N. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-14583-3_8
Piccolo, S. R., Denny, P., Luxton-Reilly, A., Payne, S. H., & Ridge, P. G. (2023) · doi:10.1371/journal.pcbi.1011511.r001
Self-citation"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2023) · arXiv
“A later study conducted by the same group found that Codex was similarly performant in Data Structures and Algorithms (CS2) exams [43].”
ResultSelf-citationKiesler, N., & Schiffner, D. (2023) · arXiv · doi:10.48550/arxiv.2308.08572
Metadata
- Authors
- Finnie-Ansley, J., Denny, P., Luxton-Reilly, A., Santos, E. A., Prather, J., & Becker, B. A.
- Affiliations
- University of Auckland (NZ)
- University College Dublin (IE)
- Abilene Christian University (US)
- ORCID
- Venue
- Proceedings of the 25th Australasian Computing Education Conference
- Publisher
- ACM
- Published
- 2023-01-30
- Pages
- 97-104
- OpenAlex
- W4317502110
- Cited by
- 186
- References
- 21
- Citations by year
- 2026: 34 · 2025: 59 · 2024: 59 · 2023: 34
- FWCI
- 55.98
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Australasian Computing Education Conference (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- OpenAI Codex, generative AI in education, data structures and algorithms, AI-generated code, GitHub Copilot, student performance comparison, undergraduate computer science education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.