The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Open accessAbstract
Recent advances in artificial intelligence have been driven by an exponential growth in digitised data. Natural language processing, in particular, has been transformed by machine learning models such as OpenAI’s GPT-3 which generates human-like text so realistic that its developers have warned of the dangers of its misuse. In recent months OpenAI released Codex, a new deep learning model trained on Python code from more than 50 million GitHub repositories. Provided with a natural language description of a programming problem as input, Codex generates solution code as output. It can also explain (in English) input code, translate code between programming languages, and more. In this work, we explore how Codex performs on typical introductory programming problems. We report its performance on real questions taken from introductory programming exams and compare it to results from students who took these same exams under normal conditions, demonstrating that Codex outscores most students. We then explore how Codex handles subtle variations in problem wording using several published variants of the well-known “Rainfall Problem” along with one unpublished variant we have used in our teaching. We find the model passes many test cases for all variants. We also explore how much variation there is in the Codex generated solutions, observing that an identical input prompt frequently leads to very different solutions in terms of algorithmic approach and code length. Finally, we discuss the implications that such technology will have for computing education as it continues to evolve, including both challenges and opportunities.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“They each focus on the capabilities of the tool they are using (ChatGPT/Codex), rather than the broader process in which marks are awarded to scripts.”
From Bob or Bot: Exploring ChatGPT's Answers to University Computer Science Assessment (2024) · cites this paper as [19]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, this approach may have become obsolete given how easily most LLMs can now solve introductory computing problems with simple prompts [85,86,167,175].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [85]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Finnie-Ansley et al. [85] published the first paper that examined the performance of large language models in solving introductory programming exercises.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [85]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the consistent pattern is that ChatGPT and other LLMs are capable of producing content sufficient to pass undergraduate level assessment in diverse curriculum areas.”
From Bob or Bot: Exploring ChatGPT's Answers to University Computer Science Assessment (2024) · cites this paper as [19]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“recent studies in the context of learning programming have found the correctness of Codex (the LLM that powers Copilot) to be around 70%-80% [34, 75], meaning that in about 20-30% of cases, the suggestion by Copilot would be incorrect.”
From “It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers (2024) · cites this paper as [34]
Experimental — accuracy not yet measured
18 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 16×
“Anticipating this shift, we put the following sentence into GPT-3, “The robots are taking over.””
BackgroundMethodInfluentialGPT-3: Its Nature, Scope, Limits, and Consequences
Floridi, L., & Chiriatti, M. (2020) · Minds and Machines · doi:10.1007/s11023-020-09548-1 · Cited by 2,519 · cited 3×
“It is considered a competitor to Alphabet’s DeepMind [11].”
BackgroundSignature pedagogies in the professions
Shulman, L. S. (2005) · Daedalus · doi:10.1162/0011526054622015 · Cited by 2,032 · cited 1×
“Such activities are so common they may form a signature pedagogy [36] of computing education.”
BackgroundEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 6×
“if the target is 21 and the parameter list is [15, 10, 5, 20], the function”
BackgroundMethodInfluentialLearning to program = learning to construct mechanisms and explanations
Soloway, E. (1986) · Communications of the ACM · doi:10.1145/6592.6594 · Cited by 567 · cited 6×
“This is a unique variant of Soloway’s Rainfall Problem [39] which has been used in a number of studies of programming ability over the past 30+ years and which has developed a reputation for being surprisingly difficult for introductory-level (CS1 [3]) students [32] with multiple studies confirming poor performance [1…”
BackgroundMethodInfluentialIntroductory programming: a systematic literature review
Luxton-Reilly, A., Simon, Albluwi, I., Becker, B. A., Giannakos, M., Kumar, A. N., Ott, L., Paterson, J., Scott, M. J., Sheard, J., et al. (2018) · Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3293881.3295779 · Cited by 485 · cited 1×
“A comprehensive review of introductory programming literature from 2018 makes no mention of tools that use AI to produce code, and only mentions a few machine learning based systems to analyse code or provide feedback along with intelligent tutoring systems that aim to provide customised learning approaches (normally,…”
BackgroundSelf-citationContract cheating by STEM students through a file sharing website: a Covid-19 pandemic perspective
Lancaster, T., & Cotarlan, C. (2021) · International Journal for Educational Integrity · doi:10.1007/s40979-021-00070-0 · Cited by 312
All I really need to know about pair programming I learned in kindergarten
Williams, L. A., & Kessler, R. R. (2000) · Communications of the ACM · doi:10.1145/332833.332848 · Cited by 293
Toward automatic program synthesis
Manna, Z., & Waldinger, R. J. (1971) · Communications of the ACM · doi:10.1145/362566.362568 · Cited by 285 · cited 1×
“The concept of computer programs being generated by computer programs goes back several decades [23], but has only gained significant ground very recently.”
BackgroundNovice programmer errors: language constructs and plan composition
Ebrahimi, A. (1994) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1994.1069 · Cited by 174
Research works printed but not linked: 2
- Paul Phillips and Luc Cohen . 2014 . Convictions of Plagiarism in Computer Science Courses on the Rise. The Daily Princetonian , March 4 (2014), 2014 . Paul Phillips and Luc Cohen. 2014. Convictions of Plagiarism in Computer Science Courses on the Rise. The Daily Princetonian, March 4 (2014), 2014.
- Judy Sheard , Angela Carbone , and Martin Dick . 2003 . Determination of Factors Which Impact on IT Students’ Propensity to Cheat. In Proceedings of the Fifth Australasian Conference on Computing Education - Volume 20(ACE ’03). Australian Computer Society, Inc., AUS, 119– 126 . Judy Sheard, Angela Carbone, and Martin Dick. 2003. Determination of Factors Which Impact on IT Students’ Propensity to Cheat. In Proceedings of the Fifth Australasian Conference on Computing Education - Volume 20(ACE ’03). Australian Computer Society, Inc., AUS, 119–126.
Cited by
Bahroun, Z., Anane, C., Ahmed, V., & Zacca, A. (2023) · Sustainability · doi:10.3390/su151712983 · Cited by 784
The promise and challenges of generative AI in education
Giannakos, M., Azevedo, R., Brusilovsky, P., Cukurova, M., Dimitriadis, Y., Hernandez-Leo, D., Järvelä, S., Mavrikis, M., & Rienties, B. (2025) · Behaviour & Information Technology · doi:10.1080/0144929x.2024.2394886 · Cited by 543
“Computer science education (CSEd) could be considered as a special domain for the educational application of LLMs.”
InfluentialAutomatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497
“Our work suggests that modern machine learning models such as OpenAI Codex provide many opportunities for programming course designers, although potential challenges outlined in prior work [27] should not be ignored.”
BackgroundMethodInfluentialSelf-citationGitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417
Yu, H. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1181712 · Cited by 409
“This innovative natural language processing technology aims to achieve language algorithm models that are more akin to human expression of thought (Finnie-Ansley et al., 2022).”
Background
Metadata
- Authors
- Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J.
- Affiliations
- University of Auckland (NZ)
- University College Dublin (IE)
- Abilene Christian University (US)
- Venue
- Proceedings of the 24th Australasian Computing Education Conference
- Publisher
- ACM
- Published
- 2022-02-14
- Pages
- 10-19
- OpenAlex
- W4211263275
- Semantic Scholar
- 979eb5c97c49d7979447ed684500895a24d75ac4
- Cited by
- 463
- References
- 29
- Citations by year
- 2026: 89 · 2025: 110 · 2024: 161 · 2023: 96 · 2022: 6
- FWCI
- 121.44
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Australasian Computing Education Conference (publishedVersion, cc-by-nd)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- OpenAI Codex, GPT-3, large language models, introductory programming, programming education, automatic code generation, student performance comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. https://app.cochat.ai/papers/10.1145/3511861.3511863
- The paper
- Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022). The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. Proceedings of the 24th Australasian Computing Education Conference. https://doi.org/10.1145/3511861.3511863
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.