The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
Recent advances in artificial intelligence have been driven by an exponential growth in digitised data. Natural language processing, in particular, has been transformed by machine learning models such as OpenAI’s GPT-3 which generates human-like text so realistic that its developers have warned of the dangers of its misuse. In recent months OpenAI released Codex, a new deep learning model trained on Python code from more than 50 million GitHub repositories. Provided with a natural language description of a programming problem as input, Codex generates solution code as output. It can also explain (in English) input code, translate code between programming languages, and more. In this work, we explore how Codex performs on typical introductory programming problems. We report its performance on real questions taken from introductory programming exams and compare it to results from students who took these same exams under normal conditions, demonstrating that Codex outscores most students. We then explore how Codex handles subtle variations in problem wording using several published variants of the well-known “Rainfall Problem” along with one unpublished variant we have used in our teaching. We find the model passes many test cases for all variants. We also explore how much variation there is in the Codex generated solutions, observing that an identical input prompt frequently leads to very different solutions in terms of algorithmic approach and code length. Finally, we discuss the implications that such technology will have for computing education as it continues to evolve, including both challenges and opportunities.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“They each focus on the capabilities of the tool they are using (ChatGPT/Codex), rather than the broader process in which marks are awarded to scripts.”
From Bob or Bot: Exploring ChatGPT's Answers to University Computer Science Assessment (2024) · cites this paper as [19]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, this approach may have become obsolete given how easily most LLMs can now solve introductory computing problems with simple prompts [85,86,167,175].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [85]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Finnie-Ansley et al. [85] published the first paper that examined the performance of large language models in solving introductory programming exercises.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [85]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the consistent pattern is that ChatGPT and other LLMs are capable of producing content sufficient to pass undergraduate level assessment in diverse curriculum areas.”
From Bob or Bot: Exploring ChatGPT's Answers to University Computer Science Assessment (2024) · cites this paper as [19]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“recent studies in the context of learning programming have found the correctness of Codex (the LLM that powers Copilot) to be around 70%-80% [34, 75], meaning that in about 20-30% of cases, the suggestion by Copilot would be incorrect.”
From “It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers (2024) · cites this paper as [34]
Experimental — accuracy not yet measured
18 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 16×
“Anticipating this shift, we put the following sentence into GPT-3, “The robots are taking over.””
BackgroundMethodInfluentialGPT-3: Its Nature, Scope, Limits, and Consequences
Floridi, L., & Chiriatti, M. (2020) · Minds and Machines · doi:10.1007/s11023-020-09548-1 · Cited by 2,519 · cited 3×
“It is considered a competitor to Alphabet’s DeepMind [11].”
BackgroundSignature pedagogies in the professions
Shulman, L. S. (2005) · Daedalus · doi:10.1162/0011526054622015 · Cited by 2,032 · cited 1×
“Such activities are so common they may form a signature pedagogy [36] of computing education.”
BackgroundEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 6×
“if the target is 21 and the parameter list is [15, 10, 5, 20], the function”
BackgroundMethodInfluentialLearning to program = learning to construct mechanisms and explanations
Soloway, E. (1986) · Communications of the ACM · doi:10.1145/6592.6594 · Cited by 567 · cited 6×
“This is a unique variant of Soloway’s Rainfall Problem [39] which has been used in a number of studies of programming ability over the past 30+ years and which has developed a reputation for being surprisingly difficult for introductory-level (CS1 [3]) students [32] with multiple studies confirming poor performance [1…”
BackgroundMethodInfluentialIntroductory programming: a systematic literature review
Luxton-Reilly, A., Simon, Albluwi, I., Becker, B. A., Giannakos, M., Kumar, A. N., Ott, L., Paterson, J., Scott, M. J., Sheard, J., et al. (2018) · Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3293881.3295779 · Cited by 485 · cited 1×
“A comprehensive review of introductory programming literature from 2018 makes no mention of tools that use AI to produce code, and only mentions a few machine learning based systems to analyse code or provide feedback along with intelligent tutoring systems that aim to provide customised learning approaches (normally,…”
BackgroundSelf-citationContract cheating by STEM students through a file sharing website: a Covid-19 pandemic perspective
Lancaster, T., & Cotarlan, C. (2021) · International Journal for Educational Integrity · doi:10.1007/s40979-021-00070-0 · Cited by 312
All I really need to know about pair programming I learned in kindergarten
Williams, L. A., & Kessler, R. R. (2000) · Communications of the ACM · doi:10.1145/332833.332848 · Cited by 293
Toward automatic program synthesis
Manna, Z., & Waldinger, R. J. (1971) · Communications of the ACM · doi:10.1145/362566.362568 · Cited by 285 · cited 1×
“The concept of computer programs being generated by computer programs goes back several decades [23], but has only gained significant ground very recently.”
BackgroundNovice programmer errors: language constructs and plan composition
Ebrahimi, A. (1994) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1994.1069 · Cited by 174
Exploring hypotheses about media computation
Guzdial, M. (2013) · Proceedings of the ninth annual international ACM conference on International computing education research · doi:10.1145/2493394.2493397 · Cited by 172 · cited 3×
“if the target is 21 and the parameter list is [15, 10, 5, 20], the function”
BackgroundIntelligent tutoring systems for programming education: a systematic review
Crow, T., Luxton-Reilly, A., & Wuensche, B. (2018) · Proceedings of the 20th Australasian Computing Education Conference · doi:10.1145/3160489.3160492 · Cited by 166 · cited 1×
“, using web-based examples), the customised solutions offered by Codex may lead students to perceive the solutions as being more credible, similar to an intelligent tutoring system [6].”
MethodSelf-citationUnderstanding the Capabilities, Limitations, and Societal Impact of Large Language Models
Tamkin, A., Brundage, M., Clark, J., & Ganguli, D. (2021) · arXiv · doi:10.48550/arxiv.2102.02503 · Cited by 135 · cited 17×
“Anticipating this shift, we put the following sentence into GPT-3, “The robots are taking over.””
BackgroundMethodInfluentialA systematic review of tools that support peer assessment
Luxton-Reilly, A. (2009) · Computer Science Education · doi:10.1080/08993400903384844 · Cited by 126 · cited 1×
“The extensive literature on peer review, including the review of code [17, 21] makes it clear that there are many benefits that arise from looking at a variety of solutions to a given problem.”
BackgroundSelf-citationGonzalez, D., Zimmermann, T., & Nagappan, N. (2020) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3379597.3387473 · Cited by 123 · cited 1×
“However, it wasn’t until the AI/ML “boom” in 2017 [13] that natural language models and other applications of AI began to make strong progress.”
BackgroundThe recurring rainfall problem
Fisler, K. (2014) · Proceedings of the tenth annual conference on International computing education research · doi:10.1145/2632320.2632346 · Cited by 112 · cited 15×
“For the Fisler [10], Simon [37], Guzdial et al. [16], Lakanen et al. [18], and apples problems, a function header was also provided as the problem descriptions prompt the solver to write a function or imply lists are provided as arguments rather than standard input.”
BackgroundMethodInfluential50 Years of CS1 at SIGCSE: A Review of the Evolution of Introductory Programming Education Research
Becker, B. A., & Quille, K. (2019) · Proceedings of the 50th ACM Technical Symposium on Computer Science Education · doi:10.1145/3287324.3287432 · Cited by 107 · cited 1×
“This is a unique variant of Soloway’s Rainfall Problem [39] which has been used in a number of studies of programming ability over the past 30+ years and which has developed a reputation for being surprisingly difficult for introductory-level (CS1 [3]) students [32] with multiple studies confirming poor performance [1…”
BackgroundSelf-citationStrategies for Maintaining Academic Integrity in First-Year Computing Courses
Sheard, J., Simon, Butler, M., Falkner, K., Morgan, M., & Weerasinghe, A. (2017) · Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3059009.3059064 · Cited by 98 · cited 3×
“Some studies have reported almost 80% of computing students to be involved [35], and there is a higher occurrence of plagiarism in computer science than in many other disciplines [20, 29, 30].”
BackgroundMethodDo we know how difficult the rainfall problem is?
Seppälä, O., Ihantola, P., Isohanni, E., Sorva, J., & Vihavainen, A. (2015) · Proceedings of the 15th Koli Calling Conference on Computing Education Research · doi:10.1145/2828959.2828963 · Cited by 93 · cited 1×
“This is a unique variant of Soloway’s Rainfall Problem [39] which has been used in a number of studies of programming ability over the past 30+ years and which has developed a reputation for being surprisingly difficult for introductory-level (CS1 [3]) students [32] with multiple studies confirming poor performance [1…”
BackgroundThe effects of pair-programming on performance in an introductory programming course
McDowell, C., Werner, L., Bullock, H., & Fernald, J. (2002) · ACM SIGCSE Bulletin · doi:10.1145/563517.563353 · Cited by 88
A Review of Peer Code Review in Higher Education
Indriasari, T. D., Luxton-Reilly, A., & Denny, P. (2020) · ACM Transactions on Computing Education · doi:10.1145/3403935 · Cited by 80 · cited 1×
“The extensive literature on peer review, including the review of code [17, 21] makes it clear that there are many benefits that arise from looking at a variety of solutions to a given problem.”
BackgroundSelf-citationPlagiarism in Programming Assessments: A Systematic Review
Albluwi, I. (2020) · ACM Transactions on Computing Education · doi:10.1145/3371156 · Cited by 63
Soloway's Rainfall Problem Has Become Harder
Simon (2013) · 2013 Learning and Teaching in Computing and Engineering · doi:10.1109/latice.2013.44 · Cited by 61
Personalized Assessment as a Means to Mitigate Plagiarism
Manoharan, S. (2017) · IEEE Transactions on Education · doi:10.1109/te.2016.2604210 · Cited by 49
Strategies for promoting academic integrity in CS courses
Roberts, E. (2003) · 32nd Annual Frontiers in Education · doi:10.1109/fie.2002.1158209 · Cited by 42 · cited 2×
“Academics have long been concerned about violations of academic integrity in computing disciplines [8, 30].”
BackgroundGuzdial, M. (2011) · Communications of the ACM · doi:10.1145/1897816.1897831 · Cited by 42 · cited 1×
“This is a unique variant of Soloway’s Rainfall Problem [39] which has been used in a number of studies of programming ability over the past 30+ years and which has developed a reputation for being surprisingly difficult for introductory-level (CS1 [3]) students [32] with multiple studies confirming poor performance [1…”
BackgroundDonaldson, J. L., Lancaster, A.-M., & Sposato, P. H. (1981) · ACM SIGCSE Bulletin · doi:10.1145/953049.800955 · Cited by 38
An Analysis of Using Many Small Programs in CS1
Allen, J. M., Vahid, F., Edgcomb, A., Downey, K., & Miller, K. (2019) · Proceedings of the 50th ACM Technical Symposium on Computer Science Education · doi:10.1145/3287324.3287466 · Cited by 36 · cited 1×
“The use of many small exercises is an approach that CS1 students can prefer and have better success with [2].”
BackgroundDesigning Programming Assignments to Reduce the Likelihood of Cheating
Simon (2017) · doi:10.1145/3013499.3013507 · Cited by 19 · cited 1×
“This is contrasted with calls for the use of more complexity in assignments to reduce plagiarism [38].”
ResultContract Cheating in Computer Science: A Case Study
Manoharan, S., & Speidel, U. (2020) · IEEE International Conference on Teaching, Assessment, and Learning for Engineering · doi:10.1109/tale48869.2020.9368454 · Cited by 19 · cited 1×
“Although this approach can prevent direct copying, students may still engage in “contract cheating” where work is outsourced to an individual or organisation [19, 25].”
BackgroundRevisiting rainfall to explore exam questions and performance on CS1
Lakanen, A.-J., Lappalainen, V., & Isomöttönen, V. (2015) · doi:10.1145/2828959.2828970 · Cited by 17 · cited 4×
“[18], and apples problems, a function header was also provided as the problem descriptions prompt the solver to write a function or imply lists are provided as arguments rather than standard input.”
BackgroundInfluentialDick, M., Sheard, J., Bareiss, C., Carter, J., Joyce, D., Harding, T. S., & Laxer, C. (2002) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/960568.783000 · Cited by 15
Research works printed but not linked: 2
- Paul Phillips and Luc Cohen . 2014 . Convictions of Plagiarism in Computer Science Courses on the Rise. The Daily Princetonian , March 4 (2014), 2014 . Paul Phillips and Luc Cohen. 2014. Convictions of Plagiarism in Computer Science Courses on the Rise. The Daily Princetonian, March 4 (2014), 2014.
- Judy Sheard , Angela Carbone , and Martin Dick . 2003 . Determination of Factors Which Impact on IT Students’ Propensity to Cheat. In Proceedings of the Fifth Australasian Conference on Computing Education - Volume 20(ACE ’03). Australian Computer Society, Inc., AUS, 119– 126 . Judy Sheard, Angela Carbone, and Martin Dick. 2003. Determination of Factors Which Impact on IT Students’ Propensity to Cheat. In Proceedings of the Fifth Australasian Conference on Computing Education - Volume 20(ACE ’03). Australian Computer Society, Inc., AUS, 119–126.
Cited by
Bahroun, Z., Anane, C., Ahmed, V., & Zacca, A. (2023) · Sustainability · doi:10.3390/su151712983 · Cited by 784
The promise and challenges of generative AI in education
Giannakos, M., Azevedo, R., Brusilovsky, P., Cukurova, M., Dimitriadis, Y., Hernandez-Leo, D., Järvelä, S., Mavrikis, M., & Rienties, B. (2025) · Behaviour & Information Technology · doi:10.1080/0144929x.2024.2394886 · Cited by 543
“Computer science education (CSEd) could be considered as a special domain for the educational application of LLMs.”
InfluentialAutomatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497
“Our work suggests that modern machine learning models such as OpenAI Codex provide many opportunities for programming course designers, although potential challenges outlined in prior work [27] should not be ignored.”
BackgroundMethodInfluentialSelf-citationGitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417
GitHub Copilot AI pair programmer: Asset or Liability?
Dakhel, A. M., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., Ming, Z., & Jiang (2022) · arXiv · doi:10.48550/arxiv.2206.15331 · Cited by 417
“Finnie-Ansley et al. [21] used Davinci (an API on a beta version of Codex) on two datasets.”
BackgroundMethod
Metadata
- Authors
- Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J.
- Affiliations
- University of Auckland (NZ)
- University College Dublin (IE)
- Abilene Christian University (US)
- Venue
- Proceedings of the 24th Australasian Computing Education Conference
- Publisher
- ACM
- Published
- 2022-02-14
- Pages
- 10-19
- OpenAlex
- W4211263275
- Semantic Scholar
- 979eb5c97c49d7979447ed684500895a24d75ac4
- Cited by
- 463
- References
- 29
- Citations by year
- 2026: 89 · 2025: 110 · 2024: 161 · 2023: 96 · 2022: 6
- FWCI
- 121.44
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Australasian Computing Education Conference (publishedVersion, cc-by-nd)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- OpenAI Codex, GPT-3, large language models, introductory programming, programming education, automatic code generation, student performance comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.