Automatic Program Repair with OpenAI's Codex: Evaluating QuixBugs
Open accessAbstract
OpenAI's Codex, a GPT-3 like model trained on a large code corpus, has made headlines in and outside of academia. Given a short user-provided description, it is capable of synthesizing code snippets that are syntactically and semantically valid in most cases. In this work, we want to investigate whether Codex is able to localize and fix bugs, a task of central interest in the field of automated program repair. Our initial evaluation uses the multi-language QuixBugs benchmark (40 bugs in both Python and Java). We find that, despite not being trained for APR, Codex is surprisingly effective, and competitive with recent state of the art techniques. Our results also show that Codex is slightly more successful at repairing Python than Java.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Prenner and Robbes (2021) explores Codex for automatic program repair; Codex outperforms numerous existing APR methods even without retraining.
“Prenner and Robbes (2021) explores the application of OpenAI's Codex to the field of automatic program repair (APR), specifically its ability to locate and fix bugs in software. They use the QuixBugs benchmark, which includes 40 bugs in Python and Java, to evaluate the effectiveness of Codex in APR tasks. Notably, Codex outperforms numerous existing APR methods even without retraining.”
From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Prenner and Robbes (2021)
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 21 · Cited by 25
Research works printed but not linked: 17
- [2][2] T. D. LaToza, G. Venolia, and R. DeLine, 'Maintaining mental models: A study of developer work habits,' in Proceeding of the 28th International Conference on Software Engineering -ICSE '06 , Shanghai, China: ACM Press, 2006, p. 492. -
- [3][3] C. L. Goues, T. Nguyen, S. Forrest, et al. , 'GenProg: A Generic Method for Automatic Software Repair,' IEEE Transactions on Software Engineering , vol. 38, no. 1, pp. 54-72, Jan. 2012. -
- [4][4] Y. Yuan and W. Banzhaf, 'ARJA: Automated Repair of Java Programs via Multi-Objective Genetic Programming,' IEEE Transactions on Software Engineering , vol. 46, no. 10, pp. 1040-1067, Oct. 2020. -
- [5][5] D. Kim, J. Nam, J. Song, et al. , 'Automatic patch generation learned from human-written patches,' in Proceedings of the 2013 International Conference on Software Engineering , ser. ICSE '13, San Francisco, CA, USA: IEEE Press, May 18, 2013, pp. 802-811. -
- [6][6] R. K. Saha, Y. Lyu, H. Yoshida, et al. , 'Elixir: Effective object-oriented program repair,' in 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE) , Oct. 2017, pp. 648-659. -
- [7][7] K. Liu, A. Koyuncu, D. Kim, et al. , 'TBar: Revisiting template-based automated program repair,' in Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis , ser. ISSTA 2019, New York, NY, USA: Association for Computing Machinery, Jul. 10, 2019, pp. 31-42. -
- [8][8] Q. Xin and S. P. Reiss, 'Leveraging syntax-related code for automated program repair,' in Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering , ser. ASE 2017, UrbanaChampaign, IL, USA: IEEE Press, Oct. 30, 2017, pp. 660-670. -
- [9][9] K. Liu, A. Koyuncu, K. Kim, et al. , 'LSRepair: Live Search of Fix Ingredients for Automated Program Repair,' in 2018 25th Asia-Pacific Software Engineering Conference (APSEC) , Dec. 2018, pp. 658-662. -
- [10][10] Z. Chen, S. Kommrusch, M. Tufano, et al. (Sep. 9, 2019). 'SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair,' [Online]. Available: http://arxiv.org/abs/1901.01808 (visited on 02/15/2021). - · arXiv
- [11][11] E. Dinella, H. Dai, Z. Li, et al. , 'HOPPITY: LEARNING GRAPH TRANSFORMATIONS TO DETECT AND FIX BUGS IN PROGRAMS,' presented at the Eighth International Conference on Learning Representations, Apr. 2020. -
- [12][12] T. Lutellier, H. V. Pham, L. Pang, et al. , 'CoCoNuT: Combining context-aware neural translation models using ensemble for program repair,' in Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis , ser. ISSTA 2020, New York, NY, USA: Association for Computing Machinery, Jul. 18, 2020, pp. 101-114. -
- [13][13] N. Jiang, T. Lutellier, and L. Tan. (Feb. 26, 2021). 'CURE: Code-Aware Neural Machine Translation for Automatic Program Repair,' [Online]. Available: http://arxiv.org/abs/2103.00073 (visited on 03/22/2021). - · arXiv
- [15][15] T. B. Brown, B. Mann, N. Ryder, et al. (Jul. 22, 2020). 'Language Models are Few-Shot Learners,' [Online]. Available: http://arxiv.org/abs/2005.14165 (visited on 03/17/2021). - · arXiv
- [18][18] D. Lin, J. Koppel, A. Chen, et al. , 'QuixBugs: A multi-lingual program repair benchmark set based on the quixey challenge,' in Proceedings Companion of the 2017 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity , ser. SPLASH Companion 2017, New York, NY, USA: Association for Computing Machinery, Oct. 22, 2017, pp. 55-56. -
- [19][19] D. Drain, C. B. Clement, G. Serrato, et al. (May 19, 2021). 'DeepDebug: Fixing Python Bugs Using Stack Traces, Backtranslation, and Code Skeletons,' [Online]. Available: http://arxiv.org/abs/2105.09352 (visited on 09/17/2021). - · arXiv
- [20][20] A. Vaswani, N. Shazeer, N. Parmar, et al. (Dec. 5, 2017). 'Attention Is All You Need,' [Online]. Available: http://arxiv.org/abs/1706.03762 (visited on 03/24/2021). - · arXiv
- [22][22] P. Gyimesi, B. Vancsics, A. Stocco, et al. , 'BugsJS: A Benchmark of JavaScript Bugs,' in 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST) , Apr. 2019, pp. 90-101. -
- Authors
- Prenner, J. A., & Robbes, R.
- Affiliations
- Free University of Bozen-Bolzano (IT)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2111.03922
- OpenAlex
- W3212260201
- Cited by
- 25
- References
- 21
- Citations by year
- 2025: 2 · 2024: 10 · 2023: 12 · 2022: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
- Keywords
- automated program repair, codices, GPT-3, bug localization, bug fixes, large language models, Python, Java
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
