The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Can OpenAI's codex fix bugs?: an evaluation on QuixBugs
Abstract
OpenAI's Codex, a GPT-3 like model trained on a large code corpus, has made headlines in and outside of academia. Given a short user-provided description, it is capable of synthesizing code snippets that are syntactically and semantically valid in most cases. In this work, we want to investigate whether Codex is able to localize and fix bugs, two important tasks in automated program repair. Our initial evaluation uses the multi-language QuixBugs benchmark (40 bugs in both Python and Java). We find that, despite not being trained for APR, Codex is surprisingly effective, and competitive with recent state of the art techniques. Our results also show that Codex is more successful at repairing Python than Java, fixing 50% more bugs in Python.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We also show that a carefully-designed demonstration based on our findings can lead to substantial improvements over widely-used demonstration construction methods”
From What Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs? (2023) · cites this paper as [19]
In the citing paper
“Existing studies [17], [19] mainly construct demonstrations by randomly selecting and arranging the demonstration examples.”
From What Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs? (2023) · cites this paper as [19]
In the citing paper
“However, these works mainly focus on the evaluation of LLMs on one or two tasks and do not discuss the construction of in-context demonstrations in-depth.”
From What Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs? (2023) · cites this paper as [17]
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“[12] observed that Transformer-based language models are subject to several scaling laws, including that the performance of a language model has a power law relationship with model size, dataset size, and amount of computing power invested in training the language model, as long as none of these factors is a bottlenec…”
BackgroundDefects4J: a database of existing faults to enable controlled testing studies for Java programs
Just, R., Jalali, D., & Ernst, M. D. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628055 · Cited by 1,447
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
GenProg: A Generic Method for Automatic Software Repair
Goues, C. L., Nguyen, T. H., Forrest, S., & Weimer, W. R. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.104 · Cited by 1,084
Maintaining mental models: a study of developer work habits
LaToza, T. D., Venolia, G., & DeLine, R. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134355 · Cited by 628
Mechtaev, S., Yi, J., & Roychoudhury, A. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2884781.2884807 · Cited by 515
CoCoNuT: combining context-aware neural translation models using ensemble for program repair
Lutellier, T., Pham, H., Pang, L., Li, Y., Wei, M., & Tan, L. (2020) · International Symposium on Software Testing and Analysis · doi:10.1145/3395363.3397369 · Cited by 463
SEQUENCER: Sequence-to-Sequence Learning for End-to-End Program Repair
Chen, Z., Kommrusch, S., Tufano, M., Pouchet, L.-N., Poshyvanyk, D., & Monperrus, M. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2940179 · Cited by 448
Flagged first, then by citations
Research works printed but not linked: 4
- Dongsun Kim , Jaechang Nam , Jaewoo Song , and Sunghun Kim . Automatic patch generation learned from human-written patches . In Proceedings of the 2013 International Conference on Software Engineering, ICSE '13 , pages 802 -- 811 . IEEE Press. ISBN 978-1-4673-3076-3. Dongsun Kim, Jaechang Nam, Jaewoo Song, and Sunghun Kim. Automatic patch generation learned from human-written patches. In Proceedings of the 2013 International Conference on Software Engineering, ICSE '13, pages 802--811. IEEE Press. ISBN 978-1-4673-3076-3.
- Hoang Duong Thien Nguyen , Dawei Qi , Abhik Roychoudhury , and Satish Chandra . SemFix : Program repair via semantic analysis . In Proceedings of the 2013 International Conference on Software Engineering, ICSE '13 , pages 772 -- 781 . IEEE Press. ISBN 978-1-4673-3076-3. Hoang Duong Thien Nguyen, Dawei Qi, Abhik Roychoudhury, and Satish Chandra. SemFix: Program repair via semantic analysis. In Proceedings of the 2013 International Conference on Software Engineering, ICSE '13, pages 772--781. IEEE Press. ISBN 978-1-4673-3076-3.
- Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , Ilya Sutskever , Language models are unsupervised multitask learners . OpenAI blog , 1 ( 8 ): 9 , 2019 . Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser and Illia Polosukhin. Attention Is All You Need. URL http://arxiv.org/abs/1706.03762. Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser and Illia Polosukhin. Attention Is All You Need. URL http://arxiv.org/abs/1706.03762. · arXiv
Cited by
Sorted by citations
Items 101–104 of 104
Messarhi, O., Kerdoudi, M. L., & Tibermacine, O. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.113101
Automated Program Repair for UI-Centric Android Bugs: How Far Are We?
Mahmud, J., Pandey, S., Silva, N. D., Dipongkor, A. K., Wu, J., Chaparro, O., Fazzini, M., & Moran, K. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832272
KnowJIT: Knowledge-Augmented Just-in-Time Defect Prediction and Repair via In-Context Learning
Wu, D., Li, W., Lv, G., Zhao, X., Yang, Y., & Dong, Z. (2026) · IEEE Transactions on Reliability · doi:10.1109/tr.2026.3744216
Evaluating Large Language Models for Web Accessibility Remediation in Real-World Websites
Aljedaani, W., Gautier, K., Parthasarathy, P. D., & Clemmer, C. (2026) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3797867.3841243
Metadata
- Authors
- Prenner, J. A., Babii, H., & Robbes, R.
- Affiliations
- Free University of Bozen-Bolzano (IT)
- Venue
- Proceedings of the Third International Workshop on Automated Program Repair
- Publisher
- ACM
- Published
- 2022-05-19
- Pages
- 69-75
- OpenAlex
- W4312970618
- Cited by
- 99
- References
- 24
- Citations by year
- 2026: 14 · 2025: 30 · 2024: 38 · 2023: 17
- FWCI
- 16.03
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/ACM International Workshop on Automated Program Repair (APR) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- OpenAI Codex, automated program repair, bug localization, bug fixes, large language models, GPT-3, Python, Java, code generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
