Cited by 104Software Testing and Debugging TechniquesSoftware Engineering ResearchTeaching and Learning Programming

The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.

Can OpenAI's codex fix bugs?: an evaluation on QuixBugs

Prenner, Julian Aron, Babii, Hlib, Robbes, Romain·2022·Proceedings of the Third International Workshop on Automated Program Repair

DOI 10.1145/3524459.3527351

Open this paper in CoChat — free account

Abstract

OpenAI's Codex, a GPT-3 like model trained on a large code corpus, has made headlines in and outside of academia. Given a short user-provided description, it is capable of synthesizing code snippets that are syntactically and semantically valid in most cases. In this work, we want to investigate whether Codex is able to localize and fix bugs, two important tasks in automated program repair. Our initial evaluation uses the multi-language QuixBugs benchmark (40 bugs in both Python and Java). We find that, despite not being trained for APR, Codex is surprisingly effective, and competitive with recent state of the art techniques. Our results also show that Codex is more successful at repairing Python than Java, fixing 50% more bugs in Python.

Integrity signals

Registry

Crossref · 10.1145/3524459.3527351

Cites

25 of 25 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“[12] observed that Transformer-based language models are subject to several scaling laws, including that the performance of a language model has a power law relationship with model size, dataset size, and amount of computing power invested in training the language model, as long as none of these factors is a bottlenec…”
Background
Defects4J: a database of existing faults to enable controlled testing studies for Java programs
Just, R., Jalali, D., & Ernst, M. D. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628055 · Cited by 1,447
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
GenProg: A Generic Method for Automatic Software Repair
Goues, C. L., Nguyen, T. H., Forrest, S., & Weimer, W. R. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.104 · Cited by 1,084
Maintaining mental models: a study of developer work habits
LaToza, T. D., Venolia, G., & DeLine, R. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134355 · Cited by 628
Angelix
Mechtaev, S., Yi, J., & Roychoudhury, A. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2884781.2884807 · Cited by 515
CoCoNuT: combining context-aware neural translation models using ensemble for program repair
Lutellier, T., Pham, H., Pang, L., Li, Y., Wei, M., & Tan, L. (2020) · International Symposium on Software Testing and Analysis · doi:10.1145/3395363.3397369 · Cited by 463
SEQUENCER: Sequence-to-Sequence Learning for End-to-End Program Repair
Chen, Z., Kommrusch, S., Tufano, M., Pouchet, L.-N., Poshyvanyk, D., & Monperrus, M. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2940179 · Cited by 448

Show all 24 references

Research works printed but not linked: 4

Dongsun Kim , Jaechang Nam , Jaewoo Song , and Sunghun Kim . Automatic patch generation learned from human-written patches . In Proceedings of the 2013 International Conference on Software Engineering, ICSE '13 , pages 802 -- 811 . IEEE Press. ISBN 978-1-4673-3076-3. Dongsun Kim, Jaechang Nam, Jaewoo Song, and Sunghun Kim. Automatic patch generation learned from human-written patches. In Proceedings of the 2013 International Conference on Software Engineering, ICSE '13, pages 802--811. IEEE Press. ISBN 978-1-4673-3076-3.
Hoang Duong Thien Nguyen , Dawei Qi , Abhik Roychoudhury , and Satish Chandra . SemFix : Program repair via semantic analysis . In Proceedings of the 2013 International Conference on Software Engineering, ICSE '13 , pages 772 -- 781 . IEEE Press. ISBN 978-1-4673-3076-3. Hoang Duong Thien Nguyen, Dawei Qi, Abhik Roychoudhury, and Satish Chandra. SemFix: Program repair via semantic analysis. In Proceedings of the 2013 International Conference on Software Engineering, ICSE '13, pages 772--781. IEEE Press. ISBN 978-1-4673-3076-3.
Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , Ilya Sutskever , Language models are unsupervised multitask learners . OpenAI blog , 1 ( 8 ): 9 , 2019 . Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser and Illia Polosukhin. Attention Is All You Need. URL http://arxiv.org/abs/1706.03762. Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser and Illia Polosukhin. Attention Is All You Need. URL http://arxiv.org/abs/1706.03762. · arXiv

Sorted by citations

Software Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573
Code Llama: Open Foundation Models for Code
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12950 · Cited by 531
“…refinement, translation (Rozière et al., 2020; 2021; Szafraniec et al., 2023) fixing bugs (Yasunaga & Liang, 2021; Zhang et al., 2022a; Prenner et al., 2022), fixing build errors (Tarlow et al., 2020) or generating unit tests (Tufano et al., 2020; Li et al., 2022; Chen et al., 2023a),…”
Background
Automated Repair of Programs from Large Language Models
Fan, Z., Gao, X., Mirchev, M., Roychoudhury, A., & Tan, S. H. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00128 · Cited by 256
Automated Program Repair via Conversation: Fixing 162 out of 337 Bugs for $0.42 Each using ChatGPT
Xia, C. S., & Zhang, L. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680323 · Cited by 468
AutoCodeRover: Autonomous Program Improvement
Zhang, Y., Ruan, H., Fan, Z., & Roychoudhury, A. (2024) · Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3650212.3680384 · Cited by 202
AutoCodeRover: Autonomous Program Improvement
Zhang, Y., Ruan, H., Fan, Z., & Roychoudhury, A. (2024) · arXiv · doi:10.48550/arxiv.2404.05427 · Cited by 202
Impact of Code Language Models on Automated Program Repair
Jiang, N., Liu, K., Lutellier, T., & Tan, L. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00125 · Cited by 202
Retrieval-Based Prompt Selection for Code-Related Few-Shot Learning
Nashid, N., Sintaha, M., & Mesbah, A. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00205 · Cited by 178
Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning
Geng, M., Wang, S., Dong, D., Wang, H., Li, G., Jin, Z. G., Mao, X., & Liao, X. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3608134 · Cited by 199
Is GitHub’s Copilot as bad as humans at introducing vulnerabilities in code?
Asare, O., Nagappan, M., & Asokan, N. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10380-1 · Cited by 128
Towards an understanding of large language models in software engineering tasks
Zheng, Z., Ning, K., Zhong, Q., Chen, J., Chen, W., Guo, L., Wang, W., & Wang, Y. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10602-0 · Cited by 212
Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repair
Wei, Y., Xia, C. S., & Zhang, L. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3616271 · Cited by 115
What Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs?
Gao, S., Wen, X.-C., Gao, C., Wang, W., Zhang, H., & Lyu, M. R. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00109 · Cited by 106
Refactoring Programs Using Large Language Models with Few-Shot Examples
Shirafuji, A., Oda, Y., Suzuki, J., Morishita, M., & Watanobe, Y. (2023) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec60848.2023.00025 · Cited by 97
Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models
Weyssow, M., Zhou, X., Kim, K., Lo, D., & Sahraoui, H. A. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3714461 · Cited by 158
ThinkRepair: Self-Directed Automated Program Repair
Yin, X., Ni, C., Wang, S., Li, Z., Zeng, L., & Yang, X. (2024) · Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3650212.3680359 · Cited by 68
Domain Knowledge Matters: Improving Prompts with Fix Templates for Repairing Python Type Errors
Peng, Y., Gao, S., Gao, C., Huo, Y., & Lyu, M. R.-T. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3608132 · Cited by 66
Does ChatGPT Generate Accessible Code? Investigating Accessibility Challenges in LLM-Generated Source Code
Aljedaani, W., Habib, A., Aljohani, A., Eler, M., & Feng, Y. (2024) · Proceedings of the 21st International Web for All Conference · doi:10.1145/3677846.3677854 · Cited by 42
LLM-Powered Static Binary Taint Analysis
Liu, P., Sun, C. P., Zheng, Y., Feng, X., Qin, C., Wang, Y., Xu, Z., Li, Z., Di, P., Jiang, Y., et al. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3711816 · Cited by 96
CREF: An LLM-Based Conversational Software Repair Framework for Programming Tutors
Yang, B., Tian, H., Pian, W., Yu, H., Wang, H., Klein, J., Bissyandé, T. F., & Jin, S. (2024) · Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3650212.3680328 · Cited by 41
The Plastic Surgery Hypothesis in the Era of Large Language Models
Xia, C. S., Ding, Y., & Zhang, L. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00047 · Cited by 67
Systematic Literature Review of Prompt Engineering Patterns in Software Engineering
Sasaki, Y., Washizaki, H., Li, J., Sander, D., Yoshioka, N., & Fukazawa, Y. (2024) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00096 · Cited by 27
LUNA: A Model-Based Universal Analysis Framework for Large Language Models
Song, D., Xie, X., Song, J., Zhu, D., Huang, Y., Juefei-Xu, F., & Ma, L. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3411928 · Cited by 33
FlakyFix: Using Large Language Models for Predicting Flaky Test Fix Categories and Test Code Repair
Fatima, S., Hemmati, H., & Briand, L. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3472476 · Cited by 35
Hybrid Automated Program Repair by Combining Large Language Models and Program Analysis
Li, F., Jiang, J., Sun, J., & Zhang, H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715004 · Cited by 53
GPT-3-Powered Type Error Debugging: Investigating the Use of Large Language Models for Code Repair
Ribeiro, F., Macedo, J. N., Tsushima, K., Abreu, R., & Saraiva, J. (2023) · ACM SIGPLAN International Conference on Software Language Engineering (SLE) · doi:10.1145/3623476.3623522 · Cited by 27
Anticipating User Needs: Insights from Design Fiction on Conversational Agents for Computational Thinking
Penney, J., Pimentel, J. F., Steinmacher, I., & Gerosa, M. A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-54975-5_12 · Cited by 23
Out of Context: How important is Local Context in Neural Program Repair?
Prenner, J. A., & Robbes, R. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639086 · Cited by 26
Self-citation
Agentless: Demystifying LLM-based Software Engineering Agents
Xia, C. S., Deng, Y., Dunn, S., & Zhang, L. (2024) · arXiv · doi:10.48550/arxiv.2407.01489 · Cited by 24
Software Vulnerability Detection with GPT and In-Context Learning
Liu, Z., Liao, Q., Gu, W., & Gao, C. (2023) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc59305.2023.00041 · Cited by 22
Mutation-based Consistency Testing for Evaluating the Code Understanding Capability of LLMs
Li, Z. H., & Shin, D. (2024) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1145/3644815.3644946 · Cited by 25
Let’s Fix this Together: Conversational Debugging with GitHub Copilot
Bajpai, Y., Chopra, B., Biyani, P., Aslan, C., Coleman, D., Gulwani, S., Parnin, C., Radhakrishna, A., & Soares, G. (2024) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc60511.2024.00011 · Cited by 18
REEF: A Framework for Collecting Real-World Vulnerabilities and Fixes
Wang, C., Li, Z., Pena, Y., Gao, S., Chen, S., Wang, S., Gao, C., & Lyu, M. R.-T. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00199 · Cited by 23
Exploring Experiences with Automated Program Repair in Practice
Meem, F. N., Smith, J., & Johnson, B. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639182 · Cited by 20
Aligning the Objective of LLM-Based Program Repair
Xu, J., Fu, Y., Tan, S. H., & He, P. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00169 · Cited by 47
Generative AI for Software Development: A Family of Studies on Code Generation
Dakhel, A. M., Nikanjam, A., Khomh, F., Desmarais, M. C., & Washizaki, H. (2024) · Springer eBooks · doi:10.1007/978-3-031-55642-5_7 · Cited by 18
APIGen: Generative API Method Recommendation
Chen, Y., Gao, C., Zhu, M., Liao, Q., Wang, Y., & Xu, G. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00025 · Cited by 25
Evaluating ChatGPT for Smart Contracts Vulnerability Correction
Napoli, E. A., & Gatteschi, V. (2023) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac57700.2023.00283 · Cited by 15
An Evaluation of the Effectiveness of OpenAI's ChatGPT for Automated Python Program Bug Fixing using QuixBugs
Wuisang, M. C., Kurniawan, M., Santosa, K. A. W., Gunawan, A. A. S., & Saputra, K. E. (2023) · International Seminar on Application for Technology of Information and Communication (iSemantic) · doi:10.1109/isemantic59612.2023.10295323 · Cited by 14
DrPlanner: Diagnosis and Repair of Motion Planners for Automated Vehicles Using Large Language Models
Lin, Y., Li, C., Ding, M., Tomizuka, M., Zhan, W., & Althoff, M. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3441493 · Cited by 22
Reality Check: Assessing GPT-4 in Fixing Real-World Software Vulnerabilities
Ságodi, Z., Antal, G., Bogenfürst, B., Isztin, M., Hegedűs, P., & Ferenć, R. (2024) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3661167.3661207 · Cited by 13
MORepair : Teaching LLMs to Repair Code via Multi-Objective Fine-Tuning
Yang, B., Tian, H., Ren, J., Zhang, H., Klein, J., Bissyandé, T. F., Goues, C. L., & Jin, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3735129 · Cited by 44
T5APR: Empowering automated program repair across languages through checkpoint ensemble
Gharibi, R., Sadreddini, M. H., & Fakhrahmad, S. M. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112083 · Cited by 15
Analyzing Developer Use of ChatGPT Generated Code in Open Source GitHub Projects
Grewal, B., Lu, W., Nadi, S., & Bezemer, C. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3645072 · Cited by 13
Navigating (in)Security of AI-Generated Code
Ambati, S. H., Ridley, N., Branca, E., & Stakhanova, N. (2024) · IEEE International Conference on Cyber Security and Resilience (CSR) · doi:10.1109/csr61664.2024.10679468 · Cited by 12
Statically Contextualizing Large Language Models with Typed Holes
Blinn, A., Li, X., Kim, J. H., & Omar, C. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689728 · Cited by 10
Context-aware prompting for LLM-based program repair
Li, Y., Cai, M., Chen, J., Xu, Y., Huang, L., & Li, J. K. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00512-w · Cited by 14
RustAssistant: Using LLMs to Fix Compilation Errors in Rust Code
Deligiannis, P., Lal, A., Mehrotra, N., Poddar, R., & Rastogi, A. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00022 · Cited by 18
Leveraging Large Language Models to Boost Dafny’s Developers Productivity
Silva, Á. F., Mendes, A. S., & Ferreira, J. F. (2024) · IEEE/ACM International Conference on Formal Methods in Software Engineering (FormaliSE) · doi:10.1145/3644033.3644374 · Cited by 10
Hybrid API Migration: A Marriage of Small API Mapping Models and Large Language Models
Zhou, B., Wang, X., Xu, S., Yao, Y., Pan, M., Xu, F., & Ma, X. (2023) · Asia-Pacific Symposium on Internetware · doi:10.1145/3609437.3609466 · Cited by 10
Automated Program Repair for Introductory Programming Assignments
Wan, H., Luo, H., Li, M., & Luo, X. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3403710 · Cited by 10
LLM Assistance for Memory Safety
Mohammed, N., Lal, A., Rastogi, A., Sharma, R., & Roy, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00023 · Cited by 9
RAPGen: An Approach for Fixing Code Inefficiencies in Zero-Shot
Garg, S., Moghaddam, R. Z., & Sundaresan, N. (2023) · 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00017 · Cited by 36
An Overview on Large Language Models
Dakhel, A. M., Nikanjam, A., Khomh, F., Desmarais, M. C., & Washizaki, H. (2024) · Springer eBooks · doi:10.1007/978-3-031-55642-5_1 · Cited by 11
The Seeds of the Future Sprout from History: Fuzzing for Unveiling Vulnerabilities in Prospective Deep-Learning Libraries
Li, Z., Wu, J., Ling, X., Luo, T., Rui, Z., & Wu, Y. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00132 · Cited by 14
Program Repair with Minimal Edits Using CodeT5
Shirafuji, A., Rahman, M. M., Amin, M. F. I., & Watanobe, Y. (2023) · International Conference on Awareness Science and Technology (iCAST) · doi:10.1109/icast57874.2023.10359288 · Cited by 10
Automated Program Repair Using Generative Models for Code Infilling
Koutcheme, C., Sarsa, S., Leinonen, J., Hellas, A., & Denny, P. C. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-36272-9_74 · Cited by 11
Evaluating AI-Generated Code for C++, Fortran, Go, Java, Julia, Matlab, Python, R, and Rust
Diehl, P., Nader, N., Brandt, S. R., & Kaiser, H. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-90200-0_20 · Cited by 12
ReAPR: Automatic program repair via retrieval-augmented large language models
Liu, Z., Du, X., & Liu, H. (2025) · Software Quality Journal · doi:10.1007/s11219-025-09728-1 · Cited by 10
RunBugRun: An executable dataset for automated program repair
Prenner, J. A., & Robbes, R. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10790-3 · Cited by 33
Self-citation
Exploring the Impact of Intervention Methods on Developers’ Security Behavior in a Manipulated ChatGPT Study
Serafini, R., Yardım, A., & Naiakshina, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713989 · Cited by 6
Detecting, Creating, Repairing, and Understanding Indivisible Multi-Hunk Bugs
Xin, Q., Wu, H., Tang, J., Liu, X., Reiss, S. P., & Xuan, J. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660828 · Cited by 6
Enhancing Automated Program Repair with Solution Design
Zhao, J., Yang, D., Zhang, L., Lian, X., Yang, Z., & Liu, F. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695537 · Cited by 6
ROSE: An IDE-Based Interactive Repair Framework for Debugging
Reiss, S. P., Wei, X., Yuan, J. P., & Xin, Q. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3705306 · Cited by 6
RetypeR: Integrated Retrieval-based Automatic Program Repair for Python Type Errors
Hao, S., Shi, X., & Liu, H. (2024) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58944.2024.00028 · Cited by 5
Automated Program Repair with the GPT Family, including GPT-2, GPT-3 and CodeX
Lajkó, M., Csuvik, V., Gyimóthy, T., & Vidács, L. (2024) · doi:10.1145/3643788.3648021 · Cited by 5
Context Matters: Investigating Its Impact on ChatGPT's Bug Fixing Performance
Qu, X., Zuo, F., Li, X., & Rhee, J. (2024) · IEEE/ACIS International Conference on Software Engineering Research, Management and Applications (SERA) · doi:10.1109/sera61261.2024.10685568 · Cited by 4
Can GPT-O1 Kill All Bugs? An Evaluation of GPT-Family LLMs on QuixBugs
Hu, H., Shang, Y., Xu, G., He, C., & Zhang, Q. (2025) · IEEE/ACM International Workshop on Automated Program Repair (APR) · doi:10.1109/apr66717.2025.00007 · Cited by 4
NIODebugger: A Novel Approach to Repair Non-Idempotent-Outcome Tests with LLM-Based Agent
Ke, K. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00226 · Cited by 4
Proceedings of the 2024 IEEE/ACM 12th International Conference on Formal Methods in Software Engineering (FormaliSE)
(2024) · doi:10.1145/3644033 · Cited by 4
Boosting Redundancy-Based Automated Program Repair by Fine-Grained Pattern Mining
Jiang, J., Li, F., Zhao, Z., Ye, Z., Liu, M., Wang, B., Zhang, H., & Chen, J. (2025) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme64153.2025.00018 · Cited by 3
Enhancing Synthetic Test Data Generation with Language Models Using a More Expressive Domain-Specific Language
Tan, C., Behjati, R., & Arisholm, E. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43240-8_2 · Cited by 3
SCC-GPT: Source Code Classification Based on Generative Pre-Trained Transformers
Alahmadi, M. D., Alshangiti, M., & Alsubhi, J. (2024) · Mathematics · doi:10.3390/math12132128 · Cited by 3
Element-Based Automated DNN Repair with Fine-Tuned Masked Language Model
Wang, X., Zhang, M., Meng, X., Zhang, J., Liu, Y., & Hu, C. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715716 · Cited by 2
Defects4C: Benchmarking Large Language Model Repair Capability with C/C++ Bugs
Wang, J., Xie, X., Hu, Q., Liu, S., Yu, J., Kong, J., & Li, Y. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00029 · Cited by 2
DLBench: A Comprehensive Benchmark for SQL Translation with Large Language Models
Lin, L., Chen, H., Zhu, Q., Chen, L., Tang, L., & Wu, R. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00076 · Cited by 3
Unveiling the power of code pre-trained models in automated program repair: A systematic review
Zhan, S., Wang, X. J., Wei, D., Xiang, X., Lin, Y., Wu, H., & Lu, J. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.113015 · Cited by 2
Robustness-Enhanced Assertion Generation Method Based on Code Mutation and Attack Defense
Li, M., Chen, S., Fan, G., Zhang, L., Wu, H., Xue, X., & Feng, Z. (2024) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-54528-3_16 · Cited by 2
Benchmarking and Categorizing the Performance of Neural Program Repair Systems for Java
Zhong, W., Li, C., Liu, K., Ge, J., Luo, B., Bissyandé, T. F., & Ng, V. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3688834 · Cited by 3
Exploring ChatGPT's Potential in Java API Method Recommendation: An Empirical Study
Wang, Y., Xue, W., Huang, Q., Jiang, B., & Zhang, H. (2025) · Journal of Software Evolution and Process · doi:10.1002/smr.2765 · Cited by 4
MultiMend: multilingual program repair with context augmentation and multi-hunk patch generation
Gharibi, R., Sadreddini, M. H., & Fakhrahmad, S. M. (2026) · Automated Software Engineering · doi:10.1007/s10515-026-00611-2 · Cited by 7
Extracting Fix Ingredients using Language Models
Prenner, J. A., & Robbes, R. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00028 · Cited by 1
Self-citation
Patch Generation in APR: A Survey from the Perspectives of Utilizing LLMs and Using APR-Specific Information
Yang, Y., Li, C., Han, Z., Li, R., Xu, K., Li, Q., Zhong, W., Shen, Z., Fei, Z., Ge, J., et al. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3764584 · Cited by 7
Specification-Guided Repair of Arithmetic Errors in Dafny Programs Using LLMs
Wu, V., Mendes, A., & Abreu, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-10444-1_16 · Cited by 2
InstructRepair: Instruct Large Language Models With Rich Bug Information for Automated Program Repair
Fu, A., Xu, P., Li, J., Kuang, B., & Gao, Y. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3618407 · Cited by 4
HFuzzer: Testing Large Language Models for Package Hallucinations via Phrase-based Fuzzing
Zhao, Y., Wu, M., Hu, X., & Xia, X. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00225 · Cited by 1
Securing AI Code Generation - A Prompt Rectification Approach for Mitigating Cyber Risks
Dong, J., Ni, Z., Sun, N., Jha, S., Zhang, Y., Bertino, E., Nepal, S., & Ma, S. (2025) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom66490.2025.00106 · Cited by 1
Towards defect-type-aware adaptive program repair: A stage-wise approach with large language models
Zhang, R., Zhang, S., & Xie, L. (2026) · Software Quality Journal · doi:10.1007/s11219-026-09751-w · Cited by 3
HAFix: history-augmented large language models for bug fixing
Shi, Y., Bangash, A. A., Fallahzadeh, E., Adams, B., & Hassan, A. E. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10833-3 · Cited by 8
Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMs
Al-Kaswan, A., Spieß, C. K., Dévanbu, P., Arie Van Deursen, A., & Izadi, M. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793341 · Cited by 1
PReMM: LLM-Based Program Repair for Multi-method Bugs via Divide and Conquer
Xie, L., Li, Z., Pei, Y., Wen, Z., Liu, K., Zhang, T., & Li, X. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3763097 · Cited by 7
Demonstration Attack against In-Context Learning for Code Intelligence
Ge, Y., Sun, W., Lou, Y., Fang, C., Zhang, Y., Li, Y., Zhang, X., Liu, Y., Zhao, Z., & Chen, Z. (2024) · arXiv · doi:10.48550/arxiv.2410.02841
“Moreover, Pren-ner et al. [31] mark, demonstrating the practical application and potential improvements of ICL in code repair tasks.”
Background
EvoAPR: Enhancing Large Language Models for Automatic Program Repair with Genetic Algorithm and Dynamic LoRA
Zhang, H., Yan, Q., Min, W., Zhang, C., Kuang, L., & Xia, Y. (2025) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws67624.2025.00123
An Empirical Study Measuring In-The-Wild Cryptographic Microarchitectural Side-Channel Patches
Deng, S., Liu, Z., Wang, S., & Zhang, Y. (2025) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3719027.3744881
Fine-Grained Bug Localization in Software Projects Using Code Graph
Sunitha, E. V. (2026) · Lecture notes in networks and systems · doi:10.1007/978-981-96-9275-0_38
How Small is Enough? Empirical Evidence of Quantized Small Language Models for Automated Program Repair
Kusama, K., Shu, H., Kondo, M., & Kamei, Y. (2025) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem64174.2025.00047
FACKT: A Fault-Aware Model for Code Knowledge Tracing
Zhou, M., Zhou, Q., Xia, W., Jiang, Y., Lin, C., & Zhao, S. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-5719-6_5
Yazılım Mühendisliği Eğitiminde Üretken Yapay Zekânın Bibliyometrik Analiz Yöntemi ile İncelenmesi
Baz, F. Ç., & Keleş, Ş. (2026) · Osmaniye Korkut Ata Üniversitesi Fen Bilimleri Enstitüsü Dergisi · doi:10.47495/okufbed.1766554
Impact of Different Buggy Code Information on Large Language Model-Based Automated Program Repair
Liu, L., Li, M., & Ren, J. (2026) · International Symposium on Computer Technology and Information Science (ISCTIS) · doi:10.1109/isctis70043.2026.11572607
Fine-Grained Repair Knowledge Organization for Knowledge-Guided Automated Program Repair
Jiang, J., Cheng, H., Liu, Y., Lin, N., Yu, S., Zhu, X., & Wang, J. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3450-9_50

Authors: Prenner, J. A., Babii, H., & Robbes, R.
Affiliations: Free University of Bozen-Bolzano (IT)
ORCID: Prenner, Julian Aron
Babii, Hlib
Robbes, Romain
Venue: Proceedings of the Third International Workshop on Automated Program Repair
Publisher: ACM
Published: 2022-05-19
Pages: 69-75
DOI: 10.1145/3524459.3527351
OpenAlex: W4312970618
Cited by: 99
References: 24
Citations by year: 2026: 14 · 2025: 30 · 2024: 38 · 2023: 17
FWCI: 16.03
Type: conference paper
Language: en
Open access: closed
Licence: www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations: IEEE/ACM International Workshop on Automated Program Repair (APR) (publishedVersion)
Topics: Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
Keywords: OpenAI Codex, automated program repair, bug localization, bug fixes, large language models, GPT-3, Python, Java, code generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Can OpenAI's codex fix bugs?: an evaluation on QuixBugs

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks