Is Self-Repair a Silver Bullet for Code Generation?
Open accessAbstract
Large language models have shown remarkable aptitude in code generation, but still struggle to perform complex tasks. Self-repair -- in which the model debugs and repairs its own code -- has recently become a popular way to boost performance in these settings. However, despite its increasing popularity, existing studies of self-repair have been limited in scope; in many settings, its efficacy thus remains poorly understood. In this paper, we analyze Code Llama, GPT-3.5 and GPT-4's ability to perform self-repair on problems taken from HumanEval and APPS. We find that when the cost of carrying out repair is taken into account, performance gains are often modest, vary a lot between subsets of the data, and are sometimes not present at all. We hypothesize that this is because self-repair is bottlenecked by the model's ability to provide feedback on its own code; using a stronger model to artificially boost the quality of the feedback, we observe substantially larger performance gains. Similarly, a small-scale study in which we provide GPT-4 with feedback from human participants suggests that even for the strongest models, self-repair still lags far behind what can be achieved with human-level debugging.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing agents struggle with the capability to perform self-reflection, tending to take all information/feedback and not filter out or correct irrelevant, incorrect, or misleading information.
“Existing agents struggle with the capability to perform self-reflection [69, 41]. That is to say they tend to take all information/feedback and do not know how to filter out or correct irrelevant, incorrect, or misleading information [84, 107].”
From Agentless: Demystifying LLM-based Software Engineering Agents (2024) · cites this paper as [69]
Cited by
Sorted by citations
Large Language Models as Optimizers
Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., & Chen, X. (2023) · arXiv · doi:10.48550/arxiv.2309.03409 · Cited by 151 · see citing works
Kamoi, R., Zhang, Y., Zhang, N., Han, J., & Zhang, R. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00713 · Cited by 82 · see citing works
Olausson, T., Gu, A., Lipkin, B., Zhang, C., Solar-Lezama, A., Tenenbaum, J., & Levy, R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.313 · Cited by 57 · see citing works
“Second, Logic-LM employs a self-refinement strategy, which has shown promise across code generation and NLP tasks (Zhang et al., 2023b; Chen et al., 2023a; Peng et al., 2023; Madaan et al., 2023; Olausson et al., 2023) but which we do not consider here.”
MethodSelf-citationClover: Closed-Loop Verifiable Code Generation
Sun, C., Sheng, Y., Padon, O., & Barrett, C. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-65112-0_7 · Cited by 43 · see citing works
Large Language Models Cannot Self-Correct Reasoning Yet
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2310.01798 · Cited by 36 · see citing works
“In contrast, external feedback incorporates inputs from humans, other models (Wang et al., 2023; Paul et al., 2023, inter alia ), or external tools and knowledge sources (Gou et al., 2023; Chen et al., 2023b; Olausson et al., 2023; Gao et al., 2023, inter alia ).”
BackgroundAgentless: Demystifying LLM-based Software Engineering Agents
Xia, C. S., Deng, Y., Dunn, S., & Zhang, L. (2024) · arXiv · doi:10.48550/arxiv.2407.01489 · Cited by 24 · see citing works
“Existing agents struggle with the capability to perform self-reflection [69, 41].”
BackgroundEvoR: Evolving Retrieval for Code Generation
Su, H., Jiang, S., Lai, Y., Wu, H., Shi, B., Liu, C., Liu, Q., & Yu, T. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.143 · Cited by 12 · see citing works
“Following Olausson et al. (2023b), we sample LLMs multiple times until the token limit, using the concatenation of the given question n and the retrieved documentation K r as the prompt.”
FeedbackEval: A Benchmark for Evaluating Large Language Models in Feedback-Driven Code Repair Tasks
Dai, D., Liu, M., Li, A., Cao, J., Wang, Y., Wang, C., Peng, X., & Zheng, Z. (2025) · arXiv · doi:10.48550/arxiv.2504.06939 · Cited by 1 · see citing works
“While some studies explore feedback-driven generation, they often restrict themselves to single-iteration corrections and lack systematic comparisons across diverse feedback modalities [7, 31, 44, 56].”
Self-citationAdversarial Review: Structured Disagreement for Grounded Agentic Code Review
Qiu, E. S., & Gill, J. (2026) · arXiv · doi:10.48550/arxiv.2608.18167
“So the critic struggles to catch what the generator missed (Olausson et al., 2024).”
LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation
Fakhoury, S., Naik, A., Sakkas, G., Chakraborty, S., & Lahiri, S. K. (2024) · arXiv · doi:10.48550/arxiv.2404.10100
“While these models are not explicitly fine-tuned for code generation, they have demonstrated strong capabilities on several related tasks [42], [43].”
Background
- Authors
- Olausson, T. X., Inala, J. P., Wang, C., Gao, J., & Solar-Lezama, A.
- Affiliations
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Microsoft (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2306.09896
- OpenAlex
- W4381252103
- Cited by
- 13
- References
- 0
- Citations by year
- 2026: 3 · 2025: 4 · 2024: 4 · 2023: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (2217064)
- Intel Corporation
- Microsoft Research
- Defense Advanced Research Projects Agency
- Advanced Research Projects Agency
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- self-healing, code generation, large language models, debugging, CodeLlama, GPT-4, GPT-3.5, HumanEval, repair costs
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
