CoChat
Sign Up
Cited by 225Software Engineering ResearchSoftware Testing and Debugging TechniquesSoftware Engineering Techniques and Practices

AutoCodeRover: Autonomous Program Improvement

Open access
Zhang, Yuntong, Ruan, Haifeng, Fan, Zhiyu, Roychoudhury, Abhik·2024·Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis
DOI 10.1145/3650212.3680384
Open this paper in CoChat — free accountOpen PDF

Abstract

Researchers have made significant progress in automating the software development process in the past decades. Automated techniques for issue summarization, bug reproduction, fault localization, and program repair have been built to ease the workload of developers. Recent progress in Large Language Models (LLMs) has significantly impacted the development process, where developers can use LLM-based programming assistants to achieve automated coding. Nevertheless, software engineering involves the process of program improvement apart from coding, specifically to enable software maintenance (e.g. program repair to fix bugs) and software evolution (e.g. feature additions). In this paper, we propose an automated approach for solving Github issues to autonomously achieve program improvement. In our approach called AutoCodeRover, LLMs are combined with sophisticated code search capabilities, ultimately leading to a program modification or patch. In contrast to recent LLM agent approaches from AI researchers and practitioners, our outlook is more software engineering oriented. We work on a program representation (abstract syntax tree) as opposed to viewing a software project as a mere collection of files. Our code search exploits the program structure in the form of classes/methods to enhance LLM’s understanding of the issue’s root cause, and effectively retrieve a context via iterative search. The use of spectrum-based fault localization using tests, further sharpens the context, as long as a test-suite is available. Experiments on the recently proposed SWE-bench-lite (300 real-life Github issues) show increased efficacy in solving Github issues (19% on SWE-bench-lite), which is higher than the efficacy of the recently reported Swe-agent. Interestingly, our approach resolved 57 GitHub issues in about 4 minutes each (pass@1), whereas developers spent more than 2.68 days on average. In addition, AutoCodeRover achieved this efficacy with significantly lower cost (on average, $0.43 USD), compared to other baselines. We posit that our workflow enables autonomous software engineering, where, in future, auto-generated code from LLMs can be autonomously improved.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3650212.3680384

Cites

34 of 35 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Memory methods maintain working state, retrieve repository evidence, store reusable experience, and support shared interaction histories.

    “memory methods that maintain working state, retrieve repository evidence, store reusable experience, and support shared interaction histories [45, 46, 47, 48]”

    From Code as Agent Harness (2026) · cites this paper as [46]

    Section 4.3, “Stratified Context Search” (page unavailable), describes retaining the current context and adding retrieved repository code snippets, supporting working state and repository evidence. Our copy does not describe storing reusable experience or using shared interaction histories as a method; the Data availability section only says the authors made LLM conversation histories public. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Both SWE-bench and Polyglot are widely used benchmarks that require the AI agent to navigate a code repository, understand the interplay between functions in different files, and spot small errors in convoluted code.

    “Both SWE-bench and Polyglot are widely used benchmarks [16, 39, 43, 137, 146, 151] that require the AI agent to navigate a code repository, understand the interplay between functions in different files, and spot small errors in convoluted code.”

    From Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents (2025) · cites this paper as [151]

    Section 2.2 (page unavailable) describes SWE-bench lite as a benchmark involving large repositories, locating bugs across files, and analyzing their root causes. Section 7 (page unavailable) also describes substantial use of SWE-bench by research agents. Our copy does not address Polyglot or establish that both benchmarks require spotting small errors in convoluted code. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Mechanisms such as AST-based structured chunking, iterative query rewriting, and retrieval strategies conditioned on current localization clues can substantially improve the utility of retrieved context for downstream generation.

    “Mechanisms such as AST-based structured chunking, iterative query rewriting, and retrieval strategies conditioned on current localization clues can substantially improve the utility of retrieved context for downstream generation.”

    From Code as Agent Harness (2026) · cites this paper as [46]

    Sections 4.2–4.3 describe AST-based retrieval of code entities and iterative API searches chosen using issue hints and previously retrieved context. Section 4.1 says the retrieved context feeds patch generation. The paper does not describe these mechanisms as structured chunking or iterative query rewriting, nor establish that each substantially improves downstream generation; page numbers are unavailable in our copy. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Repository-level coding tasks benefit not simply from retrieving more content, but from retrieving evidence aligned with program structure.

    “Representative works such as AutoCodeRover [46] and RepoCoder [47] show that repository-level coding tasks benefit not simply from retrieving more content, but from retrieving evidence aligned with program structure.”

    From Code as Agent Harness (2026) · cites this paper as [46]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 12

  1. Marcel Böhme, Cristian Cadar, and Abhik Roychoudhury. 2021. Fuzzing: Chal- lenges and Reflections. IEEE Software 38, 3 (2021).
  2. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluat- ing Large Language Models Trained on Code. arXiv.org abs/2107.03374 (7 2021). arXiv:2107.03374 https://arxiv.org/abs/2107.03374 · arXiv
  3. Claire Le Goues, Michael Pradel, and Abhik Roychoudhury. 2019. Automated program repair. Commun. ACM 62 (11 2019), 56-65. Issue 12.
  4. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. Leaderboard results on SWE-bench. Retrieved April 8, 2024 from https://www.swebench.com/
  5. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=VTF8yNQM66
  6. Wuxia Jin et al. 2024. PyAnalyzer: An Effective and Practical Approach for De- pendency Extraction from Python Code. In International Conference on Software Engineering (ICSE).
  7. Thibaud Lutellier, Hung Viet Pham, Lawrence Pang, Yitong Li, Moshi Wei, and Lin Tan. 2020. CoCoNuT: combining context-aware neural translation models us- ing ensemble for program repair., In ISSTA '20: 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, USA, July 18-22, 2020, Sarfraz Khurshid and Corina S. Pasareanu (Eds.). International Symposium on Software Testing and Analysis, 101-114.
  8. Sergey Mechtaev, Jooyong Yi, and Abhik Roychoudhury. 2016. Angelix: scalable multiline program patch synthesis via symbolic analysis., In Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, Austin, TX, USA, May 14-22, 2016, Laura K. Dillon, Willem Visser, and Laurie Williams (Eds.). International Conference on Software Engineering, 691-701.
  9. Yannic Noller, Ridwan Shariffdeen, Xiang Gao, and Abhik Roychoudhury. 2022. Trust Enhancement Issues in Program Repair. In IEEE/ACM 44th International Conference on Software Engineering (ICSE).
  10. David Williams, James Callan, Serkan Kirbas, Sergey Mechtaev, Justyna Petke, Thomas Prideaux-Ghee, and Federica Sarro. 2023. User-Centric Deployment of Automated Program Repair at Bloomberg. arXiv preprint arXiv:2311.10516 (2023). · arXiv
  11. W Eric Wong, Ruizhi Gao, Yihao Li, Rui Abreu, and Franz Wotawa. 2016. A survey on software fault localization. IEEE Transactions on Software Engineering 42, 8 (2016), 707-740.
  12. John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer In- terfaces Enable Automated Software Engineering. arXiv:2405.15793 [cs.SE] · arXiv

Cited by

Sorted by citations

Related

Authors
Zhang, Y., Ruan, H., Fan, Z., & Roychoudhury, A.
Affiliations
National University of Singapore (SG)
ORCID
Venue
Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis
Publisher
ACM
Published
2024-09-11
Pages
1592-1604
DOI
10.1145/3650212.3680384
OpenAlex
W4402442868
Cited by
202
References
34
Citations by year
2026: 123 · 2025: 70 · 2024: 9
FWCI
76.83
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
Ministry of Education - Singapore (MOE-MOET32021-0001)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
Keywords
program improvement, program repair, large language models, code search, spectrum-based fault localization, abstract syntax tree, software maintenance, software evolution, LLM agents

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review