CoChat
Sign Up
PreprintCited by 97Software Engineering ResearchSoftware Engineering Techniques and PracticesSoftware Testing and Debugging Techniques

CodePlan: Repository-level Coding using LLMs and Planning

Open access
Bairi, Ramakrishna, Sonwane, Atharv, Kanade, Aditya, C, Vageesh D, Iyer, Arun, Parthasarathy, Suresh, Rajamani, Sriram, Ashok, B. +1 more·2023·arXiv
DOI 10.48550/arxiv.2309.12499arXiv 2309.12499
Open this paper in CoChat — free account

Abstract

Software engineering activities such as package migration, fixing errors reports from static analysis or testing, and adding type annotations or other specifications to a codebase, involve pervasively editing the entire repository of code. We formulate these activities as repository-level coding tasks. Recent tools like GitHub Copilot, which are powered by Large Language Models (LLMs), have succeeded in offering high-quality solutions to localized coding problems. Repository-level coding tasks are more involved and cannot be solved directly using LLMs, since code within a repository is inter-dependent and the entire repository may be too large to fit into the prompt. We frame repository-level coding as a planning problem and present a task-agnostic framework, called CodePlan to solve it. CodePlan synthesizes a multi-step chain of edits (plan), where each step results in a call to an LLM on a code location with context derived from the entire repository, previous code changes and task-specific instructions. CodePlan is based on a novel combination of an incremental dependency analysis, a change may-impact analysis and an adaptive planning algorithm. We evaluate the effectiveness of CodePlan on two repository-level tasks: package migration (C#) and temporal code edits (Python). Each task is evaluated on multiple code repositories, each of which requires inter-dependent changes to many files (between 2-97 files). Coding tasks of this level of complexity have not been automated using LLMs before. Our results show that CodePlan has better match with the ground truth compared to baselines. CodePlan is able to get 5/6 repositories to pass the validity checks (e.g., to build without errors and make correct code edits) whereas the baselines (without planning but with the same type of contextual information as CodePlan) cannot get any of the repositories to pass them.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2309.12499

Cites

42 of 42 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Standard 'System 1' approaches that generate solutions in a single forward pass often hit a performance ceiling on complex algorithmic tasks.

    “standard 'System 1' approaches that generate solutions in a single forward pass often hit a performance ceiling on complex algorithmic tasks.”

    From ReflexiCoder: Teaching Large Language Models to Self-Reflect on Generated Code and Self-Correct It via Reinforcement Learning (2026) · cites this paper as Bairi et al., 2023

    The paper reports that planning-free baselines failed validity checks on all six repositories tested, supporting a limitation on complex repository-level coding tasks (LLM-driven Repository-level Coding Task, p. 4). It does not characterize those baselines as “System 1” or single-forward-pass approaches; its Oracles and Baselines section (p. 15) describes them as reactive methods that work step by step. The results do not establish a general performance ceiling on complex algorithmic tasks. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • In practical applications like chatbots and agents, prompts are often much larger than generated responses such as summaries and code pieces, creating significant inference latency and memory utilization overhead.

    “In practical applications like chatbots and agents, where prompts range from multi-turn conversations to extensive articles or codebases [1, 9, 10], prompts are often much larger than generated responses such as summaries and code pieces, thus creating significant inference latency and memory utilization overhead.”

    From SnapKV: LLM Knows What You are Looking for Before Generation (2024) · cites this paper as [10]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 16

  1. [9][9] Lakshya A Agrawal, Aditya Kanade, Navin Goyal, Shuvendu K. Lahiri, and Sriram K. Rajamani. 2023. Guiding Language Models of Code with Global Context using Monitors. arXiv:2306.10763 [cs.CL] - · arXiv
  2. [12][12] Alfred V Aho, Ravi Sethi, Jeffrey D Ullman, et al. 2007. Compilers: principles, techniques, and tools . Vol. 2. Addison-wesley Reading. -
  3. [15][15] Jesper Andersen and Julia L Lawall. 2010. Generic patch inference. Automated software engineering 17 (2010), 119-148. -
  4. [16][16] Taweesup Apiwattanapong, Alessandro Orso, and Mary Jean Harrold. 2004. A differencing algorithm for objectoriented programs. In Proceedings. 19th International Conference on Automated Software Engineering, 2004. IEEE, 2-13. -
  5. [19][19] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. http://arxiv.org/abs/2108.07732 arXiv:2108.07732 [cs]. - · arXiv
  6. [22][22] Bruno Blanchet. 2003. Escape analysis for JavaTM: Theory and practice. ACM Transactions on Programming Languages and Systems (TOPLAS) 25, 6 (2003), 713-775. -
  7. [24][24] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877-1901. -
  8. [26][26] Alan Bundy. 1988. The use of explicit plans to guide inductive proofs. In 9th International Conference on Automated Deduction: Argonne, Illinois, USA, May 23-26, 1988 Proceedings 9 . Springer, 111-120. -
  9. [27][27] Matteo Busi, Pierpaolo Degano, and Letterio Galletta. 2019. Using standard typing algorithms incrementally. In NASA Formal Methods: 11th International Symposium, NFM 2019, Houston, TX, USA, May 7-9, 2019, Proceedings 11 . Springer, 106-122. Ramakrishna Bairi, Atharv Sonwane, Aditya Kanade, Vageesh D C, Arun Iyer, Suresh Parthasarathy, Sriram Rajamani, B. Ashok, and Shashank Shet -
  10. [28][28] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). - · arXiv
  11. [30][30] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022). - · arXiv
  12. [32][32] Jeffrey Dean, David Grove, and Craig Chambers. 1995. Optimization of object-oriented programs using static class hierarchy analysis. In ECOOP'95-Object-Oriented Programming, 9th European Conference, Åarhus, Denmark, August 7-11, 1995 9 . Springer, 77-101. -
  13. [34][34] Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2022. CoCoMIC: Code Completion By Jointly Modeling In-file and Cross-file Context. http://arxiv.org/abs/2212.10007 arXiv:2212.10007 [cs]. - · arXiv
  14. [35][35] Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999 (2022). - · arXiv
  15. [40][40] Priyanshu Gupta, Avishree Khare, Yasharth Bajpai, Saikat Chakraborty, Sumit Gulwani, Aditya Kanade, Arjun Radhakrishna, Gustavo Soares, and Ashish Tiwari. 2023. GrACE: Generation using Associated Code Edits. arXiv:2305.14129 [cs.SE] - · arXiv
  16. [43][43] Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, and Alexey Svyatkovskiy. 2023. InferFix: End-to-End Program Repair with LLMs. arXiv preprint arXiv:2303.07263 (2023). - · arXiv

Cited by

Sorted by citations

Related

Authors
Bairi, R., Sonwane, A., Kanade, A., C, V. D., Iyer, A., Parthasarathy, S., Rajamani, S., Ashok, B., & Shet, S.
Affiliations
Microsoft Research (India) (IN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Volume
1
Issue
FSE
Pages
675-698
ISSN
2994-970X (ISSN-L)
DOI
10.48550/arxiv.2309.12499
arXiv
2309.12499
OpenAlex
W4400582844
Cited by
92
References
42
Citations by year
2026: 36 · 2025: 52 · 2024: 3 · 2023: 1
FWCI
34.32
Type
journal article
Language
en
Open access
Licence
Locations
  • Proceedings of the ACM on software engineering. (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
large language models, planning, dependency analysis, adaptive planning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review