CoChat
Sign Up
Cited by 165Software Engineering ResearchTopic ModelingTeaching and Learning Programming

Self-Planning Code Generation with Large Language Models

Open access
Jiang, Xue, Dong, Yihong, Wang, Lecheng, Fang, Zheng, Shang, Qiwei, Li, Ge, Jin, Zhi, Jiao, Wenpin·2024·ACM Transactions on Software Engineering and Methodology
DOI 10.1145/3672456
Open this paper in CoChat — free accountOpen PDF

Abstract

Although large language models (LLMs) have demonstrated impressive ability in code generation, they are still struggling to address the complicated intent provided by humans. It is widely acknowledged that humans typically employ planning to decompose complex problems and schedule solution steps prior to implementation. To this end, we introduce planning into code generation to help the model understand complex intent and reduce the difficulty of problem-solving. This paper proposes a self-planning code generation approach with large language models, which consists of two phases, namely planning phase and implementation phase. Specifically, in the planning phase, LLM plans out concise solution steps from the intent combined with few-shot prompting. Subsequently, in the implementation phase, the model generates code step by step, guided by the preceding solution steps. We conduct extensive experiments on various code-generation benchmarks across multiple programming languages. Experimental results show that self-planning code generation achieves a relative improvement of up to 25.4% in Pass@1 compared to direct code generation, and up to 11.9% compared to Chain-of-Thought of code generation. Moreover, our self-planning approach also enhances the quality of the generated code with respect to correctness, readability, and robustness, as assessed by humans.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3672456

Cites

41 of 41 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Planning methods organize long-horizon software tasks through decomposition, structural grounding, trajectory search, or workflow orchestration.

    “We therefore review planning methods that organize long-horizon software tasks through decomposition, structural grounding, trajectory search, or workflow orchestration [40, 41, 42, 43, 44]”

    From Code as Agent Harness (2026) · cites this paper as [40]

    The Introduction says planning breaks complex code-generation problems into easier subproblems, supporting the decomposition part of the collectively cited sentence. The Discussion and Future Work section says producing fully functional software remains beyond the approach, so this paper does not establish its application to long-horizon software tasks generally. The other listed methods need not each be supported by this one citation. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Standard LLMs attempt to infer missing requirements when prompts are underspecified, which leads to inconsistent compliance.

    “Standard LLMs attempt to infer missing requirements when prompts are underspecified, which leads to inconsistent compliance [42, 75].”

    From A Multi-Agent Framework for Accessible Web Code Generation (2026) · cites this paper as [42]

    The Introduction says that directly generating code satisfying complex intent is challenging for LLMs, and §6 (Threats to Validity) describes their sensitivity to prompts. Neither passage says that LLMs infer missing requirements from underspecified prompts or that this causes inconsistent compliance; searches for ambiguous or incomplete specifications did not establish that link. Page numbers are unavailable in our copy. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • The multi-agent system resolves violations in a controlled, layered manner rather than instantaneous correction.

    “This pattern confirms that the multi-agent system resolves violations in a controlled, layered manner rather than instantaneous correction [42].”

    From A Multi-Agent Framework for Accessible Web Code Generation (2026) · cites this paper as [42]

    The Readability section describes converting a coding intent into a plan and resolving each solution step, while §4.3 compares multi-turn and single-turn code generation. Neither passage addresses a multi-agent system resolving violations, or compares layered resolution with instantaneous correction. Searches for multi-agent violation resolution and iterative correction found no passage that does. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • Prompt engineering alone is insufficient to guarantee compliance with formal accessibility standards.

    “This architecture directly addresses concerns raised in prior work that prompt engineering alone is insufficient to guarantee compliance with formal accessibility standards [42, 75]”

    From A Multi-Agent Framework for Accessible Web Code Generation (2026) · cites this paper as [42]

    The Threats to Validity section discusses sensitivity to prompt design, and the Discussion and Future Work section says LLM-generated code is far from fully functional software. Neither passage addresses compliance with formal accessibility standards. Searches for accessibility standards, WCAG, and the claimed limitation of prompt engineering found no passage establishing that point. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 35

  1. Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. CoRR abs/2108.07732 (2021). · arXiv
  2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In NeurIPS 2020.
  3. Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2023. CodeT: Code Generation with Generated Tests. In ICLR.
  4. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. CoRR (2021). https://arxiv.org/abs/2107.03374 · arXiv
  5. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2022. Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks. CoRR abs/2211.12588 (2022). · arXiv
  6. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching Large Language Models to Self-Debug. CoRR abs/2304.05128 (2023). · arXiv
  7. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems. CoRR abs/2110.14168 (2021). · arXiv
  8. Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, and Ge Li. 2024. Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models. CoRR abs/2402.15938 (2024). · arXiv
  9. Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. 2023. Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution. CoRR abs/2309.16797 (2023). · arXiv
  10. Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022. InCoder: A Generative Model for Code Infilling and Synthesis. CoRR abs/2204.05999 (2022). · arXiv
  11. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2022. PAL: Program-aided Language Models. CoRR abs/2211.10435 (2022). · arXiv
  12. Zelalem Gero, Chandan Singh, Hao Cheng, Tristan Naumann, Michel Galley, Jianfeng Gao, and Hoifung Poon. 2023. Self-Verification Improves Few-Shot Clinical Information Extraction. CoRR abs/2306.00024 (2023). · arXiv
  13. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring Coding Challenge Competence With APPS. In NeurIPS Datasets and Benchmarks.
  14. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large Language Models are Zero-Shot Reasoners. In NeurIPS.
  15. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. Solving Quantitative Reasoning Problems with Language Models. In NeurIPS.
  16. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-Refine: Iterative Refinement with Self-Feedback. CoRR abs/2303.17651 (2023). · arXiv
  17. Rohan Mukherjee, Yeming Wen, Dipak Chaudhari, Thomas W. Reps, Swarat Chaudhuri, and Christopher M. Jermaine. 2021. Neural Program Generation Modulo Static Analysis. In NeurIPS. 18984-18996.
  18. OpenAI. 2023. GPT-4 Technical Report. CoRR abs/2303.08774 (2023). · arXiv
  19. Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable Code Generation from Pre-trained Language Models. In ICLR.
  20. Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. CodeBLEU: a Method for Automatic Evaluation of Code Synthesis. CoRR abs/2009.10297 (2020). · arXiv
  21. Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022. Multitask Prompted Training Enables Zero-Shot Task Generalization. In ICLR. OpenReview.net.
  22. Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, and Jonathan Berant. 2020. Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge. In NeurIPS.
  23. Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. 2023. Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents. CoRR abs/2302.01560 (2023). · arXiv
  24. Bolin Wei, Ge Li, Xin Xia, Zhiyi Fu, and Zhi Jin. 2019. Code Generation as a Dual Task of Code Summarization. In NeurIPS. 6559-6569.
  25. Yuhuai Wu, Albert Qiaochu Jiang, Wenda Li, Markus N. Rabe, Charles Staats, Mateja Jamnik, and Christian Szegedy. 2022. Autoformalization with Large Language Models. In NeurIPS.
  26. Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min-Yen Kan, Junxian He, and Qizhe Xie. 2023. Self-Evaluation Guided Beam Search for Reasoning. In Thirty-seventh Conference on Neural Information Processing Systems.
  27. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In ICLR. OpenReview.net.
  28. Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua B. Tenenbaum, and Chuang Gan. 2023. Planning with Large Language Models for Code Generation. CoRR abs/2303.05510 (2023). · arXiv
  29. Tianyi Zhang, Tao Yu, Tatsunori B. Hashimoto, Mike Lewis, Wen-tau Yih, Daniel Fried, and Sida I. Wang. 2022. Coder Reviewer Reranking for Code Generation. CoRR abs/2211.16490 (2022). · arXiv
  30. Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023. Automatic Chain of Thought Prompting in Large Language Models. In ICLR. OpenReview.net.
  31. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed H. Chi. 2022. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. CoRR abs/2205.10625 (2022). · arXiv
  32. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large Language Models are Human-Level Prompt Engineers. In ICLR. OpenReview.net.
  33. Return the result. "' G INSTANCES OF BASELINE PROMPT Instance of Chain-of-Thought Prompting with Self-planning Format def encrypt(s): "' Create a function encrypt that takes a string as an argument and returns a string encrypted with the alphabet being rotated. The alphabet should be rotated in a manner such that the letters shift down by two multiplied to two places. For example: encrypt('hi') returns 'lm' encrypt('asdfghjkl') returns 'ewhjklnop' encrypt('gf') returns 'kj' encrypt('et') returns 'ix' Let's think step by step.
  34. Create a string "alphabet" with all letters of the alphabet. 2. Assign the number of places to shift the letters to a variable "bias".
  35. Return the string "result". "' Instance of Extremely Concise Style Self-planning Prompt def encrypt(s): "' Create a function encrypt that takes a string as an argument and returns a string encrypted with the alphabet being rotated. The alphabet should be rotated in a manner such that the letters shift down by two multiplied to two places. For example: encrypt('hi') returns 'lm' encrypt('asdfghjkl') returns 'ewhjklnop' encrypt('gf') returns 'kj' encrypt('et') returns 'ix' 1. Alphabet, bias 4.

Cited by

Sorted by citations

Related

Authors
Jiang, X., Dong, Y., Wang, L., Fang, Z., Shang, Q., Li, G., Jin, Z., & Jiao, W.
Affiliations
Peking University (CN)
ORCID
Venue
ACM Transactions on Software Engineering and Methodology
Publisher
Association for Computing Machinery (ACM)
Published
2024-09-27
Volume
33
Issue
7
Pages
1-30
ISSN
  • 1049-331X (ISSN-L)
  • 1557-7392
DOI
10.1145/3672456
OpenAlex
W4399601909
Cited by
158
References
37
Citations by year
2026: 65 · 2025: 73 · 2024: 18 · 2023: 2
FWCI
58.66
Type
journal article
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM Transactions on Software Engineering and Methodology (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Key R & D Program (2023YFB4503801)
  • National Natural Science Foundation of China (62072007, 62192733, 61832009, and 62192730)
  • Key Program of Hubei (JD2023008)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
Keywords
large language models, code generation, planning, few-shot prompting, chain-of-thought, programming languages, code quality

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review