CoChat
Sign Up
PreprintCited by 36AI-based Problem Solving and PlanningNatural Language Processing TechniquesTopic Modeling

PlanGenLLMs: A Modern Survey of LLM Planning Capabilities

Open access
Wei, Hui, Zhang, Zihao, He, Shenghua, Xia, Tian, Pan, Shijia, Liu, Fei·2025·arXiv
DOI 10.48550/arxiv.2502.11221arXiv 2502.11221
Open this paper in CoChat — free accountOpen PDF

Abstract

LLMs have immense potential for generating plans, transforming an initial world state into a desired goal state. A large body of research has explored the use of LLMs for various planning tasks, from web navigation to travel planning and database querying. However, many of these systems are tailored to specific problems, making it challenging to compare them or determine the best approach for new tasks. There is also a lack of clear and consistent evaluation criteria. Our survey aims to offer a comprehensive overview of current LLM planners to fill this gap. It builds on foundational work by Kartam and Wilkins (1990) and examines six key performance criteria: completeness, executability, optimality, representation, generalization, and efficiency. For each, we provide a thorough analysis of representative works and highlight their strengths and weaknesses. Our paper also identifies crucial future directions, making it a valuable resource for both practitioners and newcomers interested in leveraging LLM planning to support agentic workflows.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2502.11221

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 131

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. 19505 · arXiv
  2. Mohamed Aghzal, Erion Plaku, and Ziyu Yao. 2023. Can large language models be good path planners? a benchmark and investigation on spatial-temporal reasoning. arXiv preprint arXiv:2310.03249. · arXiv
  3. Mohamed Aghzal, Erion Plaku, and Ziyu Yao. 2024. Look further ahead: Testing the limits of gpt-4 in path planning. arXiv preprint arXiv:2406.12000. · arXiv
  4. Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Haus- man, et al. 2022. Do as i can, not as i say: Ground- ing language in robotic affordances. arXiv preprint arXiv:2204.01691. · arXiv
  5. Ramakrishna Bairi, Atharv Sonwane, Aditya Kanade, Vageesh D C, Arun Iyer, Suresh Parthasarathy, Sri- ram Rajamani, B. Ashok, and Shashank Shet. 2023. Codeplan: Repository-level coding using llms and planning. Preprint, arXiv:2309.12499. · arXiv
  6. Ksenia Bestuzheva, Mathieu Besançon, Wei-Kun Chen, Antonia Chmiela, Tim Donkiewicz, Jasper van Doornmalen, Leon Eifler, Oliver Gaul, Gerald Gam- rath, Ambros Gleixner, et al. 2021. The scip opti- mization suite 8.0. arXiv preprint arXiv:2112.08872. · arXiv
  7. Vineet Bhat, Ali Umut Kaypak, Prashanth Krish- namurthy, Ramesh Karri, and Farshad Khorrami. 2024. Grounding llms for robot task planning using closed-loop state feedback. arXiv preprint arXiv:2402.08546. · arXiv
  8. Stephen Boyd and Lieven Vandenberghe. 2004. Convex Optimization. Cambridge University Press.
  9. Faeze Brahman, Chandra Bhagavatula, Valentina Py- atkin, Jena D Hwang, Xiang Lorraine Li, Hi- rona Jacqueline Arai, Soumya Sanyal, Keisuke Sak- aguchi, Xiang Ren, and Yejin Choi. 2024. Plasma: Procedural knowledge models for language-based planning and re-planning. In The Twelfth Interna- tional Conference on Learning Representations.
  10. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
  11. Daniel Cao, Michael Katz, Harsha Kokel, Kavitha Srini- vas, and Shirin Sohrabi. 2024. Automating thought of search: A journey towards soundness and com- pleteness. arXiv preprint arXiv:2408.11326. · arXiv
  12. Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Col- lier, Karthik Narasimhan, and Shunyu Yao. 2023a. Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:2310.05915. · arXiv
  13. Jiangjie Chen, Siyu Yuan, Rong Ye, Bodhisattwa Prasad Majumder, and Kyle Richardson. 2023b. Put your money where your mouth is: Evaluating strategic planning and execution of llm agents in an auction arena. arXiv preprint arXiv:2310.05746. · arXiv
  14. Yongchao Chen, Jacob Arkin, Charles Dawson, Yang Zhang, Nicholas Roy, and Chuchu Fan. 2024a. Auto- tamp: Autoregressive task and motion planning with llms as translators and checkers. In 2024 IEEE In- ternational conference on robotics and automation (ICRA), pages 6695-6702. IEEE.
  15. Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. 2024b. Agent-flan: Designing data and methods of effective agent tuning for large language models. arXiv preprint arXiv:2403.12881. · arXiv
  16. Zhenfang Chen, Delin Chen, Wenjun Sun, Rui abd Liu, and Chuang Gan. 2025. Autonomous agents from au- tomatic reward modeling and planning. In The Thir- teenth International Conference on Learning Repre- sentations.
  17. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. · arXiv
  18. Hongwei Cui, Yuyang Du, Qun Yang, Yulin Shao, and Soung Chang Liew. 2024. Llmind: Orchestrating ai and iot with llm for complex task execution. IEEE Communications Magazine.
  19. Aidan Curtis, Nishanth Kumar, Jing Cao, Tomás Lozano-Pérez, and Leslie Pack Kaelbling. 2024. Trust the proc3s: Solving long-horizon robotics prob- lems with llms and constraint satisfaction. arXiv preprint arXiv:2406.05572. · arXiv
  20. Gautier Dagan, Frank Keller, and Alex Lascarides. 2023. Dynamic planning with a llm. arXiv preprint arXiv:2308.06391. · arXiv
  21. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091-28114.
  22. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2024. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36.
  23. Alfonso Gerevini, Ivan Serina, et al. 2002. Lpg: A planner based on local search for planning graphs with action costs. In Aips, volume 2, pages 281-290.
  24. Gonzalo Gonzalez-Pumariega, Wayne Chen, Kushal Ke- dia, and Sanjiban Choudhury. 2024. Query-efficient planning with language models. arXiv preprint arXiv:2412.06162. · arXiv
  25. Gonzalo Gonzalez-Pumariega, Leong Su Yean, Neha Sunkara, and Sanjiban Choudhury. 2025. Robotouille: An asynchronous planning benchmark for llm agents. arXiv preprint arXiv:2502.05227. · arXiv
  26. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594. · arXiv
  27. Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati. 2023. Leveraging pre- trained large language models to construct and utilize world models for model-based task planning. Ad- vances in Neural Information Processing Systems, 36:79081-79094.
  28. Shiguang Guo, Ziliang Deng, Hongyu Lin, Yaojie Lu, Xianpei Han, and Le Sun. 2024. Open grounded planning: Challenges and benchmark construction. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, Au- gust 11-16, 2024, pages 4982-5003. Association for Computational Linguistics. · DOI
  29. Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. 2023. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992. · arXiv
  30. Yilun Hao, Yongchao Chen, Yang Zhang, and Chuchu Fan. 2024a. Large language models can plan your travels rigorously with formal verification tools. arXiv preprint arXiv:2404.11891. · arXiv
  31. Yilun Hao, Yang Zhang, and Chuchu Fan. 2024b. Plan- ning anything with rigor: General-purpose zero-shot planning with llm-based formalized programming. arXiv preprint arXiv:2410.12112. · arXiv
  32. Rishi Hazra, Pedro Zuidberg Dos Martires, and Luc De Raedt. 2024. Saycanpay: Heuristic planning with large language models using learnable domain knowl- edge. In Proceedings of the AAAI Conference on Ar- tificial Intelligence, volume 38, pages 20123-20133.
  33. Malte Helmert. 2006. The fast downward planning system. Journal of Artificial Intelligence Research, 26:191-246.
  34. Eran Hirsch, Guy Uziel, and Ateret Anaby-Tavor. 2024. What's the plan? evaluating and developing planning-aware techniques for llms. arXiv preprint arXiv:2402.11489. · arXiv
  35. Richard Howey, Derek Long, and Maria Fox. 2004. Val: Automatic plan validation, continuous effects and mixed initiative planning using pddl. In 16th IEEE International Conference on Tools with Artificial In- telligence, pages 294-301. IEEE.
  36. Hanxu Hu, Hongyuan Lu, Huajian Zhang, Yun-Ze Song, Wai Lam, and Yue Zhang. 2023a. Chain-of-symbol prompting elicits planning in large langauge models. arXiv preprint arXiv:2305.10276. · arXiv
  37. Hanxu Hu, Hongyuan Lu, Huajian Zhang, Yun-Ze Song, Wai Lam, and Yue Zhang. 2024. Chain-of-symbol prompting for spatial reasoning in large language models. In First Conference on Language Modeling.
  38. Mengkang Hu, Yao Mu, Xinmiao Yu, Mingyu Ding, Shiguang Wu, Wenqi Shao, Qiguang Chen, Bin Wang, Yu Qiao, and Ping Luo. 2023b. Tree-planner: Efficient close-loop task planning with large language models. arXiv preprint arXiv:2310.08582. · arXiv
  39. Hsiu-Yuan Huang, Yutong Yang, Zhaoxi Zhang, San- woo Lee, and Yunfang Wu. 2024a. A survey of un- certainty estimation in llms: Theory meets practice. arXiv preprint arXiv:2410.15326. · arXiv
  40. Jen-tse Huang, Eric John Li, Man Ho Lam, Tian Liang, Wenxuan Wang, Youliang Yuan, Wenxiang Jiao, Xing Wang, Zhaopeng Tu, and Michael R Lyu. 2024b. How far are we on the decision-making of llms? eval- uating llms' gaming ability in multi-agent environ- ments. arXiv preprint arXiv:2403.11807. · arXiv
  41. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232. · arXiv
  42. Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022a. Language models as zero- shot planners: Extracting actionable knowledge for embodied agents. In International conference on machine learning, pages 9118-9147. PMLR.
  43. Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. 2022b. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608. · arXiv
  44. Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruim- ing Tang, and Enhong Chen. 2024c. Understanding the planning of llm agents: A survey. arXiv preprint arXiv:2402.02716. · arXiv
  45. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richard- son, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720. · arXiv
  46. Peter A Jansen. 2020. Visually-grounded planning without vision: Language models infer detailed plans from high-level instructions. arXiv preprint arXiv:2009.14259. · arXiv
  47. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language mod- els resolve real-world github issues? arXiv preprint arXiv:2310.06770. · arXiv
  48. Da Ju, Song Jiang, Andrew Cohen, Aaron Foss, Sasha Mitts, Arman Zharmagambetov, Brandon Amos, Xian Li, Justine T Kao, Maryam Fazel-Zarandi, et al. 2024. To the globe (ttg): Towards language- driven guaranteed travel planning. arXiv preprint arXiv:2410.16456. · arXiv
  49. Subbarao Kambhampati. 2024. Can large language models reason and plan? Annals of the New York Academy of Sciences, 1534(1):15-18.
  50. Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Kaya Stechly, Mudit Verma, Siddhant Bham- bri, Lucas Saldyt, and Anil Murthy. 2024. Llms can't plan, but can help planning in llm-modulo frame- works. arXiv preprint arXiv:2402.01817. · arXiv
  51. Nabil A Kartam and David E Wilkins. 1990. Towards a foundation for evaluating ai planners. AI EDAM, 4(1):1-13.
  52. Michael Katz, Harsha Kokel, Kavitha Srinivas, and Shirin Sohrabi. 2024. Thought of search: Planning with language models through the lens of efficiency. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
  53. Jing Yu Koh, Stephen McAleer, Daniel Fried, and Rus- lan Salakhutdinov. 2024. Tree search for language model agents. arXiv preprint arXiv:2407.01476. · arXiv
  54. Kurt Konolige and Nils J Nilsson. 1980. Multiple-agent planning systems. In AAAI, volume 80, pages 138- 142.
  55. Lucas Lehnert, Sainbayar Sukhbaatar, DiJia Su, Qinqing Zheng, Paul Mcvay, Michael Rabbat, and Yuandong Tian. 2024. Beyond a*: Better planning with trans- formers via search dynamics bootstrapping. arXiv preprint arXiv:2402.14083. · arXiv
  56. Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhat- tacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. 2024a. From generation to judgment: Op- portunities and challenges of llm-as-a-judge. arXiv preprint arXiv:2411.16594. · arXiv
  57. Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yu- jia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024b. Llms-as-judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579. · arXiv
  58. Haoming Li, Zhaoliang Chen, Songyuan Liu, Yiming Lu, and Fei Liu. 2024c. Systematic analysis of llm contributions to planning: Solver, verifier, heuristic. Preprint, arXiv:2412.09666. · arXiv
  59. Haoming Li, Zhaoliang Chen, Jonathan Zhang, and Fei Liu. 2024d. Lasp: Surveying the state-of-the-art in large language model-assisted ai planning. arXiv preprint arXiv:2409.01806. · arXiv
  60. Haoming Li, Zhaoliang Chen, Jonathan Zhang, and Fei Liu. 2025. Planet: A collection of benchmarks for evaluating llms' planning capabilities. Preprint, arXiv:2504.14773. · arXiv
  61. Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. 2024e. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36.
  62. Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, et al. 2024f. Embod- ied agent interface: Benchmarking llms for embodied decision making. arXiv preprint arXiv:2410.07166. · arXiv
  63. Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. Api-bank: A comprehensive benchmark for tool-augmented llms. arXiv preprint arXiv:2304.08244. · arXiv
  64. Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clin- ton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, et al. 2022. Pre- trained language models for interactive decision- making. Advances in Neural Information Processing Systems, 35:31199-31212.
  65. Bill Yuchen Lin, Chengsong Huang, Qian Liu, Wenda Gu, Sam Sommerer, and Xiang Ren. 2023a. On grounded planning for embodied tasks with language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13192- 13200.
  66. Fangru Lin, Emanuele La Malfa, Valentin Hofmann, Elle Michelle Yang, Anthony Cohn, and Janet B Pierrehumbert. 2024. Graph-enhanced large lan- guage models in asynchronous plan reasoning. arXiv preprint arXiv:2402.02805. · arXiv
  67. Han Lin, Abhay Zala, Jaemin Cho, and Mohit Bansal. 2023b. Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning. arXiv preprint arXiv:2309.15091. · arXiv
  68. Nir Lipovetzky, Miquel Ramirez, Christian Muise, and Hector Geffner. 2014. Width and inference based planners: Siw, bfs (f), and probe. Proceedings of the 8th International Planning Competition (IPC-2014), page 43.
  69. Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. 2023a. Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.11477. · arXiv
  70. Gang Liu, Michael Sun, Wojciech Matusik, Meng Jiang, and Jie Chen. 2024. Multimodal large language mod- els for inverse molecular design with retrosynthetic planning. arXiv preprint arXiv:2410.04223. · arXiv
  71. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xu- anyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2023b. Agent- bench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688. · arXiv
  72. Yujie Lu, Pan Lu, Zhiyu Chen, Wanrong Zhu, Xin Eric Wang, and William Yang Wang. 2023. Multimodal procedural planning via dual text-image prompting. arXiv preprint arXiv:2305.01795. · arXiv
  73. Amit Moryossef, Yoav Goldberg, and Ido Dagan. 2019. Step-by-step: Separating planning from realization in neural data-to-text generation. arXiv preprint arXiv:1904.03396. · arXiv
  74. Allen Newell, John Calman Shaw, and Herbert A Si- mon. 1958. Elements of a theory of human problem solving. Psychological review, 65(3):151.
  75. Odhran O'Donoghue, Aleksandar Shtedritski, John Gin- ger, Ralph Abboud, Ali Essa Ghareeb, Justin Booth, and Samuel G Rodriques. 2023. Bioplanner: au- tomatic evaluation of llms on protocol planning in biology. arXiv preprint arXiv:2310.10632. · arXiv
  76. Jiefu Ou, Arda Uzunoglu, Benjamin Van Durme, and Daniel Khashabi. 2024. Worldapis: The world is worth how many apis? a thought experiment. arXiv preprint arXiv:2407.07778. · arXiv
  77. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in neural in- formation processing systems, 35:27730-27744.
  78. Jiayi Pan, Glen Chou, and Dmitry Berenson. 2023. Data-efficient learning of natural language to linear temporal logic translators for robot task specification. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11554-11561. IEEE.
  79. Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. 2023. Adapt: As-needed decompo- sition and planning with language models. arXiv preprint arXiv:2311.05772. · arXiv
  80. Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. 2018. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8494-8502.
  81. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  82. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the lim- its of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67.
  83. Shreyas Sundara Raman, Vanya Cohen, Ifrah Idrees, Eric Rosen, Raymond Mooney, Stefanie Tellex, and David Paulius. 2024. Cape: Corrective actions from precondition errors using large language models. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 14070-14077. IEEE.
  84. Shreyas Sundara Raman, Vanya Cohen, Eric Rosen, Ifrah Idrees, David Paulius, and Stefanie Tellex. 2022. Planning with large language models via corrective re-prompting. In NeurIPS 2022 Foundation Models for Decision Making Workshop.
  85. Amirhossein Razavi, Mina Soltangheis, Negar Arabzadeh, Sara Salamat, Morteza Zihayat, and Ebrahim Bagheri. 2025. Benchmarking prompt sensitivity in large language models. Preprint, arXiv:2502.06065. · arXiv
  86. Allen Z Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, et al. 2023. Robots that ask for help: Uncertainty alignment for large language model planners. arXiv preprint arXiv:2307.01928. · arXiv
  87. Jingqing Ruan, Yihong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, Guoqing Du, Shiwei Shi, Hangyu Mao, Ziyue Li, Xingyu Zeng, and Rui Zhao. 2023. Tptu: Large language model-based ai agents for task planning and tool usage. Preprint, arXiv:2308.03427. · arXiv
  88. Abulhair Saparov and He He. 2022. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought. arXiv preprint arXiv:2210.01240. · arXiv
  89. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying language models' sensitiv- ity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. Preprint, arXiv:2310.11324. · arXiv
  90. Pratyusha Sharma, Antonio Torralba, and Jacob An- dreas. 2021. Skill induction and planning with latent language. arXiv preprint arXiv:2110.01517. · arXiv
  91. Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020a. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 10740-10749.
  92. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2020b. Alfworld: Aligning text and em- bodied environments for interactive learning. arXiv preprint arXiv:2010.03768. · arXiv
  93. Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B Tenenbaum, Leslie Kaelbling, and Michael Katz. 2024. Generalized planning in pddl domains with pretrained large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 20256-20264.
  94. Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. 2023. Prog- prompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11523-11530. IEEE.
  95. Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. 2023. Llm-planner: Few-shot grounded planning for em- bodied agents with large language models. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 2998-3009.
  96. Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. 2024. Trial and error: Exploration-based trajectory optimization for llm agents. arXiv preprint arXiv:2403.02502. · arXiv
  97. Siddharth Srivastava, Neil Immerman, and Shlomo Zil- berstein. 2011. A new representation and associated algorithms for generalized planning. Artificial Intel- ligence, 175(2):615-647.
  98. Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. 2024. Adaplanner: Adaptive plan- ning from feedback with language models. Advances in Neural Information Processing Systems, 36.
  99. Marcus Tantakoun, Xiaodan Zhu, and Christian Muise. 2025. Llms as planning modelers: A survey for lever- aging large language models to construct automated planning models. Preprint, arXiv:2503.18971. · arXiv
  100. Alejandro Torreno, Eva Onaindia, Antonín Komenda, and Michal Štolba. 2017. Cooperative multi-agent planning: A survey. ACM Computing Surveys (CSUR), 50(6):1-32.
  101. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. · arXiv
  102. Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2023a. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. Advances in Neural Infor- mation Processing Systems, 36:38975-38987.
  103. Karthik Valmeekam, Matthew Marquez, Sarath Sreed- haran, and Subbarao Kambhampati. 2023b. On the planning abilities of large language models-a criti- cal investigation. Advances in Neural Information Processing Systems, 36:75993-76005.
  104. Karthik Valmeekam, Kaya Stechly, and Subbarao Kamb- hampati. 2024. Llms still can't plan; can lrms? a preliminary evaluation of openai's o1 on planbench. arXiv preprint arXiv:2409.13373. · arXiv
  105. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Man- dlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and An- ima Anandkumar. 2023a. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291. · arXiv
  106. Renxi Wang, Haonan Li, Xudong Han, Yixuan Zhang, and Timothy Baldwin. 2024. Learning from fail- ure: Integrating negative examples when fine-tuning large language models as agents. arXiv preprint arXiv:2402.11651. · arXiv
  107. Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. 2023b. Describe, ex- plain, plan and select: Interactive planning with large language models enables open-worldzhao2024large multi-task agents. arXiv preprint arXiv:2302.01560. · arXiv
  108. Hui Wei, Shenghua He, Tian Xia, Andy Wong, Jingyang Lin, and Mei Han. 2024. Systematic evaluation of llm-as-a-judge in llm alignment tasks: Explainable metrics and diverse prompt templates. arXiv preprint arXiv:2408.13006. · arXiv
  109. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits rea- soning in large language models. Advances in neural information processing systems, 35:24824-24837.
  110. Xixi Wu, Yifei Shen, Caihua Shan, Kaitao Song, Siwei Wang, Bohang Zhang, Jiarui Feng, Hong Cheng, Wei Chen, Yun Xiong, et al. 2024. Can graph learning improve planning in llm-based agents? In The Thirty- eighth Annual Conference on Neural Information Processing Systems.
  111. Yue Wu, Xuan Tang, Tom M Mitchell, and Yuanzhi Li. 2023a. Smartplay: A benchmark for llms as intelligent agents. arXiv preprint arXiv:2310.01557. · arXiv
  112. Zhenyu Wu, Ziwei Wang, Xiuwei Xu, Jiwen Lu, and Haibin Yan. 2023b. Embodied task plan- ning with large language models. arXiv preprint arXiv:2307.01848. · arXiv
  113. Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. 2024. Travelplanner: A benchmark for real- world planning with language agents. arXiv preprint arXiv:2402.01622. · arXiv
  114. Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh. 2023. Translating natural language to planning goals with large-language models. arXiv preprint arXiv:2302.05128. · arXiv
  115. Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. 2023. On the tool manipulation capability of open-source large lan- guage models. arXiv preprint arXiv:2305.16504. · arXiv
  116. Ruihan Yang, Jiangjie Chen, Yikai Zhang, Siyu Yuan, Aili Chen, Kyle Richardson, Yanghua Xiao, and De- qing Yang. 2024. Selfgoal: Your language agents already know how to achieve high-level goals. arXiv preprint arXiv:2406.04784. · arXiv
  117. Ryan Yang, Tom Silver, Aidan Curtis, Tomas Lozano- Perez, and Leslie Pack Kaelbling. 2022. Pg3: Policy- guided planning for generalized policy generation. arXiv preprint arXiv:2204.10420. · arXiv
  118. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36.
  119. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, et al. 2024. Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2410.02736. · arXiv
  120. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingn- ing Yao, Shanelle Roman, et al. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint arXiv:1809.08887. · arXiv
  121. Quan Yuan, Mehran Kazemi, Xin Xu, Isaac Noble, Vaiva Imbrasaite, and Deepak Ramachandran. 2024. Tasklama: probing the complex task understanding of language models. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 38, pages 19468-19476.
  122. Siyu Yuan, Jiangjie Chen, Ziquan Fu, Xuyang Ge, So- ham Shah, Charles Robert Jankowski, Yanghua Xiao, and Deqing Yang. 2023. Distilling script knowledge from large language models for constrained language planning. arXiv preprint arXiv:2305.05252. · arXiv
  123. Abhay Zala, Han Lin, Jaemin Cho, and Mohit Bansal. 2023. Diagrammergpt: Generating open-domain, open-platform diagrams via llm planning. arXiv preprint arXiv:2310.12128. · arXiv
  124. Jianguo Zhang, Tian Lan, Rithesh Murthy, Zhiwei Liu, Weiran Yao, Juntao Tan, Thai Hoang, Liangwei Yang, Yihao Feng, Zuxin Liu, et al. 2024. Agentohana: Design unified data and training pipeline for effective agent learning. arXiv preprint arXiv:2402.15506. · arXiv
  125. Zirui Zhao, Wee Sun Lee, and David Hsu. 2024. Large language models as commonsense knowledge for large-scale task planning. Advances in Neural Infor- mation Processing Systems, 36.
  126. Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng-Tze Cheng, Quoc V Le, Ed H Chi, et al. 2024. Natural plan: Benchmarking llms on natural lan- guage planning. arXiv preprint arXiv:2406.04520. · arXiv
  127. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623.
  128. Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. 2023a. Lan- guage agent tree search unifies reasoning acting and planning in language models. arXiv preprint arXiv:2310.04406. · arXiv
  129. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. 2023b. We- barena: A realistic web environment for building au- tonomous agents. arXiv preprint arXiv:2307.13854. · arXiv
  130. Zhehua Zhou, Jiayang Song, Kunpeng Yao, Zhan Shu, and Lei Ma. 2024. Isr-llm: Iterative self-refined large language model for long-horizon sequential task planning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 2081-2088. IEEE.
  131. Yuchen Zhuang, Xiang Chen, Tong Yu, Saayan Mitra, Victor Bursztyn, Ryan A Rossi, Somdeb Sarkhel, and Chao Zhang. 2023. Toolchain*: Efficient action space navigation in large language models with a* search. arXiv preprint arXiv:2310.13227. · arXiv

Cited by

Sorted by citations

Authors
Wei, H., Zhang, Z., He, S., Xia, T., Pan, S., & Liu, F.
Affiliations
  • University of California, Merced (US)
  • Emory University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
Pages
19497-19521
ISSN
0736-587X (ISSN-L)
DOI
10.48550/arxiv.2502.11221
arXiv
2502.11221
OpenAlex
W4412944859
Cited by
36
References
1
Citations by year
2026: 32 · 2025: 4
FWCI
34.32
Type
conference paper
Language
en
Open access
Licence
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (2303655, CAREER)
  • U.S. Department of Commerce
  • University of California Merced
  • Economic Development Administration
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language models, LLM-based planning, plan generation, completeness, executability, optimality, representation, generalization, efficiency, agentic workflows, web navigation, tourism planning, database queries

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review