CoChat
Sign Up
Cited by 80AI-based Problem Solving and PlanningTopic ModelingLogic, Reasoning, and Knowledge

PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change

Open access
Valmeekam, Karthik, Marquez, Matthew, Olmo, Alberto, Sreedharan, Sarath, Kambhampati, Subbarao·2023·Advances in Neural Information Processing Systems 36
DOI 10.52202/075280-1693
Open this paper in CoChat — free accountOpen PDF

Abstract

Generating plans of action, and reasoning about change have long been considered a core competence of intelligent agents. It is thus no surprise that evaluating the planning and reasoning capabilities of large language models (LLMs) has become a hot topic of research. Most claims about LLM planning capabilities are however based on common sense tasks-where it becomes hard to tell whether LLMs are planning or merely retrieving from their vast world knowledge. There is a strong need for systematic and extensible planning benchmarks with sufficient diversity to evaluate whether LLMs have innate planning capabilities. Motivated by this, we propose PlanBench, an extensible benchmark suite based on the kinds of domains used in the automated planning community, especially in the International Planning Competition, to test the capabilities of LLMs in planning or reasoning about actions and change. PlanBench provides sufficient diversity in both the task domains and the specific planning capabilities. Our studies also show that on many critical capabilities-including plan generation-LLM performance falls quite short, even with the SOTA models. PlanBench can thus function as a useful marker of progress of LLMs in planning and reasoning.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.52202/075280-1693

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • PlanBench provides an extensible framework for evaluating LLMs on planning and reasoning about change but is limited to generating action sequences without considering other crucial aspects of embodied decision-making.

    “PlanBench [99] provides an extensible framework for evaluating LLMs on planning and reasoning about change but is limited to generating action sequences without considering other crucial aspects of embodied decision-making.”

    From Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making (2024) · cites this paper as [99]

    The Abstract (p. 1) supports the description of PlanBench as an extensible benchmark for planning and reasoning about change. But §5, “Current Curriculum for Testing” (p. 7), explicitly includes plan verification and predicting the state after executing actions—not just generating action sequences. The paper does not claim to evaluate every aspect of embodied decision-making, but its stated scope contradicts the claimed action-sequence-only limitation. (LLM, conf=0.89)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Works claiming planning capabilities for LLMs either work in domains where subgoal interactions can be safely ignored or delegate interaction resolution to humans in the loop.

    “our close examination of several works claiming planning capabilities for LLMs (Kambhampati et al., 2023) suggests that they either work in domains/tasks where subgoal interactions can be safely ignored”

    From LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks (2024) · cites this paper as Kambhampati et al., 2023

    The Abstract (p. 1) says most claims about LLM planning capabilities are based on commonsense tasks, and Related Work (p. 2) describes some reasoning tasks as shallow and notes systems that use user feedback. The paper does not say that subgoal interactions can safely be ignored in those tasks or that humans resolve those interactions for the LLMs. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

  • Reasoning tests that break common co-occurrence patterns in the input or require multi-step operations lead to model failure.

    “Reasoning tests that break common co-occurrence patterns in the input or require multi-step operations also lead to model failure [144, 145].”

    From Dissociating language and thought in large language models (2024) · cites this paper as [144]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 31

  1. [1][1] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691 , 2022. - · arXiv
  2. [2][2] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022. - · arXiv
  3. [3][3] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021. - · arXiv
  4. [4][4] Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. GLaM: Efficient Scaling of Language Models with Mixture-of-Experts. pages 5547-5569, 2022. -
  5. [5][5] Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics , 9:346-361, 2021. -
  6. [6][6] Malik Ghallab, Dana S. Nau, and Paolo Traverso. Automated Planning and Acting . Cambridge University Press, 2016. -
  7. [7][7] Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati. Leveraging pre-trained large language models to construct and utilize world models for model-based task planning. arXiv preprint arXiv:2305.14909 , 2023. - · arXiv
  8. [8][8] Malte Helmert. The fast downward planning system. Journal of Artificial Intelligence Research , 26:191-246, 2006. -
  9. [9][9] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022. - · arXiv
  10. [10][10] Richard Howey, Derek Long, and Maria Fox. VAL: Automatic plan validation, continuous effects and mixed initiative planning using PDDL. In 16th IEEE International Conference on Tools with Artificial Intelligence , pages 294-301. IEEE, 2004. -
  11. [11][11] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning , pages 9118-9147. PMLR, 2022. -
  12. [12][12] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608 , 2022. - · arXiv
  13. [13][13] IPC. International planning competition. https://www.icaps-conference.org/competitions, 1998. -
  14. [14][14] Liwei Jiang, Jena D. Hwang, Chandrasekhar Bhagavatula, Ronan Le Bras, Maxwell Forbes, Jon Borchardt, Jenny Liang, Oren Etzioni, Maarten Sap, and Yejin Choi. Delphi: Towards Machine Ethics and Norms. ArXiv , abs/2110.07574, 2021. - · arXiv
  15. [16][16] Subbarao Kambhampati, Karthik Valmeekam, Matthew Marquez, and Lin Guan. On the role of large language models in planning, July 2023. Tutorial presented at the International Conference on Automated Planning and Scheduling (ICAPS), Prague. https://yochan-lab.github. io/tutorial/ICAPS-2023/ . -
  16. [17][17] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large Language Models are Zero-Shot Reasoners. arXiv preprint arXiv:2205.11916 , 2022. - · arXiv
  17. [18][18] Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146 , 2017. - · arXiv
  18. [22][22] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 , 2022. - · arXiv
  19. [23][23] Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP Models really able to Solve Simple Math Word Problems? arXiv preprint arXiv:2103.07191 , 2021. - · arXiv
  20. [24][24] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446 , 2021. - · arXiv
  21. [25][25] Shreyas Sundara Raman, Vanya Cohen, Eric Rosen, Ifrah Idrees, David Paulius, and Stefanie Tellex. Planning with large language models via corrective re-prompting. arXiv preprint arXiv:2211.09935 , 2022. - · arXiv
  22. [27][27] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 8732-8740, 2020. -
  23. [28][28] Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201.11990 , 2022. - · arXiv
  24. [29][29] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 , 2022. - · arXiv
  25. [30][30] Siddharth Srivastava, Shlomo Zilberstein, Neil Immerman, and Hector Geffner. Qualitative numeric planning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 25, pages 1010-1016, 2011. -
  26. [31][31] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937 , 2018. - · arXiv
  27. [32][32] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, HengTze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. LaMDA: Language Models for Dialog Applications. arXiv preprint arXiv:2201.08239 , 2022. - · arXiv
  28. [33][33] Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models-a critical investigation. arXiv preprint arXiv:2305.15771 , 2023. - · arXiv
  29. [34][34] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 , 2022. - · arXiv
  30. [35][35] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations , 2023. -
  31. [36][36] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022. · arXiv

Cited by

Sorted by citations

Related

Authors
Valmeekam, K., Marquez, M., Olmo, A., Sreedharan, S., & Kambhampati, S.
ORCID
Sreedharan, Sarath
Venue
Advances in Neural Information Processing Systems 36
Publisher
Neural Information Processing Systems Foundation, Inc. (NeurIPS)
Published
2023
Pages
38975-38987
ISSN
1049-5258 (ISSN-L)
DOI
10.52202/075280-1693
OpenAlex
W7133237354
Cited by
70
References
0
Citations by year
2026: 18 · 2025: 12 · 2024: 24 · 2023: 16
FWCI
12.40
Type
conference paper
Language
en
Open access
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (2303019, W911NF-19)
  • Defense Advanced Research Projects Agency (W911NF19-2-0006)
  • Office of Naval Research (N00014-18, N00014-18-1-2442, N0001423‐12409, N00014-18-12840, N00014-19, FA9550, N00014-19-1-2119, N00014)
  • Air Force Office of Scientific Research (FA9550-, FA9550‐18‐1‐0067)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
Keywords
large language models, planning, reasoning about change, benchmark, temporal reasoning, extensibility

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review