CoChat
Sign Up
PreprintCited by 70AI-based Problem Solving and PlanningTopic ModelingLogic, Reasoning, and Knowledge

PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change

Open access
Valmeekam, Karthik, Marquez, Matthew, Olmo, Alberto, Sreedharan, Sarath, Kambhampati, Subbarao·2022·arXiv
DOI 10.48550/arxiv.2206.10498arXiv 2206.10498
Open this paper in CoChat — free accountOpen PDF

Abstract

Generating plans of action, and reasoning about change have long been considered a core competence of intelligent agents. It is thus no surprise that evaluating the planning and reasoning capabilities of large language models (LLMs) has become a hot topic of research. Most claims about LLM planning capabilities are however based on common sense tasks-where it becomes hard to tell whether LLMs are planning or merely retrieving from their vast world knowledge. There is a strong need for systematic and extensible planning benchmarks with sufficient diversity to evaluate whether LLMs have innate planning capabilities. Motivated by this, we propose PlanBench, an extensible benchmark suite based on the kinds of domains used in the automated planning community, especially in the International Planning Competition, to test the capabilities of LLMs in planning or reasoning about actions and change. PlanBench provides sufficient diversity in both the task domains and the specific planning capabilities. Our studies also show that on many critical capabilities-including plan generation-LLM performance falls quite short, even with the SOTA models. PlanBench can thus function as a useful marker of progress of LLMs in planning and reasoning.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2206.10498

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The opaqueness of LLMs coupled with inherent randomness makes it challenging to analyze or control their behaviors.

    “This opaqueness coupled with inherent randomness from their probabilistic formulation - makes it challenging to analyze or control their behaviors (Romero et al., 2023; Valmeekam et al., 2022).”

    From Cognitive Architectures for Language Agents (2023) · cites this paper as Valmeekam et al., 2022

    The Introduction (p. 2) describes a subjective aspect of earlier analyses of LLM reasoning and proposes a mechanistic framework, supporting the idea that analyzing their behavior poses challenges. It does not attribute that challenge to opaqueness or inherent randomness; the experimental details (p. 14) say the authors set temperature to zero to make the tested LLMs deterministic. The paper does not address the claimed difficulty of controlling their behavior. (LLM, conf=0.79)

    Experimental — accuracy not yet measured

  • Vanilla transformers and variants trained for next-token predictions do not learn an explicit world model of their environment that can be flexibly queried and cannot be used for planning or navigation.

    “vanilla transformers and variants trained for next-token predictions (a) do not learn an explicit world model of their environment which can be flexibly queried and (b) cannot be used for planning or navigation.”

    From Learning Cognitive Maps from Transformer Representations for Efficient Planning in Partially Observed Environments (2024) · cites this paper as Valmeekam et al., 2022

    The Introduction (p. 1) describes LLMs as next-word predictors, and the specimen evaluation (p. 9) finds that GPT-4 and InstructGPT3 perform poorly on most Blocksworld tests. The paper does not establish that such models lack a flexibly queryable explicit world model, nor does its limited planning evaluation justify the categorical claim that they cannot be used for planning or navigation. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • LLMs have limited ability to generate and validate plans on their own, even for simple planning tasks.

    “One of the main findings is that LLMs have limited ability to generate and validate plans on their own, even for simple planning tasks (Valmeekam et al., 2022;Silver et al., 2022;Valmeekam et al., 2023b).”

    From Planetarium: A Rigorous Benchmark for Translating Text to Structured Planning Languages (2025) · cites this paper as Valmeekam et al., 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 31

  1. [1][1] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691 , 2022. - · arXiv
  2. [2][2] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022. - · arXiv
  3. [3][3] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021. - · arXiv
  4. [4][4] Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. GLaM: Efficient Scaling of Language Models with Mixture-of-Experts. pages 5547-5569, 2022. -
  5. [5][5] Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics , 9:346-361, 2021. -
  6. [6][6] Malik Ghallab, Dana S. Nau, and Paolo Traverso. Automated Planning and Acting . Cambridge University Press, 2016. -
  7. [7][7] Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati. Leveraging pre-trained large language models to construct and utilize world models for model-based task planning. arXiv preprint arXiv:2305.14909 , 2023. - · arXiv
  8. [8][8] Malte Helmert. The fast downward planning system. Journal of Artificial Intelligence Research , 26:191-246, 2006. -
  9. [9][9] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022. - · arXiv
  10. [10][10] Richard Howey, Derek Long, and Maria Fox. VAL: Automatic plan validation, continuous effects and mixed initiative planning using PDDL. In 16th IEEE International Conference on Tools with Artificial Intelligence , pages 294-301. IEEE, 2004. -
  11. [11][11] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning , pages 9118-9147. PMLR, 2022. -
  12. [12][12] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608 , 2022. - · arXiv
  13. [13][13] IPC. International planning competition. https://www.icaps-conference.org/competitions, 1998. -
  14. [14][14] Liwei Jiang, Jena D. Hwang, Chandrasekhar Bhagavatula, Ronan Le Bras, Maxwell Forbes, Jon Borchardt, Jenny Liang, Oren Etzioni, Maarten Sap, and Yejin Choi. Delphi: Towards Machine Ethics and Norms. ArXiv , abs/2110.07574, 2021. - · arXiv
  15. [16][16] Subbarao Kambhampati, Karthik Valmeekam, Matthew Marquez, and Lin Guan. On the role of large language models in planning, July 2023. Tutorial presented at the International Conference on Automated Planning and Scheduling (ICAPS), Prague. https://yochan-lab.github. io/tutorial/ICAPS-2023/ . -
  16. [17][17] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large Language Models are Zero-Shot Reasoners. arXiv preprint arXiv:2205.11916 , 2022. - · arXiv
  17. [18][18] Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146 , 2017. - · arXiv
  18. [22][22] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 , 2022. - · arXiv
  19. [23][23] Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP Models really able to Solve Simple Math Word Problems? arXiv preprint arXiv:2103.07191 , 2021. - · arXiv
  20. [24][24] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446 , 2021. - · arXiv
  21. [25][25] Shreyas Sundara Raman, Vanya Cohen, Eric Rosen, Ifrah Idrees, David Paulius, and Stefanie Tellex. Planning with large language models via corrective re-prompting. arXiv preprint arXiv:2211.09935 , 2022. - · arXiv
  22. [27][27] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 8732-8740, 2020. -
  23. [28][28] Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201.11990 , 2022. - · arXiv
  24. [29][29] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 , 2022. - · arXiv
  25. [30][30] Siddharth Srivastava, Shlomo Zilberstein, Neil Immerman, and Hector Geffner. Qualitative numeric planning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 25, pages 1010-1016, 2011. -
  26. [31][31] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937 , 2018. - · arXiv
  27. [32][32] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, HengTze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. LaMDA: Language Models for Dialog Applications. arXiv preprint arXiv:2201.08239 , 2022. - · arXiv
  28. [33][33] Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models-a critical investigation. arXiv preprint arXiv:2305.15771 , 2023. - · arXiv
  29. [34][34] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 , 2022. - · arXiv
  30. [35][35] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations , 2023. -
  31. [36][36] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022. · arXiv

Cited by

Sorted by citations

Authors
Valmeekam, K., Marquez, M., Olmo, A., Sreedharan, S., & Kambhampati, S.
ORCID
Sreedharan, Sarath
Venue
arXiv
Publisher
arXiv
Published
2022
Pages
38975-38987
ISSN
1049-5258 (ISSN-L)
DOI
10.48550/arxiv.2206.10498
arXiv
2206.10498
OpenAlex
W7133237354
Cited by
70
References
0
Citations by year
2026: 18 · 2025: 12 · 2024: 24 · 2023: 16
FWCI
12.40
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (2303019, W911NF-19)
  • Defense Advanced Research Projects Agency (W911NF19-2-0006)
  • Office of Naval Research (N00014-18, N00014-18-1-2442, N0001423‐12409, N00014-18-12840, N00014-19, FA9550, N00014-19-1-2119, N00014)
  • Air Force Office of Scientific Research (FA9550-, FA9550‐18‐1‐0067)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
Keywords
large language models, planning, reasoning about change, benchmark, temporal reasoning, extensibility

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review