The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.
On the Planning Abilities of Large Language Models - A Critical Investigation
Abstract
Intrigued by the claims of emergent reasoning capabilities in LLMs trained on general web corpora, in this paper, we set out to investigate their planning capabilities. We aim to evaluate (1) the effectiveness of LLMs in generating plans autonomously in commonsense planning tasks and (2) the potential of LLMs in LLM-Modulo settings where they act as a source of heuristic guidance for external planners and verifiers. We conduct a systematic study by generating a suite of instances on domains similar to the ones employed in the International Planning Competition and evaluate LLMs in two distinct modes: autonomous and heuristic. Our findings reveal that LLMs' ability to generate executable plans autonomously is rather limited, with the best model (GPT-4) having an average success rate of ~12% across the domains. However, the results in the LLM-Modulo setting show more promise. In the LLM-Modulo setting, we demonstrate that LLM-generated plans can improve the search process for underlying sound planners and additionally show that external verifiers can help provide feedback on the generated plans and back-prompt the LLM for better plan generation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs may not perform well in commonsense planning tasks, even in simple planning tasks where humans excel.
“In commonsense planning tasks, LLMs may not perform well, even in simple planning tasks where humans excel [194, 195].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [194]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Inner Monologue: Embodied Reasoning through Planning with Language Models
Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P., Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2207.05608 · Cited by 287 · cited 1×
“One other mode of evaluation of planning capabilities in the literature involves the user incrementally interacting with the LLM, and re-prompting it to point out flaws in its plans, with the hope that the LLM eventually reaches an executable plan [13, 39, 28].”
Background
Flagged first, then by citations
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
Can large language models reason and plan?
Kambhampati, S. (2024) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15125 · Cited by 83
Self-citationLLexus: an AI agent system for incident management
Las-Casas, P., Kumbhare, A. G., Fonseca, R., & Agarwal, S. (2024) · ACM SIGOPS Operating Systems Review · doi:10.1145/3689051.3689056 · Cited by 14
LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks
Kambhampati, S., Valmeekam, K., Guan, L., Verma, M., Stechly, K., Bhambri, S., Saldyt, L., & Murthy, A. (2024) · arXiv · doi:10.48550/arxiv.2402.01817 · Cited by 237
“For example, in Valmeekam et al. (2023c,b), the authors evaluate LLMs ability to generate correct plans on a suite of planning problem instances based on the kinds of domains employed in the International Planning Competition IPC (1998).”
BackgroundMethodResultInfluentialSelf-citationAgarwal, R., Singh, A., Zhang, L. M., Bohnet, B., Rosias, L., Chan, S., Zhang, B., Anand, A., Abbas, Z., Nova, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.11018 · Cited by 35
“As a reference, we report 1-shot GPT-4 results from Valmeekam et al. [58].”
Collaborative Large Language Models for Automated Data Extraction in Living Systematic Reviews
Khan, M. A., Ayub, U., Naqvi, S. A. A., Khakwani, K. Z. R., Sipra, Z. B. R., Raina, A., Zou, S., He, H., Hossein, S. A., Hasan, B., et al. (2024) · doi:10.1101/2024.09.20.24314108 · Cited by 54
“[56 57] Moreover, a key future direction involves further exploring the reasoning chains of LLMs, which could not only enhance confidence in their use but also improve trackability throughout the data extraction process.”
BackgroundCan Large Language Models Really Improve by Self-critiquing Their Own Plans?
Valmeekam, K., Marquez, M., & Kambhampati, S. (2023) · arXiv · doi:10.48550/arxiv.2310.08118 · Cited by 90
Self-citationLarge Language Models as Biomedical Hypothesis Generators: A Comprehensive Evaluation
Qi, B., Zhang, K., Tian, K., Li, H., Chen, Z.-R., Zeng, S., Hua, E., Jinfang, H., & Zhou, B. (2024) · arXiv · doi:10.48550/arxiv.2407.08940 · Cited by 5
“These agents are equipped with reasoning (Wei et al., 2023a; Yao et al., 2023a), planning (Shen et al., 2023; Valmeekam et al., 2023), decision-making (Yang et al., 2023a; Kang et al., 2023), and long-term memory capabilities (Zhu et al., 2023a; Hu et al., 2023), and they are constructed upon the…”
BackgroundPlanetarium: A Rigorous Benchmark for Translating Text to Structured Planning Languages
Zuo, M., Velez, F. P., Li, X., Littman, M., & Bach, S. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.560 · Cited by 5
“One of the main findings is that LLMs have limited ability to generate and validate plans on their own, even for simple planning tasks (Valmeekam et al., 2022; Silver et al., 2022; Valmeekam et al., 2023b).”
On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks
Stechly, K., Valmeekam, K., & Kambhampati, S. (2024) · arXiv · doi:10.48550/arxiv.2402.08115 · Cited by 120
“…anecdotal accounts of performance on reasoning tasks [4] spurred much research into the capabilities of LLMs on various reasoning domains, from a menagerie of benchmarks covering basic problems [19] to planning [32], logic and arithmetic [9], analogical reasoning [33], and even math puzzles [37].”
BackgroundSelf-citationAutoHarness: improving LLM agents by automatically synthesizing a code harness
Lou, X., Lázaro-Gredilla, M., Dedieu, A., Wendelken, C., Lehrach, W., & Murphy, K. P. (2026) · arXiv · doi:10.48550/arxiv.2603.03329
“However, their planning and reasoning performance can be brittle (see e.g., (Valmeekam et al., 2023a; Petrov et al., 2025).”
Metadata
- Authors
- Valmeekam, K., Marquez, M., Sreedharan, S., & Kambhampati, S.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- arXiv
- 2305.15771
- References
- 1
- Type
- Preprint
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
