Cited by 87Topic ModelingAI-based Problem Solving and PlanningNatural Language Processing Techniques

Can large language models reason and plan?

Open access
Kambhampati, Subbarao·2024·Annals of the New York Academy of Sciences

Abstract

While humans sometimes do show the capability of correcting their own erroneous guesses with self-critiquing, there seems to be no basis for that assumption in the case of LLMs.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1111/nyas.15125

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The parameters of modern LLMs encode a remarkable amount of world knowledge.

    “It is well-accepted, even among skeptical researchers, that the parameters of modern LLMs encode a remarkable amount of world knowledge [2, 12, 23, 28, 31, 39, 62, 79], which can be leveraged by an LLM in flexible and robust ways [81, 83, 93].”
  • There is a considerable body of work suggesting that the reasoning process in LLMs is fragile and not formal.

    “There is a considerable body of work suggesting that the reasoning process in LLMs is fragile and not formal (Kambhampati, 2024; Valmeekam et al., 2022; 2024; Nezhurina et al., 2024; McCoy et al., 2023; Zhang et al., 2023), even though it appears that these models understand symbols and can work with them to some limited degree (Boix-Adserà et al., 2024).”
  • AGI represents visionary goal of crafting AI systems that emulate comprehensive and multifaceted aspects of human cognition, focused on developing AI with capability for holistic understanding and complex reasoning closely aligning with depth and breadth of human cognitive abilities.

    “AGI, representing the visionary goal of crafting AI systems that emulate the comprehensive and multifaceted aspects of human cognition, is a subdomain focused on developing AI with the capability for holistic understanding and complex reasoning that closely aligns with the depth and breadth of human cognitive abilities [282], [283], [32].”
  • AGI Safety and Containment acknowledges potential risks associated with highly advanced AI systems, focused on ensuring these systems are technically proficient and ethically aligned with human values and societal norms.

    “AGI Safety and Containment acknowledges the potential risks associated with highly advanced AI systems, focused on ensuring that these advanced systems are not only technically proficient but also ethically aligned with human values and societal norms [15], [32], [11].”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 6

  1. Kambhampati, S. (2022). AI as (an ersatz) natural science? Communica- tions of the ACM, 65(9), 8-9.
  2. Valmeekam, K., Olmo, A., Sreedharan, S., & Kambhampati, S. (2022). Large language models still can't plan (a benchmark for LLMs on planning and reasoning about change). arXiv preprint arXiv:2206.104 98.
  3. Kambhampati, S., Valmeekam, K., Guan, L., Stechly, K., Verma, M., Bhambri, S., Saldyt, L., & Murthy, A. (2024). LLMs can't plan, but can help planning in LLM-Modulo frameworks. arXiv preprint 2402. 01817.
  4. Valmeekam, K., Marquez, M., & Kambhampati, S. (2023). Can large language models really improve by self-critiquing their own plans? In NeurIPS 2023 Foundation Models for Decision Making Workshop.
  5. Stechly, K., Marquez, M., & Kambhampati, S. (2023). GPT-4 Doesn't Know It's Wrong: An Analysis of Iterative Prompting for Reasoning Problems. In NeurIPS 2023 Foundation Models for Decision Making Workshop.
  6. Kambhampati, S., Valmeekam, K., Marquez, M., & Guan, L. (2023). On the role of large language models in planning. Tutorial presented at the International Conference on Automated Planning and Scheduling (ICAPS), Prague.

Cited by

Sorted by citations

Authors
Kambhampati, S.
Affiliations
School of Computing & Augmented Intelligence Arizona State University
Venue
Annals of the New York Academy of Sciences
Publisher
Wiley
Published
2024-03-06
Volume
1534
Issue
1
Pages
15-18
ISSN
  • 0077-8923 (ISSN-L)
  • 1749-6632
OpenAlex
W4392517661
Cited by
83
References
6
Citations by year
2026: 36 · 2025: 37 · 2024: 10
FWCI
12.30
Type
journal article
Language
en
Open access
Locations
  • Annals of the New York Academy of Sciences (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
large language models, reasoning, planning, self-criticism, error correction, metacognition
MeSH terms
Artificial Intelligence, Humans, Language

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review