Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index; none of the 30 sources its claims cite is checked for support yet.

Evaluating Cognitive Maps and Planning in Large Language Models with CogEval

Open access
Momennejad, Ida, Hasanbeig, Hosein, Frujeri, Felipe Vieira, Sharma, Hiteshi, Jojic, Nebojsa, Palangi, Hamid, Ness, Robert, Larson, Jonathan · 2023 · Advances in Neural Information Processing Systems 36

Abstract

Recently an influx of studies claim emergent cognitive abilities in large language models (LLMs). Yet, most rely on anecdotes, overlook contamination of training sets, or lack systematic Evaluation involving multiple tasks, control conditions, multiple iterations, and statistical robustness tests. Here we make two major contributions. First, we propose CogEval, a cognitive science-inspired protocol for the systematic evaluation of cognitive capacities in Large Language Models. The CogEval protocol can be followed for the evaluation of various abilities. Second, here we follow CogEval to systematically evaluate cognitive maps and planning ability across eight LLMs (OpenAI GPT-4, GPT-3.5-turbo-175B, davinci-003-175B, Google Bard, Cohere-xlarge-52.4B, Anthropic Claude-1-52B, LLaMA-13B, and Alpaca-7B). We base our task prompts on human experiments, which offer both established construct validity for evaluating planning, and are absent from LLM training sets. We find that, while LLMs show apparent competence in a few planning tasks with simpler structures, systematic evaluation reveals striking failure modes in planning tasks, including hallucinations of invalid trajectories and getting trapped in loops. These findings do not support the idea of emergent out-of-the-box planning ability in LLMs. This could be because LLMs do not understand the latent relational structures underlying planning problems, known as cognitive maps, and fail at unrolling goal-directed trajectories based on the underlying structure. Implications for application and future directions are discussed.

Overview

CoChat Summary

TL;DR

The paper introduces CogEval, a cognitive-science-inspired protocol, and applies it to evaluate cognitive maps and planning across eight LLMs. It finds apparent competence on simple tasks but systematic failures on planning tasks with denser graph structures, including hallucinations, suboptimally long routes, and loops, arguing against emergent out-of-the-box planning.

Key findings

  • Across LLMs, performance varied significantly by LLM (χ²=2357.87, p<.001), graph (χ²=3431.53, p<.001), condition (χ²=2080.04, p<.001), and domain (χ²=304.06, p<.001); temperature was non-significant (χ²=1.28, p=.53), as was the LLM×temperature interaction (χ²=10.69, p=.71).
  • Three main failure modes appeared on dense community graphs: hallucinating edges that do not exist, falling into longer trajectories instead of shortest paths, and getting trapped in loops. For finding the shortest path to a state 1 cluster away, GPT-4 had a success rate of 0 at temperature 0 out of 30 runs; even at temperatures 0.5 or 1, success did not exceed 10%.
  • Experiment 2 found that BFS/DFS Chain-of-Thought instructions improved some complex community-graph tasks but effects were inconsistent and non-monotonic: cluster counting on graph D improved at temperatures 0 and 0.5 but less at 1; shortest paths within a cluster improved across all temperatures with BFS or DFS, with BFS most effective at temperature 0; for shortest path 1-cluster away, only BFS at temperature 0.5 improved performance.
  • LLMs could list pairwise (state, action, state) tuples but failed to use them for adaptive planning; apparent competence in simpler tasks may reflect route memorization or cached routes rather than understanding cognitive maps.

Methods

  • CogEval operationalized cognitive maps and planning with tasks adapted from non-linguistic human experiments, varying graph structure (six graphs based on human experiments), domain (spatial ordered/unordered, social ties, object relations), and task conditions (valuePath, 1/2/3/n-step paths, reward/policy/transition revaluation, shortcut, detour, with/without teleportation and CoT).
  • Evaluated LLMs: GPT-4, GPT-3.5-turbo-175B, davinci-003-175B, Bard, Cohere-xlarge-52.4B, Claude-1-52B, LLaMA-13B, and Alpaca-7B. Responses were generated 30 times per task prompt and temperature for the three OpenAI models and once per task and temperature for other LLMs, across temperatures 0, 0.5, and 1.
  • Main analysis used logistic regression with domain, condition, and graph as categorical regressors, including second- and third-order interactions, with model version and temperature as separate independent variables; results were summarized in an analysis of deviance table. Experiment 2 tested BFS/DFS CoT instructions on GPT-4 with 30 repeats per prompt and temperature.

Limitations

  • The paper states it lacks knowledge of LLMs like GPT-4’s architecture or training, so it avoided existing text-based benchmarks and generated novel prompts not in training sets.
  • The paper states that human experiments involve gradual learning, whereas LLMs were presented the environment’s structure in linguistic format; it argues this difference is sufficiently addressed but acknowledges it.
  • Evident from the text: only the three OpenAI models had 30 repeats per task and temperature, while other LLMs were evaluated once per task and temperature due to lack of API access; the paper also notes it could not set a seed in these LLMs.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.52202/075280-3056

Cites

55 of 55 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Image 1
CC-BY-NC-ND · From the PDF
Image 2
CC-BY-NC-ND · From the PDF
VI. Drawing a map.
VI. Drawing a map.CC-BY-NC-ND · From the PDF
Figure 2 Experiment 1 results. (top) Mean and standard error of performance on all tasks for each of the different graphs (see Figure 1 for graph details) across different LLMs studied in this work. (bottom) Mean performance compared across per main task category (see Table 3 for details).
Figure 2 Experiment 1 results. (top) Mean and standard error of performance on all tasks for each of the different graphs (see Figure 1 for graph details) across different LLMs studied in this work. (bottom) Mean performance compared across per main task category (see Table 3 for details).CC-BY-NC-ND · From the PDF
Image 5
CC-BY-NC-ND · From the PDF
Figure 3 Experiment 2 results. (Bottom) BFS and DFS instructions marginally enhance performance on community graphs. In the Cluster counting task (graph D) adding BFS or DFS is beneficial at temperatures 0 and 0.5 but less at 1. For finding shortest paths within a cluster, BFS or DFS help with BFS being effective at temperature 0. However, for finding the shortest path 1-cluster away, only BFS at temperature 0.5 yields slight improvements.
Figure 3 Experiment 2 results. (Bottom) BFS and DFS instructions marginally enhance performance on community graphs. In the Cluster counting task (graph D) adding BFS or DFS is beneficial at temperatures 0 and 0.5 but less at 1. For finding shortest paths within a cluster, BFS or DFS help with BFS being effective at temperature 0. However, for finding the shortest path 1-cluster away, only BFS at temperature 0.5 yields slight improvements.CC-BY-NC-ND · From the PDF
Figure 4 Examples of three failure modes. (left) Edge hallucination. (middle) Failure at finding a 1-step policy within the same cluster. (right) Failure at multi-hop path by both getting trapped in a loop and hallucinating edges. In each example the blue box is the task prompt , the grey box shows the model response , and the green arrows demonstrate the correct response on the graph.
Figure 4 Examples of three failure modes. (left) Edge hallucination. (middle) Failure at finding a 1-step policy within the same cluster. (right) Failure at multi-hop path by both getting trapped in a loop and hallucinating edges. In each example the blue box is the task prompt , the grey box shows the model response , and the green arrows demonstrate the correct response on the graph.CC-BY-NC-ND · From the PDF
Image 8
CC-BY-NC-ND · From the PDF
x axis: The graph's block density
x axis: The graph's block densityCC-BY-NC-ND · From the PDF
Image 10
CC-BY-NC-ND · From the PDF
Figure 4 Results of systematic investigation of line graphs to full-clique structures. TopP = 0.5.
Figure 4 Results of systematic investigation of line graphs to full-clique structures. TopP = 0.5.CC-BY-NC-ND · From the PDF
Figure 5 Results of systematic investigation of line graphs to full-clique structures. TopP = 0.95.
Figure 5 Results of systematic investigation of line graphs to full-clique structures. TopP = 0.95.CC-BY-NC-ND · From the PDF

Verification

Claims the paper makes

0 Valid · 0 Not valid · 12 Not checked · 87 Own results and interpretations

41 more claims

Create a free account to see all 99 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 30
  • The quote only partly states this claim. 2
  • Probably not a substantial claim. 27

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

4 of 4

Citations

Cites

Part of this paper’s reference list could not be split into single entries.

Show all 12 references

Research works printed but not linked: 8

  1. The nobel prize in physiology or medicine 2014. https://www.nobelprize.org/prizes/ medicine/2014/press-release/. Accessed: 2023-5-10.
  2. Rosa Cao. Multiple realizability and the spirit of functionalism. Synthese, 200(6):506, December 2022.
  3. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8086-8098, 2022.
  4. Ida Momennejad. A rubric for human-like agents and NeuroAI. Philos. Trans. R. Soc. Lond. B Biol. Sci., 378(1869):20210446, December 2022.
  5. J O'Keefe. Place units in the hippocampus of the freely moving rat. Exp. Neurol., 51(1):78-109, April 1976.
  6. K L Stachenfeld, M Botvinick, and others. Design principles of the hippocampal cognitive map. Adv. Neural Inf. Process. Syst., 2014.
  7. Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press, November 2018.
  8. E C Tolman. Cognitive maps in rats and men. Psychol. Rev., 55(4):189-208, July 1948.

Cited by

See all 60

Metadata

Authors
Momennejad, I., Hasanbeig, H., Frujeri, F. V., Sharma, H., Jojic, N., Palangi, H., Ness, R., & Larson, J.
Venue
Advances in Neural Information Processing Systems 36
Published
2023
Pages
69736-69751
ISSN
1049-5258 (ISSN-L)
OpenAlex
W7133238287
Cited by
27
References
12
Citations by year
2026: 5 · 2025: 8 · 2024: 14
FWCI
5.10
Type
conference paper
Language
en
Open access
Licence
cc-by-nc-nd
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
large language models, cognitive maps, planning, evaluation benchmark

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Evaluating Cognitive Maps and Planning in Large Language Models with CogEval. https://app.cochat.ai/papers/10.52202/075280-3056
The paper
Momennejad, I., Hasanbeig, H., Frujeri, F. V., Sharma, H., Jojic, N., Palangi, H., Ness, R., & Larson, J. (2023). Evaluating Cognitive Maps and Planning in Large Language Models with CogEval. Advances in Neural Information Processing Systems 36. https://doi.org/10.52202/075280-3056

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.