The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index; none of the 30 sources its claims cite is checked for support yet.
Evaluating Cognitive Maps and Planning in Large Language Models with CogEval
Open accessAbstract
Recently an influx of studies claim emergent cognitive abilities in large language models (LLMs). Yet, most rely on anecdotes, overlook contamination of training sets, or lack systematic Evaluation involving multiple tasks, control conditions, multiple iterations, and statistical robustness tests. Here we make two major contributions. First, we propose CogEval, a cognitive science-inspired protocol for the systematic evaluation of cognitive capacities in Large Language Models. The CogEval protocol can be followed for the evaluation of various abilities. Second, here we follow CogEval to systematically evaluate cognitive maps and planning ability across eight LLMs (OpenAI GPT-4, GPT-3.5-turbo-175B, davinci-003-175B, Google Bard, Cohere-xlarge-52.4B, Anthropic Claude-1-52B, LLaMA-13B, and Alpaca-7B). We base our task prompts on human experiments, which offer both established construct validity for evaluating planning, and are absent from LLM training sets. We find that, while LLMs show apparent competence in a few planning tasks with simpler structures, systematic evaluation reveals striking failure modes in planning tasks, including hallucinations of invalid trajectories and getting trapped in loops. These findings do not support the idea of emergent out-of-the-box planning ability in LLMs. This could be because LLMs do not understand the latent relational structures underlying planning problems, known as cognitive maps, and fail at unrolling goal-directed trajectories based on the underlying structure. Implications for application and future directions are discussed.
Overview
CoChat Summary
TL;DR
The paper introduces CogEval, a cognitive-science-inspired protocol, and applies it to evaluate cognitive maps and planning across eight LLMs. It finds apparent competence on simple tasks but systematic failures on planning tasks with denser graph structures, including hallucinations, suboptimally long routes, and loops, arguing against emergent out-of-the-box planning.
Key findings
- Across LLMs, performance varied significantly by LLM (χ²=2357.87, p<.001), graph (χ²=3431.53, p<.001), condition (χ²=2080.04, p<.001), and domain (χ²=304.06, p<.001); temperature was non-significant (χ²=1.28, p=.53), as was the LLM×temperature interaction (χ²=10.69, p=.71).
- Three main failure modes appeared on dense community graphs: hallucinating edges that do not exist, falling into longer trajectories instead of shortest paths, and getting trapped in loops. For finding the shortest path to a state 1 cluster away, GPT-4 had a success rate of 0 at temperature 0 out of 30 runs; even at temperatures 0.5 or 1, success did not exceed 10%.
- Experiment 2 found that BFS/DFS Chain-of-Thought instructions improved some complex community-graph tasks but effects were inconsistent and non-monotonic: cluster counting on graph D improved at temperatures 0 and 0.5 but less at 1; shortest paths within a cluster improved across all temperatures with BFS or DFS, with BFS most effective at temperature 0; for shortest path 1-cluster away, only BFS at temperature 0.5 improved performance.
- LLMs could list pairwise (state, action, state) tuples but failed to use them for adaptive planning; apparent competence in simpler tasks may reflect route memorization or cached routes rather than understanding cognitive maps.
Methods
- CogEval operationalized cognitive maps and planning with tasks adapted from non-linguistic human experiments, varying graph structure (six graphs based on human experiments), domain (spatial ordered/unordered, social ties, object relations), and task conditions (valuePath, 1/2/3/n-step paths, reward/policy/transition revaluation, shortcut, detour, with/without teleportation and CoT).
- Evaluated LLMs: GPT-4, GPT-3.5-turbo-175B, davinci-003-175B, Bard, Cohere-xlarge-52.4B, Claude-1-52B, LLaMA-13B, and Alpaca-7B. Responses were generated 30 times per task prompt and temperature for the three OpenAI models and once per task and temperature for other LLMs, across temperatures 0, 0.5, and 1.
- Main analysis used logistic regression with domain, condition, and graph as categorical regressors, including second- and third-order interactions, with model version and temperature as separate independent variables; results were summarized in an analysis of deviance table. Experiment 2 tested BFS/DFS CoT instructions on GPT-4 with 30 repeats per prompt and temperature.
Limitations
- The paper states it lacks knowledge of LLMs like GPT-4’s architecture or training, so it avoided existing text-based benchmarks and generated novel prompts not in training sets.
- The paper states that human experiments involve gradual learning, whereas LLMs were presented the environment’s structure in linguistic format; it argues this difference is sufficiently addressed but acknowledges it.
- Evident from the text: only the three OpenAI models had 30 repeats per task and temperature, while other LLMs were evaluated once per task and temperature due to lack of API access; the paper also notes it could not set a seed in these LLMs.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures












Verification
Claims the paper makes
0 Valid · 0 Not valid · 12 Not checked · 87 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that, while LLMs show apparent competence in a few planning tasks with simpler structures, systematic evaluation reveals striking failure modes in planning tasks, including hallucinations of invalid trajectories and getting trapped in loops.”
- Also stated in: § 1 Introduction · plain
“We found that LLMs only show apparent competence in simpler tasks, where route memorization was sufficient to find a solution, but fail on closer systematic observation.”
§ Evaluating Cognitive Maps and Planning in Large Language Models with CogEval > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even with changing the temperature to 0.5 or 1 and repeating the same 30 runs its success rate can not exceed 10%.”
§ Evaluating Cognitive Maps and Planning in Large Language Models with CogEval > 3.3 Failure modes
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, when we probe the LLMs to list connected rooms or items as (state, actions, state) tuples, they do well (e.g., (room1, opendoor, room3) is a tuple for graph A in Figure 1).”
§ Evaluating Cognitive Maps and Planning in Large Language Models with CogEval > 3.3 Failure modes
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, when asked to do any tasks with a community graph structure using this tuple knowledge, LLMs display the following failure modes; (1) hallucinate edges that do not exist, or (2) fall into longer trajectories instead of shortest paths, or (3) get trapped in loops.”
§ Evaluating Cognitive Maps and Planning in Large Language Models with CogEval > 3.3 Failure modes
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We explored the impact of instructing GPT-4 with graph traversal methods-Breadth First Search (BFS) and Depth First Search (DFS).Even though explained at a high level, they resulted in performance improvement on several experiments with complex community graphs.”
§ Evaluating Cognitive Maps and Planning in Large Language Models with CogEval > 3.2 Experiment 2: The effect of Chain of Thought (CoT) Instructions
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
41 more claims
Create a free account to see all 99 claims and every source verdict.
- It may state two separate claims as one. 30
- The quote only partly states this claim. 2
- Probably not a substantial claim. 27
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Evaluating Cognitive Maps and Planning in Large Language Models with CogEval > 3.1 Experiment 1: Repeated measures comparison of planning across LLMs
“domain (χ 2 (11) = 304.06,p < .001)”
§ Evaluating Cognitive Maps and Planning in Large Language Models with CogEval > 100
“2357.87 7 <0.001 3431.53 5 <0.001 458.74 2 <0.001”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Evaluating Cognitive Maps and Planning in Large Language Models with CogEval > 3.1 Experiment 1: Repeated measures comparison of planning across LLMs
“The results in Table 2 indicate that the LLM (χ 2 (11) = 2357.87,p < .001),graph (χ 2 (11) = 3431.53,p < .001),condition (χ 2 (11) = 2080.04,p < .001)”
§ Evaluating Cognitive Maps and Planning in Large Language Models with CogEval > 100
“Table 2 indicate that the model engine (χ 2 (11) = 689.36,p < .001.), graph (χ 2 (11) = 101 7247.30,p < .001.), condition (χ 2 (11) = 1475.03,p < .001.)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Evaluating Cognitive Maps and Planning in Large Language Models with CogEval > Abstract
“Second, here we follow CogEval to systematically evaluate cognitive maps and planning ability across eight LLMs (OpenAI GPT-4, GPT-3.5-turbo-175B,davinci-003-175B, Google Bard, Cohere-xlarge-52.4B,Anthropic Claude-1-52B, LLaMA-13B, and Alpaca-7B).”
§ Evaluating Cognitive Maps and Planning in Large Language Models with CogEval > 2.1 Experiment 1: A cognitive science inspired evaluation of cognitive maps and planning capacity in LLMs
“We compared the results across 10 LLMs.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Evaluating Cognitive Maps and Planning in Large Language Models with CogEval > 3.1 Experiment 1: Repeated measures comparison of planning across LLMs
“The results in Table 2 indicate that the LLM (χ 2 (11) = 2357.87,p < .001),graph (χ 2 (11) = 3431.53,p < .001),condition (χ 2 (11) = 2080.04,p < .001),and domain (χ 2 (11) = 304.06,p < .001) each yielded significant chi-squared statistics.”
§ Evaluating Cognitive Maps and Planning in Large Language Models with CogEval > 100
“2357.87 7 <0.001 3431.53 5 <0.001 458.74 2 <0.001 1.28 2 0.53 2080.04 4 <0.001 10.69 14 0.71”
Citations
Cites
Part of this paper’s reference list could not be split into single entries.
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat
O'Keefe, J., & Dostrovsky, J. (1971) · Brain Research · doi:10.1016/0006-8993(71)90358-1 · Cited by 6,151
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Microstructure of a spatial map in the entorhinal cortex
Hafting, T., Fyhn, M., Molden, S., Moser, M.-B., & Moser, E. I. (2005) · Nature · doi:10.1038/nature03721 · Cited by 4,068
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
Large Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
Place Cells, Grid Cells, and the Brain's Spatial Representation System
Moser, E. I., Kropff, E., & Moser, M.-B. (2008) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.31.061307.090723 · Cited by 1,934
Research works printed but not linked: 8
- The nobel prize in physiology or medicine 2014. https://www.nobelprize.org/prizes/ medicine/2014/press-release/. Accessed: 2023-5-10.
- Rosa Cao. Multiple realizability and the spirit of functionalism. Synthese, 200(6):506, December 2022.
- Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8086-8098, 2022.
- Ida Momennejad. A rubric for human-like agents and NeuroAI. Philos. Trans. R. Soc. Lond. B Biol. Sci., 378(1869):20210446, December 2022.
- J O'Keefe. Place units in the hippocampus of the freely moving rat. Exp. Neurol., 51(1):78-109, April 1976.
- K L Stachenfeld, M Botvinick, and others. Design principles of the hippocampal cognitive map. Adv. Neural Inf. Process. Syst., 2014.
- Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press, November 2018.
- E C Tolman. Cognitive maps in rats and men. Psychol. Rev., 55(4):189-208, July 1948.
Cited by
Building machines that learn and think with people
Collins, K. M., Sucholutsky, I., Bhatt, U., Chandra, K., Wong, L., Lee, M., Zhang, C. E., Zhi‐Xuan, T., Ho, M. K., Mansinghka, V. K., et al. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01991-9 · Cited by 216
Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces
Yang, J., Yang, S., Gupta, A. W., Han, R., Fei-Fei, L., & Xie, S. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.00994 · Cited by 120
Beyond Preferences in AI Alignment
Zhi‐Xuan, T., Carroll, M., Franklin, M., & Ashton, H. (2024) · Philosophical Studies · doi:10.1007/s11098-024-02249-w · Cited by 127
Pallagani, V., Muppasani, B. C., Roy, K., Fabiano, F., Loreggia, A., Murugesan, K., Srivastava, B., Rossi, F., Lior Horesh, C., & Sheth, A. (2024) · Proceedings of the International Conference on Automated Planning and Scheduling · doi:10.1609/icaps.v34i1.31503 · Cited by 124
Strong and weak alignment of large language models with human values
Khamassi, M., Nahon, M., & Chatila, R. (2024) · Scientific Reports · doi:10.1038/s41598-024-70031-3 · Cited by 101
Metadata
- Authors
- Momennejad, I., Hasanbeig, H., Frujeri, F. V., Sharma, H., Jojic, N., Palangi, H., Ness, R., & Larson, J.
- Venue
- Advances in Neural Information Processing Systems 36
- Published
- 2023
- Pages
- 69736-69751
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133238287
- Cited by
- 27
- References
- 12
- Citations by year
- 2026: 5 · 2025: 8 · 2024: 14
- FWCI
- 5.10
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- cc-by-nc-nd
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, cognitive maps, planning, evaluation benchmark
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluating Cognitive Maps and Planning in Large Language Models with CogEval. https://app.cochat.ai/papers/10.52202/075280-3056
- The paper
- Momennejad, I., Hasanbeig, H., Frujeri, F. V., Sharma, H., Jojic, N., Palangi, H., Ness, R., & Larson, J. (2023). Evaluating Cognitive Maps and Planning in Large Language Models with CogEval. Advances in Neural Information Processing Systems 36. https://doi.org/10.52202/075280-3056
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.