Evaluating Cognitive Maps and Planning in Large Language Models with CogEval
Open accessThe DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index; none of the 30 sources its claims cite is checked for support yet.
Abstract
Recently an influx of studies claim emergent cognitive abilities in large language models (LLMs). Yet, most rely on anecdotes, overlook contamination of training sets, or lack systematic Evaluation involving multiple tasks, control conditions, multiple iterations, and statistical robustness tests. Here we make two major contributions. First, we propose CogEval, a cognitive science-inspired protocol for the systematic evaluation of cognitive capacities in Large Language Models. The CogEval protocol can be followed for the evaluation of various abilities. Second, here we follow CogEval to systematically evaluate cognitive maps and planning ability across eight LLMs (OpenAI GPT-4, GPT-3.5-turbo-175B, davinci-003-175B, Google Bard, Cohere-xlarge-52.4B, Anthropic Claude-1-52B, LLaMA-13B, and Alpaca-7B). We base our task prompts on human experiments, which offer both established construct validity for evaluating planning, and are absent from LLM training sets. We find that, while LLMs show apparent competence in a few planning tasks with simpler structures, systematic evaluation reveals striking failure modes in planning tasks, including hallucinations of invalid trajectories and getting trapped in loops. These findings do not support the idea of emergent out-of-the-box planning ability in LLMs. This could be because LLMs do not understand the latent relational structures underlying planning problems, known as cognitive maps, and fail at unrolling goal-directed trajectories based on the underlying structure. Implications for application and future directions are discussed.
Overview
CoChat Summary
TL;DR
The paper introduces CogEval, a cognitive-science-inspired protocol, and applies it to evaluate cognitive maps and planning across eight LLMs. It finds apparent competence on simple tasks but systematic failures on planning tasks with denser graph structures, including hallucinations, suboptimally long routes, and loops, arguing against emergent out-of-the-box planning.
Key findings
- Across LLMs, performance varied significantly by LLM (χ²=2357.87, p<.001), graph (χ²=3431.53, p<.001), condition (χ²=2080.04, p<.001), and domain (χ²=304.06, p<.001); temperature was non-significant (χ²=1.28, p=.53), as was the LLM×temperature interaction (χ²=10.69, p=.71).
- Three main failure modes appeared on dense community graphs: hallucinating edges that do not exist, falling into longer trajectories instead of shortest paths, and getting trapped in loops. For finding the shortest path to a state 1 cluster away, GPT-4 had a success rate of 0 at temperature 0 out of 30 runs; even at temperatures 0.5 or 1, success did not exceed 10%.
- Experiment 2 found that BFS/DFS Chain-of-Thought instructions improved some complex community-graph tasks but effects were inconsistent and non-monotonic: cluster counting on graph D improved at temperatures 0 and 0.5 but less at 1; shortest paths within a cluster improved across all temperatures with BFS or DFS, with BFS most effective at temperature 0; for shortest path 1-cluster away, only BFS at temperature 0.5 improved performance.
- LLMs could list pairwise (state, action, state) tuples but failed to use them for adaptive planning; apparent competence in simpler tasks may reflect route memorization or cached routes rather than understanding cognitive maps.
Methods
- CogEval operationalized cognitive maps and planning with tasks adapted from non-linguistic human experiments, varying graph structure (six graphs based on human experiments), domain (spatial ordered/unordered, social ties, object relations), and task conditions (valuePath, 1/2/3/n-step paths, reward/policy/transition revaluation, shortcut, detour, with/without teleportation and CoT).
- Evaluated LLMs: GPT-4, GPT-3.5-turbo-175B, davinci-003-175B, Bard, Cohere-xlarge-52.4B, Claude-1-52B, LLaMA-13B, and Alpaca-7B. Responses were generated 30 times per task prompt and temperature for the three OpenAI models and once per task and temperature for other LLMs, across temperatures 0, 0.5, and 1.
- Main analysis used logistic regression with domain, condition, and graph as categorical regressors, including second- and third-order interactions, with model version and temperature as separate independent variables; results were summarized in an analysis of deviance table. Experiment 2 tested BFS/DFS CoT instructions on GPT-4 with 30 repeats per prompt and temperature.
Limitations
- The paper states it lacks knowledge of LLMs like GPT-4’s architecture or training, so it avoided existing text-based benchmarks and generated novel prompts not in training sets.
- The paper states that human experiments involve gradual learning, whereas LLMs were presented the environment’s structure in linguistic format; it argues this difference is sufficiently addressed but acknowledges it.
- Evident from the text: only the three OpenAI models had 30 repeats per task and temperature, while other LLMs were evaluated once per task and temperature due to lack of API access; the paper also notes it could not set a seed in these LLMs.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures












Verification
Claims the paper makes
0 Valid · 0 Not valid · 12 Not checked · 87 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that, while LLMs show apparent competence in a few planning tasks with simpler structures, systematic evaluation reveals striking failure modes in planning tasks, including hallucinations of invalid trajectories and getting trapped in loops.”
- Also stated in: § 1 Introduction · plain
“We found that LLMs only show apparent competence in simpler tasks, where route memorization was sufficient to find a solution, but fail on closer systematic observation.”
§ Evaluating Cognitive Maps and Planning in Large Language Models with CogEval > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even with changing the temperature to 0.5 or 1 and repeating the same 30 runs its success rate can not exceed 10%.”
§ Evaluating Cognitive Maps and Planning in Large Language Models with CogEval > 3.3 Failure modes
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, when we probe the LLMs to list connected rooms or items as (state, actions, state) tuples, they do well (e.g., (room1, opendoor, room3) is a tuple for graph A in Figure 1).”
§ Evaluating Cognitive Maps and Planning in Large Language Models with CogEval > 3.3 Failure modes
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, when asked to do any tasks with a community graph structure using this tuple knowledge, LLMs display the following failure modes; (1) hallucinate edges that do not exist, or (2) fall into longer trajectories instead of shortest paths, or (3) get trapped in loops.”
§ Evaluating Cognitive Maps and Planning in Large Language Models with CogEval > 3.3 Failure modes
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We explored the impact of instructing GPT-4 with graph traversal methods-Breadth First Search (BFS) and Depth First Search (DFS).Even though explained at a high level, they resulted in performance improvement on several experiments with complex community graphs.”
§ Evaluating Cognitive Maps and Planning in Large Language Models with CogEval > 3.2 Experiment 2: The effect of Chain of Thought (CoT) Instructions
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
41 more claims
Create a free account to see all 99 claims and every source verdict.
- It may state two separate claims as one. 30
- The quote only partly states this claim. 2
- Probably not a substantial claim. 27
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Evaluating Cognitive Maps and Planning in Large Language Models with CogEval > 3.1 Experiment 1: Repeated measures comparison of planning across LLMs
“domain (χ 2 (11) = 304.06,p < .001)”
§ Evaluating Cognitive Maps and Planning in Large Language Models with CogEval > 100
“2357.87 7 <0.001 3431.53 5 <0.001 458.74 2 <0.001”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Evaluating Cognitive Maps and Planning in Large Language Models with CogEval > 3.1 Experiment 1: Repeated measures comparison of planning across LLMs
“The results in Table 2 indicate that the LLM (χ 2 (11) = 2357.87,p < .001),graph (χ 2 (11) = 3431.53,p < .001),condition (χ 2 (11) = 2080.04,p < .001)”
§ Evaluating Cognitive Maps and Planning in Large Language Models with CogEval > 100
“Table 2 indicate that the model engine (χ 2 (11) = 689.36,p < .001.), graph (χ 2 (11) = 101 7247.30,p < .001.), condition (χ 2 (11) = 1475.03,p < .001.)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Evaluating Cognitive Maps and Planning in Large Language Models with CogEval > Abstract
“Second, here we follow CogEval to systematically evaluate cognitive maps and planning ability across eight LLMs (OpenAI GPT-4, GPT-3.5-turbo-175B,davinci-003-175B, Google Bard, Cohere-xlarge-52.4B,Anthropic Claude-1-52B, LLaMA-13B, and Alpaca-7B).”
§ Evaluating Cognitive Maps and Planning in Large Language Models with CogEval > 2.1 Experiment 1: A cognitive science inspired evaluation of cognitive maps and planning capacity in LLMs
“We compared the results across 10 LLMs.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Evaluating Cognitive Maps and Planning in Large Language Models with CogEval > 3.1 Experiment 1: Repeated measures comparison of planning across LLMs
“The results in Table 2 indicate that the LLM (χ 2 (11) = 2357.87,p < .001),graph (χ 2 (11) = 3431.53,p < .001),condition (χ 2 (11) = 2080.04,p < .001),and domain (χ 2 (11) = 304.06,p < .001) each yielded significant chi-squared statistics.”
§ Evaluating Cognitive Maps and Planning in Large Language Models with CogEval > 100
“2357.87 7 <0.001 3431.53 5 <0.001 458.74 2 <0.001 1.28 2 0.53 2080.04 4 <0.001 10.69 14 0.71”
Citations
Cites
Part of this paper’s reference list could not be split into single entries.
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat
O'Keefe, J., & Dostrovsky, J. (1971) · Brain Research · doi:10.1016/0006-8993(71)90358-1 · Cited by 6,151
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Microstructure of a spatial map in the entorhinal cortex
Hafting, T., Fyhn, M., Molden, S., Moser, M.-B., & Moser, E. I. (2005) · Nature · doi:10.1038/nature03721 · Cited by 4,068
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
Large Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
Place Cells, Grid Cells, and the Brain's Spatial Representation System
Moser, E. I., Kropff, E., & Moser, M.-B. (2008) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.31.061307.090723 · Cited by 1,934
Research works printed but not linked: 8
- The nobel prize in physiology or medicine 2014. https://www.nobelprize.org/prizes/ medicine/2014/press-release/. Accessed: 2023-5-10.
- Rosa Cao. Multiple realizability and the spirit of functionalism. Synthese, 200(6):506, December 2022.
- Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8086-8098, 2022.
- Ida Momennejad. A rubric for human-like agents and NeuroAI. Philos. Trans. R. Soc. Lond. B Biol. Sci., 378(1869):20210446, December 2022.
- J O'Keefe. Place units in the hippocampus of the freely moving rat. Exp. Neurol., 51(1):78-109, April 1976.
- K L Stachenfeld, M Botvinick, and others. Design principles of the hippocampal cognitive map. Adv. Neural Inf. Process. Syst., 2014.
- Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press, November 2018.
- E C Tolman. Cognitive maps in rats and men. Psychol. Rev., 55(4):189-208, July 1948.
Cited by
Building machines that learn and think with people
Collins, K. M., Sucholutsky, I., Bhatt, U., Chandra, K., Wong, L., Lee, M., Zhang, C. E., Zhi-Xuan, T., Ho, M., Mansinghka, V., et al. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01991-9 · Cited by 73
Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces
Yang, J., Yang, S., Gupta, A. W., Han, R., Fei-Fei, L., & Xie, S. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.00994 · Cited by 120
Beyond Preferences in AI Alignment
Zhi-Xuan, T., Carroll, M., Franklin, M., & Ashton, H. (2025) · Philosophical Studies · doi:10.1007/s11098-024-02249-w · Cited by 34
Pallagani, V., Muppasani, B. C., Roy, K., Fabiano, F., Loreggia, A., Murugesan, K., Srivastava, B., Rossi, F., Horesh, L., & Sheth, A. (2024) · Proceedings of the International Conference on Automated Planning and Scheduling · doi:10.1609/icaps.v34i1.31503 · Cited by 125
Strong and weak alignment of large language models with human values
Khamassi, M., Nahon, M., & Chatila, R. (2024) · Scientific Reports · doi:10.1038/s41598-024-70031-3 · Cited by 101
Can large language models explore in-context?
Krishnamurthy, A., Harris, K., Foster, D. J., Zhang, C., & Slivkins, A. (2024) · neural information processing systems · doi:10.52202/079017-3818 · Cited by 118
Evaluating Large Language Models as Generative User Simulators for Conversational Recommendation
Yoon, S.-E., He, Z., Echterhoff, J. M., & McAuley, J. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.83 · Cited by 110
Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models
Wu, W., Mao, S., Zhang, Y., Xia, Y., Dong, L., Cui, L., & Wei, F. (2024) · neural information processing systems · doi:10.52202/079017-2866 · Cited by 175
A brain-inspired agentic architecture to improve planning with LLMs
Webb, T. W., Mondal, S. S., & Momennejad, I. (2025) · Nature Communications · doi:10.1038/s41467-025-63804-5 · Cited by 59
Self-citationSix Fallacies in Substituting Large Language Models for Human Participants
Lin, Z. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251357566 · Cited by 64
Zeng, Q., Yang, Q., Dong, S., Du, H., Zheng, L., Xu, F., & Li, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.04168 · Cited by 42
Xu, W., Lyu, D., Wang, W., Feng, J., Gao, C., & Li, Y. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.567 · Cited by 34
Integration of cognitive tasks into artificial general intelligence test for large models
Qu, Y., Wei, C., Du, P., Che, W., Zhang, C., Ouyang, W., Bian, Y., Xu, F., Hu, B., Du, K., et al. (2024) · iScience · doi:10.1016/j.isci.2024.109550 · Cited by 25
Structured World Representations in Maze-Solving TransformersPreprint
Ivanitskiy, M., Spies, A. F., Räuker, T., Corlouer, G., Mathwin, C., Quirke, L., Rager, C., Shah, R., Valentine, D., Behn, C. D., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.02566 · Cited by 23
TALES: Text Adventure Learning Environment SuitePreprint
Cui, C., Yuan, X., Xiao, Z., Ammanabrolu, P., & Côté, M.-A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.14128 · Cited by 14
Ito, S., Muraoka, K., & Nakayama, A. (2025) · Chemistry of Materials · doi:10.1021/acs.chemmater.4c02726 · Cited by 18
What Is It for a Machine Learning Model to Have a Capability?
Harding, J., & Sharadin, N. (2024) · The British Journal for the Philosophy of Science · doi:10.1086/732153 · Cited by 20
Systematic Human Learning and Generalization From a Brief Tutorial With Explanatory Feedback
Nam, A., & McClelland, J. L. (2024) · Open Mind · doi:10.1162/opmi_a_00123 · Cited by 15
Reconciling shared versus context-specific information in a neural network model of latent causes
Lu, Q., Nguyen, T. T., Zhang, Q., Hasson, U., Griffiths, T. L., Zacks, J. M., Gershman, S. J., & Norman, K. A. (2024) · Scientific Reports · doi:10.1038/s41598-024-64272-5 · Cited by 15
Yousefi, S., Betthauser, L., Hasanbeig, H., Millière, R., & Momennejad, I. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.00313 · Cited by 11
Self-citationChun, J. (2024) · Frontiers in Computer Science · doi:10.3389/fcomp.2024.1444549 · Cited by 12
Structural knowledge: from brain to artificial intelligence
Yu, Y., Yan, Y., & Jin, Y. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11270-2 · Cited by 12
Stop Acting Like Language Model Agents Are Normal AgentsPreprint
Perrier, E., & Bennett, M. T. (2025) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/3tpq4_v1 · Cited by 10
Are transformers truly foundational for robotics?
Marshall, J. A. R., & Barron, A. (2024) · npj Robotics · doi:10.1038/s44182-025-00025-4 · Cited by 9
How language models extrapolate outside the training data: A case study in Textualized GridworldPreprint
Kim, D., Lee, J., Park, J., & Seo, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.15275 · Cited by 7
Planning with the ViewsPreprint
Wang, K., Li, L., Yang, Z., Chen, S., Wang, Z., Fei-Fei, L., Wu, J., Guibas, L. J., Wang, L., & Li, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.29563 · Cited by 6
Bio-inspired cognitive navigation for robots
Hao, Z., Guo, B., Ding, Y., Li, L., Liu, S., Gao, T., Liu, S., & Yu, Z. (2026) · Nature Reviews Electrical Engineering · doi:10.1038/s44287-026-00294-7 · Cited by 7
Evaluation of geographical distortions in language models
Decoupes, R., Interdonato, R., Roche, M., Teisseire, M., & Valentin, S. (2025) · Machine Learning · doi:10.1007/s10994-025-06916-9 · Cited by 8
AriGraph: Learning Knowledge Graph World Models with Episodic Memory for LLM Agents
Anokhin, P., Semenov, N., Sorokin, A., Evseev, D., Kravchenko, A., Burtsev, M., & Burnaev, E. (2024) · arXiv · doi:10.48550/arxiv.2407.04363 · Cited by 8
Symbolic Task Inference in Deep Reinforcement Learning
Hasanbeig, H., Jeppu, N. Y., Abate, A., Melham, T., & Kroening, D. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14063 · Cited by 12
Self-citationA surprising link between cognitive maps, successor-relation based reinforcement learning, and BTSPPreprint
Yang, Y., Stöckl, C., & Maass, W. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.04.22.650046 · Cited by 5
Schepens, J. J., Woloszyn, H., Marx, N., & Gagl, B. (2025) · Open Mind · doi:10.1162/opmi.a.30 · Cited by 5
Evaluation of Geographical Distortions in Language ModelsPreprint
Decoupes, R., Interdonato, R., Roche, M., Teisseire, M., & Valentin, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.17401 · Cited by 5
Jeoung, S., Ge, Y., Wang, H., & Diesner, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.14294 · Cited by 6
Dedieu, A., Lehrach, W., Zhou, G., George, D., & Lázaro-Gredilla, M. (2024) · arXiv · doi:10.48550/arxiv.2401.05946 · Cited by 2
Ito, S., Muraoka, K., & Nakayama, A. (2024) · ChemRxiv · doi:10.26434/chemrxiv-2024-wqxl7 · Cited by 5
Fine-Tuning Large Language Models Using EEG Microstate Features for Mental Workload AssessmentPreprint
Raufi, B. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.07283 · Cited by 2
The recipe for intelligence in natural and artificial systems
Summerfield, C., & Stachenfeld, K. (2026) · Neuron · doi:10.1016/j.neuron.2026.08.013 · Cited by 1
Minds of their own? Decoding free will in large language models
Lavazza, A., Sartori, G., Farina, M., Melis, G., & Orrù, G. (2026) · AI & Society · doi:10.1007/s00146-025-02809-6 · Cited by 3
Ito, S., Muraoka, K., & Nakayama, A. (2024) · ChemRxiv · doi:10.26434/chemrxiv-2024-wqxl7-v2 · Cited by 1
Are Transformers Truly Foundational for Robotics?Preprint
Marshall, J. A. R., & Barron, A. B. (2024) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · doi:10.48550/arxiv.2411.16917 · Cited by 1
Sidhu, R., & Hockenmaier, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.17333 · Cited by 1
Proactive models for open-domain conversational search
Erbacher, P. (2024) · theses.fr (ABES) · doi:10.70675/f7d68861zc694z49f1zb130z675cbf38ed1c · Cited by 1
Maproute-Bench: Evaluating Spatial Reasoning on Top-View Maps in Vision-Language Models
Cheng, A., Ma, J., Yang, F., Luo, Z., Chen, J., & Qiu, C. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11460536
Intentionality All-Stars Redux: Do Language Models Know What They Are Talking About?
Browning, J. (2026) · Oxford University Press eBooks · doi:10.1093/9780191998317.003.0002
Lin, Y., & Xiao, N. (2026) · Spatial Cognition and Computation · doi:10.1080/13875868.2026.2730164
Does Spatial Cognition Emerge in Frontier Models?
Ramakrishnan, S. K., Wijmans, E., Kraehenbuehl, P., & Koltun, V. (2024) · arXiv · doi:10.48550/arxiv.2410.06468 · Cited by 49
Ito, S., Muraoka, K., & Nakayama, A. (2024) · ChemRxiv · doi:10.26434/chemrxiv-2024-wqxl7-v3
Ito, S., Muraoka, K., & Nakayama, A. (2024) · ChemRxiv · doi:10.26434/chemrxiv-2024-wqxl7-v4
Ito, S., Muraoka, K., & Nakayama, A. (2024) · ChemRxiv · doi:10.26434/chemrxiv-2024-wqxl7-v5
Assessing the Performance of Human-Capable LLMs -- Are LLMs Coming for Your Job?Preprint
Mavi, J., Summers, N., & Coronado, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.16285
Are Transformers Truly Foundational for Robotics?Preprint
Marshall, J. A. R., & Barron, A. B. (2024) · Qeios · doi:10.32388/eh2gbo
Ito, S., Muraoka, K., & Nakayama, A. (2025) · ChemRxiv · doi:10.26434/chemrxiv-2024-wqxl7-v6
Schöne neue Textwelt. Vertrauen in KI-generierte Texte
Knopp, M., & Schindler, K. (2025) · Deutschdidaktik · doi:10.1007/978-3-662-71769-1_7
TurnBack: A Geospatial Route Cognition Benchmark for Large Language Models through Reverse Route
Luo, H., Cheng, Q., Matos, D., Gadi, H. K., Zhang, Y., Liu, L., Wang, Y., Zeller, N., Cremers, D., & Meng, L. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.440 · Cited by 1
Conversations on the Web: Prediction, Recommendation, and Simulation
Yoon, S.-E. (2025) · eScholarship (California Digital Library) · Cited by 1
Ding, Y., Han, X., Yang, J., Wang, T., Bi, Z., Song, X., Hao, J., Song, J., Ge, E., Peng, B., et al. (2026) · doi:10.36227/techrxiv.176784493.33333564/v1
World Agent: Can Language Models Keep a World Running?Preprint
Chen, W., Zhang, W., An, N., Liu, Y., & Lin, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.32692
Examining large language models for safety and robustness through the lens of social science
Jeoung, S. (2025) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 2
Generalized Planning Using Language Models and Its Applications
Pallagani, V. (2026) · Scholar Commons (University of South Carolina)
Metadata
- Authors
- Momennejad, I., Hasanbeig, H., Frujeri, F. V., Sharma, H., Jojic, N., Palangi, H., Ness, R., & Larson, J.
- Venue
- Advances in Neural Information Processing Systems 36
- Published
- 2023
- Pages
- 69736-69751
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133238287
- Cited by
- 27
- References
- 12
- Citations by year
- 2026: 5 · 2025: 8 · 2024: 14
- FWCI
- 5.10
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- cc-by-nc-nd
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, cognitive maps, planning, evaluation benchmark
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.