Evaluating the World Model Implicit in a Generative Model
Open accessThe DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index; 8 of the 10 sources its claims cite were checked for support.
Abstract
Recent work suggests that large language models may implicitly learn world models. How should we assess this possibility? We formalize this question for the case where the underlying reality is governed by a deterministic finite automaton. This includes problems as diverse as simple logical reasoning, geographic navigation, game-playing, and chemistry. We propose new evaluation metrics for world model recovery inspired by the classic Myhill-Nerode theorem from language theory. We illustrate their utility in three domains: game playing, logic puzzles, and navigation. In all domains, the generative models we consider do well on existing diagnostics for assessing world models, but our evaluation metrics reveal their world models to be far less coherent than they appear. Such incoherence creates fragility: using a generative model to solve related but subtly different tasks can lead to failures. Building generative models that meaningfully capture the underlying logic of the domains they model would be immensely valuable; our results suggest new ways to assess how close a given model is to that goal.
Overview
CoChat Summary
TL;DR
The paper formalizes world-model recovery when the underlying reality is a deterministic finite automaton (DFA) and proposes Myhill-Nerode-inspired metrics for sequence compression and sequence distinction. Across navigation, Othello, and logic puzzles, generative models often pass existing diagnostics while their implicit world models remain incoherent and fragile under detours.
Key findings
- Existing diagnostics can be misleading: in the NYC taxi map setting, trained transformers score 1.00 on the next-token test and 0.91–0.99 on the current-state probe, but proposed metrics reveal much weaker recovery, e.g. compression precision 0.05–0.50 and distinction recall 0.20–1.00 depending on training data.
- Detours expose fragility: for shortest-paths models, valid traversals fall from 0.99 at 0% detour probability to 0.69 at 1%, 0.08 at 10%, and 0.00 at 50%/75%; the random-walks model is more robust, remaining at 0.99, 0.99, 1.00, 0.97, and 0.74 across those detour levels.
- In Othello, the synthetic-game model scores compression precision 0.98, distinction precision 0.99, and distinction recall 1.00, while the championship-game model scores 0.00, 0.65, and 0.27; detour tests show the synthetic model produces near-perfect games while the championship model fails immediately.
- In logic puzzles, GPT-4 reaches 1.00 task accuracy but only 0.21 compression precision and 0.56 distinction recall; no evaluated model exceeds 40% compression precision or 0.60 distinction recall.
Methods
- The paper models the true world as a DFA and defines recovery as matching the DFA’s language. It uses the Myhill-Nerode theorem to define interior and boundary sets, then builds sequence compression metrics from equal-state pairs and sequence distinction metrics from distinct-state pairs, comparing model-implied boundaries to true DFA boundaries.
- Navigation data come from 2014 NYC taxi rides in Manhattan, represented as a graph with 4,580 nodes and 9,846 edges. Three traversal datasets are used: shortest paths (2.9M sequences, 120M tokens), noisy shortest paths (31M sequences, 1.7B tokens), and random walks (91M sequences, 4.7B tokens). GPT-2-style transformers of 89.3M and 1.5B parameters are trained from scratch.
- Additional applications use Othello sequence models from Li et al. and prompted LLMs on seating-arrangement logic puzzles. Metrics are approximated with Monte Carlo sampling, an acceptance threshold ε = 0.01, graph reconstruction, and detour robustness tests.
Limitations
- Stated: the primary limitation is the focus on DFAs; extending the approach to more complicated or unknown world models is left to future work.
- Evident from the text: the metrics require query access to the true DFA and tractable boundary computation, and the LLM logic-puzzle evaluation lacks token-level probability access, relying on prompting with small samples (5 continuations, 100 states) and reporting compression as an overestimate.
- Evident from the text: results depend on choices such as ε, decoding acceptance rule, Monte Carlo sample count, and reconstruction constraints; the paper reports ablations suggesting conclusions are stable, but the reconstruction algorithm also gives the model the benefit of the doubt and can fail to reconstruct some sequences.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
5 Valid · 2 Not valid · 0 Not checked · 88 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The classic Myhill-Nerode theorem [26, 27] provides intuition: every pair of distinct states can be distinguished by some sequence (admitted by one state but not the other). Unless those minimal distinguishing sequences are of length one, looking at the next single token outputted will not reliably assess whether the generative model has an accurate model of the underlying state.”
What the sources say
- Not matchedcites [26] [26] Myhill, J. Finite automata and the representation of events. WADD Technical Report , 57: 112-137, 1957. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites [27] [27] Nerode, A. Linear automaton transformations. Proceedings of the American Mathematical Society , 9(4):541-544, 1958. - sourceExperimental — accuracy not yet measured
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Large language models (LLMs) appear to have capacities that far exceed the next-token prediction task they were trained to perform [17, 39, 35].”
What the sources say
- Supportedcites [17] [17] Kıcıman, E., Ness, R., Sharma, A., and Tan, C. Causal reasoning and large language models: Opening a new frontier for causality. arXiv preprint arXiv:2305.00050 , 2023. - source
Section 2.3 describes next-word completion as LLMs’ primary task. The Introduction says they also develop emergent abilities, including writing code and performing reasoning tasks, supporting the claim that their capacities extend beyond that task. (LLM, conf=0.91)
- Partly supportedcites [39] [39] Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 , 2022. - sourceExperimental — accuracy not yet measured
The Discussion (p. 6) says large models exhibit emergent abilities on tasks not explicitly included in pre-training, supporting the claim that their capacities extend beyond their training tasks. It does not characterize next-token prediction as the training task for all the models discussed; Section 5.2 (p. 7) also discusses a different pre-training objective. (LLM, conf=0.82)
- Partly supportedcites [35] [35] Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. Challenging BIG-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261 , 2022. - sourceExperimental — accuracy not yet measured
The Introduction reports that large language models perform challenging BIG-Bench tasks and that chain-of-thought prompting helped a model surpass the average human-rater on 17 of 23 tasks. This supports the claim that their capabilities extend beyond simple text prediction. Our copy does not establish the claim’s specific premise that the models studied were trained only to predict the next token. (LLM, conf=0.82)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We note that this distribution of (origin, destination) pairs varies from the distributions used to train and evaluate each model”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Cumulative Connect-4 exhibits this behavior; any board for which there are less than 𝑘 disks in each column will have the same set of valid moves for the next 𝑛 -𝑘 moves.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is much closer to the true world model than the transformer (which makes errors at the same rate).”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
56 more claims
Create a free account to see all 95 claims and every source verdict.
- Probably not a substantial claim. 9
- It may state two separate claims as one. 26
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4 Other Applications: Othello and Logic Puzzles · p. 10
“no model has a compression precision higher than 40%.”
§ 4 Other Applications: Othello and Logic Puzzles · p. 10
“| Qwen 1.5 (110B) | 0.98 (0.00) | 0.53 (0.05) | 0.53 (0.04) |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.1 Data and models · p. 6
“Weinclude all sequences containing less than 100 directions.”
§ F Rides data construction and training · p. 20
“We also remove sequences with more than 100 tokens.”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Evolutionary-scale prediction of atomic level protein structure with a language model
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. (2022) · doi:10.1101/2022.07.20.500902 · Cited by 5,990
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242
Autonomous chemical research with large language models
Boiko, D. A., MacKnight, R., Kline, B., & Gomes, G. (2023) · Nature · doi:10.1038/s41586-023-06792-0 · Cited by 1,050
Linear automaton transformations
Nerode, A. (1958) · Proceedings of the American Mathematical Society · doi:10.1090/s0002-9939-1958-0135681-9 · Cited by 541
Research works printed but not linked: 3
- [28][28] Patel, R. and Pavlick, E. Mapping language models to grounded conceptual spaces. In International Conference on Learning Representations , 2021. -
- [29][29] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019. -
- [33][33] Sipser, M. Introduction to the Theory of Computation, Third Edition . Cengage Learning, 2013. -
Cited by
Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation
Eriksson, M., Purificato, E., Noroozian, A., Vinagre, J., Chaslot, G., Gomez, E., & Fernandez-Llorca, D. (2025) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1609/aies.v8i1.36595 · Cited by 26
ICLR: In-Context Learning of Representations
Park, C. F., Lee, A., Lubana, E. S., Yang, Y., Okawa, M., Nishi, K., WATTENBERG, M. P., & Tanaka, H. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2501.00070 · Cited by 47
How close is AI to human-level intelligence?
Ananthaswamy, A. (2024) · Nature · doi:10.1038/d41586-024-03905-1 · Cited by 32
A Survey on Large Language Models for Automated PlanningPreprint
Aghzal, M., Plaku, E., Stein, G. J., & Yao, Z. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.12435 · Cited by 49
Opening the AI Black Box: Distilling Machine-Learned Algorithms into Code
Michaud, E. J., Liao, I., Lad, V., Liu, Z., Mudide, A., Loughridge, C., Guo, Z. C., Kheirkhah, T. R., Vukelić, M., & Tegmark, M. (2024) · Entropy · doi:10.3390/e26121046 · Cited by 30
Potemkin Understanding in Large Language ModelsPreprint
Mancoridis, M., Weeks, B., Vafa, K., & Mullainathan, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.21521 · Cited by 47
Self-citationText2World: Benchmarking Large Language Models for Symbolic World Model Generation
Hu, M., Chen, T., Zou, Y., Lei, Y., Chen, Q., Li, M., Mu, Y., Zhang, H., Shao, W., & Luo, P. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.1337 · Cited by 50
Ramoglou, S. E., Chandra, Y., & Jin, Q. (2025) · Journal of Management Studies · doi:10.1111/joms.70011 · Cited by 31
Looking Inward: Language Models Can Learn About Themselves by Introspection
Binder, F. J., Chua, J., Korbak, T., Sleight, H., Hughes, J., Long, R., Perez, E., Turpin, M., & Evans, O. (2024) · arXiv · doi:10.48550/arxiv.2410.13787 · Cited by 20
Modeling Open-World Cognition as On-Demand Synthesis of Probabilistic ModelsPreprint
Wong, L., Collins, K. M., Ying, L., Zhang, C. E., Weller, A., Gerstenberg, T., O’Donnell, T. J., Lew, A. K., Andreas, J., Tenenbaum, J. B., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.12547 · Cited by 29
Zhang, Y., Du, W., Jin, D., Fu, J., & Jin, Z.-J. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.668 · Cited by 22
Yu, Y., Zhang, S., Sheng, Y., Ren, H., & Lin, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.15032 · Cited by 16
Yager, K. G. (2024) · Digital Discovery · doi:10.1039/d4dd00178h · Cited by 12
Navigating the knowledge problem in the era of open AI and polycentric governance
Makridis, C. A., & Lazanski, D. (2025) · The Review of Austrian Economics · doi:10.1007/s11138-025-00708-z · Cited by 10
Language Models’ Hall of Mirrors Problem: Why AI Alignment Requires Peircean Semiosis
Manheim, D. (2026) · Philosophy & Technology · doi:10.1007/s13347-025-00975-5 · Cited by 13
What Does it Mean for a Neural Network to Learn a "World Model"?Preprint
Li, K., Viégas, F. B., & Wattenberg, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.21513 · Cited by 10
AI and actor-specific decisions
Felin, T., Sako, M., & Hullman, J. (2026) · AI & Society · doi:10.1007/s00146-026-03202-7 · Cited by 8
Economic Surveillance using Corporate Text
Hassan, T. A., Hollander, S., Kalyani, A., van Lent, L., Schwedeler, M., & Tahoun, A. (2024) · National Bureau of Economic Research · doi:10.3386/w33158 · Cited by 16
Agentic AI Needs a Systems Theory
Miehling, E., Ramamurthy, K. N., Varshney, K. R., Riemer, M., Bouneffouf, D., Richards, J. T., Dhurandhar, A., Daly, E. M., Hind, M., Sattigeri, P., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.00237 · Cited by 6
The Philosophy of Language Models
Millière, R., & Buckner, C. J. (2026) · Philosophy Compass · doi:10.1111/phc3.70095 · Cited by 9
Dutch Books for Language ModelsPreprint
Andrews, I., & Sarkar, S. (2026) · arXiv (Cornell University) · Cited by 6
Geo-hallucination in urban analytics: What it is and why it matters
Huang, X. (2025) · Environment and Planning B Urban Analytics and City Science · doi:10.1177/23998083251409526 · Cited by 9
AI needs physics more than physics needs AI
Coveney, P. V., & Highfield, R. (2026) · Frontiers in Physics · doi:10.3389/fphy.2025.1731777 · Cited by 5
Worker Rights in Collective Bargaining
Arold, B. W., Ash, E., MacLeod, W. B., & Naidu, S. (2025) · National Bureau of Economic Research · doi:10.3386/w33605 · Cited by 5
Mining the gaps: Deciphering Alzheimer’s biology through AI-driven reconciliation
Funk, C. C., Paterson, T., Bangs, A., Cannon, D. M., Savage, G., Ringger, E. K., & Hood, L. (2025) · The Journal of Prevention of Alzheimer s Disease · doi:10.1016/j.tjpad.2025.100402 · Cited by 6
Behavioural vs. Representational Systematicity in End-to-End Models: An Opinionated Survey
Vegner, I., de Souza, S., Forch, V., Lewis, M., & Doumas, L. A. A. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1537 · Cited by 7
Quo Vadis, World Modeling?Preprint
Yang, Y., Yang, X., Wen, L., Kong, L., Hu, X., Lu, D., Chow, W., Huang, X., Feng, Y., Liao, Y., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.02713 · Cited by 4
CheckersGPT: Learning World Models through Language Modeling
Joshi, A., Sharma, V., & Modi, A. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-srw.48 · Cited by 6
Reasoning about Affordances: Causal and Compositional Reasoning in LLMsPreprint
Gjerde, M. F., Cheung, V., & Lagnado, D. A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.16606 · Cited by 4
Chen, H., Grimm, S. R., Russakovsky, O., & Lombrozo, T. (2026) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2026.04.003 · Cited by 4
Guidotti, R., Schmid, U., & Longo, L. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-08330-2 · Cited by 2
Will AI ever win its own Nobel? Some predict a prize-worthy science discovery soon
Ahart, J. (2025) · Nature · doi:10.1038/d41586-025-03223-0 · Cited by 2
Vowels, M. J., Rochat, M., & Akbari, S. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2410.14485 · Cited by 2
A Hierarchical Language Model For Interpretable Graph Reasoning
Khurana, S., Li, X., Gui, S., & Ji, S. (2024) · Transactions on machine learning research · doi:10.48550/arxiv.2410.22372 · Cited by 4
The Impossibility of Inverse Permutation Learning in Transformer ModelsPreprint
Alur, R., Hays, C., Raghavan, M., & Shah, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.24125 · Cited by 2
Behavioural vs. Representational Systematicity in End-to-End Models: An Opinionated SurveyPreprint
Vegner, I., de Souza, S., Forch, V., Lewis, M., & Doumas, L. A. A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.04461 · Cited by 3
GPT-4o Lacks Core Features of Theory of MindPreprint
Muchovej, J., Royka, A. L., Lee, S., & Jara‐Ettinger, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.12150 · Cited by 2
BigBang-Proton Technical Report: Next-Word-Prediction is Scientific Multitask LearnerPreprint
Wu, H., Liu, L., He, J., Wang, Q., Zhao, K., Hu, S., Fu, R., Liang, D., Zeng, L., Liu, B. X., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.00129 · Cited by 2
Measuring Harness-Induced Belief Divergence in Multi-Step LLM AgentsPreprint
Yi, H.-W., & Song, X.-Y. (2026) · arXiv.org · doi:10.48550/arxiv.2607.04528
CyberGeoAI For Intelligent Spatial Decision Support
Zhang, Z., Huang, X., Ye, X., Yuan, M., Jain, D., Li, W., Gao, S., Shekhar, S., He, Z., Liu, H., et al. (2026) · Transactions in GIS · doi:10.1111/tgis.70362 · Cited by 1
Hierarchical learning creates invariant schema within plastic neural networks
Elder, J. R., Zheng, J., Shimelis, L. B., Rutishauser, U., & Lin, M. M. (2026) · Journal of Computational Neuroscience · doi:10.1007/s10827-026-00940-x · Cited by 1
Simulating strategic interactions with AI agents
Tranchero, M., Brenninkmeijer, C.-F., Murugan, A., & Nagaraj, A. (2026) · Strategic Management Journal · doi:10.1002/smj.70112 · Cited by 5
How Can We Characterize Human Generalization and Distinguish It From Generalization in Machines?
Thalmann, M., & Schulz, E. (2025) · Current Directions in Psychological Science · doi:10.1177/09637214251336212 · Cited by 1
Wahidi, M., Rizk, A., & Imad, R. (2025) · IEEE Access · doi:10.1109/access.2025.3589514 · Cited by 3
Information Density of Objects in Digital Environment: Theoretical Foundations
Bondareva, N. A., Bondarev, A. E., Andreev, S. V., & Ryzhova, I. G. (2025) · Scientific Visualization · doi:10.26583/sv.17.4.09 · Cited by 1
Planning in the LLM Era: Building for Reliability and Efficiency
Katz, M., Kokel, H., Srinivas, K., & Sohrabi, S. (2026) · Proceedings of the International Conference on Automated Planning and Scheduling · doi:10.1609/icaps.v36i1.42851 · Cited by 1
Emergent Stack Representations in Modeling Counter Languages Using TransformersPreprint
Tiwari, U., Gupta, A., & Hahn, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.01432 · Cited by 1
Performance Gains of LLMs With Humans in a World of LLMs Versus HumansPreprint
McCullum, L. B., Agassi, P. A., Celi, L. A., Ebner, D. K., Fernandes, C. O., Hicklen, R. S., Koumbia, M., Lehmann, L. S., & Restrepo, D. S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.08902 · Cited by 1
The Limits of Predicting Agents from BehaviourPreprint
Bellot, A., Richens, J. G., & Everitt, T. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.02923 · Cited by 1
Vashistha, S., Bibhuti, A., Naik, A., Tutek, M., & Aditya, S. (2025) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.48550/arxiv.2511.13021 · Cited by 1
World-Time Compute with Verified Code World ModelsPreprint
Schwoebel, J., Semenec, I., Rousseva, J., Ortiz, M., Overbay, C., Klaus, C., Edmond, A., Bhatt, M., Thorstenson, R., Tsai, J., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.09163 · Cited by 1
MultiEcho: An Experimental Science of Learned WorldsPreprint
Zhu, M.-Y., & Zhang, A.-R. (2026) · arXiv.org · doi:10.48550/arxiv.2609.33347 · Cited by 1
Open-Endedness Bench: Measuring Epistemic Process from Agent RecordsPreprint
Shi, C., Ji, X., Huang, J., Wang, J., He, Y., & Liu, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.02588 · Cited by 1
Building Digital Twins of Cities via Urban Theory-Informed Neural Networks
Leung, Y. (2026) · Advances in spatial science · doi:10.1007/978-3-032-07966-4_5
Staying in the Loop: The Role of Judgement in Generative AI–Enabled Leadership
Breede, H. C. (2026) · Oxford University Press eBooks · doi:10.1093/9780198945215.003.0237
The paradox of intelligence failures in the age of artificial intelligence
Lonergan, E. D. (2026) · Intelligence & National Security · doi:10.1080/02684527.2026.2690392 · Cited by 1
Brain's world model is smarter than we think
Yang, M. A., Kim, H., Kang, J., & Lee, S. W. (2026) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2026.101672
Aurnob, A. R., Tanim, S. A., & Sattar, F. (2026) · Frontiers in Signal Processing · doi:10.3389/frsip.2026.1769553
LLMs Do Not Emulate PopulationsPreprint
Green, B. S., & Leo, G. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7352138
Is Kant’s Transcendental Logic a Logic?
Schumann, A. (2026) · Foundations of Science · doi:10.1007/s10699-026-10048-1 · Cited by 1
KilometerVision: A New Frontier for Large-Scale Spatial Intelligence in VLMs
Mahendran, A., King, M., Grimes, M. K., Yang, A., Zhu, T., Heyward, J., Han, T., Ginosar, S., Sun, C., Damen, D., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37020-4_11 · Cited by 1
Abraham, B. (2026) · International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW) · doi:10.1109/aciiw71585.2026.11712079
From Text to Space: Mapping Abstract Spatial Models in LLMs during a Grid-World Navigation Task
Martorell, N. (2025) · arXiv · doi:10.48550/arxiv.2502.16690
$\textit{New News}$: System-2 Fine-tuning for Robust Integration of New Knowledge
Park, C. F., Zhang, Z., & Tanaka, H. (2025) · arXiv · doi:10.48550/arxiv.2505.01812
Text as Data in Economic Analysis
Kalyani, A., van Lent, L., Tahoun, A., Schwedeler, M., & Hassan, T. A. (2024) · doi:10.20955/wp.2024.022
Worker Rights in Collective BargainingPreprint
Arold, B. W., Ash, E., MacLeod, W. B., & Naidu, S. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5201664
Music Psychology and Complex Systems
Sawicki, J. (2025) · Current research in systematic musicology · doi:10.1007/978-3-031-97003-0_2
Bayesian Optimization and Large Language Models
Candelieri, A., Ponti, A., & Archetti, F. (2025) · SpringerBriefs in optimization · doi:10.1007/978-3-031-97965-1_4
Li, Y., & McClelland, J. L. (2025) · Open Mind · doi:10.1162/opmi.a.256
Semler, J. (2025) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.5287/ora-rday5gd7d · Cited by 1
Bondareva, N. A. (2025) · International Conference on Computer Graphics and Vision (GraphiCon) · doi:10.25686/978-5-8158-2474-4-2025-544-554
Masís, J. (2026) · Revista de Filosofía de la Universidad de Costa Rica · doi:10.15517/revfil.2026.2022
Scaling Next-Brain-Token Prediction for MEGPreprint
Csáky, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.20138
Understanding the Emergence of Seemingly Useless Features in Next-Token Predictors
Rofin, M., Naghiyev, J., & Hahn, M. (2026) · International Conference on Learning Representations · doi:10.48550/arxiv.2603.14087
Don’t Judge a Book by its Cover: Testing LLMs’ Robustness Under Logical Obfuscation
Borah, A., Ghosh, S., Joshi, K., Guru, A. K., & Ghosh, K. (2026) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2026.eacl-long.95 · Cited by 3
Symbolizer: Symbolic Model-Free Task Planning With VLMs
Azirar, S., Ajanović, Z., & Blum, H. (2026) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2026.3726377
A Close Look At World Model Recovery In Supervised Fine-Tuned LLM PlannersPreprint
Emami, P., Qiang, N., & Graf, P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.03685
Synthetic Audiences as Proxies for Human Users: A Systematic Literature Review
Lappas, T., & Filippas, A. (2026) · IEEE Access · doi:10.1109/access.2026.3703706
The CRISTAL Method: Neurosymbolic analysis from AI-synthesized world modelsPreprint
Kaufmann, R., Neubürger, F., Walters, M., Kopinski, T., & Marković, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.29799
A Calibrated Test of Internal Action Maps: State Signals Without Global Affine ClosurePreprint
Yang, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.13626
How AI Prompts Can Teach Us About the Structure of Human BehaviorPreprint
Jackson, M. O., Manning, B. S., Xie, Y., Yuan, W., & Mei, Q. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7308598
World Modeling in TransformersPreprint
Beckmann, P., Queloz, M., & Freitas, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.21748
Where Does Jagged Competence Come From?Preprint
Tsiokos, I. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.03831
Two errors in attributing intelligence and a Bayesian Turing testPreprint
Gilestro, G. F. (2026) · PhilSci-Archive (University of Pittsburgh)
Thinking out of distribution: what a PhD must train in the age of AI
Gago, J. E., Tella, R. D., & Shi, J. (2026) · AJE Advances Research in Epidemiology · doi:10.1093/ajeadv/uuag039
CivWorldBench: Evaluating Language World Models in Structured Multi-Agent Sandbox WorldsPreprint
Li, A., Yu, H., Tao, Z., Chen, L., Li, K., Li, M., Cai, C., Liang, B., You, J., & Wong, K. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.16450
Dual Exam: Controlled Evaluation of Multimodal Agents and World ModelsPreprint
Rao, Z.-F., Chu, M., Zhang, X., Chen, J., Cai, G.-H., Li, R.-Y., Du, S.-Y., Huang, X.-W., Lu, N., Chen, T., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.31066
Self-citationMaterials world models: Foundation for scientific continuity, emergence, and creation
熊婕, Liu, Y., & Zhang, T.-Y. (2026) · Originality · doi:10.1016/j.orig.2026.11.001
Metadata
- Authors
- Vafa, K., Chen, J., Rambachan, A., Kleinberg, J., & Mullainathan, S.
- Venue
- Advances in Neural Information Processing Systems 37
- Published
- 2024
- Pages
- 26941-26975
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W4415798308
- Cited by
- 22
- References
- 19
- Citations by year
- 2026: 13 · 2025: 7 · 2024: 2
- FWCI
- 6.46
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- cc-by
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- world models, generative models, model evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.