Evaluating the World Model Implicit in a Generative Model
Open accessThe DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index; 8 of the 10 sources its claims cite were checked for support.
Abstract
Recent work suggests that large language models may implicitly learn world models. How should we assess this possibility? We formalize this question for the case where the underlying reality is governed by a deterministic finite automaton. This includes problems as diverse as simple logical reasoning, geographic navigation, game-playing, and chemistry. We propose new evaluation metrics for world model recovery inspired by the classic Myhill-Nerode theorem from language theory. We illustrate their utility in three domains: game playing, logic puzzles, and navigation. In all domains, the generative models we consider do well on existing diagnostics for assessing world models, but our evaluation metrics reveal their world models to be far less coherent than they appear. Such incoherence creates fragility: using a generative model to solve related but subtly different tasks can lead to failures. Building generative models that meaningfully capture the underlying logic of the domains they model would be immensely valuable; our results suggest new ways to assess how close a given model is to that goal.
Overview
CoChat Summary
TL;DR
The paper formalizes world-model recovery when the underlying reality is a deterministic finite automaton (DFA) and proposes Myhill-Nerode-inspired metrics for sequence compression and sequence distinction. Across navigation, Othello, and logic puzzles, generative models often pass existing diagnostics while their implicit world models remain incoherent and fragile under detours.
Key findings
- Existing diagnostics can be misleading: in the NYC taxi map setting, trained transformers score 1.00 on the next-token test and 0.91–0.99 on the current-state probe, but proposed metrics reveal much weaker recovery, e.g. compression precision 0.05–0.50 and distinction recall 0.20–1.00 depending on training data.
- Detours expose fragility: for shortest-paths models, valid traversals fall from 0.99 at 0% detour probability to 0.69 at 1%, 0.08 at 10%, and 0.00 at 50%/75%; the random-walks model is more robust, remaining at 0.99, 0.99, 1.00, 0.97, and 0.74 across those detour levels.
- In Othello, the synthetic-game model scores compression precision 0.98, distinction precision 0.99, and distinction recall 1.00, while the championship-game model scores 0.00, 0.65, and 0.27; detour tests show the synthetic model produces near-perfect games while the championship model fails immediately.
- In logic puzzles, GPT-4 reaches 1.00 task accuracy but only 0.21 compression precision and 0.56 distinction recall; no evaluated model exceeds 40% compression precision or 0.60 distinction recall.
Methods
- The paper models the true world as a DFA and defines recovery as matching the DFA’s language. It uses the Myhill-Nerode theorem to define interior and boundary sets, then builds sequence compression metrics from equal-state pairs and sequence distinction metrics from distinct-state pairs, comparing model-implied boundaries to true DFA boundaries.
- Navigation data come from 2014 NYC taxi rides in Manhattan, represented as a graph with 4,580 nodes and 9,846 edges. Three traversal datasets are used: shortest paths (2.9M sequences, 120M tokens), noisy shortest paths (31M sequences, 1.7B tokens), and random walks (91M sequences, 4.7B tokens). GPT-2-style transformers of 89.3M and 1.5B parameters are trained from scratch.
- Additional applications use Othello sequence models from Li et al. and prompted LLMs on seating-arrangement logic puzzles. Metrics are approximated with Monte Carlo sampling, an acceptance threshold ε = 0.01, graph reconstruction, and detour robustness tests.
Limitations
- Stated: the primary limitation is the focus on DFAs; extending the approach to more complicated or unknown world models is left to future work.
- Evident from the text: the metrics require query access to the true DFA and tractable boundary computation, and the LLM logic-puzzle evaluation lacks token-level probability access, relying on prompting with small samples (5 continuations, 100 states) and reporting compression as an overestimate.
- Evident from the text: results depend on choices such as ε, decoding acceptance rule, Monte Carlo sample count, and reconstruction constraints; the paper reports ablations suggesting conclusions are stable, but the reconstruction algorithm also gives the model the benefit of the doubt and can fail to reconstruct some sequences.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
5 Valid · 2 Not valid · 0 Not checked · 88 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The classic Myhill-Nerode theorem [26, 27] provides intuition: every pair of distinct states can be distinguished by some sequence (admitted by one state but not the other). Unless those minimal distinguishing sequences are of length one, looking at the next single token outputted will not reliably assess whether the generative model has an accurate model of the underlying state.”
What the sources say
- Not matchedcites [26] [26] Myhill, J. Finite automata and the representation of events. WADD Technical Report , 57: 112-137, 1957. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites [27] [27] Nerode, A. Linear automaton transformations. Proceedings of the American Mathematical Society , 9(4):541-544, 1958. - sourceExperimental — accuracy not yet measured
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Large language models (LLMs) appear to have capacities that far exceed the next-token prediction task they were trained to perform [17, 39, 35].”
What the sources say
- Supportedcites [17] [17] Kıcıman, E., Ness, R., Sharma, A., and Tan, C. Causal reasoning and large language models: Opening a new frontier for causality. arXiv preprint arXiv:2305.00050 , 2023. - source
Section 2.3 describes next-word completion as LLMs’ primary task. The Introduction says they also develop emergent abilities, including writing code and performing reasoning tasks, supporting the claim that their capacities extend beyond that task. (LLM, conf=0.91)
- Partly supportedcites [39] [39] Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 , 2022. - sourceExperimental — accuracy not yet measured
The Discussion (p. 6) says large models exhibit emergent abilities on tasks not explicitly included in pre-training, supporting the claim that their capacities extend beyond their training tasks. It does not characterize next-token prediction as the training task for all the models discussed; Section 5.2 (p. 7) also discusses a different pre-training objective. (LLM, conf=0.82)
- Partly supportedcites [35] [35] Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. Challenging BIG-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261 , 2022. - sourceExperimental — accuracy not yet measured
The Introduction reports that large language models perform challenging BIG-Bench tasks and that chain-of-thought prompting helped a model surpass the average human-rater on 17 of 23 tasks. This supports the claim that their capabilities extend beyond simple text prediction. Our copy does not establish the claim’s specific premise that the models studied were trained only to predict the next token. (LLM, conf=0.82)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We note that this distribution of (origin, destination) pairs varies from the distributions used to train and evaluate each model”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Cumulative Connect-4 exhibits this behavior; any board for which there are less than 𝑘 disks in each column will have the same set of valid moves for the next 𝑛 -𝑘 moves.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is much closer to the true world model than the transformer (which makes errors at the same rate).”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
56 more claims
Create a free account to see all 95 claims and every source verdict.
- Probably not a substantial claim. 9
- It may state two separate claims as one. 26
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4 Other Applications: Othello and Logic Puzzles · p. 10
“no model has a compression precision higher than 40%.”
§ 4 Other Applications: Othello and Logic Puzzles · p. 10
“| Qwen 1.5 (110B) | 0.98 (0.00) | 0.53 (0.05) | 0.53 (0.04) |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.1 Data and models · p. 6
“Weinclude all sequences containing less than 100 directions.”
§ F Rides data construction and training · p. 20
“We also remove sequences with more than 100 tokens.”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Evolutionary-scale prediction of atomic level protein structure with a language model
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. (2022) · doi:10.1101/2022.07.20.500902 · Cited by 5,990
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242
Autonomous chemical research with large language models
Boiko, D. A., MacKnight, R., Kline, B., & Gomes, G. (2023) · Nature · doi:10.1038/s41586-023-06792-0 · Cited by 1,050
Linear automaton transformations
Nerode, A. (1958) · Proceedings of the American Mathematical Society · doi:10.1090/s0002-9939-1958-0135681-9 · Cited by 541
Single-sequence protein structure prediction using a language model and deep learning
Chowdhury, R., Bouatta, N., Biswas, S., Floristean, C., Kharkar, A., Roy, K., Rochereau, C., Ahdritz, G., Zhang, J., Church, G. M., et al. (2022) · Nature Biotechnology · doi:10.1038/s41587-022-01432-w · Cited by 455
Leveraging large language models for predictive chemistry
Jablonka, K. M., Schwaller, P., Ortega-Guerrero, A., & Smit, B. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-023-00788-1 · Cited by 378
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. D. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.06825 · Cited by 340
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16609 · Cited by 332
Transformers Learn Shortcuts to AutomataPreprint
Liu, B., Ash, J. T., Goel, S., Krishnamurthy, A., & Zhang, C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.10749 · Cited by 214
Causal Reasoning and Large Language Models: Opening a New Frontier for Causality
Kıcıman, E., Ness, R., Sharma, A., & Tan, C. (2023) · arXiv · doi:10.48550/arxiv.2305.00050 · Cited by 210
DNA language models are powerful predictors of genome-wide variant effects
Benegas, G., Batra, S. S., & Song, Y. S. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2311219120 · Cited by 157
Designing and Interpreting Probes with Control Tasks
Hewitt, J., & Liang, P. (2019) · arXiv · doi:10.48550/arxiv.1909.03368 · Cited by 110
Implicit Representations of Meaning in Neural Language Models
Li, B. Z., Nye, M., & Andreas, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.143 · Cited by 84
Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color
Abdou, M., Kulmizev, A., Hershcovich, D., Frank, S., Pavlick, E., & Søgaard, A. (2021) · arXiv · doi:10.48550/arxiv.2109.06129 · Cited by 70
Truncation Sampling as Language Model Desmoothing
Hewitt, J., Manning, C. D., & Liang, P. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.249 · Cited by 70
VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View
Schumann, R., Zhu, W., Feng, W., Fu, T.-J., Riezler, S., & Wang, W. Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29858 · Cited by 68
Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color
Abdou, M., Kulmizev, A., Hershcovich, D., Frank, S., Pavlick, E., & Søgaard, A. (2021) · Proceedings of the 25th Conference on Computational Natural Language Learning · doi:10.18653/v1/2021.conll-1.9 · Cited by 64
Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task
Li, K., Hopkins, A. K., Bau, D., Viégas, F., Pfister, H., & Wattenberg, M. (2022) · arXiv · doi:10.48550/arxiv.2210.13382 · Cited by 61
Guan, L., Valmeekam, K., Sreedharan, S., & Subbarao Kambhampati, B. (2023) · neural information processing systems · doi:10.52202/075280-3459 · Cited by 52
On the Ability and Limitations of Transformers to Recognize Formal Languages
Bhattamishra, S., Ahuja, K., & Goyal, N. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.576 · Cited by 48
Emergent Representations of Program Semantics in Language Models Trained on ProgramsPreprint
Jin, C., & Rinard, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.11169 · Cited by 45
Analyzing Generalization of Vision and Language Navigation to Unseen Outdoor Areas
Schumann, R., & Riezler, S. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.518 · Cited by 45
Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.09261 · Cited by 44
Generating Landmark Navigation Instructions from Maps as a Graph-to-Text Problem
Schumann, R., & Riezler, S. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.41 · Cited by 44
On Evaluating the Generalization of LSTM Models in Formal LanguagesPreprint
Süzgün, M., Belinkov, Y., & Shieber, S. M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.01001 · Cited by 42
Linear Latent World Models in Simple Transformers: A Case Study on Othello-GPTPreprint
Hazineh, D., Zhang, Z., & Chiu, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.07582 · Cited by 25
The Parallelism Tradeoff: Limitations of Log-Precision Transformers
Merrill, W., & Sabharwal, A. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00562 · Cited by 22
Kuo, M.-T., Hsueh, C.-C., & Tsai, R. T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.15118 · Cited by 14
On the Ability of Self-Attention Networks to Recognize Counter Languages
Bhattamishra, S., Ahuja, K., & Goyal, N. (2020) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2009.11264 · Cited by 11
Chess as a Testbed for Language Model State Tracking
Toshniwal, S., Wiseman, S., Livescu, K., & Gimpel, K. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21390 · Cited by 9
The Illusion of State in State-Space Models
Merrill, W., Petty, J., & Sabharwal, A. (2024) · arXiv · doi:10.48550/arxiv.2404.08819 · Cited by 5
Truncation Sampling as Language Model DesmoothingPreprint
Hewitt, J., Manning, C. D., & Liang, P. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.15191
Implicit Representations of Meaning in Neural Language Models
Li, B. Z., Nye, M., & Andreas, J. (2021) · arXiv · doi:10.48550/arxiv.2106.00737
Research works printed but not linked: 3
- [28][28] Patel, R. and Pavlick, E. Mapping language models to grounded conceptual spaces. In International Conference on Learning Representations , 2021. -
- [29][29] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019. -
- [33][33] Sipser, M. Introduction to the Theory of Computation, Third Edition . Cengage Learning, 2013. -
Cited by
Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation
Eriksson, M., Purificato, E., Noroozian, A., Vinagre, J., Chaslot, G., Gomez, E., & Fernandez-Llorca, D. (2025) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1609/aies.v8i1.36595 · Cited by 26
ICLR: In-Context Learning of Representations
Park, C. F., Lee, A., Lubana, E. S., Yang, Y., Okawa, M., Nishi, K., WATTENBERG, M. P., & Tanaka, H. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2501.00070 · Cited by 47
How close is AI to human-level intelligence?
Ananthaswamy, A. (2024) · Nature · doi:10.1038/d41586-024-03905-1 · Cited by 32
A Survey on Large Language Models for Automated PlanningPreprint
Aghzal, M., Plaku, E., Stein, G. J., & Yao, Z. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.12435 · Cited by 49
Opening the AI Black Box: Distilling Machine-Learned Algorithms into Code
Michaud, E. J., Liao, I., Lad, V., Liu, Z., Mudide, A., Loughridge, C., Guo, Z. C., Kheirkhah, T. R., Vukelić, M., & Tegmark, M. (2024) · Entropy · doi:10.3390/e26121046 · Cited by 30
Metadata
- Authors
- Vafa, K., Chen, J., Rambachan, A., Kleinberg, J., & Mullainathan, S.
- Venue
- Advances in Neural Information Processing Systems 37
- Published
- 2024
- Pages
- 26941-26975
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W4415798308
- Cited by
- 22
- References
- 19
- Citations by year
- 2026: 13 · 2025: 7 · 2024: 2
- FWCI
- 6.46
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- cc-by
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- world models, generative models, model evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.