CoChat

Evaluating the World Model Implicit in a Generative Model

Open access
Vafa, Keyon, Chen, Justin, Rambachan, Ashesh, Kleinberg, Jon, Mullainathan, Sendhil · 2024 · Advances in Neural Information Processing Systems 37

The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index; 8 of the 10 sources its claims cite were checked for support.

Abstract

Recent work suggests that large language models may implicitly learn world models. How should we assess this possibility? We formalize this question for the case where the underlying reality is governed by a deterministic finite automaton. This includes problems as diverse as simple logical reasoning, geographic navigation, game-playing, and chemistry. We propose new evaluation metrics for world model recovery inspired by the classic Myhill-Nerode theorem from language theory. We illustrate their utility in three domains: game playing, logic puzzles, and navigation. In all domains, the generative models we consider do well on existing diagnostics for assessing world models, but our evaluation metrics reveal their world models to be far less coherent than they appear. Such incoherence creates fragility: using a generative model to solve related but subtly different tasks can lead to failures. Building generative models that meaningfully capture the underlying logic of the domains they model would be immensely valuable; our results suggest new ways to assess how close a given model is to that goal.

Overview

CoChat Summary

TL;DR

The paper formalizes world-model recovery when the underlying reality is a deterministic finite automaton (DFA) and proposes Myhill-Nerode-inspired metrics for sequence compression and sequence distinction. Across navigation, Othello, and logic puzzles, generative models often pass existing diagnostics while their implicit world models remain incoherent and fragile under detours.

Key findings

  • Existing diagnostics can be misleading: in the NYC taxi map setting, trained transformers score 1.00 on the next-token test and 0.91–0.99 on the current-state probe, but proposed metrics reveal much weaker recovery, e.g. compression precision 0.05–0.50 and distinction recall 0.20–1.00 depending on training data.
  • Detours expose fragility: for shortest-paths models, valid traversals fall from 0.99 at 0% detour probability to 0.69 at 1%, 0.08 at 10%, and 0.00 at 50%/75%; the random-walks model is more robust, remaining at 0.99, 0.99, 1.00, 0.97, and 0.74 across those detour levels.
  • In Othello, the synthetic-game model scores compression precision 0.98, distinction precision 0.99, and distinction recall 1.00, while the championship-game model scores 0.00, 0.65, and 0.27; detour tests show the synthetic model produces near-perfect games while the championship model fails immediately.
  • In logic puzzles, GPT-4 reaches 1.00 task accuracy but only 0.21 compression precision and 0.56 distinction recall; no evaluated model exceeds 40% compression precision or 0.60 distinction recall.

Methods

  • The paper models the true world as a DFA and defines recovery as matching the DFA’s language. It uses the Myhill-Nerode theorem to define interior and boundary sets, then builds sequence compression metrics from equal-state pairs and sequence distinction metrics from distinct-state pairs, comparing model-implied boundaries to true DFA boundaries.
  • Navigation data come from 2014 NYC taxi rides in Manhattan, represented as a graph with 4,580 nodes and 9,846 edges. Three traversal datasets are used: shortest paths (2.9M sequences, 120M tokens), noisy shortest paths (31M sequences, 1.7B tokens), and random walks (91M sequences, 4.7B tokens). GPT-2-style transformers of 89.3M and 1.5B parameters are trained from scratch.
  • Additional applications use Othello sequence models from Li et al. and prompted LLMs on seating-arrangement logic puzzles. Metrics are approximated with Monte Carlo sampling, an acceptance threshold ε = 0.01, graph reconstruction, and detour robustness tests.

Limitations

  • Stated: the primary limitation is the focus on DFAs; extending the approach to more complicated or unknown world models is left to future work.
  • Evident from the text: the metrics require query access to the true DFA and tractable boundary computation, and the LLM logic-puzzle evaluation lacks token-level probability access, relying on prompting with small samples (5 continuations, 100 states) and reporting compression as an overestimate.
  • Evident from the text: results depend on choices such as ε, decoding acceptance rule, Monte Carlo sample count, and reconstruction constraints; the paper reports ablations suggesting conclusions are stable, but the reconstruction algorithm also gives the model the benefit of the doubt and can fail to reconstruct some sequences.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.52202/079017-0846

Cites

40 of 40 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

5 Valid · 2 Not valid · 0 Not checked · 88 Own results and interpretations

56 more claims

Create a free account to see all 95 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 9
  • It may state two separate claims as one. 26

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Research works printed but not linked: 3

  1. [28][28] Patel, R. and Pavlick, E. Mapping language models to grounded conceptual spaces. In International Conference on Learning Representations , 2021. -
  2. [29][29] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019. -
  3. [33][33] Sipser, M. Introduction to the Theory of Computation, Third Edition . Cengage Learning, 2013. -

Cited by

See all 88

Metadata

Authors
Vafa, K., Chen, J., Rambachan, A., Kleinberg, J., & Mullainathan, S.
Venue
Advances in Neural Information Processing Systems 37
Published
2024
Pages
26941-26975
ISSN
1049-5258 (ISSN-L)
OpenAlex
W4415798308
Cited by
22
References
19
Citations by year
2026: 13 · 2025: 7 · 2024: 2
FWCI
6.46
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords
world models, generative models, model evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-11-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.