Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 304 of the 304 references it cites were found in a registry or an index; 48 of the 72 sources its claims cite were checked for support.

Understanding World or Predicting Future? A Comprehensive Survey of World Models

Open access

Abstract

The concept of world models has garnered significant attention due to advancements in multimodal large language models such as GPT-4 and video generation models such as Sora, which are central to the pursuit of artificial general intelligence. This survey offers a comprehensive review of the literature on world models. Generally, world models are regarded as tools for either understanding the present state of the world or predicting its future dynamics. This review presents a systematic categorization of world models, emphasizing two primary functions: (1) constructing internal representations to understand the mechanisms of the world, and (2) predicting future states to simulate and guide decision-making. Initially, we examine the current progress in these two categories. We then explore the application of world models in key domains, including autonomous driving, robotics, and social simulacra, with a focus on how each domain utilizes these aspects. Finally, we outline key challenges and provide insights into potential future research directions. We summarize the representative articles along with their code repositories in https://github.com/tsinghua-fib-lab/World-Model .

Overview

CoChat Summary

TL;DR

This survey categorizes world models by two primary functions: constructing implicit internal representations to understand the world’s mechanisms and predicting future states to simulate and guide decision-making. It reviews progress in decision-making, video generation, embodied environments, and applications in autonomous driving, robotics, and social simulacra, then outlines open challenges and future directions.

Key findings

  • The paper proposes a taxonomy of world models: implicit representation of the external world (e.g., model-based RL and LLM/MLLM world knowledge) and future prediction of the physical world (e.g., video generation and embodied environments). It reports that DreamerV3 solves over 150 tasks—including diamond collection in Minecraft—without human data or domain-specific tuning.
  • Video world models such as Sora can generate high-quality, temporally consistent video sequences up to one minute long, but the survey notes persistent limitations in causal reasoning and physical-law simulation. It cites evidence that scaling diffusion video models yields perfect in-distribution fidelity yet breaks down on out-of-distribution or combinatorial tests, indicating “case-based” rather than rule-based generalization.
  • Applications are organized across autonomous driving, robotics, and social simulacra. For robotics, DayDreamer allows robots to learn locomotion in real-world environments within hours, and SWIM enables task generalization with less than 30 minutes of interaction; for social simulacra, AI Town is described as a world model composed of 25 generative agents.
  • Open challenges include physical rules and counterfactual simulation, enriching the social dimension, bridging simulation and reality with embodied intelligence, simulation efficiency, ethical/safety concerns, and benchmarking. Benchmark examples include WorldScore across 3,000 camera-specified scenes, PhysBench with 10k video-image-text triplets, and UrbanVideo-Bench with 5.2k drone clips.

Methods

  • Design: a comprehensive literature survey that formally defines and categorizes world models around two functions—implicit representation and future prediction—and summarizes representative articles plus code repositories.
  • Data/sample: no primary dataset or experimental sample; the paper synthesizes existing literature across model-based RL, LLM/MLLM world knowledge, video generation, embodied environments, autonomous driving, robotics, and social simulacra, with summary tables of representative works.
  • Main analysis: qualitative synthesis and taxonomy development; it compares technical approaches, application emphases, and open problems rather than conducting meta-analysis or empirical evaluation.

Limitations

  • The paper states that benchmarking world models is challenging because the community pursues divergent goals, uses heterogeneous technical approaches, and spans wide-ranging application domains, so there is no single canonical task or metric. It also states that evaluating the realism of generated human behavior still largely relies on subjective human assessment, which is hard to scale.
  • The paper states that current video generators achieve visual realism but largely fail on tasks requiring understanding of optics, fluid dynamics, or magnetism, and that data-driven scaling alone is insufficient to recover robust physical laws. It also identifies a longstanding simulation-to-reality gap for embodied agents.
  • Evident from the text: as a survey, it presents no primary experiments or quantitative validation of its own; its coverage is organized around representative works and categories rather than a systematic search protocol with inclusion/exclusion criteria.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3746449

Cites

304 of 304 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 The overall framework of this survey. We systematically define the essential purpose of a world model as understanding the dynamics of the external world and predicting future scenarios. The timeline illustrates the development of key definitions and applications.
Fig. 1 The overall framework of this survey. We systematically define the essential purpose of a world model as understanding the dynamics of the external world and predicting future scenarios. The timeline illustrates the development of key definitions and applications.CC-BY · From the PDF
Fig. 2 Two schemes of utilizing world model in decision-making.
Fig. 2 Two schemes of utilizing world model in decision-making.CC-BY · From the PDF
Fig. 3 World knowledge in large language models for world model.
Fig. 3 World knowledge in large language models for world model.CC-BY · From the PDF
Fig. 4 Classification of world models as interactive embodied environments, including indoor, outdoor and dynamic environments. The modeling of the outside world is evolving from constructing static, current environments to predicting dynamic, future environments.
Fig. 4 Classification of world models as interactive embodied environments, including indoor, outdoor and dynamic environments. The modeling of the outside world is evolving from constructing static, current environments to predicting dynamic, future environments.CC-BY · From the PDF
Fig. 5 Application of world model in autonomous driving.
Fig. 5 Application of world model in autonomous driving.CC-BY · From the PDF
Fig. S1 The development of the robotic world model.
Fig. S1 The development of the robotic world model.CC-BY · From the PDF
Fig. S2 World model and social simulacra.
Fig. S2 World model and social simulacra.CC-BY · From the PDF

Verification

Claims the paper makes

33 Valid · 8 Not valid · 13 Not checked · 31 Own results and interpretations

10 more claims · 4 with their cited sources checked

Create a free account to see all 85 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 35
  • Probably not a substantial claim. 53

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 5

Citations

Cites

Show all 48 references

Research works printed but not linked: 26

  1. Yifan Cui, Xinyi Shan, and Jeanhun Chung. 2024. A feasibility study on RUNWAY GEN-2 for generating realistic style images. International Journal of Internet, Broadcasting and Communication 16, 1 (2024), 99–105.
  2. Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. 2022. Glam: Efficient scaling of language models with mixture-of-experts. In Proceedings of the International Conference on Machine Learning. PMLR, 5547–5569.
  3. Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. 2024. Learning universal policies via text-guided video generation. Advances in Neural Information Processing Systems 36 (2024).
  4. Ignat Georgiev, Varun Giridhar, Nicklas Hansen, and Animesh Garg. 2025. PWM: Policy learning with multi-task world models. In Proceedings of the 13th International Conference on Learning Representations.
  5. David Ha and Jürgen Schmidhuber. 2018. Recurrent world models facilitate policy evolution. Advances in Neural Information Processing Systems 31 (2018).
  6. Haoran He, Chenjia Bai, Ling Pan, Weinan Zhang, Bin Zhao, and Xuelong Li. 2024. Learning an actionable discrete diffusion policy via large-scale actionless video pre-training. In Proceedings of the 38th Annual Conference on Neural Information Processing Systems.
  7. Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems 33 (2020), 6840–6851.
  8. Anthony Hu Lloyd Russell Hudson Yeo Zak Murez George Fedoseev Alex Kendall Jamie Shotton and Gianluca Corrado. 2023. GAIA-1: A Generative World Model for Autonomous Driving. arxiv:2309.17080. Retrieved from https://arxiv.org/abs/2309.17080 · arXiv
  9. Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. 2022. Bc-z: Zero-shot task generalization with robotic imitation learning. In Proceedings of the Conference on Robot Learning. PMLR, 991–1002.
  10. 10.5555/3692070.3692961 · doi:10.5555/3692070.3692961
  11. Philip Nicholas Johnson-Laird. 1983. Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness. Number 6. Harvard University Press.
  12. Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Paul Saldyt, and Anil B. Murthy. 2024. Position: LLMs can’t plan, but can help planning in LLM-modulo frameworks. In Proceedings of the 41st International Conference on Machine Learning.
  13. Kuaishou. 2024. Kling AI: Next-Generation AI Creative Studio. Retrieved May 06, 2025 from https://www.klingai.com/global/.
  14. Yann LeCun. 2022. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review 62, 1 (2022), 1–62.
  15. Junhyuk Oh, Satinder Singh, and Honglak Lee. 2017. Value prediction network. Advances in Neural Information Processing Systems 30 (2017).
  16. OpenAI. 2024. Sora: Creating video from text. Retrieved May 06, 2025 from https://openai.com/sora.
  17. Aravind Rajeswaran, Igor Mordatch, and Vikash Kumar. 2020. A game theoretic framework for model based reinforcement learning. In Proceedings of the International Conference on Machine Learning. PMLR, 7953–7963.
  18. Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards real-time object detection with region proposal networks. In Proceedings of the Advances in Neural Information Processing Systems.C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.), Vol. 28, Curran Associates, Inc. Retrieved from https://proceedings.neurips.cc/paper_files/paper/2015/file/14bfa6bb14875e45bba028a21ed38046-Paper.pdf
  19. Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. 2022. Learning to walk in minutes using massively parallel deep reinforcement learning. In Proceedings of the Conference on Robot Learning. PMLR, 91–100.
  20. Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. 2020. Learning to simulate complex physics with graph networks. In Proceedings of the International Conference on Machine Learning. PMLR, 8459–8468.
  21. Hongzhi Shi, Jingtao Ding, Yufan Cao, Li Liu, Yong Li, et al. 2022. Learning symbolic models for graph-structured physical mechanism. In Proceedings of the 11th International Conference on Learning Representations.
  22. A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems 30 (2017).
  23. Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. 2023. Daydreamer: World models for physical robot learning. In Proceedings of the Conference on Robot Learning. PMLR, 2226–2240.
  24. Wilson Yan, Danijar Hafner, Stephen James, and Pieter Abbeel. 2023. Temporally consistent transformers for video generation. In Proceedings of the International Conference on Machine Learning. PMLR, 39062–39098.
  25. Sherry Yang, Yilun Du, Seyed Kamyar Seyed Ghasemipour, Jonathan Tompson, Leslie Pack Kaelbling, Dale Schuurmans, and Pieter Abbeel. 2024. Learning interactive real-world simulators. In Proceedings of the 12th International Conference on Learning Representations.
  26. 10.1007/978-3-031-72624-8_4 · DOI · Paper page

Cited by

See all 169

Metadata

Authors
Ding, J., Zhang, Y., Shang, Y., Zhang, Y., Zong, Z., Feng, J., Yuan, Y., Su, H., Li, N., Sukiennik, N., Xu, F., & Li, Y.
Affiliations
Tsinghua University (CN)
Venue
ACM Computing Surveys
Published
2025-09-09
Volume
58
Issue
3
Pages
1-38
ISSN
  • 0360-0300 (ISSN-L)
  • 1557-7341
OpenAlex
W4411735734
Cited by
169
References
48
Citations by year
2026: 122 · 2025: 46
FWCI
43.25
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • ACM Computing Surveys (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Key Research and Development Program of China (2024YFC3307603)
  • National Natural Science Foundation of China (62476152 and U24B20180)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
Keywords
world models, internal representations, future state prediction, artificial general intelligence, autonomous driving, robotics, social simulation, multimodal large language models, video generation models, decision-making

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Understanding World or Predicting Future? A Comprehensive Survey of World Models. https://app.cochat.ai/papers/10.1145/3746449
The paper
Ding, J., Zhang, Y., Shang, Y., Zhang, Y., Zong, Z., Feng, J., Yuan, Y., Su, H., Li, N., Sukiennik, N., Xu, F., & Li, Y. (2026). Understanding World or Predicting Future? A Comprehensive Survey of World Models. ACM Computing Surveys. https://doi.org/10.1145/3746449

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.