The DOI matches its Crossref record; 304 of the 304 references it cites were found in a registry or an index; 48 of the 72 sources its claims cite were checked for support.
Understanding World or Predicting Future? A Comprehensive Survey of World Models
Open accessAbstract
The concept of world models has garnered significant attention due to advancements in multimodal large language models such as GPT-4 and video generation models such as Sora, which are central to the pursuit of artificial general intelligence. This survey offers a comprehensive review of the literature on world models. Generally, world models are regarded as tools for either understanding the present state of the world or predicting its future dynamics. This review presents a systematic categorization of world models, emphasizing two primary functions: (1) constructing internal representations to understand the mechanisms of the world, and (2) predicting future states to simulate and guide decision-making. Initially, we examine the current progress in these two categories. We then explore the application of world models in key domains, including autonomous driving, robotics, and social simulacra, with a focus on how each domain utilizes these aspects. Finally, we outline key challenges and provide insights into potential future research directions. We summarize the representative articles along with their code repositories in https://github.com/tsinghua-fib-lab/World-Model .
Overview
CoChat Summary
TL;DR
This survey categorizes world models by two primary functions: constructing implicit internal representations to understand the world’s mechanisms and predicting future states to simulate and guide decision-making. It reviews progress in decision-making, video generation, embodied environments, and applications in autonomous driving, robotics, and social simulacra, then outlines open challenges and future directions.
Key findings
- The paper proposes a taxonomy of world models: implicit representation of the external world (e.g., model-based RL and LLM/MLLM world knowledge) and future prediction of the physical world (e.g., video generation and embodied environments). It reports that DreamerV3 solves over 150 tasks—including diamond collection in Minecraft—without human data or domain-specific tuning.
- Video world models such as Sora can generate high-quality, temporally consistent video sequences up to one minute long, but the survey notes persistent limitations in causal reasoning and physical-law simulation. It cites evidence that scaling diffusion video models yields perfect in-distribution fidelity yet breaks down on out-of-distribution or combinatorial tests, indicating “case-based” rather than rule-based generalization.
- Applications are organized across autonomous driving, robotics, and social simulacra. For robotics, DayDreamer allows robots to learn locomotion in real-world environments within hours, and SWIM enables task generalization with less than 30 minutes of interaction; for social simulacra, AI Town is described as a world model composed of 25 generative agents.
- Open challenges include physical rules and counterfactual simulation, enriching the social dimension, bridging simulation and reality with embodied intelligence, simulation efficiency, ethical/safety concerns, and benchmarking. Benchmark examples include WorldScore across 3,000 camera-specified scenes, PhysBench with 10k video-image-text triplets, and UrbanVideo-Bench with 5.2k drone clips.
Methods
- Design: a comprehensive literature survey that formally defines and categorizes world models around two functions—implicit representation and future prediction—and summarizes representative articles plus code repositories.
- Data/sample: no primary dataset or experimental sample; the paper synthesizes existing literature across model-based RL, LLM/MLLM world knowledge, video generation, embodied environments, autonomous driving, robotics, and social simulacra, with summary tables of representative works.
- Main analysis: qualitative synthesis and taxonomy development; it compares technical approaches, application emphases, and open problems rather than conducting meta-analysis or empirical evaluation.
Limitations
- The paper states that benchmarking world models is challenging because the community pursues divergent goals, uses heterogeneous technical approaches, and spans wide-ranging application domains, so there is no single canonical task or metric. It also states that evaluating the realism of generated human behavior still largely relies on subjective human assessment, which is hard to scale.
- The paper states that current video generators achieve visual realism but largely fail on tasks requiring understanding of optics, fluid dynamics, or magnetism, and that data-driven scaling alone is insufficient to recover robust physical laws. It also identifies a longstanding simulation-to-reality gap for embodied agents.
- Evident from the text: as a survey, it presents no primary experiments or quantitative validation of its own; its coverage is organized around representative works and categories rather than a systematic search protocol with inclusion/exclusion criteria.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures







Verification
Claims the paper makes
33 Valid · 8 Not valid · 13 Not checked · 31 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“After pretraining on large-scale web text and books [145,201], large language models attain extensive knowledge about the real-world and common sense relevant to daily life.”
§ Understanding World or Predicting Future? A Comprehensive Survey of World Models > 3.2 World Knowledge Learned by Models
What the sources say
- Not matchedcites [145] 145. OpenAI. 2022. Introducing ChatGPT. Retrieved May 06, 2025 from https://openai.com/blog/chatgpt.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites [201] 201. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv:2302.13971. Retrieved from https://arxiv.org/abs/2302.13971 source
The citation follows the pretraining clause, so the relevant attribution is that LLMs are pretrained on large-scale web text and books. Section 2.1 (p. 2) reports CommonCrawl and C4 web data and Gutenberg and Books3 book corpora in LLaMA’s pretraining mixture; it also says these data sources have been used to train other LLMs. (LLM, conf=0.90)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Ha et al. [65,66] significantly revived and popularized the term”
§ Understanding World or Predicting Future? A Comprehensive Survey of World Models > 2 Background and Categorization
What the sources say
- Not matchedcites [65] 65. David Ha and Jürgen Schmidhuber. 2018. Recurrent world models facilitate policy evolution. Advances in Neural Information Processing Systems 31 (2018).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites [66] 66. David Ha and Jürgen Schmidhuber. 2018. World models. arXiv:1803.10122. Retrieved from https://arxiv.org/abs/1803. 10122 sourceExperimental — accuracy not yet measured
The Abstract (p. 1) describes a neural-network world model that learns compressed spatial and temporal representations, supporting the model-description part of the claim. The Introduction (p. 2) discusses earlier RNN-based world models, but our copy does not establish that this 2018 paper significantly revived or popularized the term. (LLM, conf=0.85)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“early work by Ha and Schmidhuber [66] focused on abstracting the external world to gain a deep understanding of its underlying mechanisms.”
§ Understanding World or Predicting Future? A Comprehensive Survey of World Models > 1 Introduction
What the sources say
- Partly supportedcites [66] 66. David Ha and Jürgen Schmidhuber. 2018. World models. arXiv:1803.10122. Retrieved from https://arxiv.org/abs/1803. 10122 sourceExperimental — accuracy not yet measured
Section 3.1 (p. 4) says the world model learns a low-dimensional representation of observations, supporting the abstraction part of the claim. It describes using that representation to train a controller, however, not gaining a deep understanding of the external world's underlying mechanisms. The Introduction (p. 2) likewise frames the work around world models and agent training. (LLM, conf=0.82)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, by generating video-like simulations that capture continuous spatial and temporal dynamics, world models [146, 233] have evolved to address more complex, dynamic environments.”
§ Understanding World or Predicting Future? A Comprehensive Survey of World Models > 4 Future Prediction of the Physical World 4.1 World Model as Video Generation
What the sources say
- Not matchedcites [146] 146. OpenAI. 2024. Sora: Creating video from text. Retrieved May 06, 2025 from https://openai.com/sora.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites [233] 233. Deshun Yang, Luhui Hu, Yu Tian, Zihao Li, Chris Kelly, Bang Yang, Cindy Yang, and Yuexian Zou. 2024. WorldGPT: A sora-inspired video AI agent as rich world models from text and image inputs. arXiv:2403.07944. Retrieved from https://arxiv.org/abs/2403.07944 sourceExperimental — accuracy not yet measured
The Introduction (page not specified) describes video generation using dynamic scene modeling and spatiotemporal prediction, and Section 2.3 describes continuous video sequences with time-and-space consistency. These passages support the video and dynamics aspect, but they do not establish the broader claim that world models have evolved to address more complex, dynamic environments. (LLM, conf=0.82)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“For example, the emergence of deepfake technology gives rise to large-scale misuses that have widespread negative effects on social, economic, and political systems [214].”
§ Understanding World or Predicting Future? A Comprehensive Survey of World Models > Simulating Unsafe Scenario.
What the sources say
- Supportedcites [214] 214. Mika Westerlund. 2019. The emergence of deepfake technology: A review. Technology Innovation Management Review 9, 11 (2019), 39-52. source
The Introduction says deepfakes can quickly reach millions of people and negatively affect society. The Discussion and Conclusion describes them as a major threat to society, the political system, and businesses, detailing disinformation, election interference, and cybersecurity harms. (LLM, conf=0.89)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
10 more claims · 4 with their cited sources checked
Create a free account to see all 85 claims and every source verdict.
- It may state two separate claims as one. 35
- Probably not a substantial claim. 53
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding World or Predicting Future? A Comprehensive Survey of World Models > 5.1.1 Learning Implicit Representations.
“OmniDrive [107] sets up llm-based agents and covers multiple tasks including scene description, counterfactual reasoning and decision making through visual question-answering.”
§ Understanding World or Predicting Future? A Comprehensive Survey of World Models > References
“107. Varun Ravi Kumar, Senthil Yogamani, Hazem Rashed, Ganesh Sistu, Christian Witt, Isabelle Leang, Stefan Milz, and Patrick Mäder. 2023. OmniDet: Surround View Cameras based Multi-task Visual Perception Network for Au- tonomous Driving.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding World or Predicting Future? A Comprehensive Survey of World Models > 5.1.2 World Simulators.
“Copilot4D [248] constructs a world model by predicting changes in radar point cloud data.”
§ Understanding World or Predicting Future? A Comprehensive Survey of World Models > 5.1.1 Learning Implicit Representations.
“Copilot4D [248] 2024 Lidar Diffusion Point Cloud Generation”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding World or Predicting Future? A Comprehensive Survey of World Models > 2 Background and Categorization
“foundational work such as Marvin Minsky's frame representation in the 1960s [136]”
§ Understanding World or Predicting Future? A Comprehensive Survey of World Models > References
“136. Marvin Minsky. 1974. A framework for representing knowledge. MIT, Cambridge.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding World or Predicting Future? A Comprehensive Survey of World Models > 7 Conclusion
“| Indoor | Matterport 3D [18] | Home | 2018 | 90 | V | ✗ | ✗ |”
§ Understanding World or Predicting Future? A Comprehensive Survey of World Models > References
“18. Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. 2017. Matterport3D: Learning from RGB-D data in indoor environments.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding World or Predicting Future? A Comprehensive Survey of World Models > 5.1.1 Learning Implicit Representations.
“Pointnet [158], introduced in 2017, was the first to employ deep learning methods for processing point cloud data.”
§ Understanding World or Predicting Future? A Comprehensive Survey of World Models > References
“158. Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas. 2017. PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space.”
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 2×
“To help a robot understand the objects in the world, visual models such as convolutional neural networks (CNNs) [77, 132, 141] and vision transformers (ViT) [45, 259] integrate visual characteristics of entities into representations, making it possible for robots to recognize critical objects for…”
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,667
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612 · cited 1×
“The two schemes of leveraging world models in decision-making tasks are shown in Figure 2.”
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625
(1995) · Cited by 10,558
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
Charles, R. Q., Su, H., Kaichun, M., & Guibas, L. J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.16 · Cited by 9,237
Self-citationMastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084 · cited 1×
“The two schemes of leveraging world models in decision-making tasks are shown in Figure 2.”
Does the chimpanzee have a theory of mind?
Premack, D., & Woodruff, G. (1978) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00076512 · Cited by 8,743 · cited 1×
“Predictive [59] 2024 NMI Vision Learning Emergent [93] 2024 ICML Language Learning E2WM [224] 2023 NeurIPS Language Learning Dynalang [119] 2024 ICML LanguageandVision Learning”
Cognitive maps in rats and men
Tolman, E. C. (1948) · Psychological Review · doi:10.1037/h0061626 · Cited by 6,641 · cited 1×
“Predictive [59] 2024 NMI Vision Learning Emergent [93] 2024 ICML Language Learning E2WM [224] 2023 NeurIPS Language Learning Dynalang [119] 2024 ICML LanguageandVision Learning”
Dynamic models of segregation†
Schelling, T. C. (1971) · The Journal of Mathematical Sociology · doi:10.1080/0022250x.1971.9989794 · Cited by 4,915 · cited 1×
“Traditionally, such systems have been constructed using expert-defined rules [20, 222], which encode domain knowledge into explicit behavior specifications, or through reinforcement learning [331], which optimizes agent strategies via trial-and-error in simulated environments.”
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
A FRAMEWORK FOR REPRESENTING KNOWLEDGE
Minsky, M. (1988) · Readings in Cognitive Science · doi:10.1016/b978-1-4832-1446-7.50018-2 · Cited by 4,536
Simultaneous localization and mapping: part I
Durrant-Whyte, H., & Bailey, T. (2006) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2006.1638022 · Cited by 4,199
Microscopic Traffic Simulation using SUMO
Lopez, P. A., Wiessner, E., Behrisch, M., Bieker-Walz, L., Erdmann, J., Flotterod, Y.-P., Hilbrich, R., Lucken, L., Rummel, J., & Wagner, P. (2018) · 2018 21st International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2018.8569938 · Cited by 3,297
Khatib, O. (1987) · IEEE Journal on Robotics and Automation · doi:10.1109/jra.1987.1087068 · Cited by 2,940 · cited 1×
“For instance, training a robot to solve a Rubik’s Cube in the real world can require tens of thousands of simulated years [4].”
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1516 · Cited by 2,428
Causal inference in statistics: An overview
Pearl, J. (2009) · Statistics Surveys · doi:10.1214/09-ss057 · Cited by 2,336 · cited 1×
“A key objective of world models is to capture the causal structure of their environments – especially the underlying physical rules – so they can reason about counterfactuals beyond the data distribution [193].”
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187 · cited 5×
“Recent studies suggest that theories of human behavior patterns and cognitive processes can inform the design of agentic workflows, which in turn enhance the human behavior simulation capabilities of LLMs [189, 229], representing an important direction for future research.”
InfluentialPointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space
Qi, C. R., Yi, L., Su, H., & Guibas, L. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.02413 · Cited by 2,114
Self-citationScore-Based Generative Modeling through Stochastic Differential Equations
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2020) · arXiv · doi:10.48550/arxiv.2011.13456 · Cited by 1,953 · cited 1×
“In recent years, with the rapid advancement of vision-based generative models [21, 99, 239] and multimodal large language models [1 ?”
Matterport3D: Learning from RGB-D Data in Indoor Environments
Chang, A., Dai, A., Funkhouser, T., Halber, M., Niebner, M., Savva, M., Song, S., Zeng, A., & Zhang, Y. (2017) · 2017 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2017.00081 · Cited by 1,858
Self-citationHeterogeneous beliefs and routes to chaos in a simple asset pricing model
Brock, W. A., & Hommes, C. H. (1998) · Journal of Economic Dynamics and Control · doi:10.1016/s0165-1889(98)00011-6 · Cited by 1,693 · cited 1×
“Traditionally, such systems have been constructed using expert-defined rules [20, 222], which encode domain knowledge into explicit behavior specifications, or through reinforcement learning [331], which optimizes agent strategies via trial-and-error in simulated environments.”
Li, Z., Wang, W., Li, H., Xie, E., Sima, C., Lu, T., Yu, Q., & Dai, J. (2022) · arXiv · doi:10.48550/arxiv.2203.17270 · Cited by 1,419
Habitat: A Platform for Embodied AI Research
Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., et al. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00943 · Cited by 1,237 · cited 2×
“ConsistI2V [166] Enhances visual consistency in image-to-video generation.”
The Emergence of Deepfake Technology: A Review
Westerlund, M. (2019) · Technology Innovation Management Review · doi:10.22215/timreview/1282 · Cited by 1,138 · cited 1×
“Furthermore, in economic contexts, works such as EconAgent [114] reveal that collective behaviors emerging from individual economic reasoning by LLM agents can convincingly reproduce macroeconomic trends and regularities.”
A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly
Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., & Zhang, Y. (2024) · High-Confidence Computing · doi:10.1016/j.hcc.2024.100211 · Cited by 1,083 · cited 1×
“The recent trend of building world models with big generative AIs raises significant concerns of privacy risk, largely due to the massive and often opaque training data [301].”
Self-citationTransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers
Bai, X., Hu, Z., Zhu, X., Huang, Q., Chen, Y., Fu, H., & Tai, C.-L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00116 · Cited by 990 · cited 2×
“Additionally, Transfusion [10] enhances perceptual accuracy by fusing lidar and camera data through a cross-attention approach.”
Planning-oriented Autonomous Driving
Hu, Y., Yang, J., Chen, L., Li, K., Sima, C., Zhu, X., Chai, S., Du, S., Lin, T., Wang, W., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01712 · Cited by 931
Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning
Nagabandi, A., Kahn, G., Fearing, R. S., & Levine, S. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8463189 · Cited by 848 · cited 1×
“Nagabandi et al.[182] adopt a simple Monte Carlo method to sample action sequences.”
Offline Reinforcement Learning as One Big Sequence Modeling ProblemPreprint
Janner, M., Li, Q., & Levine, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02039 · Cited by 767
Unsupervised Learning for Physical Interaction through Video Prediction
Finn, C., Goodfellow, I., & Levine, S. (2016) · arXiv · doi:10.48550/arxiv.1605.07157 · Cited by 759
Rep ViT: Revisiting Mobile CNN From ViT Perspective
Wang, A., Chen, H., Lin, Z., Han, J., & Ding, G. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01506 · Cited by 752 · cited 2×
“…a robot understand the objects in the world, visual models such as convolutional neural networks (CNNs) [77, 132, 141] and vision transformers (ViT) [45, 259] integrate visual characteristics of entities into representations, making it possible for robots to recognize critical objects for tasks.”
An LSTM network for highway trajectory prediction
Altché, F., & Fortelle, A. D. L. (2017) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2017.8317913 · Cited by 714 · cited 1×
“More recently, with the development of LLMs, some works [21, 48, 238] seek to provide a flexible environment generation pipeline, supporting the generation of arbitrary indoor environments with language instructions.”
Learning Latent Dynamics for Planning from Pixels
Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., & Davidson, J. (2018) · arXiv · doi:10.48550/arxiv.1811.04551 · Cited by 708
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., & Finn, C. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2338 · Cited by 686
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · arXiv · doi:10.48550/arxiv.2304.08485 · Cited by 682
MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving
Teichmann, M., Weber, M., Zollner, M., Cipolla, R., & Urtasun, R. (2018) · 2018 IEEE Intelligent Vehicles Symposium (IV) · doi:10.1109/ivs.2018.8500504 · Cited by 681 · cited 3×
“These works build indoor environments by providing photorealistic settings where agents can practice visual navigation and engage in interactive tasks that mimic real-life home activities.”
A Survey on Trajectory-Prediction Methods for Autonomous Driving
Huang, Y., Du, J., Yang, Z., Zhou, Z., Zhang, L., & Chen, H. (2022) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2022.3167103 · Cited by 677 · cited 1×
“Conversely, when humans operate vehicles, they typically observe and predict the current and future states of other traffic participants to determine their own driving strategies [108].”
Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks
Cui, H., Radosavljevic, V., Chou, F.-C., Lin, T.-H., Nguyen, T., Huang, T.-K., Schneider, J., & Djuric, N. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8793868 · Cited by 676 · cited 1×
“More recently, with the development of LLMs, some works [21, 48, 238] seek to provide a flexible environment generation pipeline, supporting the generation of arbitrary indoor environments with language instructions.”
DriveDreamer: Towards Real-World-Drive World Models for Autonomous Driving
Wang, X., Zhu, Z. H., Huang, G., Chen, X., Zhu, J., & Lu, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73195-2_4 · Cited by 666
Toward large reasoning models: A survey of reinforced reasoning with large language models
Xu, F., Hao, Q., Shao, C., Zong, Z., Li, Y., Wang, J., Zhang, Y., Wang, J., Lan, X., Gong, J., et al. (2025) · Patterns · doi:10.1016/j.patter.2025.101370 · Cited by 646
Self-citationDiffusion policy: Visuomotor policy learning via action diffusion
Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., & Song, S. (2025) · The International Journal of Robotics Research · doi:10.1177/02783649241273668 · Cited by 613
Solving Rubik's Cube with a Robot Hand
OpenAI, Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.07113 · Cited by 602
Deep visual foresight for planning robot motion
Finn, C., & Levine, S. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989324 · Cited by 599 · cited 1×
“This shift provides agents with richer, more varied training experiences, improving their adaptability and generalization in complex, unpredictable real-world situations.”
Dense reinforcement learning for safety validation of autonomous vehicles
Feng, S., Sun, H., Yan, X., Zhu, H., Zou, Z., Shen, S., & Liu, H. X. (2023) · Nature · doi:10.1038/s41586-023-05732-2 · Cited by 587 · cited 1×
“This capacity is crucial for handling rare, mission-critical events (e.g., autonomous-driving corner cases [62]) and for narrowing sim-to-real gaps.”
AI2-THOR: An Interactive 3D Environment for Visual AI
Kolve, E., Mottaghi, R., Han, W., VanderBilt, E., Weihs, L., Herrasti, A., Deitke, M., Ehsani, K., Gordon, D., Zhu, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1712.05474 · Cited by 578 · cited 2×
“Early works on establishing indoor environments like AI2-THOR [129] and Matterport 3D [26] focus on providing only visual information.”
Deepfake Detection: A Systematic Literature Review
Rana, M. S., Nobi, M. N., Murali, B., & Sung, A. H. (2022) · IEEE Access · doi:10.1109/access.2022.3154404 · Cited by 573 · cited 1×
“Thus, detecting AI-generated content has been a key research problem in addressing these risks [208].”
Dream to Control: Learning Behaviors by Latent Imagination
Hafner, D., Lillicrap, T., Ba, J., & Norouzi, M. (2019) · arXiv · doi:10.48550/arxiv.1912.01603 · Cited by 558
Deep reinforcement learning in a handful of trials using probabilistic dynamics models
Chua, K., Calandra, R., McAllister, R., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1805.12114 · Cited by 525
Kouvaritakis, B., & Cannon, M. (2016) · Advanced Textbooks in Control and Signal Processing · doi:10.1007/978-3-319-24853-0 · Cited by 515
RMA: Rapid Motor Adaptation for Legged Robots
Kumar, A., Fu, Z., Pathak, D., & Malik, J. (2021) · Robotics: Science and Systems XVII · doi:10.15607/rss.2021.xvii.011 · Cited by 513 · cited 1×
“…Space Rule-based Traffic Simulation Metadrive [116] 2022 Geometric Space Data-driven Traffic Simulation Trafficbots [252] 2023 Geometric Space Transformer Traffic Simulation Waymax [63] 2024 Geometric Space Data-driven Traffic Simulation End to End Sensor Simulation GAIA-1 [78] 2023 Camera…”
MAGVIT: Masked Generative Video Transformer
Yu, L., Cheng, Y., Sohn, K., Lezama, J., Zhang, H., Chang, H., Hauptmann, A. G., Yang, M., Yuan, H., Essa, I. A., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01008 · Cited by 507 · cited 1×
“A key recent insight is the use of generative video models—particularly those leveraging diffusion [18, 33, 54, 96] and transformer architectures [287, 307]—to implicitly learn environmental dynamics directly from visual data.”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability
Gao, S., Yang, J., Chen, L., Chitta, K., Qiu, Y., Geiger, A., Zhang, J., & Li, H. (2024) · neural information processing systems · doi:10.52202/079017-2906 · Cited by 467
YOLOP: You Only Look Once for Panoptic Driving Perception
Wu, D., Liao, M.-W., Zhang, W.-T., Wang, X.-G., Bai, X., Cheng, W.-Q., & Liu, W.-Y. (2022) · Machine Intelligence Research · doi:10.1007/s11633-022-1339-y · Cited by 466 · cited 2×
“By integrating spatial representations and transitioning from video-based simulations to immersive, embodied environments, world models can now provide a more comprehensive platform for developing agents capable of interacting with complex real-world environments.”
CoverNet: Multimodal Behavior Prediction Using Trajectory Sets
Phan-Minh, T., Grigore, E. C., Boulton, F. A., Beijbom, O., & Wolff, E. M. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01408 · Cited by 438 · cited 1×
“More recently, with the development of LLMs, some works [21, 48, 238] seek to provide a flexible environment generation pipeline, supporting the generation of arbitrary indoor environments with language instructions.”
Guan, L., Valmeekam, K., Sreedharan, S., & Kambhampati, S. (2023) · neural information processing systems · doi:10.52202/075280-3459 · Cited by 413
PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies
Qian, G., Li, Y., Peng, H., Mai, J., Hammoud, H. A. A. K., Elhoseiny, M., & Ghanem, B. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.04670 · Cited by 411
Self-citationSAPIEN: A SimulAted Part-Based Interactive ENvironment
Xiang, F., Qin, Y., Mo, K., Xia, Y., Zhu, H., Liu, F., Liu, M., Jiang, H., Yuan, Y., Wang, H., et al. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01111 · Cited by 406 · cited 1×
“ConsistI2V [166] Enhances visual consistency in image-to-video generation.”
Self-citationStructure and Content-Guided Video Synthesis with Diffusion Models
Esser, P., Chiu, J., Atighehchian, P., Granskog, J., & Germanidis, A. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00675 · Cited by 391 · cited 1×
“A key recent insight is the use of generative video models—particularly those leveraging diffusion [18, 33, 54, 96] and transformer architectures [287, 307]—to implicitly learn environmental dynamics directly from visual data.”
VirtualHome: Simulating Household Activities Via Programs
Puig, X., Ra, K., Boben, M., Li, J., Wang, T., Fidler, S., & Torralba, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00886 · Cited by 378 · cited 2×
“ConsistI2V [166] Enhances visual consistency in image-to-video generation.”
When to Trust Your Model: Model-Based Policy Optimization
Janner, M., Fu, J., Zhang, M., & Levine, S. (2019) · arXiv · doi:10.48550/arxiv.1906.08253 · Cited by 351 · cited 1×
“The two schemes of leveraging world models in decision-making tasks are shown in Figure 2.”
Learning Interactive Real-World Simulators
Yang, S., Du, Y., Ghasemipour, K., Tompson, J., Kaelbling, L., Schuurmans, D., & Abbeel, P. (2023) · arXiv · doi:10.48550/arxiv.2310.06114 · Cited by 346
Structured control for autonomous robots
Simmons, R. G. (1994) · IEEE Transactions on Robotics and Automation · doi:10.1109/70.285583 · Cited by 344 · cited 1×
“Autonomous vehicles typically utilize cameras, radar, and lidar to perceive the real-world, gathering information through images, video data, and point cloud data.”
OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving
Zheng, W., Chen, W., Huang, Y., Zhang, B., Duan, Y., & Lu, J. (2023) · arXiv · doi:10.48550/arxiv.2311.16038 · Cited by 342
Query-Centric Trajectory Prediction
Zhou, Z., Wang, J., Li, Y., & Huang, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01713 · Cited by 333 · cited 3×
“In autonomous driving, for example, world models need to perceive road conditions in real-time [250, 272] and accurately predict their evolution [184, 234, 336], with a particular focus on immediate environmental awareness and forecasting of complex trends.”
Self-citationThreeDWorld: A Platform for Interactive Multi-Modal Physical SimulationPreprint
Gan, C., Schwartz, J., Alter, S., Mrowca, D., Schrimpf, M., Traer, J. A., Freitas, J. D., Kubilius, J., Bhandwaldar, A., Haber, N., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.04954 · Cited by 320
Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology ViewPreprint
Zhang, J., Xu, X., Liu, R., & Deng, S.-M. (2023) · arXiv.org · doi:10.48550/arxiv.2310.02124 · Cited by 300
Testing theory of mind in large language models and humans
Strachan, J. W. A., Albergo, D., Borghini, G., Pansardi, O., Scaliti, E., Gupta, S., Saxena, K., Rufo, A., Panzeri, S., Manzi, G., et al. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01882-z · Cited by 299 · cited 3×
“Testing ToM [240] 2024 NHB Language Benchmark High-order ToM [241] 2024 arxiv Language Benchmark COKE [274] 2024 ACL Language Learning MuMA-ToM [232] 2024 ACL Language&Vision Benchmark SimToM [271] 2024 ACL Language Learning EAI [179] 2024 NeurIPS Language Benchmark LLM-ToM [130] 2024 PNAS Lanauge…”
DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation
Zhao, G., Wang, X., Zhu, Z., Chen, X., Huang, G., Bao, X., & Wang, X. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i10.33130 · Cited by 297
Model-free Deep Reinforcement Learning for Urban Autonomous Driving
Chen, J., Yuan, B., & Tomizuka, M. (2019) · 2019 IEEE Intelligent Transportation Systems Conference (ITSC) · doi:10.1109/itsc.2019.8917306 · Cited by 296
Large language models empowered agent-based modeling and simulation: a survey and perspectives
Gao, C., Lan, X., Li, N., Yuan, Y., Ding, J., Zhou, Z., Xu, F., & Li, Y. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-03611-3 · Cited by 294 · cited 1×
“Simulating the physical elements alone is not sufficient for an advanced world model, since human behavior and social interaction also play a crucial role in many important scenarios [68, 78, 310].”
Self-citationA Survey on Learning-Based Robotic Grasping
Kleeberger, K., Bormann, R., Kraus, W., & Huber, M. F. (2020) · Current Robotics Reports · doi:10.1007/s43154-020-00021-6 · Cited by 290 · cited 1×
“Unlike predefined environments that require manual adjustments, these models allow for the dynamic creation of a wide variety of scenarios, enabling agents to experience diverse, first-person perspectives.”
Inner Monologue: Embodied Reasoning through Planning with Language Models
Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P., Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2207.05608 · Cited by 287 · cited 1×
“For instance, researchers leverage the common sense of large language models for task planning [327], robot control [107], and image understanding [157].”
Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion ModelsPreprint
Black, K., Nakamoto, M., Atreya, P., Walke, H., Finn, C., Kumar, A., & Levine, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.10639 · Cited by 266
Wayformer: Motion Forecasting via Simple & Efficient Attention Networks
Nayakanti, N., Al-Rfou, R., Zhou, A., Goel, K., Refaat, K. S., & Sapp, B. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10160609 · Cited by 261 · cited 1×
“Furthermore, realistic scene rendering typically requires substantial computational resources, which limits the efficiency of virtual traffic scenario generation.”
Learning to Walk in the Real World with Minimal Human EffortPreprint
Ha, S., Xu, P., Tan, Z., Levine, S., & Tan, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.08550 · Cited by 261
A survey on model-based reinforcement learning
Luo, F.-M., Xu, T., Lai, H., Chen, X., Zhang, W., & Yu, Y. (2024) · Science China Information Sciences · doi:10.1007/s11432-022-3696-5 · Cited by 259 · cited 1×
“The learning labels are the trajectories derived from real interaction environments, also called the simulation data [164].”
PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation
Zhang, T.-Y., Yu, H.-X., Wu, R.-D., Feng, B. Y., Zheng, C.-X., Snavely, N., Wu, J., & Freeman, W. T. (2024) · European Conference on Computer Vision · doi:10.48550/arxiv.2404.13026 · Cited by 258
Model-Ensemble Trust-Region Policy Optimization
Kurutach, T., Clavera, I., Duan, Y., Tamar, A., & Abbeel, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.10592 · Cited by 247
Mastering Atari with Discrete World Models
Hafner, D., Lillicrap, T., Norouzi, M., & Ba, J. (2020) · arXiv · doi:10.48550/arxiv.2010.02193 · Cited by 247 · cited 3×
“For high-dimensional environments, representation learning is essential for effective world-model training in MBRL. Early work by Ha and Schmidhuber[88] reconstructs images through an autoencoder–latent-state pipeline, whereas Hafner et al. [91, 93] couple a visual encoder with latent dynamics to master pixel-based co…”
Motion Transformer with Global Intention Localization and Local Movement Refinement
Shi, S., Jiang, L., Dai, D., & Schiele, B. (2022) · neural information processing systems · doi:10.52202/068431-0473 · Cited by 247
MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning
Li, Q., Peng, Z., Feng, L., Zhang, Q., Xue, Z., & Zhou, B. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3190471 · Cited by 246 · cited 2×
“Motion Simulation SUMO [162] 2000 Geometric Space Rule-based Traffic Simulation Metadrive [150] 2022 Geometric Space Data-driven Traffic Simulation Trafficbots [320] 2023 Geometric Space Transformer Traffic Simulation Waymax [84] 2024 Geometric Space Data-driven Traffic Simulation End to End Sensor…”
DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video GenerationPreprint
Zhao, G., Wang, X.-F., Zhu, Z., Chen, X., Huang, G., Bao, X.-Y., & Wang, X. (2024) · arXiv.org · doi:10.48550/arxiv.2403.06845 · Cited by 246
Do generative video models understand physical principles?
Motamed, S., Culp, L., Swersky, K., Jaini, P., & Geirhos, R. (2026) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00099 · Cited by 244
PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation
Zhang, T., Yu, H.-X., Wu, R., Feng, B. Y., Zheng, C., Snavely, N., Wu, J., & Freeman, W. T. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72627-9_22 · Cited by 238
iVideoGPT: Interactive VideoGPTs are Scalable World Models
Wu, J., Yin, S., Feng, N., He, X., Li, D., Hao, J., & Long, M. (2024) · neural information processing systems · doi:10.52202/079017-2173 · Cited by 231
Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical GuaranteesPreprint
Luo, Y., Xu, H., Li, Y., Tian, Y., Darrell, T., & Ma, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.03858 · Cited by 222
Self-citationPhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation
Liu, S., Ren, Z., Gupta, S., & Wang, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73007-8_21 · Cited by 222
Naturalistic Driver Intention and Path Prediction Using Recurrent Neural Networks
Zyner, A., Worrall, S., & Nebot, E. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2913166 · Cited by 219 · cited 1×
“Building on the perceptual results, a series of techniques such as RNNs [7, 127, 339], CNNs [36, 40, 194], and Transformers [109, 184, 234, 336] have been employed to encode historical scene information and predict the future behaviors of traffic participants.”
Text2Motion: from natural language instructions to feasible plans
Lin, K., Agia, C., Migimatsu, T., Pavone, M., & Bohg, J. (2023) · Autonomous Robots · doi:10.1007/s10514-023-10131-7 · Cited by 215 · cited 2×
“Text2Motion [155] splits the natural language instruction into task-level and motion-level plans with LLM to handle complex sequential manipulation tasks.”
IRASim: A Fine-Grained World Model for Robot Manipulation
Zhu, F., Wu, H., Guo, S., Liu, Y., Cheang, C., & Kong, T. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00917 · Cited by 214
Social Simulacra: Creating Populated Prototypes for Social Computing Systems
Park, J. S., Popowski, L., Cai, C., Morris, M. R., Liang, P., & Bernstein, M. S. (2022) · Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3526113.3545616 · Cited by 212 · cited 1×
“One effective way to operationalize societal intelligence in silico is through social simulacra [190], which are virtual social computing systems populated by diverse agents capable of realistic, human-like behaviors.”
Agha–mohammadi, A., Otsu, K., Morrell, B. J., Fan, D. D., Thakker, R., Santamaria‐Navarro, À., Kim, S.-K., Bouman, A., Lei, X., Edlund, J. A., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.11470 · Cited by 212
Uncertainty-Aware Reinforcement Learning for Collision Avoidance
Kahn, G., Villaflor, A., Pong, V. H., Abbeel, P., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.01182 · Cited by 209
Discuss Before Moving: Visual Language Navigation via Multi-expert Discussions
Long, Y., Li, X., Cai, W., & Dong, H. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611565 · Cited by 205 · cited 1×
“For example, Long et al. [161] propose a multi-expert scheme to handle visual language navigation tasks.”
Motion Transformer with Global Intention Localization and Local Movement Refinement
Shi, S., Jiang, L., Dai, D., & Schiele, B. (2022) · arXiv · doi:10.48550/arxiv.2209.13508 · Cited by 195 · cited 3×
“In autonomous driving, for example, world models need to perceive road conditions in real-time [250, 272] and accurately predict their evolution [184, 234, 336], with a particular focus on immediate environmental awareness and forecasting of complex trends.”
Multi-modal Motion Prediction with Transformer-based Neural Network for Autonomous Driving
Huang, Z., Mo, X., & Lv, C. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9812060 · Cited by 194 · cited 1×
“Building on the perceptual results, a series of techniques such as RNNs [7, 127, 339], CNNs [36, 40, 194], and Transformers [109, 184, 234, 336] have been employed to encode historical scene information and predict the future behaviors of traffic participants.”
Mastering diverse control tasks through world models
Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. (2025) · Nature · doi:10.1038/s41586-025-08744-2 · Cited by 189 · cited 12×
“The robotics domain has similarly advanced, exemplified by DayDreamer in 2023 [275] and continuing with recent developments in 2025 for robotic manipulation tasks [163].”
InfluentialHa, D., & Schmidhuber, J. (2018) · Zenodo · doi:10.5281/zenodo.1207631 · Cited by 187
GenSim: Generating Robotic Simulation Tasks via Large Language ModelsPreprint
Wang, L., Ling, Y., Yuan, Z., Shridhar, M., Chen, B., Qin, Y., Wang, B., Xu, H., & Wang, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.01361 · Cited by 180
MgNet: A unified framework of multigrid and convolutional neural network
He, J., & Xu, J. (2019) · Science China Mathematics · doi:10.1007/s11425-019-9547-2 · Cited by 179 · cited 1×
“As convolutional neural networks advanced, perception techniques based on image data, exemplified by YOLOP [272] and MultiNet [250], emerged and excelled in driving scene understanding tasks [97, 136, 257, 334].”
iVideoGPT: Interactive VideoGPTs are Scalable World ModelsPreprint
Wu, J., Yin, S.-F., Feng, N.-Y., et al. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2405.15223 · Cited by 179
iGibson 1.0: A Simulation Environment for Interactive Tasks in Large Realistic Scenes
Shen, B., Xia, F., Li, C., Martin-Martin, R., Fan, L., Wang, G., Perez-D'Arpino, C., Buch, S., Srivastava, S., Tchapmi, L., et al. (2021) · 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636667 · Cited by 178 · cited 1×
“ConsistI2V [166] Enhances visual consistency in image-to-video generation.”
Structured World Models from Human Videos
Mendonca, R., Bahl, S., & Pathak, D. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.012 · Cited by 177 · cited 3×
“Moreover, the concept of world models has expanded beyond pure imagination, finding application in diverse scenario-specific simulations [19, 23, 101, 156, 173, 174, 265, 267], encompassing natural environments, games, autonomous driving, and robotics.”
NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation
Yin, S., Wu, C., Yang, H., Wang, J., Wang, X., Ni, M., Yang, Z., Li, L., Liu, S., Yang, F., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.73 · Cited by 167
Tamburri, D. A. (2019) · Information Systems · doi:10.1016/j.is.2019.101469 · Cited by 162 · cited 1×
“To be compliant with privacy regulations like GDPR [244], it is important to improve the transparency of the life cycle of generative AIs, helping the public understand how data is collected, stored, and used in these AI models.”
MotionDiffuser: Controllable Multi-Agent Motion Prediction Using Diffusion
Jiang, C. “., Cornman, A., Park, C., Sapp, B., Zhou, Y., & Anguelov, D. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00930 · Cited by 162
SE-ResUNet: A Novel Robotic Grasp Detection Method
Yu, S., Zhai, D.-H., Xia, Y., Wu, H., & Liao, J. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3145064 · Cited by 159 · cited 1×
“Deep reinforcement learning has demonstrated remarkable capabilities in robotics, enabling autonomous performance in complex tasks such as stable locomotion [135, 238], precise object manipulation [44, 308], and intricate activities like tying shoelaces [6].”
VideoGPT: Video Generation using VQ-VAE and Transformers
Yan, W., Zhang, Y., Abbeel, P., & Srinivas, A. (2021) · arXiv · doi:10.48550/arxiv.2104.10157 · Cited by 156 · cited 1×
“We consider this process as it manifests in how autonomous vehicles learn a world model in latent space.”
Self-citationProcTHOR: Large-Scale Embodied AI Using Procedural Generation
Deitke, M., VanderBilt, E., Herrasti, A., Weihs, L., Salvador, J., Ehsani, K., Han, W., Kolve, E., Farhadi, A., Kembhavi, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.06994 · Cited by 155
(2022) · doi:10.52202/068431-0433 · Cited by 155
Sim2Real in Robotics and Automation: Applications and Challenges
Hofer, S., Bekris, K., Handa, A., Gamboa, J. C., Mozifian, M., Golemo, F., Atkeson, C., Fox, D., Goldberg, K., Leonard, J., et al. (2021) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2021.3064065 · Cited by 154 · cited 1×
“Social simulacra can be seen as a form of world model that mirrors real-world social computing systems.”
Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimization
Zhang, W., Tang, K., Wu, H. H., Wang, M., Shen, Y., Hou, G., Tan, Z., Li, P., Zhuang, Y., & Lü, W. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.292 · Cited by 153
A Survey on the Memory Mechanism of Large Language Model based Agents
Zhang, Z., Bo, X., Ma, C., Li, R., Chen, X., Dai, Q., Zhu, J., Dong, Z., & Wen, J.-R. (2024) · arXiv · doi:10.48550/arxiv.2404.13501 · Cited by 149 · cited 1×
“These models provide the cognitive substrate for societal intelligence, allowing agents not only to recall past interactions but also to reason about other agents and the broader environment when making decisions [319].”
HybridNets: End-to-End Perception Network
Dat, V. T., Bao, N. V. H., & Hung, P. D. (2025) · Pattern Recognition and Image Analysis · doi:10.1134/s1054661825700038 · Cited by 148
International Conference on 3D Vision (3DV 2018)
(2018) · The Photogrammetric Record · doi:10.1111/phor.3_12246 · Cited by 146
Digital twins in city planning
Batty, M. (2024) · Nature Computational Science · doi:10.1038/s43588-024-00606-7 · Cited by 146 · cited 1×
“For example, the behavior of urban dwellers is particularly important for building world models of the urban environment [16, 284].”
Exploring Model-based Planning with Policy NetworksPreprint
Wang, T., & Ba, J. L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.08649 · Cited by 140
GPT4GEO: How a Language Model Sees the World's GeographyPreprint
Roberts, J., Lüddecke, T., Das, S., Han, K., & Albanie, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.00020 · Cited by 137
CityGPT: Empowering Urban Spatial Cognition of Large Language Models
Feng, J., Liu, T., Du, Y., Guo, S., Lin, Y., & Li, Y. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3736878 · Cited by 137
Self-citationDriving in Dense Traffic with Model-Free Reinforcement Learning
Saxena, D. M., Bae, S., Nakhaei, A., Fujimura, K., & Likhachev, M. A. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9197132 · Cited by 136 · cited 1×
“In the initial decision-making paradigm [29, 221], models often take perceptual data as input and directly output motion planning results for the autonomous vehicle.”
A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement LearningPreprint
Smith, L. E., Kostrikov, I., & Levine, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.07860 · Cited by 133
Scalable watermarking for identifying large language model outputs
Dathathri, S., See, A., Ghaisas, S., Huang, P.-S., McAdam, R., Welbl, J., Bachani, V., Kaskasoli, A., Stanforth, R., Matejovicova, T., et al. (2024) · Nature · doi:10.1038/s41586-024-08025-4 · Cited by 132 · cited 1×
“Technology like watermarking could help improve the accountability of world model usage [41].”
Agent-Pro: Learning to Evolve via Policy-Level Reflection and OptimizationPreprint
Zhang, W., Tang, K., Wu, H., et al. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2402.17574 · Cited by 132
LLMs achieve adult human performance on higher-order theory of mind tasks
Street, W., Siy, J. O., Keeling, G., Baranès, A., Barnett, B., McKibben, M., Kanyere, T., Lentz, A., y Arcas, B. A., & Dunbar, R. (2026) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2025.1633272 · Cited by 132
Video Prediction Models as Rewards for Reinforcement Learning
Escontrela, A., Adeniji, A., Yan, W., Jain, A., Peng, X. B., Goldberg, K., Lee, Y. S., Hafner, D., & Abbeel, P. (2023) · neural information processing systems · doi:10.52202/075280-3009 · Cited by 128
AnyHome: Open-Vocabulary Generation of Structured and Textured 3D Homes
Fu, R., Wen, Z., Liu, Z., & Sridhar, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72933-1_4 · Cited by 124
OmniDet: Surround View Cameras Based Multi-Task Visual Perception Network for Autonomous Driving
Ravi Kumar, V., Yogamani, S., Rashed, H., Sitsu, G., Witt, C., Leang, I., Milz, S., & Mader, P. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3062324 · Cited by 121
Wang, Y., He, J., Fan, L., Li, H., Chen, Y., & Zhang, Z. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01397 · Cited by 119
Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs
Sap, M., LeBras, R., Fried, D., & Choi, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.13312 · Cited by 118
Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs
Sap, M., Le Bras, R., Fried, D., & Choi, Y. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.248 · Cited by 118
WorldDreamer: Towards General World Models for Video Generation via Predicting Masked TokensPreprint
Wang, X., Zhu, Z. D., Huang, G., Wang, B.-Y., Chen, X., & Lu, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.09985 · Cited by 115
S3: Social-network Simulation System with Large Language Model-Empowered Agents
Gao, C., Lan, X., Lu, Z., Mao, J., Piao, J., Wang, H., Jin, D., & Li, Y. (2023) · doi:10.2139/ssrn.4607026 · Cited by 113
Self-citationVideo as the New Language for Real-World Decision MakingPreprint
Yang, S. X., Walker, J., Parker-Holder, J., Du, Y., Bruce, J., Barreto, A., Abbeel, P., & Schuurmans, D. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.17139 · Cited by 106
The AI economist: taxation policy design via two-level deep multiagent reinforcement learning
Zheng, S., Trott, A., Srinivasa, S., Parkes, D. C., & Socher, R. (2022) · Science Advances · doi:10.1126/sciadv.abk2607 · Cited by 105 · cited 1×
“Traditionally, such systems have been constructed using expert-defined rules [20, 222], which encode domain knowledge into explicit behavior specifications, or through reinforcement learning [331], which optimizes agent strategies via trial-and-error in simulated environments.”
GRUtopia: Dream General Robots in a City at ScalePreprint
Wang, H., Chen, J., Huang, W., Ben, Q., Wang, T., Mi, B., Huang, T., Zhao, S., Chen, Y.-L., Yang, S., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.10943 · Cited by 103
Peekaboo: Interactive Video Generation via Masked-Diffusion
Jain, Y., Nasery, A., Vineet, V., & Behl, H. S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00772 · Cited by 102 · cited 2×
“Transformer PEEKABOO [113] Enhances interactivity with spatiotemporal control without extra training.”
OpenEQA: Embodied Question Answering in the Era of Foundation Models
Majumdar, A., Ajay, A., Zhang, X., Putta, P., Yenamandra, S., Henaff, M., Silwal, S., Mcvay, P., Maksymets, O., Arnaud, S., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01560 · Cited by 101 · cited 1×
“DayDreamer [275] 2023 Video RSSM SWIM [173] 2023 Video Transfer Learning CoSTAR [3] 2021 Multimodal Belief Space Evaluation OpenEQA [167] 2024 Image& Text LLM First, embodied world models can synthesize high-quality robot action videos to augment real-world collected data, thereby enhancing the…”
Holodeck: Language Guided Generation of 3D Embodied AI Environments
Yang, Y., Sun, F.-Y., Weihs, L., Vanderbilt, E., Herrasti, A., Han, W., Wu, J., Haber, N., Krishna, R., Liu, L., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01536 · Cited by 101 · cited 1×
“Training in such environments can help agents learn knowledge across a broad range of tasks and terrains, enabling them to operate effectively in various outdoor environments.”
TrafficBots: Towards World Models for Autonomous Driving Simulation and Motion Prediction
Zhang, Z., Liniger, A., Dai, D., Yu, F., & Gool, L. V. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10161243 · Cited by 100 · cited 2×
“These simulations represent future states of the scenario in the form of geometric trajectories [84, 150, 162, 320], which require further modeling and rendering to produce outputs suitable for vehicle perception.”
Urban Generative Intelligence (UGI): A Foundational Platform for Agents in Embodied City EnvironmentPreprint
Xu, F., Zhang, J., Gao, C., Feng, J., & Li, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.11813 · Cited by 100
Self-citationOccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Driving
Wang, L., Zheng, W., Ren, Y., Jiang, H., Cui, Z., Yu, H., & Lu, J. (2026) · IEEE Transactions on Image Processing · doi:10.1109/tip.2026.3687468 · Cited by 99
MuMA-ToM: Multi-modal Multi-Agent Theory of Mind
Shi, H., Ye, S., Fang, X., Jin, C., Işık, L., Kuo, Y., & Shu, T. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i2.32142 · Cited by 98
CityBench: Evaluating the Capabilities of Large Language Models for Urban Tasks
Feng, J., Zhang, J., Liu, T., Zhang, X., Ouyang, T., Yan, J., Du, Y., Guo, S., & Li, Y. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737375 · Cited by 97
Self-citationLarge language models as commonsense knowledge for large-scale task planning
Zhao, Z., Lee, W. S., & Hsu, D. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1387 · Cited by 95
Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion
Zhang, L., Xiong, Y., Yang, Z., Casas, S., Hu, R., & Urtasun, R. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2311.01017 · Cited by 92
NEWTON: Are Large Language Models Capable of Physical Reasoning?
Wang, Y., Duan, J., Fox, D., & Srinivasa, S. S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.652 · Cited by 90
Derivative-Free Optimization via Classification
Yu, Y., Qian, H., & Hu, Y.-Q. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10289 · Cited by 90 · cited 1×
“Further literature also improves the optimization efficiency by leveraging the world model usage [91, 103, 264, 309].”
ClarifyGPT: Empowering LLM-based Code Generation with Intention ClarificationPreprint
Mu, F.-W., Shi, L., Wang, S., Yu, Z.-H., Zhang, B., Wang, C.-X., Liu, S., & Wang, Q. (2023) · arXiv.org · doi:10.48550/arxiv.2310.10996 · Cited by 87
GNM: A General Navigation Model to Drive Any Robot
Shah, D., Sridhar, A., Bhorkar, A., Hirose, N., & Levine, S. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10161227 · Cited by 85 · cited 1×
“For robotics, world models are essential for tasks such as navigation [225], object detection [259], and task planning [92], requiring a precise understanding of external dynamics [69] and the ability to generate interactive and embodied environments [189].”
Scene Transformer: A unified architecture for predicting multiple agent trajectories
Ngiam, J., Caine, B., Vasudevan, V., Zhang, Z., Chiang, H.-T. L., Ling, J., Roelofs, R., Bewley, A., Liu, C., Venugopal, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2106.08417 · Cited by 84
Smith, L., Kostrikov, I., & Levine, S. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.056 · Cited by 82
MuMA-ToM: Multi-modal Multi-Agent Theory of MindPreprint
Shi, H.-J., Ye, S.-Y., Fang, X., Jin, C.-Y., Isik, L., Kuo, Y.-L., & Shu, T.-M. (2024) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2408.12574 · Cited by 81
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
Shridhar, M., Yuan, X., Côté, M.-A., Bisk, Y., Trischler, A., & Hausknecht, M. (2020) · arXiv · doi:10.48550/arxiv.2010.03768 · Cited by 80 · cited 1×
“Furthermore, Li et al. [117] discover brain-like structures of world knowledge embedded in the high-dimensional vectors that represent the universe of concepts in large language models.”
Learning to Model the World with LanguagePreprint
Lin, J., Du, Y., Watkins, O., Hafner, D., Abbeel, P., Klein, D., & Dragan, A. D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.01399 · Cited by 80
The Geometry of Concepts: Sparse Autoencoder Feature Structure
Li, Y., Michaud, E. J., Baek, D. D., Engels, J., Sun, X., & Tegmark, M. (2025) · Entropy · doi:10.3390/e27040344 · Cited by 80
Self-citationWorld Models for Autonomous Driving: An Initial Survey
Guan, Y., Liao, H., Li, Z., Hu, J., Yuan, R., Zhang, G., & Xu, C. (2025) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3398357 · Cited by 76
MatrixVT: Efficient Multi-Camera to BEV Transformation for 3D Perception
Zhou, H., Ge, Z., Li, Z., & Zhang, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00785 · Cited by 75 · cited 1×
“These simulations represent future states of the scenario in the form of geometric trajectories [63, 116, 127, 252], which require further modeling and rendering to produce outputs suitable for vehicle perception.”
Predicting Motion of Vulnerable Road Users using High-Definition Maps and Efficient ConvNets
Chou, F., Lin, T.-H., Cui, H., Radosavljević, V., Nguyen, T., Huang, T.-K., Niedoba, M., Schneider, J., & Djuric, N. (2020) · IEEE Intelligent Vehicles Symposium · doi:10.1109/iv47402.2020.9304564 · Cited by 74 · cited 1×
“Building on the perceptual results, a series of techniques such as RNNs [7, 127, 339], CNNs [36, 40, 194], and Transformers [109, 184, 234, 336] have been employed to encode historical scene information and predict the future behaviors of traffic participants.”
ClarifyGPT: A Framework for Enhancing LLM-Based Code Generation via Requirements Clarification
Mu, F., Shi, L., Wang, S., Yu, Z., Zhang, B., Wang, C., Liu, S., & Wang, Q. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660810 · Cited by 74
Multi-robot grasp planning for sequential assembly operations
Doğar, M. R., Spielberg, A., Baker, S., & Rus, D. L. (2018) · Autonomous Robots · doi:10.1007/s10514-018-9748-z · Cited by 73 · cited 1×
“A central challenge in this domain is equipping robots with the ability to reason about their environment’s dynamics to support robust, real-time decision-making.”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge
Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., Tang, A., Huang, D.-A., Zhu, Y., & Anandkumar, A. (2022) · arXiv · doi:10.48550/arxiv.2206.08853 · Cited by 72 · cited 2×
“In addition to the above real open-world generation works, there are also some virtual open-world platforms like MineDOJO [55] that extend these challenges even further by simulating procedurally generated, sandbox-like environments.”
RoboCraft: Learning to see, simulate, and shape elasto-plastic objects in 3D with graph networks
Shi, H., Xu, H., Huang, Z., Li, Y., & Wu, J. (2023) · The International Journal of Robotics Research · doi:10.1177/02783649231219020 · Cited by 72 · cited 2×
“RoboCraft [231] transfers visual observation into particles and captures the structure of the underlying system through a graph neural network.”
Self-citationEmbodiedCity: A Benchmark Platform for Embodied Agent in Real-world City EnvironmentPreprint
Gao, C., Zhao, B., Zhang, W., Mao, J., Zhang, J., Zheng, Z., Man, F., Fang, J., Zhou, Z., Cui, J., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.09604 · Cited by 71
Self-citationCai, S., Chan, E. R., Peng, S., Shahbazi, M., Obukhov, A., Gool, L. V., & Wetzstein, G. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00204 · Cited by 69 · cited 2×
“DiffDreamer [24] Long-range scene extrapolation with improved consistency.”
OpenCity: A Scalable Platform to Simulate Urban Activities with Massive LLM AgentsPreprint
Yan, Y., Zeng, Q., Zheng, Z., Yuan, J., Feng, J., Zhang, J., Xu, F., & Li, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.21286 · Cited by 67
Self-citationHybridNets: End-to-End Perception NetworkPreprint
Vu, D. T., Ngo, B., & Phan, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.09035 · Cited by 65
Chain-of-Planned-Behaviour Workflow Elicits Few-Shot Mobility Generation in LLMsPreprint
Shao, C., Xu, F., Fan, B., Ding, J., Yuan, Y., Wang, M., & Li, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.09836 · Cited by 65
Self-citationGuan, L., Valmeekam, K., Sreedharan, S., & Kambhampati, S. (2023) · arXiv · doi:10.48550/arxiv.2305.14909 · Cited by 64 · cited 2×
“This capability has been harnessed for model-based task planning, where pre-trained language models serve as the foundation for constructing world models that can reason about complex sequential tasks [82].”
Network formation and dynamics among multi-LLMs
Papachristou, M., & Yuan, Y. (2025) · PNAS Nexus · doi:10.1093/pnasnexus/pgaf317 · Cited by 62
Self-citationLiu, Z., Hu, H., Zhang, S., Guo, H., Ke, S., Liu, B., & Wang, Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.17382 · Cited by 61
Multi-robot grasp planning for sequential assembly operations
Doğar, M. R., Spielberg, A., Baker, S., & Rus, D. L. (2015) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2015.7138999 · Cited by 60
EconAgent: Large Language Model-Empowered Agents for Simulating Macroeconomic Activities
Li, N., Gao, C., Li, M., Li, Y., & Liao, Q. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.829 · Cited by 59 · cited 3×
“…Social networks Predictions Papachristou et al. [188] Network formation Stylized facts Xu et al. [286] Social games Strategic patterns EconAgent [148] Macroeconomics Stylized facts SRAP-Agent [118] Resource allocation Stylized facts Project Sid [5] Collective rules Stylized facts OASIS [300]…”
Self-citationMastering Memory Tasks with World Models
Samsami, M. R., Zholus, A., Rajendran, J., & Chandar, S. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2403.04253 · Cited by 58
Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusion
Deng, B., Tucker, R. P., Li, Z., Guibas, L. J., Snavely, N., & Wetzstein, G. (2024) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/3641519.3657513 · Cited by 57 · cited 1×
“As for the generation of open-world dynamic environments, Streetscapes [43] employs autoregressive video diffusion models to simulate urban environments where agents must navigate dynamic challenges like changing weather and traffic.”
Sequential Classification-Based Optimization for Direct Policy Search
Hu, Y.-Q., Qian, H., & Yu, Y. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10927 · Cited by 56 · cited 1×
“Further literature also improves the optimization efficiency by leveraging the world model usage [91, 103, 264, 309].”
Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
Inan, H., Upasani, K., Chi, J., Rungta, R., Iyer, K., Mao, Y., Tontchev, M., Hu, Q., Fuller, B., Testuggine, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.06674 · Cited by 56 · cited 1×
“Previous studies on LLMs found they can be misled to generate unsafe content with adversarial prompting [111, 134].”
MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representations
Bogdoll, D., Yang, Y., Joseph, T., Yazgan, M., & Zöllner, J. M. (2025) · IEEE Intelligent Vehicles Symposium · doi:10.1109/iv64158.2025.11097718 · Cited by 55
UrbanWorld: An Urban World Model for 3D City GenerationPreprint
Shang, Y., Lin, Y., Zheng, Y., Fan, H., Ding, J., Feng, J., Chen, J., Tian, L., & Li, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.11965 · Cited by 53
Self-citationExploring Collaboration Mechanisms for LLM Agents: A Social Psychology View
Zhang, J., Xu, X., Zhang, N., Liu, R., Hooi, B., & Deng, S. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.782 · Cited by 53
UniWorld: Autonomous Driving Pre-training via World ModelsPreprint
Min, C., Zhao, D., Xiao, L., Nie, Y., & Dai, B. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.07234 · Cited by 51
Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research
Gulino, C., Fu, J., Luo, W., Tucker, G., Bronstein, E., Lu, Y., Harb, J., Pan, X., Wang, Y., Chen, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.08710 · Cited by 48
Language Models Represent Space and Time
Gurnee, W., & Tegmark, M. (2023) · arXiv · doi:10.48550/arxiv.2310.02207 · Cited by 46 · cited 5×
“Gurnee et al. [87] present the first evidence that large language models genuinely acquire spatial and temporal knowledge of the world, rather than merely collecting superficial statistics.”
InfluentialVirtualHome: Simulating Household Activities via ProgramsPreprint
Puig, X., Ra, K., Boben, M., Li, J., Wang, T., Fidler, S., & Torralba, A. B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07011 · Cited by 46
Automated construction of cognitive maps with visual predictive coding
Gornet, J., & Thomson, M. W. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00863-1 · Cited by 45 · cited 4×
“PointNet [199] 2017 3D point clouds MLP Predictive Coding [79] 2024 Image ResNet SpatialLM [249] 2025 3D point clouds MLLM Task Representation BC-Z [114] 2022 Text& Video LLM& ResNet Text2Motion [155] 2023 Text LLM Gensim [263] 2023 Text LLM PredictingFutureEnvironment”
InfluentialEmergent Representations of Program Semantics in Language Models Trained on ProgramsPreprint
Jin, C., & Rinard, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.11169 · Cited by 45
COKE: A Cognitive Knowledge Graph for Machine Theory of Mind
Wu, J., Chen, Z., Deng, J., Sabour, S., Meng, H. M. L., & Huang, M. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.848 · Cited by 45
A Generalist Dynamics Model for ControlPreprint
Schubert, I., Zhang, J., Bruce, J., Bechtle, S., Parisotto, E., Riedmiller, M., Springenberg, J. T., Byravan, A., Hasenclever, L., & Heess, N. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.10912 · Cited by 44
AVID: Adapting Video Diffusion Models to World ModelsPreprint
Rigter, M., Gupta, T., Hilmkil, A., & Ma, C. (2024) · arXiv.org · doi:10.48550/arxiv.2410.12822 · Cited by 44
AVLEN: Audio-Visual-Language Embodied Navigation in 3D Environments
Paul, S., Roy-chowdhury, A. K., & Cherian, A. (2022) · neural information processing systems · doi:10.52202/068431-0451 · Cited by 44
StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text
Henschel, R., Khachatryan, L., Poghosyan, H., Hayrapetyan, D., Tadevosyan, V., Wang, Z., Navasardyan, S., & Shi, H. (2024) · arXiv · doi:10.48550/arxiv.2403.14773 · Cited by 44
Piao, J., Yan, Y., Zhang, J., Li, N., Yan, J., Lan, X., Lu, Z., Zheng, Z., Wang, J. Y., Zhou, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2502.08691 · Cited by 43 · cited 4×
“Among them, AgentSociety [195] stands out as the most advanced effort to date, providing a large-scale and versatile environment for examining polarization, policy interventions, and other phenomena central to societal intelligence.”
InfluentialSelf-citation
Research works printed but not linked: 26
- Yifan Cui, Xinyi Shan, and Jeanhun Chung. 2024. A feasibility study on RUNWAY GEN-2 for generating realistic style images. International Journal of Internet, Broadcasting and Communication 16, 1 (2024), 99–105.
- Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. 2022. Glam: Efficient scaling of language models with mixture-of-experts. In Proceedings of the International Conference on Machine Learning. PMLR, 5547–5569.
- Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. 2024. Learning universal policies via text-guided video generation. Advances in Neural Information Processing Systems 36 (2024).
- Ignat Georgiev, Varun Giridhar, Nicklas Hansen, and Animesh Garg. 2025. PWM: Policy learning with multi-task world models. In Proceedings of the 13th International Conference on Learning Representations.
- David Ha and Jürgen Schmidhuber. 2018. Recurrent world models facilitate policy evolution. Advances in Neural Information Processing Systems 31 (2018).
- Haoran He, Chenjia Bai, Ling Pan, Weinan Zhang, Bin Zhao, and Xuelong Li. 2024. Learning an actionable discrete diffusion policy via large-scale actionless video pre-training. In Proceedings of the 38th Annual Conference on Neural Information Processing Systems.
- Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems 33 (2020), 6840–6851.
- Anthony Hu Lloyd Russell Hudson Yeo Zak Murez George Fedoseev Alex Kendall Jamie Shotton and Gianluca Corrado. 2023. GAIA-1: A Generative World Model for Autonomous Driving. arxiv:2309.17080. Retrieved from https://arxiv.org/abs/2309.17080 · arXiv
- Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. 2022. Bc-z: Zero-shot task generalization with robotic imitation learning. In Proceedings of the Conference on Robot Learning. PMLR, 991–1002.
- 10.5555/3692070.3692961 · doi:10.5555/3692070.3692961
- Philip Nicholas Johnson-Laird. 1983. Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness. Number 6. Harvard University Press.
- Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Paul Saldyt, and Anil B. Murthy. 2024. Position: LLMs can’t plan, but can help planning in LLM-modulo frameworks. In Proceedings of the 41st International Conference on Machine Learning.
- Kuaishou. 2024. Kling AI: Next-Generation AI Creative Studio. Retrieved May 06, 2025 from https://www.klingai.com/global/.
- Yann LeCun. 2022. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review 62, 1 (2022), 1–62.
- Junhyuk Oh, Satinder Singh, and Honglak Lee. 2017. Value prediction network. Advances in Neural Information Processing Systems 30 (2017).
- OpenAI. 2024. Sora: Creating video from text. Retrieved May 06, 2025 from https://openai.com/sora.
- Aravind Rajeswaran, Igor Mordatch, and Vikash Kumar. 2020. A game theoretic framework for model based reinforcement learning. In Proceedings of the International Conference on Machine Learning. PMLR, 7953–7963.
- Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards real-time object detection with region proposal networks. In Proceedings of the Advances in Neural Information Processing Systems.C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.), Vol. 28, Curran Associates, Inc. Retrieved from https://proceedings.neurips.cc/paper_files/paper/2015/file/14bfa6bb14875e45bba028a21ed38046-Paper.pdf
- Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. 2022. Learning to walk in minutes using massively parallel deep reinforcement learning. In Proceedings of the Conference on Robot Learning. PMLR, 91–100.
- Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. 2020. Learning to simulate complex physics with graph networks. In Proceedings of the International Conference on Machine Learning. PMLR, 8459–8468.
- Hongzhi Shi, Jingtao Ding, Yufan Cao, Li Liu, Yong Li, et al. 2022. Learning symbolic models for graph-structured physical mechanism. In Proceedings of the 11th International Conference on Learning Representations.
- A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems 30 (2017).
- Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. 2023. Daydreamer: World models for physical robot learning. In Proceedings of the Conference on Robot Learning. PMLR, 2226–2240.
- Wilson Yan, Danijar Hafner, Stephen James, and Pieter Abbeel. 2023. Temporally consistent transformers for video generation. In Proceedings of the International Conference on Machine Learning. PMLR, 39062–39098.
- Sherry Yang, Yilun Du, Seyed Kamyar Seyed Ghasemipour, Jonathan Tompson, Leslie Pack Kaelbling, Dale Schuurmans, and Pieter Abbeel. 2024. Learning interactive real-world simulators. In Proceedings of the 12th International Conference on Learning Representations.
- 10.1007/978-3-031-72624-8_4 · DOI · Paper page
Cited by
Nie, T., Sun, J., & Ma, W. (2025) · Artificial Intelligence for Transportation · doi:10.1016/j.ait.2025.100003 · Cited by 116
CityGPT: Empowering Urban Spatial Cognition of Large Language Models
Feng, J., Liu, T., Du, Y., Guo, S., Lin, Y., & Li, Y. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3736878 · Cited by 137
Self-citationZhang, Y., Tian, J., & Xiong, Q. (2025) · Frontiers in Robotics and AI · doi:10.3389/frobt.2025.1668910 · Cited by 33
Self-citationZhao, C., Liu, G., Zhang, R., Liu, Y., Wang, J., Kang, J., Niyato, D. T., Li, Z., Shen, X., Han, Z., et al. (2026) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2026.3658762 · Cited by 40
“Most world models are implemented as neural architectures with three core components that achieve the perception-prediction cycle [20]: • Encoder: The encoder network compresses high-dimensional observations, such as images, sensor data, etc., into a compact latent state.”
InfluentialCityBench: Evaluating the Capabilities of Large Language Models for Urban Tasks
Feng, J., Zhang, J., Liu, T., Zhang, X., Ouyang, T., Yan, J., Du, Y., Guo, S., & Li, Y. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737375 · Cited by 97
Self-citation
Metadata
- Authors
- Ding, J., Zhang, Y., Shang, Y., Zhang, Y., Zong, Z., Feng, J., Yuan, Y., Su, H., Li, N., Sukiennik, N., Xu, F., & Li, Y.
- Affiliations
- Tsinghua University (CN)
- ORCID
- Venue
- ACM Computing Surveys
- Published
- 2025-09-09
- Volume
- 58
- Issue
- 3
- Pages
- 1-38
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W4411735734
- Cited by
- 169
- References
- 48
- Citations by year
- 2026: 122 · 2025: 46
- FWCI
- 43.25
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- ACM Computing Surveys (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Key Research and Development Program of China (2024YFC3307603)
- National Natural Science Foundation of China (62476152 and U24B20180)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Keywords
- world models, internal representations, future state prediction, artificial general intelligence, autonomous driving, robotics, social simulation, multimodal large language models, video generation models, decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.