The DOI matches its DataCite record; 70 of the 73 references it cites were found in a registry or an index; none of the 23 sources its claims cite is checked for support yet.
OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving
Open accessAbstract
Understanding how the 3D scene evolves is vital for making decisions in autonomous driving. Most existing methods achieve this by predicting the movements of object boxes, which cannot capture more fine-grained scene information. In this paper, we explore a new framework of learning a world model, OccWorld, in the 3D Occupancy space to simultaneously predict the movement of the ego car and the evolution of the surrounding scenes. We propose to learn a world model based on 3D occupancy rather than 3D bounding boxes and segmentation maps for three reasons: 1) expressiveness. 3D occupancy can describe the more fine-grained 3D structure of the scene; 2) efficiency. 3D occupancy is more economical to obtain (e.g., from sparse LiDAR points). 3) versatility. 3D occupancy can adapt to both vision and LiDAR. To facilitate the modeling of the world evolution, we learn a reconstruction-based scene tokenizer on the 3D occupancy to obtain discrete scene tokens to describe the surrounding scenes. We then adopt a GPT-like spatial-temporal generative transformer to generate subsequent scene and ego tokens to decode the future occupancy and ego trajectory. Extensive experiments on the widely used nuScenes benchmark demonstrate the ability of OccWorld to effectively model the evolution of the driving scenes. OccWorld also produces competitive planning results without using instance and map supervision. Code: https://github.com/wzzheng/OccWorld.
Overview
CoChat Summary
TL;DR
OccWorld is a 3D occupancy world model that jointly forecasts future 3D semantic occupancy and ego trajectory in an autoregressive GPT-like framework. It reports non-trivial 4D occupancy forecasting and competitive motion planning on nuScenes/Occ3D without instance or map annotations.
Key findings
- For 4D occupancy forecasting, OccWorld-O achieves average IoU 26.63 and mIoU 17.14 (abstract: 17.13) over 1s/2s/3s future given 2s history; it outperforms Copy&Paste and shows the model learns underlying scene evolution.
- For motion planning, OccWorld-O reports average L2 error 1.17 m (abstract: 1.16) and average collision rate 0.60%; it outperforms OccNet by a large margin without map or bounding-box supervision, though UniAD achieves the best overall performance.
- End-to-end variants using camera inputs and TPVFormer occupancy—OccWorld-D, OccWorld-T, and self-supervised OccWorld-S—deliver non-trivial results; OccWorld-S uses no supervision other than the future trajectory.
- Ablations show spatial aggregation and temporal attention are vital: removing spatial attention drops forecast mIoU to 10.07 and IoU to 21.44; removing temporal attention drops mIoU to 8.98 and IoU to 20.10. Removing ego temporal attention worsens planning (L2 5.89, collision 6.23) and also hurts 3D forecasting.
Methods
- Design: Uses 3D semantic occupancy as the scene representation, a VQ-VAE scene tokenizer to obtain discrete scene tokens, and a GPT-like spatial-temporal generative transformer with spatial mixing, masked temporal attention, U-net aggregation, and an ego token for trajectory prediction.
- Training: Two-stage training—first trains the scene tokenizer/decoder with softmax and Lovasz-softmax losses; second trains the world model and ego decoder with softmax over the codebook and L2 trajectory loss. Uses AdamW, cosine annealing, initial LR 1e-3, weight decay 0.01, batch size 1 per GPU on 8 RTX 4090 GPUs.
- Data/evaluation: Evaluates 4D occupancy forecasting on Occ3D and motion planning on nuScenes, using 2s history to predict 3s future; metrics are mIoU/IoU for occupancy and L2 error/collision rate for planning. Scene tokenizer uses downsampling factor 4, codebook 512, 128-dim features; transformer uses 3 scales with 6 layers of spatial-wise temporal attention for scene tokens and 2 layers of spatial/temporal cross-attention for ego tokens.
Limitations
- The paper states OccWorld fails to forecast new vehicles entering the sight, which is difficult given their absence in the inputs.
- The paper states collision rate slightly falls behind because learning safe trajectories is harder without freespace or bounding-box guidance; short-term planning is strong but worsens quickly, and diverse future generations may deviate from the ground-truth trajectory.
- Evident from the text: OccWorld-O uses ground-truth 3D occupancy as input, so it is not a fully deployable perception-to-planning system; the end-to-end variants show lower performance, and the paper does not provide a dedicated limitations section beyond these result/analysis statements.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 1 Not valid · 11 Not checked · 44 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and achieves an average IoU of 26.63 and mIoU of 17.13 for 3s future given 2s history”
§ OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving > 1. Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Still, OccWorld-O demonstrates comparable collision rates with OccNet which exploits map and box supervision, showing that OccWorld can learn the concept of freespace with 3D occupancy.”
§ OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving > Motion planning.
What the sources say
- Not matchedcites [53]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observe that using spatial aggregation to model spatial dependencies and using temporal attention to integrate history information is vital to the performance of both 4D occupancy forecasting and motion planning tasks.”
§ OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving > Analysis of the spatial-temporal generative transformer.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Using self-supervised learned 3D occupancy from camera inputs [20], our method achieves non-trivial 4D occupancy forecasting and planning results”
§ OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving > 1. Introduction
What the sources say
- Not matchedcites [20]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“OccWorld can also produce planning trajectories with an L2 error of 1.16 without using any instance and map annotations.”
§ OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving > 1. Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
31 more claims
Create a free account to see all 56 claims and every source verdict.
- It may state two separate claims as one. 15
- Probably not a substantial claim. 6
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving > 3.2. 3D Occupancy Scene Tokenizer
“We then integrate the quantized features {z ij } to obtain the final scene representation z ∈ R H×W ×C .”
§ OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving > 3.2. 3D Occupancy Scene Tokenizer
“We then adopt a lightweight encoder composed of 2D convolution layers to obtain down-sampled features ẑ ∈ R H d × W d ×C of the scene, where d is the down-sampling factor.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving > 3.2. 3D Occupancy Scene Tokenizer
“z ij = N (ẑ ij , C) = min c∈C ||ẑ ij -c|| 2 , (4”
§ OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving > 3.2. 3D Occupancy Scene Tokenizer
“We then integrate the quantized features {z ij } to obtain the final scene representation z ∈ R H×W ×C .”
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
nuScenes: A Multimodal Dataset for Autonomous Driving
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., & Beijbom, O. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01164 · Cited by 7,152
SGDR: Stochastic Gradient Descent with Warm Restarts
Loshchilov, I., & Hutter, F. (2016) · arXiv · doi:10.48550/arxiv.1608.03983 · Cited by 5,309
Neural Discrete Representation Learning
Oord, A. V. D., Vinyals, O., & Kavukcuoglu, K. (2017) · arXiv · doi:10.48550/arxiv.1711.00937 · Cited by 2,729
Research works printed but not linked: 6
- Ran Cheng, Ryan Razani, Ehsan Taghavi, Enxu Li, and Bingbing Liu. 2-s3net: Attentive feature fusion with adap- tive feature selection for sparse semantic segmentation net- work. In CVPR, pages 12547-12556, 2021.
- Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. In ICLR, 2020. 5
- Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 27, 2014. 2
- Benedikt Mersch, Xieyuanli Chen, Jens Behley, and Cyrill Stachniss. Self-supervised point cloud prediction using 3d spatio-temporal convolutional networks. In CoRL, pages 1444-1454. PMLR, 2022. 2
- Xinshuo Weng, Jianren Wang, Sergey Levine, Kris Kitani, and Nicholas Rhinehart. Inverting the pose forecasting pipeline with spf2: Sequential pointcloud forecasting for se- quential pose forecasting. In Conference on robot learning, pages 11-20, 2021. 2
- Yunpeng Zhang, Zheng Zhu, Wenzhao Zheng, Junjie Huang, Guan Huang, Jie Zhou, and Jiwen Lu. Beverse: Unified per- ception and prediction in birds-eye-view for vision-centric autonomous driving. arXiv preprint arXiv:2205.09743, 2022. 1, 2 · arXiv
Cited by
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability
Gao, S., Yang, J., Chen, L., Chitta, K., Qiu, Y., Geiger, A., Zhang, J., & Li, H. (2024) · neural information processing systems · doi:10.52202/079017-2906 · Cited by 467
GenAD: Generative End-to-End Autonomous Driving
Zheng, W., Song, R., Guo, X., Zhang, C., & Chen, L. F. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73650-6_6 · Cited by 350
Self-citationGaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Prediction
Huang, Y., Zheng, W., Zhang, Y., Zhou, J. N., & Lu, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73383-3_22 · Cited by 213
Self-citationEpona: Autoregressive Diffusion World Model for Autonomous Driving
Zhang, K., Tang, Z., Hu, X., Pan, X., Guo, X., Liu, Y., Huang, J., Yuan, L., Zhang, Q., Long, X., et al. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02527 · Cited by 180
Understanding World or Predicting Future? A Comprehensive Survey of World Models
Ding, J., Zhang, Y., Shang, Y., Zhang, Y., Zong, Z., Feng, J., Yuan, Y., Su, H., Li, N., Sukiennik, N., et al. (2026) · ACM Computing Surveys · doi:10.1145/3746449 · Cited by 169
Metadata
- Authors
- Zheng, W., Chen, W., Huang, Y., Zhang, B., Duan, Y., & Lu, J.
- Affiliations
- University of California, Berkeley (US)
- Tsinghua University (CN)
- Corresponding author
- Lu, Jiwen
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 55-72
- ISSN
- 0302-9743 (ISSN-L)
- arXiv
- 2311.16038
- OpenAlex
- W4403771534
- Cited by
- 342
- References
- 73
- Citations by year
- 2026: 186 · 2025: 120 · 2024: 33 · 2023: 3
- FWCI
- 88.70
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Lecture notes in computer science (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- world models, autonomous driving, occupancy grid, LiDAR, nuScenes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-19, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-19 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving. https://app.cochat.ai/papers/10.48550/arxiv.2311.16038
- The paper
- Zheng, W., Chen, W., Huang, Y., Zhang, B., Duan, Y., & Lu, J. (2023). OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving. arXiv. https://doi.org/10.48550/arxiv.2311.16038
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.