Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its DataCite record; 70 of the 73 references it cites were found in a registry or an index; none of the 23 sources its claims cite is checked for support yet.

OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving

Open access

Abstract

Understanding how the 3D scene evolves is vital for making decisions in autonomous driving. Most existing methods achieve this by predicting the movements of object boxes, which cannot capture more fine-grained scene information. In this paper, we explore a new framework of learning a world model, OccWorld, in the 3D Occupancy space to simultaneously predict the movement of the ego car and the evolution of the surrounding scenes. We propose to learn a world model based on 3D occupancy rather than 3D bounding boxes and segmentation maps for three reasons: 1) expressiveness. 3D occupancy can describe the more fine-grained 3D structure of the scene; 2) efficiency. 3D occupancy is more economical to obtain (e.g., from sparse LiDAR points). 3) versatility. 3D occupancy can adapt to both vision and LiDAR. To facilitate the modeling of the world evolution, we learn a reconstruction-based scene tokenizer on the 3D occupancy to obtain discrete scene tokens to describe the surrounding scenes. We then adopt a GPT-like spatial-temporal generative transformer to generate subsequent scene and ego tokens to decode the future occupancy and ego trajectory. Extensive experiments on the widely used nuScenes benchmark demonstrate the ability of OccWorld to effectively model the evolution of the driving scenes. OccWorld also produces competitive planning results without using instance and map supervision. Code: https://github.com/wzzheng/OccWorld.

Overview

CoChat Summary

TL;DR

OccWorld is a 3D occupancy world model that jointly forecasts future 3D semantic occupancy and ego trajectory in an autoregressive GPT-like framework. It reports non-trivial 4D occupancy forecasting and competitive motion planning on nuScenes/Occ3D without instance or map annotations.

Key findings

  • For 4D occupancy forecasting, OccWorld-O achieves average IoU 26.63 and mIoU 17.14 (abstract: 17.13) over 1s/2s/3s future given 2s history; it outperforms Copy&Paste and shows the model learns underlying scene evolution.
  • For motion planning, OccWorld-O reports average L2 error 1.17 m (abstract: 1.16) and average collision rate 0.60%; it outperforms OccNet by a large margin without map or bounding-box supervision, though UniAD achieves the best overall performance.
  • End-to-end variants using camera inputs and TPVFormer occupancy—OccWorld-D, OccWorld-T, and self-supervised OccWorld-S—deliver non-trivial results; OccWorld-S uses no supervision other than the future trajectory.
  • Ablations show spatial aggregation and temporal attention are vital: removing spatial attention drops forecast mIoU to 10.07 and IoU to 21.44; removing temporal attention drops mIoU to 8.98 and IoU to 20.10. Removing ego temporal attention worsens planning (L2 5.89, collision 6.23) and also hurts 3D forecasting.

Methods

  • Design: Uses 3D semantic occupancy as the scene representation, a VQ-VAE scene tokenizer to obtain discrete scene tokens, and a GPT-like spatial-temporal generative transformer with spatial mixing, masked temporal attention, U-net aggregation, and an ego token for trajectory prediction.
  • Training: Two-stage training—first trains the scene tokenizer/decoder with softmax and Lovasz-softmax losses; second trains the world model and ego decoder with softmax over the codebook and L2 trajectory loss. Uses AdamW, cosine annealing, initial LR 1e-3, weight decay 0.01, batch size 1 per GPU on 8 RTX 4090 GPUs.
  • Data/evaluation: Evaluates 4D occupancy forecasting on Occ3D and motion planning on nuScenes, using 2s history to predict 3s future; metrics are mIoU/IoU for occupancy and L2 error/collision rate for planning. Scene tokenizer uses downsampling factor 4, codebook 512, 128-dim features; transformer uses 3 scales with 6 layers of spatial-wise temporal attention for scene tokens and 2 layers of spatial/temporal cross-attention for ego tokens.

Limitations

  • The paper states OccWorld fails to forecast new vehicles entering the sight, which is difficult given their absence in the inputs.
  • The paper states collision rate slightly falls behind because learning safe trajectories is harder without freespace or bounding-box guidance; short-term planning is strong but worsens quickly, and diverse future generations may deviate from the ground-truth trajectory.
  • Evident from the text: OccWorld-O uses ground-truth 3D occupancy as input, so it is not a fully deployable perception-to-planning system; the end-to-end variants show lower performance, and the paper does not provide a dedicated limitations section beyond these result/analysis statements.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2311.16038

Cites

70 of 70 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 1 Not valid · 11 Not checked · 44 Own results and interpretations

31 more claims

Create a free account to see all 56 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 15
  • Probably not a substantial claim. 6

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 73 references

Research works printed but not linked: 6

  1. Ran Cheng, Ryan Razani, Ehsan Taghavi, Enxu Li, and Bingbing Liu. 2-s3net: Attentive feature fusion with adap- tive feature selection for sparse semantic segmentation net- work. In CVPR, pages 12547-12556, 2021.
  2. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. In ICLR, 2020. 5
  3. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 27, 2014. 2
  4. Benedikt Mersch, Xieyuanli Chen, Jens Behley, and Cyrill Stachniss. Self-supervised point cloud prediction using 3d spatio-temporal convolutional networks. In CoRL, pages 1444-1454. PMLR, 2022. 2
  5. Xinshuo Weng, Jianren Wang, Sergey Levine, Kris Kitani, and Nicholas Rhinehart. Inverting the pose forecasting pipeline with spf2: Sequential pointcloud forecasting for se- quential pose forecasting. In Conference on robot learning, pages 11-20, 2021. 2
  6. Yunpeng Zhang, Zheng Zhu, Wenzhao Zheng, Junjie Huang, Guan Huang, Jie Zhou, and Jiwen Lu. Beverse: Unified per- ception and prediction in birds-eye-view for vision-centric autonomous driving. arXiv preprint arXiv:2205.09743, 2022. 1, 2 · arXiv

Cited by

See all 342

Metadata

Authors
Zheng, W., Chen, W., Huang, Y., Zhang, B., Duan, Y., & Lu, J.
Affiliations
  • University of California, Berkeley (US)
  • Tsinghua University (CN)
Corresponding author
Lu, Jiwen
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
55-72
ISSN
0302-9743 (ISSN-L)
OpenAlex
W4403771534
Cited by
342
References
73
Citations by year
2026: 186 · 2025: 120 · 2024: 33 · 2023: 3
FWCI
88.70
Type
conference paper
Language
en
Open access
Locations
  • Lecture notes in computer science (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
world models, autonomous driving, occupancy grid, LiDAR, nuScenes

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-19, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-19 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving. https://app.cochat.ai/papers/10.48550/arxiv.2311.16038
The paper
Zheng, W., Chen, W., Huang, Y., Zhang, B., Duan, Y., & Lu, J. (2023). OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving. arXiv. https://doi.org/10.48550/arxiv.2311.16038

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.