GAIA-1: A Generative World Model for Autonomous Driving

“Its training objective is to predict the next image token in the sequence conditioned on all past tokens, using causal masking in the attention matrix of the transformer blocks [35].”

“The discrete autoencoder is a fully convolutional 2D U-Net [31].”

“Because these text sources are imperfect, to improve the alignment between generated futures and the text prompt, we employ classifier-free guidance [45, 46] at inference time.”

“Large language models have shown clear benefits in scaling model size and data [90, 24, 26, 20, 21, 22, 23].”

“The advantage of such a perspective is that world models can benefit from scaling properties of high-capacity sequence model architectures applied to large-scale unsupervised training [26].”

“Following the recent advances in image [36, 37] and video generation [16, 18] we use denoising video diffusion models for the GAIA-1 decoder.”

“Most language models [25, 26, 24, 21, 27, 22] use byte-pair encoding (or equivalent) as a trade-off between character-level and word-level tokenization.”

“The image reconstruction loss is a weighted sum of L 1 , L 2 , perceptual loss L perceptual [32], and GAN loss L GAN [33].”

“Large language models have shown clear benefits in scaling model size and data [90, 24, 26, 20, 21, 22, 23].”

“The models used to fit the power-law ranged from 10,000x to 10x smaller models in terms of parameters (0.65M to 650M), as visualized in Figure 8b. Similarly to [49], the compute was estimated as a function of the parameter count.”

“GAIA-1 [1] and DriveDreamer [22] generate driving video diffusion.”

“In video prediction, a world model learns the temporal evolution of visual scenes, often conditioned on actions that describe how the state changes over time [16, 17, 23, 24].”

“Generative world models have emerged as a paradigm shift in autonomous driving simulation, enabling the synthesis of diverse driving scenarios for closed-loop training [15, 33, 43, 56].”

“Driving-behavior research requires observing drivers’ perception, decision-making, and control in environments that are both controllable and realistic.”

“GAIA-1 (Hu, 2023) builds a driving world model using autoregressive video generation.”

“These models can synthesize temporally and spatially consistent futures from short visual contexts and support controllable generation, making them attractive foundations for autonomous decision-making and interactive perception systems [4, 7, 19, 20].”

“GAIA (Hu et al. 2023) designs an autoregressive model to generate future frames of driving videos.”

“For instance, the Orbis [22] world model aligns the tokenizer’s semantic branch with DINOv2 features via cosine similarity during training while GAIA-1 [26] increases semantic content of image tokens predicted by the world model through DINO [24] distillation.”

“…have used diffusion-based techniques to demonstrate long-horizon rollouts by conditioning on history and forecasting both future scene content and ego motion (Hu et al., 2023a; Gao et al., 2024; Jia et al., 2023; Wang et al., 2024; Chen et al., 2025b; Zheng et al., 2024b; Zhang et al., 2025b).”

“…on these frameworks, Feng et al. categorized world-model research into three major families [ 12 ] : (1) Generative World Models (e.g., GAIA-1 [ 15 ] ): Learn future scene evolution—including images and occupancy grids—through multimodal generative modeling for simulation, planning, and safety…”

“Specifically, the V-JEPA 2 aim at learning the essential knowledge on enviroment interaction as the motion, object movement , without focusing on the stochasitic and noisy pixel detail as reconstruction attemps as DreamerV3-based encoders [22, 63].”

“BEVGen[35] Transformer nuScenes,Argoverse2 FUTURIS[19] Transformer Cityscapes GAIA-1[13] Transformer Internaldataset NeMo[18] Transformer nuScenes ability to capture the complexity, diversity, and unpredictability of real-world scenarios.”

“200 FieldSeer waveguides The integration of machine learning into nanophotonics has enabled fast surrogate models that replace costly full-wave simulations by learning mappings from geometry and excitation conditions to electromagnetic responses [1–4,7, 10 ].”

“GAIA-1 [22] and DriveDreamer [23] use the world model’s imagination to generate future driving videos.”

“These methods are increasingly leveraging foundation models and Vision-Language Models (VLMs) to bridge simulation with natural scene understanding, allowing scenario synthesis from abstract or rare prompts (e.g., “child crossing in fog”) [17, 20, 21].”

“More recently, world models that are largely built on pretrained video generation models [2, 30, 35, 36] have been promoted as a path to realistic, controllable data.”

“Recently, generative modeling on autonomous driving tasks has also drawn great attention and yields impressive effects [5, 7, 8, 12, 13, 22, 24, 26].”

“These approaches include simulation [8, 27, 1], Domain Adaptation (DA) [24, 20] with GANs [12] or Stable Diffusion (SD) [32], Generative AI (GenAI) [18, 38], and Neural Rendering [40, 45, 41, 4].”

“Wayvehas proposedthegenerativeworldmodelGaia-1[24],capableof generatingrealisticdrivingscenariosthroughenteringinputs suchasvideosandtext.”

“Many works [3, 4, 6, 22, 25, 26, 31] extent video diffusion models into autonomous driving to simulate different driving scenarios.”

“Current world models are primarily focused on real-world scenarios, such as robotic manipulation [3], autonomous driving [4] and virtual environments [5].”

“This has opened up several avenues for application in fields such as autonomous driving Hu et al. (2023); Tian et al. (2024), finance Wang et al. (2023); Bhatia et al. (2024), medical research Delbrouck et al. (2022); Hartsock & Rasool (2024) and robotics Black et al. (2024); Brohan et al. (2023).”

“The software stack of AVs are framed either modular based or end-to-end control based on sensors [21], [22], [23], [24].”

“GAIA-1 represents a novel autonomous generative artificial intelligence model that can create realistic driving videos using video, text, and action inputs [14].”

“In this regard, large diffusion models paired with temporal models offer novel avenues for data creation[1][5][7].”

“Agents must share what they see , what they believe will happen , and how their actions may jointly influence future states [24, 25].”

“Transformer-based architectures, pre-trained on vast datasets, have shown remarkable potential for learning generalized visual representations [13], [14].”

“Although a primary prospect of world models is to enable the generalization ability to novel environments, existing driving world models are still constrained by data scale [90, 125, 127, 143, 147] and geographical coverage [54, 61].”

“Animated visualizations of this procedure can be found at https://diamond-wm.github.io . required for the task is less well-defined, such as training autonomous vehicles (Hu et al., 2023).”

“Some recent endeavors have introduced diffusion models [54] into the domain of autonomous driving to build world models as neural simulators to generate requisite autonomous 2D driving videos [28], [29], [180], [181].”

“Besides the robotics domain, world models are also widely used in autonomous driving [76, 75, 35, 45, 87, 84, 89], where they mainly focus on path planning and real-time decision-making, which is pivotal in enabling vehicles to navigate complex environments safely and efficiently.”

“This enables efficient coordination among large teams of aerial robots.”

“Therefore, world models for autonomous driving gained prominent attention in recent years A. Hu et al. (2023); Jia et al. (2023), since they have the potential to pave the way to end-to-end autonomous driving Y. Hu et al. (2023); B. Jiang et al. (2022).”

“In autonomous driving fields, GAIA-1 [ Hu et al. , 2023 ] and DriveDreamer [ Wang et al. , 2023 ] are representative works for data generation based on world models.”

“World Models now under-pin applications ranging from autonomous driving (Wang et al., 2024c; Jia et al., 2023; Hu et al., 2023; Wang et al., 2024e; Zhang et al., 2023) to embodied navigation (Zhong et al., 2025; Yao et al., 2025; Bar et al., 2025; Nie et al., 2025; Jin and Jia, 2025; Long et al.,…”