GAIA-1: A Generative World Model for Autonomous Driving

“Its training objective is to predict the next image token in the sequence conditioned on all past tokens, using causal masking in the attention matrix of the transformer blocks [35].”

“The discrete autoencoder is a fully convolutional 2D U-Net [31].”

“Because these text sources are imperfect, to improve the alignment between generated futures and the text prompt, we employ classifier-free guidance [45, 46] at inference time.”

“Large language models have shown clear benefits in scaling model size and data [90, 24, 26, 20, 21, 22, 23].”

“The advantage of such a perspective is that world models can benefit from scaling properties of high-capacity sequence model architectures applied to large-scale unsupervised training [26].”

“Following the recent advances in image [36, 37] and video generation [16, 18] we use denoising video diffusion models for the GAIA-1 decoder.”

“Most language models [25, 26, 24, 21, 27, 22] use byte-pair encoding (or equivalent) as a trade-off between character-level and word-level tokenization.”

“The image reconstruction loss is a weighted sum of L 1 , L 2 , perceptual loss L perceptual [32], and GAN loss L GAN [33].”

“Large language models have shown clear benefits in scaling model size and data [90, 24, 26, 20, 21, 22, 23].”

“The models used to fit the power-law ranged from 10,000x to 10x smaller models in terms of parameters (0.65M to 650M), as visualized in Figure 8b. Similarly to [49], the compute was estimated as a function of the parameter count.”

“Computer vision applications of word models have produced stunning success in video generation [3, 4].”

“This method for synthetic data creation is now being replaced by more flexible, machine learning driven workflows [15, 37], which synthesize visual data for robustness testing or training data augmentation.”

“…planning in complex interactive environments, and underpin advances in neural game engines (Kan-ervisto et al., 2025; Guo et al., 2025; Gao et al., 2025; Chen et al., 2025), embodied AI (Du et al.; Yang et al., 2024), and autonomous driving (Russell et al., 2025; Hu et al., 2023a; Ni et al., 2025).”

“World models [7, 8, 9] have emerged as a promising solution, enabling agents to learn in simulated environments, with applications spanning robotics [10, 11, 12], autonomous driving [13], and video games [14].”

“Closed-loop sensor simulation on controllable simulators with reactive agents allows for dynamic evaluation of the ADS with its environment and actors [46][15], although realism of the world has been questioned [10].”

“However, most of them cannot maintain view consistency [5, 14, 26, 30, 35, 39] or can only generate data without producing corresponding bbox annotations [9, 13].”

“First, unlike GAIA-1 [12] which relies heavily on large-scale pre-training with paired video–text data, our approach eliminates the need for language supervision by leveraging purely visual self-supervision.”

“Recently, the community has started investigating scaling world models (Ha & Schmidhuber, 2018), for example, Hu et al. (2023a); Pearce et al. (2024); Wu et al. (2025); Agarwal et al. (2025); Mereu et al. (2025) train world models with Diffusion Models or Transformers.”

“GAIA-1 [42] utilizes self-supervised learning methods, such as autoregressive and autoencoding techniques similar to those in LLM, to train a world model.”

“The work [53] divides the state into images and texts as two separate inputs.”

“For larger world models [9], the gap widens even further, such that we expect our approach to scale better to real-world driving.”

“To enable LLM-based embodied AI to function as a world model, it is necessary to clarify future predictions utilizing the world model, similar to other existing world models (Hu, 2023).”

“…gained significant attention in recent years, offering a transformative approach to various domains, ranging from content creation (Bar-Tal et al. 2024) to robotics (Guo et al. 2023a), autonomous driving 1 (Hu et al. 2023; Zhang et al. 2024; Gao et al. 2024) and video games (Menapace et al. 2024).”

“World Model Simulation for Autonomous Driving: Generative AI has shown great capabilities as simulators for autonomous driving through the so-called world models, for example, in the project GAIA-1 [8] as it creates realistic driving environments.”

“There are many synthetic driving datasets that have been proposed, such as GTA-V [20], Virtual KITTI [4], SYNTHIA [21], SHIFT [27], OPV2V [32] or most recently GAIA-1 [6].”

“In the context of autonomous driving, several models have introduced task-specific conditioning mechanisms [5, 6, 7, 8, 9], enabling some control over scenario elements.”

“Further advances in video world modelling (Yang et al., 2024; Wang et al., 2024b; Hu et al., 2023a) could lead to similar techniques being scaled towards real-world settings.”

“These 2D generative models are particularly useful as interactive environments for video games (Bruce et al., 2024) or controlled experimental environments in real-world simulators for autonomous systems (Hu et al., 2023).”

“…et al., 2026a), video games (Fan et al., 2022; Alonso et al., 2024; Wang et al., 2026; Valevski et al., 2025; Xiao et al., 2026), autonomous driving (Ren et al., 2025; Hu et al., 2023), and physical simulation (Yuan et al., 2026), demonstrating their potential as general-purpose visual simulators.”

“Prior work has explored training navigation world models (Hu et al., 2023; Bar et al., 2025; Gao et al., 2024; Hassan et al., 2025) with well-defined action spaces (e.g., velocity).”

“Recent advancements, such as the development of world models [29], [30], have shown potential in addressing these issues, but continued research and innovation are necessary to fully close these gaps and ensure the reliability of AV testing.”

“…for anticipating scene changes and exploring possible outcomes beyond the observed frames, supporting scenario generation for autonomous driving (Hu et al., 2023; Wang et al., 2024a), visual planning for robotic manipulation (Dharmarajan et al., 2026; Zhou et al., 2024), and action-controllable…”

“GAIA-1 uses video, text, and action inputs to steer driving scenarios (Hu et al., 2023), while DriveDreamer learns structured traffic constraints for controllable generation (Wang et al., 2024).”

“…2023; Bruce et al., 2024; Parker-Holder et al., 2024; 2025; Decart et al., 2024; Alonso et al., 2024; Agarwal et al., 2025; Alhaija et al., 2025; Hu et al., 2023; Che et al., 2025; Yu et al., 2025b; He et al., 2025b; Li et al., 2025a; Ma et al., 2024b; Ji et al., 2026), and extend the horizon…”

“Prior work makes the latent more diffusable by distilling a frozen pretrained vision encoder into the autoencoder bottleneck, matching each latent position to its co-located patch feature [Peng et al., 2022, Hu et al., 2023, Russell et al., 2025, Yao and Wang, 2025, Bi et al., 2026].”

“…being used as visual simulators, with systems spanning learned latent dynamics (Ha & Schmidhuber, 2018; Hafner et al., 2023), interactive environment generation (Bruce et al., 2024), video prediction for autonomous driving (Hu et al., 2023), and long-term spatial memory (Wu et al., 2026b).”

“Three lines of work address this by predicting future frames with generative world models [20, 21, 22, 14, 23, 24, 25, 26], synthesizing novel views from reconstructed dynamic 3D scenes [27, 8, 9, 10], and re-rendering physics-based simulator state with large pretrained video models [28, 16, 12,…”

“GAIA-1 [17] models video, text, and actions jointly.”

“From early latent-dynamics methods [10, 12] to the Dreamer family and its embodied extensions [11, 13, 14, 32], and to broader generative or driving environments [7, 29, 30, 18, 15], this paradigm enables comparing multiple candidate futures before acting.”

“World models have recently emerged as a prominent approach for end-to-end autonomous driving (Zhang et al. 2025; Zhao et al. 2025; Xu et al. 2026; Liu et al. 2026; Li et al. 2026b; Zheng et al. 2024a; Hu et al. 2023a; Wang et al. 2024; Gao et al. 2024).”

“…foundation models (Kirillov et al. 2023; Oquab et al. 2023; Wang et al. 2025a, 2024a; Leroy, Cabon, and Revaud 2024) and generative world models (Hu et al. 2023a; Wang et al. 2024b; Li et al. 2025b; Zuo et al. 2025; Bogdoll et al. 2025) increasingly call for shared representa-a tions that…”

“Autoregressive examples include GAIA-1 [Hu et al., 2023], which predicts image tokens from video, text, and action context; OccWorld [Zheng et al., 2024], which tokenizes 3D occupancy and forecasts future scene and ego states; and DrivingWorld [Hu et al., 2024], which employs a pose-conditioned…”

“Early works such as GAIA-1 [21] and CommaVQ [8] are often restricted to a single camera view and limited text-or action-based control, while later works like Drive-WM [56] and UniMLVG [7] enable multi-view generation conditioned on a wide range of inputs, and MaskGWM [41] and Vista [13] focus on…”

“A first line of work models this prediction in discrete token space : GAIA-1 [22], DrivingWorld [23], and DrivingGPT [7] employ autoregressive transformers over quantized visual tokens, while Cosmos [1] scales this paradigm to large-scale video pretraining.”

“Vision-centric autonomous driving has emerged as a scalable and cost-effective paradigm for perception, thereby garnering escalating scholarly and industrial interest [1]–[5].”

“…body of work leverages large-scale video generative models as world models for robot con-trol [11, 41, 42, 56], motivated by the observation that video prediction implicitly captures physical dynamics, object interactions, and causal structure that can be repurposed for policy learning [12, 28].”

“Early approaches such as Dreamer [8] learned latent dynamics in simpli-fied settings, while recent large scale models including Ge-nie [2], RT-2 [30], OpenVLA [11], and GAIA-1 [9] expand this idea to multimodal, physically grounded domains.”

“Recent models, including GAIA-1 [29], GAIA-2 [47], Vista [15], DriveDreamer [54], DriveDreamer-2 [58], UniDriveDreamer [59], and VaViM/VaVAM [2], have shown strong capability in realistic driving simulation.”

“Complementary directions include world models for predictive simulation (Hu et al., 2023a; Gao et al., 2024; Liu et al., 2026b) and reinforcement learning for driving policy optimization (Kiran et al., 2021; Sallab et al., 2017; Wang et al., 2018).”

“Another direction of work addresses sensor simulation, either via generative models [31, 32, 33, 34, 35] or via Gaussian splatting [36, 8, 37].”

“Representative examples include large-scale video generation models such as Sora[45], interactive environment models trained from unlabeled video such as Genie[11], and action-conditioned simulators in domains such as GAIA-1[36].”

“Visual world models [14], [15] using image representations have been successful in predicting images or videos of driving scenes and simulating driving environments.”

“World-model-based approaches, including Vista [29], Drive-WM [30], and GAIA-1 [31], further model temporal traffic evolution and interactive driving dynamics for long-horizon simulation and prediction.”

“GAIA-1[20] cast autonomous driving world modeling as action-conditioned generative sequence modeling over multimodal signals.”

“GAIA-1 [11] presents a token-based generative world model that takes video, text, and action as input and generates controllable driving scenarios.”

“In parallel, driving world models [14– 17, 39, 40] model the temporal evolution of driving scenes conditioned on trajectories, actions, or multi-view observations for future prediction and planning.”

“To connect fidelity to world model usefulness, we train a lightweight next-token predictor [5], [6], [13] and observe lower dynamic-position perplexity at comparable bitrate (206.0 → 193.1 at ∼ 0.024 Mb/s; 158.9 → 155.6 at ∼ 0.036 Mb/s).”

“In autonomous driving, three mainstream world models have emerged: vision-based [12, 43, 44], occupancy-based [9, 41, 54], and LiDAR-based [13, 20, 45, 46, 52].”

“World models have progressed from narrow settings like simple games [14, 18, 37] to diverse environments, with actions that span text [1], navigation [4, 21], and whole-body joint control [3].”

“Representative vehicle-centric world models include GAIA-1 [14], which generates realistic driving videos conditioned on actions and text; Vista [10], a generalizable driving world model with versatile controllability; DriveDreamer [36], which leverages structured traffic constraints; and Epona…”

“Recent breakthroughs have demonstrated highly successful applications across specific domains, including autonomous driving ( Hu et al. [2023], Feng et al. [2023]), robotics ( Agarwal et al. [2025]), gaming ( Hafner et al. [2025], Kanervisto et al. [2025]), and navigation ( Bar et al. [2025]).”

“GAIA-1 [21] demonstrates controllable multi-camera gen-2 eration, while subsequent work scales to longer horizons (LongDWM) and improves multi-view consistency (GAIA-2) [41, 50].”

“GAIA-1 [138] demonstrated that large-scale generative models can produce controllable scenario generation with emergent understanding of scene dynamics, geometry, and agent interactions.”

“One line of methods aligns VAE latent spaces with VFM features through distillation losses [12,31,47,60,86].”

“For instance, GAIA-1 [14] employs a multi-modal transformer architecture with language and action conditioning to autoregressively generate minute-long video clips.”

“For explicit scene-level prediction, many approaches rely on diffusion [12, 40] or autoregressive models [16, 58] to synthesize future observations.”

“This transition is largely driven by vision-language-action models (VLAs) [23,27, 45,63,64] and world models [15,27,48,50,59,61], which are increasingly applied to enhance the cognitive capabilities of autonomous systems.”

“Recent studies [23, 20] have shown that BEV perception pipelines are vulnerable to such corruptions, especially in modules such as view transformers.”

“Minor inaccuracies or suboptimal choices in generating a single frame can propagate and amplify over time, leading to a degradation of quality, loss of coherence, and ”drifting” from the intended content in longer video sequences (Hu et al., 2023).”

“While early work focused on image/video prediction [16, 45, 53, 62, 63], recent studies extend to structured 3D data.”

“…2025a), they have only been scaled up recently (Agarwal et al., 2025; Ball et al., 2025; Bruce et al., 2024; Decart et al., 2024; Guo et al., 2025b; Hafner et al., 2025b; Hu et al., 2023; Li et al., 2025; Parker-Holder et al., 2024; Pearce et al., 2025; Seid and Hojel, 2024; Valevski et al., 2025).”

“Recent work has also developed generative models for autonomous driving, aiming to address large-scale real-world challenges [42], [43].”

“In scenario generation, GAIA-1 [18], DriveDreamer [19], and DriveWorld [20] develop LLM-based frameworks that synthesize realistic driving content from multimodal inputs such as video, text, and actions, expanding data diversity for training.”

“Large-scale generative video models [45, 46, 47] push realism further but rely on expensive diffusion backbones and language prompts, limiting their use for fine-grained control.”

“…test coverage and applied in various fields of AD function validation, ranging from the generation of synthetic trajectories [12], [13] over the creation of synthetic training data such as traffic signs [14] and camera images [15] to the setup of world models such as GAIA-1 [16] or Genie [17].”

“Table 1 summarizes each R-CNN version's main characteristics and performance for AV applications [18] .”

“In the domain of autonomous driving, Hu et al. [23] proposed GAIA-1, an innovative generative world model.”

“…perception models such as HiLMD [70] and Talk2BEV [71] utilize video and natural language input to generate visuo-linguistic responses related to driving, while generative models such as GAIA-1 [72] and UniSim [73] can anticipate environmental dynamics based on previous driving maneuvers.”

“NUWA-XL (Yin et al., 2023) and GAIA-1 (Hu et al., 2023) adopt structured diffusion or world model approaches, while FreeNoise (Qiu et al., 2023) and Gen-L-Video (Wang et al., 2023a) extend generation by aggregating noise-sampled or overlapping segments.”

“…that single configurations with no hyperparameter tuning can outperform specialized methods across diverse benchmark tasks, complete farsighted tasks such as collecting diamonds in Minecraft without human data or curricula, and capture expectations of future events during autonomous driving [3–5].”

“Thus, generative world models, similarly based upon frame generation but explicitly conditioned on state-action information, have shown to be an increasingly powerful paradigm; enabling generalist agents to learn, plan, reason, and act upon their surroundings [9, 10].”

“Many of them rely on discrete representations of actions and/or images (Hafner et al., 2023; Hu et al., 2023) in order to leverage the techniques of autoregressive inference that are extensively developed in language modeling.”

“Specifically, front-view driving video continuation demands that models maintain long-term temporal coherence for global scene structure and object dynamics (Hu et al., 2023a; Russell et al., 2025; Wang et al., 2026; Xie et al., 2026; Zhang et al., 2026; Mittal et al., 2026).”

“There is also growing interest in utilizing their generative capabilities for photo-realistic simulation [60, 61].”

“Owing to their training stability, these models are versatile enough to be conditioned on data from a range of modalities, including images [22], text [44, 47], and audio [50].”

“GAIA-1(Hu et al. 2023) utilizes DINOv2(Oquab et al. 2024) to integrate high-level semantic features.”

“For instance, GAIA-1 [14] requires training on 420M images to achieve desired performance in complex scenarios.”

“Predictive learning, an unsupervised method focused on predicting future events based on past observations, plays a critical role in the world models [11, 17, 47].”

“The emergence of LLMs has sparked adaptations for autonomous driving [10, 17, 26, 28].”

“While many recent works do show successful RGB video generation [24,34], they typically train on far more data than us (days of compute on 10 million videos vs 7 hours of compute on 1000 videos).”

“…game imagery, producing datasets like DITM (Johnson-Roberson et al., 2016) and VIPER (Richter et al., 2017); (3) Generative AI employs techniques like GANs (Wang et al., 2017), diffusion models (Zhang et al., 2023), and un-supervised learning (Hu et al., 2023) to synthesize high-quality images.”

“GAIA-I [14] further expands this paradigm by incorporating multiple conditioning inputs, such as video, text, lay-outs, and actions, enabling the generation of realistic and diverse driving scenarios with fine-grained control over vehicle behavior and scene features.”

“The development of realistic “world models” has become a critical foundation for embodied AI and autonomous driving Wang et al. [2023b], Zhao et al. [2024], Li et al. [2023], Hu et al. [2023], Yang et al. [2024a].”

“Large open-ended world models [61, 23, 6] may potentially enable the training of highly generally capable agents .”

“Finally, some recent AD research has harnessed multi-a modal foundation models and encompassed both large language models (LLMs) and vision-language models (VLMs) to enhance world modeling, reasoning, and planning [19], [7], [20], [21].”

“Recent work in world models focuses on learning the environment’s transition dynamics, often by predicting future sensor data or occupancy grids [9], [14].”

“This capability is pivotal for downstream applications spanning interactive entertainment and embodied AI (Zhu et al., 2025; Yang et al., 2023; Hu et al., 2023; Zhou et al., 2024; Kanervisto et al., 2025; He et al., 2026; Tang et al., 2025).”

“They have been applied across robotics [Yang et al., 2023, Agarwal et al., 2025], autonomous driving [Hu et al., 2024], and video games [Bruce et al., 2024], with the promise of serving as a scalable substitute for costly real-world interaction in reinforcement learning [Hafner et al., 2020, 2025].”

“Across these approaches, world models offer promising applications in autonomous driving (Hu et al., 2023; Wang et al., 2023; Ren et al., 2025a; Du et al., 2023; NVIDIA et al., 2025), robotics and action-conditioned simulation (Yang et al., 2023; Zhou et al., 2024; Gao et al., 2026a; NVIDIA et al.,…”

“Applications include video games, where models generate playable frames from controller input [2, 4, 6, 9–12, 22, 41, 48, 56], autonomous driving [13, 27, 36], robotic control and planning [8, 23, 51, 53], and generative computer interfaces [58].”

“In autonomous driving, GAIA-1 [26], Drive-WM [27], and DrivingWorld [28] generate controllable future scenes for trajectory planning.”

“Driving models add other road users and an environment that changes without the agent’s action (Hu et al., 2023).”