GAIA-1: A Generative World Model for Autonomous Driving

“Its training objective is to predict the next image token in the sequence conditioned on all past tokens, using causal masking in the attention matrix of the transformer blocks [35].”

“The discrete autoencoder is a fully convolutional 2D U-Net [31].”

“Because these text sources are imperfect, to improve the alignment between generated futures and the text prompt, we employ classifier-free guidance [45, 46] at inference time.”

“Large language models have shown clear benefits in scaling model size and data [90, 24, 26, 20, 21, 22, 23].”

“The advantage of such a perspective is that world models can benefit from scaling properties of high-capacity sequence model architectures applied to large-scale unsupervised training [26].”

“Following the recent advances in image [36, 37] and video generation [16, 18] we use denoising video diffusion models for the GAIA-1 decoder.”

“Most language models [25, 26, 24, 21, 27, 22] use byte-pair encoding (or equivalent) as a trade-off between character-level and word-level tokenization.”

“The image reconstruction loss is a weighted sum of L 1 , L 2 , perceptual loss L perceptual [32], and GAN loss L GAN [33].”

“Large language models have shown clear benefits in scaling model size and data [90, 24, 26, 20, 21, 22, 23].”

“The models used to fit the power-law ranged from 10,000x to 10x smaller models in terms of parameters (0.65M to 650M), as visualized in Figure 8b. Similarly to [49], the compute was estimated as a function of the parameter count.”

“However, most existing approaches predict videos only from past frames [20], [21] or condition on limited motion information such as ego actions [22], [23] or initial traffic layouts [24], [25].”

“Wayve’s GAIA-1 synthesizes driving scenarios at 288×512 resolution using discrete token sequences, demonstrating the potential to generate large quantities of edge cases that could substantially reduce the reliance on real world test miles [69], [103].”

“In this work, we explore how two data driven simulators can be used to train an E2E driving policy: a repro-jective novel view synthesis simulator [27], and a learned World Model [10, 11, 25].”

“Some of them introduce this paradigm into autonomous driving [5, 6, 10, 15, 18, 29, 30, 45] for simulating the plausible visual outcomes under versatile driving controls.”

“GAIA-1 [23] 2023 Video-Prompt Video Realistic traffic scenarios with safety risks and CLIP to align images and text.”

“Through various frameworks and simulations, RL is shown to effectively teach machines to navigate and make decisions like human drivers, achieving human-like car-following behaviors, and handling diverse driving conditions with improved accuracy [23].”

“The advent of large-scale generative models has catalyzed the development of world models for autonomous driving (AD), aiming to synthesize infinite driving scenarios for data augmentation and policy learning [12, 30, 40].”

“Environmental Consistency (E) matters for synthetic data augmentation [61, 28]: hallucinated background changes induce shortcut learning [22].”

“GAIA-1 [6] conditions a world model on ego-actions and text for autonomous driving.”

“GAIA-1 [90] formulates world modeling as an autoregressive sequence prediction task over tokenized video, text, and action inputs, combining large-scale Transformers with video di ff usion decoders to produce realistic, controllable driving rollouts.”

“This approach is widely applied across different visual domains (Mizrahi et al., 2023; Chang et al., 2022; 2023; Li et al., 2023b; Villegas et al., 2022; Hu et al., 2023), with (Junke et al., 2024; Lu et al., 2025; Ma et al., 2025) developing unified tokenizers across multiple modalities.”

“Image-based methods leverage powerful generative architectures to synthesize high-fidelity driving videos, primarily for data generation and scene simulation [14, 18, 54, 64, 65, 78].”

“…2024; Renz et al., 2025; Fu et al., 2025); World Models that act as data engines generating diverse traffic scenarios for training and evaluation (Hu et al., 2023; Russell et al., 2025; Ren et al., 2025; Wang et al., 2024); and simulation at scale providing infrastructure for closed-loop…”

“• Section G presents the additional experiment results and analysis of COW .”

“Such capabilities are central not only to creative media (Ye et al., 2025; Yuan et al., 2025), but also to emerging world models for gaming, simulation and embodied AI (Ha and Schmidhuber, 2018; Yang et al., 2023; Hu et al., 2023; Google DeepMind, 2024; Hafner et al., 2025).”

“They hold broad application prospects in fields such as autonomous driving [25, 37], navigation [4], and robotic manipulation [2, 29, 48] and games [9, 16, 18, 44, 53].”

“Inspired by these advances, researchers have applied world models to autonomous driving [17]–[20].”

“For world models, generative approaches like DriveDreamer [52] and GAIA-1 [19] generate realistic driving scenarios for data augmentation and training.”

“In GAIA-1 [5], a generative world model creates realistic driving scenarios with video, text, and action prompts.”

“Therefore, the normalization ratio S is, Published as a conference paper at ICLR 2025 - Outstanding paper at the World Models Workshop”

“This unique prospect makes our model a promising approach to building world models (Hafner et al., 2020; 2023; Hu et al., 2023; Sekar et al., 2020; Yang et al., 2024) for vision.”

“Recent generative simulators and action-conditioned video models scale this idea to robotics, driving, games, and interactive environments (Hu et al., 2023; Bruce et al., 2024; AI, 2024; Sun et al., 2025; Robbyant Team et al., 2026; Shen et al., 2026; Wang et al., 2026b; Mao et al., 2026; Wu et…”

“Evaluating robustness in this context is challenging because attack efficacy is highly sensitive to both environment dynamics and adversarial hyperparameters [51, 24, 38].”

“Video models like GAIA-1 [18], Drive-WM [49], and Drive-JEPA [47] construct predictive world representations from visual histories, with Drive-JEPA further combining video pretraining and trajectory distillation to support end-to-end planning.”

“Furthermore, generative world models [11, 28, 48, 57, 58] have been developed to synthe-size future driving scenes or Bird’s-Eye View (BEV) representations [31, 41] for trajectory evaluation.”

“Its growing importance stems from promising applications in areas such as video or game scene generation [5]–[7] and world simulation [8]–[10].”

“World models have been widely used for learning environment dynamics across domains such as robotics [44, 12], autonomous driving [45, 46], and video games [47, 48].”

“Modern video generation models have emerged as a powerful candidate of the world models [14], [20], [27], [28], [29], [30], [31], [32].”

“Consequently, world models are widely applied in fields such as navigation [7–11], autonomous driving [12–16], robotics [17–22], and are considered a cornerstone of embodied artificial intelligence.”

“Recent models [10], [11], [12], [13] have focused on driving video generation.”

“Initial works in the AV domain focused on AR video generation [17] and predicting text explanations alongside driving actions [18], [19] from a front-facing camera, tokenizing images with the patch-based embedding scheme from ViT [7].”

“…to the human driver, several works use controllable video generation technology to build the driving world model and predict the future as video, which has made great progress in the past few years [9, 10, 11, 12], and video generators also achieved promising results as world simulators [13, 14].”

“Frequently, text prompts provide context [6] or driving instructions [50], [56].”

“GAIA-1 [43] is an autonomous driving world model that combines video, text, and action inputs to generate realistic driving scenarios.”

“Similarly, world models such as DriveDreamer [321], GAIA [322], and ADriver [323] simulate plausible traffic scenes informed by real-world interactions.”

“GAIA-1 (Hu et al., 2023), DriveDreamer (Wang et al., 2023), DrivingWorld (Hu et al., 2024), and Vista (Gao et al., 2024) generate future observations for scenario simulation, planning, and policy evaluation.”

“Generative approaches synthesize controllable future driving videos from observations, actions, or language (Hu et al. 2023; Wang et al. 2023a,b; Gao et al. 2024), while structured alternatives forecast future occupancy or point clouds (Zheng et al. 2023; Yang et al. 2023).”

“World models learn a predictive model of the environment and roll out future states conditioned on text [65], driving commands [34], or robot actions [69]; trained on large video corpora, world foundation models [58] generate high-fidelity future observations, recover action-controllable dynamics…”

“UniSim [1], GAIA-1 [21], Genie [22], and Cosmos [23] treat the world as a video that unrolls in response to actions or text, and a growing line of work repurposes pretrained video diffusion models as interactive simulators [24]–[26].”

“These advances have driven world model deployment into high-stakes domains: autonomous driving [4, 5, 6], robotics [7, 8], video generation [9], and LLM-based agentic systems that use world models for multi-step planning and counterfactual simulation [10, 11].”

“Early efforts [23, 64] demonstrate action-conditioned video generation but are constrained primarily to a single front-view perspective.”

“Wu et al. (Wu et al., 2022) proposed a human-guided RL framework, with the core being the Prioritized Experience Replay (PER) mechanism, which significantly improved the learning efficiency and performance of RL.”

“Domain-specific systems like GAIA [33] and VAVAM [5] focus on driving or embodied navigation [4].”

“These models show promise in domains like robotics [47, 54, 55] and autonomous driving [30, 58, 70, 71].”

“Image-based approaches [9], [10], [11], [12], [13], [14], [15] generate future driving videos using diffusion or transformer architectures, often conditioned on actions, BEV layouts, or HD maps to enhance realism.”

“Different models represent the scene in various representations spaces, which can be divided into 2D image representations [15, 22, 35, 63, 66, 73, 86], 3D point clouds representations [30, 31, 46, 69, 83], and 3D occupancy representations [1, 61, 67, 87].”

“Autoregressive models [ Hu et al. , 2023; Agarwal et al. , 2025 ] In what follows, we will focus mostly on diffusion models [ Dhariwal and Nichol, 2021 ] , which have emerged as a prominent architectural framework in both the domains of autonomous driving and robotics scene generation.”

“Generative systems [36], [37] have achieved controllable, photorealistic synthesis of physical scenarios, while vision-language-action models [38], [39] integrate natural language instructions with continuous physical manipulation, bridging the gap from abstract reasoning to embodied interaction.”

“Recently, attempts to use large language models [115] or generative models [187] for automatic annotation and to expand data samples have provided potential solutions to this problem.”

“In the autonomous driving domain, successful works like [36, 18] sample trajectories using diffusion models while [21] formulate Constrained MDP (CMDP) to incorporate constraints.”

“…some research optimistically suggests that large generative models and their world representations are robust enough to be used as world simulators in scenarios such as autonomous driving (Hu et al. 2023), robotics (Yang et al. 2023) and for training generalist agents (Bruce et al. 2024).”

“In autonomous driving [48, 10], models predict based on driver actions, while in robotics [41, 18], predictions are conditioned on control signals of robots.”

“GAIA-1 [16], introduced by Wayve, is a generative world model that utilizes video, text, and action inputs to create realistic driving scenarios.”

“Some of them introduce this paradigm into autonomous driving [5, 6, 10, 15, 18, 29, 30, 45] for simulating the plausible visual outcomes under versatile driving controls.”

“For example, in autonomous driving (Hu et al., 2023; Jia et al., 2023; Gao et al., 2024; Zheng et al., 2024), the condition is the driver’s operations, while in robot world models (Yang et al., 2023; Black et al., 2023; 1xw, 2024), the condition is often the control signals.”

“DriveDreamer [174], GAIA-1 [68], DriveDreamer-2 [177], GAIA-2 [179], Driving-World [181], and DrivingGen [183] use this capability for data augmentation, scenario coverage, or policy-conditioned evaluation.”

“More recent systems, including Genie, GAIA-1, and interactive video simulators, have broadened this line of research towards generative and interactive environments 14,75–77 .”

“At the other end, photorealistic simulators render high-fidelity sensor data to minimize the sim-to-real gap, either via handcrafted assets [61, 55, 62], neural reconstruction methods [63, 64, 27, 65], or generative models [66, 67, 68, 69, 70].”

“However, as shown in Figure 2 (b), current methods [20, 21, 22, 23] that attempt to incorporate world models into E2E-AD largely rely on diffusion-based future scene generation, which is computationally intensive and unsuitable for real-time applications.”

“Compared to pixel-level autoregressive predictors, diffusion models tend to be more stable to train and provide stronger perceptual quality, which is attractive for long-horizon rollout and planning [1,18,23,48].”

“Other methods use ego-motion or trajectory cues to control ego-vehicle behavior [Hu et al., 2023, Lu et al., 2024, Gao et al., 2024b, Zhang et al., 2025, Russell et al., 2025, Zhan et al., 2026].”

“World models (WMs) are powerful tools designed to predict plausible future states of the world based on past observations Ha & Schmidhuber (2018b;a); Hafner et al. (2019); Hu et al. (2023); Bruce et al. (2024).”

“Existing research primarily focuses on predicting scene evolution, targeting either visual appearance changes [2], [5] or 3D geometric deformations [6], [7].”

“2) Interactive Simulation: As an interactive environment simulator, a video-based world model can support stable action-conditioned rollout generation for reinforcement learning and facilitate real-time evaluation of generated trajectories, allowing safe and reproducible policy testing and…”

“…distills multimodal large-model knowledge to improve planning robustness to rare events while enabling efficient inference, and generative world-model efforts such as GAIA-1 [39] and DriveDreamer-2 [40] enable controllable synthesis of rare hazards for scalable stress testing and data augmentation.”

“GAIA-1 [19] and Vista [24] generate the future world with video diffusion models [3], [17] conditioning on text prompts and driving actions.”

“VLMs have the potential to transform multiple domains, including autonomous driving, where they can analyze real-time imagery for rapid decision-making [11]; medicine, where they assist in pathology image classification [13]; and law, where they simplify complex legal documents for non-experts [24].”

“Another notable model is GAIA-1 [10], which focuses on planning by integrating visual and language inputs to generate realistic driving scenarios.”

“Artificially generated synthetic data, produced by platforms such as Wayve’s GAIA-1 [23] and the CARLA simulator [15], offers an alternative source of multimodal driving data.”

“While generative world models such as GAIA-1 (Hu et al., 2023) offer photorealistic closed-loop simulation, their computational cost remains a barrier to large-scale RL training.”

“There are a variety of ways to create such an embedding (see, e.g., [41, 42, 43, 39, 44, 45]); BAMS does not have any restrictions for this embedding, although smaller dimensionality d improves computational efficiency.”

“Additionally, VLMs are instrumental in scene generation, creating detailed and realistic driving videos or scenarios that factor in specific environmental conditions [33].”

“Generative world models have shown promising results in autonomous driving [9, 21, 26, 27, 53].”

“In in the real world, this oversight would pose safety concerns and the virtual testing should flag such issues (i.e., the two cars should also be missed in virtual testing).”

“The emergence of LLMs has sparked adaptations for autonomous driving [10, 17, 26, 28].”

“For instance, GAIA-1 [29] generates realistic driving scenarios by integrating video, text, and action inputs and showcases generative models in adapting to the changing dynamics of the real world.”

“…world models like Genie (Bruce et al., 2024), and text-to-video generation models like Sora (Brooks et al., 2024) demonstrate that foundation video models can be used as learned simulators, including in real-world settings like robotics (Yang et al., 2023b) and autonomous driving (Hu et al., 2023).”

“World models learn environment dynamics from past observations, in autonomous driving, they are predominantly formulated as future-video generation [9, 11, 12, 35].”

“GAIA-1 [7] and DriveDreamer [25] generate controllable futures from video, text, and action conditions.”

“UniSim [57] and GAIA-1 [18] study action-conditioned simulation in robotics and driving, while GameNGen [43], Oasis [8], MineWorld [13], and related systems [32, 23] demonstrate controllable generation in game environments.”

“Related action-conditioned video world models have also been developed for autonomous driving (Hu et al. 2023; Wang et al. 2024; Gao et al. 2024; Zhang et al. 2025a).”

“A complementary thread grounds control in human embodiment instead: PlayerOne [48] and PEVA [2] drive generation from SMPL or whole-body kinematics, but capture only coarse, large-scale motion without finger articulation.”

“Representative works such as GAIA-1 (Hu et al. 2023a”

“This paradigm was subsequently extended to complex real-world domains by GAIA-1 [10] and UniSim [11], focusing on autonomous driving and robotic manipulation.”

“This ability makes them useful for interactive games [8, 24, 46, 61], robotics [39, 7, 4, 14], and autonomous driving [9, 16, 28, 29, 60, 66].”

“…reconstruction perspective (Yan et al., 2024; Chen et al., 2024b; Yang et al., 2025; Chen et al., 2025; Tan et al., 2026) and the generation side (Hu et al., 2023; Wang et al., 2024a; Gao et al., 2024b; Russell et al., 2025; Guo et al., 2025), converging toward a hybrid reconstruction-generation…”