GAIA-1: A Generative World Model for Autonomous Driving

“Its training objective is to predict the next image token in the sequence conditioned on all past tokens, using causal masking in the attention matrix of the transformer blocks [35].”

“The discrete autoencoder is a fully convolutional 2D U-Net [31].”

“Because these text sources are imperfect, to improve the alignment between generated futures and the text prompt, we employ classifier-free guidance [45, 46] at inference time.”

“Large language models have shown clear benefits in scaling model size and data [90, 24, 26, 20, 21, 22, 23].”

“The advantage of such a perspective is that world models can benefit from scaling properties of high-capacity sequence model architectures applied to large-scale unsupervised training [26].”

“Following the recent advances in image [36, 37] and video generation [16, 18] we use denoising video diffusion models for the GAIA-1 decoder.”

“Most language models [25, 26, 24, 21, 27, 22] use byte-pair encoding (or equivalent) as a trade-off between character-level and word-level tokenization.”

“The image reconstruction loss is a weighted sum of L 1 , L 2 , perceptual loss L perceptual [32], and GAN loss L GAN [33].”

“Large language models have shown clear benefits in scaling model size and data [90, 24, 26, 20, 21, 22, 23].”

“The models used to fit the power-law ranged from 10,000x to 10x smaller models in terms of parameters (0.65M to 650M), as visualized in Figure 8b. Similarly to [49], the compute was estimated as a function of the parameter count.”

“Beyond artistic content creation, these models are increasingly envisioned as world simulators [12] for robotics [1] and interactive agents [39, 3, 8, 30].”

“A world model must often represent facts that are not visible in the current observation: object permanence, prior locations, hidden goals, map structure, delayed effects, or previously observed scene details.”

“Some train high-fidelity pixel-level world models as data engines to generate rich synthetic experiences, especially for long-tail scenarios [11, 13, 21, 31, 37, 38].”

“…over multiple seconds of driving history [19], tokenized/diffused representations of scenes that enable attention across spatial and temporal dimensions, and the use of imitation learning and reinforcement learning to align model behavior with human driver preferences and safety constraints.”

“It demonstrates the ability to comprehend and infer driving concepts not present in the training data, exhibiting counterfactual inference capabilities [15].”

“Furthermore, to enhance 4D reconstruction stability, recent generative world models [11,36] incorporate probabilistic state transitions.”

“Diffusion-based approaches have been applied to sensor simulation or data augmentation, leveraging layout conditioning such as HD maps or object bounding boxes to produce realistic urban scenes [16, 24, 37, 53, 63, 68, 70, 77].”

“Some like GAIA [26, 48] train at scale on specific domains such as autonomous driving, while Cosmos [3] uses large-scale data to handle heterogeneous domains.”

“However, such generative models suffer from generalization issues required for real life driving scenarios [18].”

“Existing text-based image generation methods (Hu et al. 2023; Souˇcek et al. 2024) and world models (Liu et al. 2024; Wang et al. 2024b) utilize visual and textual information to predict future frames but lack action guidance.”

“GAIA-1 (Hu et al. 2023a) leverages sequence learning to anticipate future events.”

“…beyond conditional video generation, particularly where underlying laws are not directly observable from image pixel space, such as simulating fluid dynamics [11], weather forecasting [12], robot motion planning [13], future scenarios for autonomous driving [14], [15], and tra ffi c prediction [16].”

“The generative approach enables autonomous vehicles to predict the intentions and likely behaviors of other vehicles, pedestrians, and other traffic participants [64, 176].”

“Video diffusion in driving scenes has garnered widespread attention [20, 26, 41, 73, 79] for safe driving by frame or view synthesis.”

“GAIA-1 [34] generates realistic and diverse driving videos by integrating video, text, and action inputs, demonstrating a strong understanding of contextual information and physical principles.”

“For example, works like Menapace et al. (2021), Yang et al. (2023), Bruce et al. (2024), Hu et al. (2023), simulate real-world environments.”

“More recent approaches leverage large generative backbones and represent control implicitly [34, 28, 3], which enables test-time action inference via latent-space search or optimization [37, 66, 18].”

“Recent video-generation models turn this prediction into high-fidelity, controllable simulation [12, 18], with the potential to reshape how robotic systems are developed and tested.”

“Recent advances in generative world models have made synthetic video a promising interface for embodied AI, enabling robot-centric rollouts for manipulation prediction, demonstration synthesis, and policy evaluation [1, 2, 3, 4, 5, 6, 7].”

“• Autonomousdrivingsimulation [49, 2]: The causally relevant events are vehicle dynamics and inter-agent interactions (e.g., a forward collision, a lane-merge maneuver, or a pedestrian-crossing response) which manifest as large-scale, visually obvious changes in the scene.”

“A robot must predict what happens after pushing a cup along a tabletop [24, 27]; a driving simulator must model a pedestrian stepping into the road [20, 6]; an interactive game must allow users to move entities, not just the viewpoint.”

“Prominent examples, such as Wayve’s GAIA-1 [19] and the NVIDIA Cosmos [2] platform, primarily target scenario generation, future prediction, and planning for closed-loop simulation [53].”

“In brief, 2D video-generative models are strong at perceptual realism and flexible conditional generation, but they often lack explicit physical structure and may drift under long rollouts [Hu et al., 2023, Bruce et al., 2024].”

“GAIA [20] conditions generation on image, text, and action inputs.”

“…including mobile robots with highly complex and difficult-to-control dynamics [49, 62], and autonomous driving systems operating in vastly broader environments [45, 60, 69, 104, 127, 155, 158, 159, 168], thereby advancing the closed-loop interactive capabilities of models in the real world.”

“In autonomous driving, GAIA-1 [20] and Drive-WM [47] forecast driving scenes conditioned on planned actions and map context.”

“16,22,36,49].”

“…complex settings where providing domain knowledge is prohibitively difficult, such as climate systems (Alet et al., 2025; Lam et al., 2023), biological systems (Abramson et al., 2024), and interactive environments like video games and autonomous driving (Hafner et al., 2025; Hu et al., 2023).”

“Semantic-guided driving scene generation remains under-explored, with existing approaches primarily relying on coarse-grained controls such as text prompts [23,52] or weather tags.”

“Recent work extends world models to video prediction [5, 65], autonomous driving [25], and robotics [61], suggesting their potential as general-purpose simulators for complex real-world dynamics.”

“Traditionally, high-cost interaction with physical hardware (Black et al.; Hu et al., 2023) is the ultimate goal for world models to engage with the real world, however, acquiring training data at scale remains a significant challenge.”

“In recent years, autonomous driving has emerged as a major frontier for world model applications, with numerous methods [20,37] proposed to evaluate their effectiveness in driving scenarios.”

“Some studies employ world models to predict future real-world scenes [14]–[16], where historical latent information is propagated to estimate future states and decode them into future images.”

“…across various domains, ranging from video generation (e.g., Sora [OpenAI, 2024], Veo [Google DeepMind, 2024]) and world simulators (e.g., Genie series [Bruce et al., 2024]) to autonomous driving systems (e.g., FSD V12 [Elluswamy, 2023], GAIA-1 [Hu et al., 2023a], UniAD [Hu et al., 2023b]).”

“With the continuous advancement of video generation models, interactive generative video (IGV) has emerged as a research focus, demonstrating broad application prospects in domains such as gaming, embodied AI, and autonomous driving[6, 16, 27, 30, 38, 47, 51].”

“…(Chi et al., 2025; Yang et al., 2025a) and evaluation (Ljungbergh et al., 2024; Yan et al., 2025; Cao et al., 2025a), for which the models generating AD data, such as AD world models (Hu et al., 2023a; Wang et al., 2024b; Gao et al., 2024b; Wen et al., 2024b; Zhao et al., 2025a;c), can be helpful.”

“This arises in applications such as one-shot cinematography (Failes, 2020; Yates, 2023) and synthetic scenario generation for autonomous driving (Hu et al., 2023; Russell et al., 2025), which entail camera-guided video generation with a predefined camera trajectory.”

“GAIA-1 [14], DriveDreamer-2 [48], and Drive-WM [40] use spatio-temporal attention or recurrent architectures, while Panacea [43] and DrivingDiffusion [21] employ feature propagation between frames.”

“Similarly, GAIA-1 [71] adopts an object-centric perspective, explicitly modeling key scene entities (e.g., vehicles, pedestrians) with integrated geometric constraints, aligning more closely with human scene cognition.”

“Video-based meth-ods [6, 19, 35] leverage autoregression and richer conditioning, while BEV-based approaches such as Magic-Drive [4] enforce temporal consistency across frames [26, 36].”

“As video generation extends into domains such as autonomous driving, recent works have explored using structured inputs like driving commands or actions to guide the generative process [75, 89, 18, 23, 84, 30].”

“Generative world models are already being applied in practical domains such as autonomously driving cars (Russell et al, 2025; Hu et al, 2023).”

“GAIA-1 [50] leverages video, text, and action inputs to generate realistic driving scenarios with fine-grained control over ego-vehicle behavior and scene features.”

“For the term “slower”, the model might consider various degrees of “slowness”:—for calls inject sound dimensions to the model, offering insights into ambient and environmental conditions like traffic noise or mechanical failures.”

“As shown in Fig.2, we reorganize and define safety-critical scenarios, including static scenario generation, i.e., adversarial attack and natural distribution shift scenarios, and dynamic scenario generation, i.e., accident-prone scenarios.”

“…abilities To further enhance the alignment of vision and language features, many works have incorporated VisionLLMs with diffusion models to construct world models grounded in language, as seen in projects like DriveDreamer [17], [18], GALA [19], DrivingDiffusion [20], and DriveGenVLM [21].”

“Recent empirical studies have also demonstrated the capacity of world models to generalize to unseen noisy states and dynamics in complex environments, such as autonomous driving [9].”

“An important goal in generative AI is to develop dynamic and accurate 3D environments, allowing for dynamic simulations consisting of objects, spaces, and agents [Kaur et al., 2023, Hu et al., 2023].”

“They can be trained to be as general as needed by training on a large dataset such as GAIA-1, which comprises 4,700 hours of autonomous driving [15].”

“Recently, a subset of transfer learning known as domain adaptive learning has grown in popularity [28].”

“[28] LLaMA-7b [193] Custom”

“GAIA-1 [27] tackle autonomous driving in unstructured settings by encoding multi-modal inputs into a unified representation and predicting image tokens based on prior inputs, using an autoregressive transformer.”

“Visual models such as GAIA-1, Drive-WM, and Epona generate future observations from scene, trajectory, or action conditions [24], [16], [17].”

“Generative world models [1, 12] instead decode the predicted latent into a predicted observation.”

“Visual-action models [24, 56, 6, 11] and large-scale unified backbones [32, 89, 2, 50, 44, 76, 66, 67] provide tighter coupling but typically at billions of parameters for general-purpose manipulation; DreamZero [79], for instance, uses a 14B-parameter WAM as a zero-shot policy on physical…”

“In driving, GAIA-1 (Hu et al., 2023), DriveDreamer-2 (Zhao et al., 2025b), and Epona (Zhang et al., 2025) generate future driving video, with Epona additionally modeling future trajectories.”

“…Agarwal et al. [2025], Kong et al. [2024], Gao et al. [2025] are intuitive and demonstrate remarkable physical consistency—such as understanding gravity and collisions—highlighting their potential as implicit world models Hu et al. [2023], Du et al. [2023], Ren et al. [2025], Kang et al. [2024].”

“Large video models provide temporal and motion priors for this purpose, and driving-specific world models adapt such priors to controllable future prediction, simulation, and planning (Wan et al. 2025; Hu et al. 2023a; Gao et al. 2024a,b).”

“The typical work GAIA-1 [113] captures spatiotemporal dynamics and high-level structures in driving scenes through a diffusion decoder, modeling the world as an unsupervised sequence modeling problem, and then predicts the next token in the sequence.”

“AR-Diffusion [27] proposes non-decreasing noise schedules to better align training with autoregressive inference.”

“…generation technology, video world models have demonstrated tremendous influence and promising prospects in domains including autonomous driving (Hu et al., 2023; Ren et al., 2025), navigation (Bar et al., 2024), robotic manipulation (Wu et al., 2024; Azzolini et al., 2025; Liao et al., 2025;…”

“The first introduces structured conditions over scene layouts, weather, agents, and ego trajectories to enable realistic and controllable generation [13, 18, 36, 51].”

“Specifically, they usually either lack cross-view consistency or rely on conditions like 3D bounding boxes and LiDAR, which are absent in heterogeneous videos (Gao et al., 2023; Wang et al., 2024a; Zhao et al., 2025; Ren et al., 2025a; Hu et al., 2023; Russell et al., 2025; Wang et al., 2024b).”

“Generative world models [26, 57, 59, 61, 71] have emerged as promising tools for scalable simulation in autonomous driving, offering a pathway to synthe-size safety-critical corner cases that are rare in the real world [1, 63].”

“In autonomous driving, generative world models such as DriveDreamer [34] and GAIA-1 [17] focus on modeling future external driving scenes.”

“Recent progress has demonstrated strong performance in environments with relatively simple action spaces, such as navigation with discrete actions (e.g., forward, backward, and turning commands) and low-DoF robotic manipulation [4, 5, 6].”

“Video-generation-based world models are widely used for scene generation, data augmentation, and closed-loop simulation [5], [6], [7], [8], [9], [10], [20], [21].”

“GAIA-1 exemplifies an explicit generative instantiation of internal world modeling for autonomous driving [24].”

“In general, E2E autonomous driving systems operate in continuous latent spaces [26, 51, 97], where representations are highly entangled, providing strong continuity and interpolation capability but lacking explicit compositional semantics [56].”

“Contemporary world models often condition their predictions on explicit action labels [15, 19, 20], which limits generalization and transfer between embodiments when actions are agent-specific [21, 22].”

“In the same spirit, earlier efforts cast driving world modeling as controllable sequence generation and enable planning over multiple futures [13, 33], and more recent end-to-end frameworks incorporate intention-aware latent world representations for trajectory evaluation and selection [42].”

“The first focuses on image-domain generators, driven by latent-diffusion or autoregressive mechanisms [2, 20–23], which aim to synthesize multi-view future scenes directly in pixel space.”

“These models typically build on modern multimodal foundation models [34, 35, 36, 37, 38, 39, 40, 41, 42, 43], while a parallel line of work uses generative world models [44, 45, 46] to capture multi-view scene dynamics.”

“World models support planning and control from compact latent states [1, 2, 3], and generative video models have pushed this idea toward interactive, visually rich world simulation [4, 5, 6].”

“…[1,13,22,42,53,65] trained on large-scale datasets with structured inputs ( e.g ., actions and camera movements) have emerged as promising world simulators [32, 51, 52], with applications in autonomous driving [10, 15, 36], embodied intelligence [7, 61], and interactive game engines [12,29,41,57].”

“Early efforts like GAIA-1 [5] generated single-view videos.”

“Although some recent attempts like DriveDreamer-2 [29] combine large language models (LLMs) [30] for more diverse driving video generation, maintaining the spatiotemporal consistency for multiview video generation still remains challenging, especially for long driving video generation.”

“…interpretability and interoperability to the human driver, several works use controllable video generation technology to build driving world models (Hu et al. 2023; Wang et al. 2023b,c; Gao et al. 2024b), and video generators also achieved promising results as world simulators (OpenAI 2024; Kong…”

“Previous research [1, 13, 15, 24, 36, 49, 50, 70–72, 78, 92, 96– 98, 102] has mainly leveraged world models to generate controllable future driving trajectories (e.g., RGB images and 3D/4D representations) conditioned on specific actions and scene descriptions.”

“…to improve the world model in its ability to predict the next state and reward. tion as world models—systems that predict future observations, dynamics, and outcomes conditioned on agentic actions in domains such as autonomous driving (Hu et al., 2023), robotics (Yang et al., 2023; Quevedo et al.)”

“In the following, we shed light on two generative applications: In Section 2 we take a look under the surface of a large language model (LLM) application, and in Section 4 we do the same for a video generation AI such as GAIA-1 [4].”

“Existing world models predict future states either directly in the observable space (Brooks et al. 2024; Bruce et al. 2024; Hu et al. 2023; Agarwal et al. 2025) or through latent transition dynamics (Hafner et al. 2023; Assran et al. 2023).”

“In interactive settings, WFMs increasingly accept action or multimodal prompts, e.g., GAIA-1 [14] for autonomous driving and DeepMind’s GENIE [5] for action-conditioned virtual environments.”

“Recent approaches tackle this issue by: MonoWAD (Oh et al. 2025) adopts weather-adaptive diffusion models to revert weather conditions to ideal situations, whereas MonoTTA (Lin et al. 2025b) improves model robustness via online test-time adaptation.”

“With generative AI projected to influence more than 40 percent of in-cabin digital features by 2030 (SAE EDGE Report, 2024), early integration of verifiable AI-safety practices and lifecycle management frameworks will determine the technology’s long-term success and regulatory acceptance.”

“As such, trajectory con-trol [12, 16, 42] or the use of 3D bounding boxes and scene description text [10, 11] have been introduced to guide autonomous driving scene generation.”

“…of the environment to enable efficient policy learning and long-horizon planning, but prior works such as Dreamer and V-JEPA 2 have relied on ground truth reward signals or actions (Ha and Schmidhuber, 2018; Hafner et al., 2019, 2020; Schwarzer et al., 2021; Hu et al., 2023; Assran et al., 2025).”

“Since 2024, the rapid evolution of video diffusion models, such as GAIA-1 [38], DriveDreamer [39], Vista [40], and GenAD [41], has shown great promise in generating photo-realistic driving videos.”

“End-to-end (E2E) models [12], [31], [40] address these limitations by directly mapping sensor data to control commands through unified, fully differentiable architectures trained on extensive driving data.”