GAIA-1: A Generative World Model for Autonomous Driving

“Its training objective is to predict the next image token in the sequence conditioned on all past tokens, using causal masking in the attention matrix of the transformer blocks [35].”

“The discrete autoencoder is a fully convolutional 2D U-Net [31].”

“Because these text sources are imperfect, to improve the alignment between generated futures and the text prompt, we employ classifier-free guidance [45, 46] at inference time.”

“Large language models have shown clear benefits in scaling model size and data [90, 24, 26, 20, 21, 22, 23].”

“The advantage of such a perspective is that world models can benefit from scaling properties of high-capacity sequence model architectures applied to large-scale unsupervised training [26].”

“Following the recent advances in image [36, 37] and video generation [16, 18] we use denoising video diffusion models for the GAIA-1 decoder.”

“Most language models [25, 26, 24, 21, 27, 22] use byte-pair encoding (or equivalent) as a trade-off between character-level and word-level tokenization.”

“The image reconstruction loss is a weighted sum of L 1 , L 2 , perceptual loss L perceptual [32], and GAN loss L GAN [33].”

“Large language models have shown clear benefits in scaling model size and data [90, 24, 26, 20, 21, 22, 23].”

“The models used to fit the power-law ranged from 10,000x to 10x smaller models in terms of parameters (0.65M to 650M), as visualized in Figure 8b. Similarly to [49], the compute was estimated as a function of the parameter count.”

“…robotics (Hafner et al., 2019; Wu et al., 2023; Mendonca et al., 2023; Zhou et al., 2024b) with manipulation-based actions, autonomous driving (Hu et al., 2023; Guan et al., 2024; Wang et al., 2024a,b) with vehicle control actions, and navigation (Koh et al., 2021; Shah et al., 2023; Bar et…”

“In response, world model systems [12, 20, 29, 30, 39, 40, 47, 52, 54] capable of understanding and predicting future environmental states have emerged as a promising solu-tion.”

“Drive-Dreamer [71] and GAIA-1 [72] employ diffusion models to generate real-world driving scenarios.”

“Retrieval is natural in world models when geometry, camera pose, or scene coordinates are available [20, 13, 30, 56, 47, 40, 50, 45].”

“Representative applications span robotics [14, 31, 32, 39, 56, 59], video games [3, 10, 19, 51, 55], autonomous driving [1, 22], and physical simulation [29, 64], suggesting their potential as general-purpose simulators for dynamic visual environments.”

“…AI and autonomous systems (Mirowski et al., 2016; Chaplot et al., 2020; Fu et al., 2022; Sridhar et al., 2024), enabling agents to interpret egocentric observations and sequentially choose actions to reach goals in complex environments (Karnan et al., 2022; Yue et al., 2025; Hu et al., 2023).”

“Domain-specific systems for driving [51–55] and instructional video generation [56–61] demonstrate utility but lack generality.”

“These advances enable video generation models to serve as learnable simulators, powering physics-grounded simulation in applications such as game environments [6, 56, 37, 7, 69, 14, 65], autonomous driving [33, 35, 44, 63, 18, 19], and robotic control [3, 16, 8, 28, 23, 73].”

“Effect of tokenizer design For the discrete model, adding DINO distillation to the image tokenizer, similar to GAIA-I [28], leads to lower FVD, as shown in Table 4.”

“These include 2D videos [6, 11, 25, 26], 3D lidar point clouds [1, 9, 11, 32, 38], and 3D occupancy grids [36, 8, 4, 34, 28, 11, 30, 27].”

“In the context of autonomous driving, several studies have been proposed to predict multiple plausible future trajectories conditioned on various prompts, including weather conditions, surrounding traffic participants, and vehicle actions [25, 42, 64, 19, 63].”

“This hinders their application in open-world scenarios, such as autonomous driving [27, 29] and embodied intelligence [23, 55].”

“For instance, in autonomous driving, a cloud twin provides city-scale priors, an LLM planner defines route-level intents, and a local world model predicts sub-second collision risks, together realizing real-time, context-aware cognition without overloading the communication backbone [193].”

“World models invert this relationship by generating future states that respond to continuous user or agent inputs [1, 12, 14, 25, 35, 44, 54].”

“Spatial understanding is an area of increasing interest in vision-language, vision-language-action, vision-action, and world models [17, 20, 28, 62, 86, 94, 122, 127, 163, 235].”

“In addition, such a real-world grounded simulation would enable urban planning visualization, autonomous driving scenario generation, and location-based exploration [9,17,39].”

“Recent work spans driving [12], robotics [34], and interactive gaming; Genie [4], GameNGen [29], and Oasis [7] generate playable environments in real-time.”

“Among these, BEV-based methods, including Carformer [364], Genad [346], BEV-World [365], and DrivingDi ff usion [366], provide better spatial consistency compared to image-based [360], point cloud-based [367], and occupancy-based methods [368].”

“Together with other advanced neural techniques, NeRF and 3DGS exemplify the rise of learning-based 3D reconstruction methods that overcome many limitations of traditional approaches while o ff ering improved fidelity and e ffi ciency.”

“Recently, some approaches (Gao et al., 2023; Yang et al., 2023b; Swerdlow et al., 2024; Li et al., 2023a; Jia et al., 2023; Hu et al., 2023) attempt to employ Stable Diffusion (Rombach et al., 2022) to generate the synthetic data of driving scenes.”

“Wayve is a software development company that uses AI to pioneer a next-generation approach to self-driving [ 9].”

“0 pioneers a new era of autonomous driving by developing end-to-end AI foundation models, especially the language model LINGO[39, 40] to increase the transparency in end-to-end reasoning and decision-making, as well as the generative world model GAIA-1[18] to generate realistic driving videos from text, action, and vi…”

“For example, GAIA-1 [16] leverages a generative model conditioned on video, text, and action inputs to produce realistic driving videos.”

“Representative works such as GAIA [21], DriveDreamer [22], DrivingWorld [23], and Vista [24] leverage action-conditioned future frame prediction to support planning and simulation.”

“A local world model, such as NIO WorldModel, forecasts sub-second collision risks, and an LLM planner narrates route-level intents—together realising real-time cognition without saturating the link [44], [170].”

“…to the actions it can take, it helps planning and policy learning in control tasks, e.g., autonomous driving (Hu et al., 2022; Pan et al., 2022; Hu et al., 2023; Zhang et al., 2023; Wang et al., 2023), gaming (Schrittwieser et al., 2020; Hafner et al., 2019; 2020; 2023), and robotics (Black et…”

“…work explicitly models future scene evolution in the observation space, including multi-view driving video generation and world simulation methods [24–32], which emphasize high-fidelity future rollout, controllable scene evolution, long-horizon generation, and reliable simulation for downstream…”

“Generative world models are redefining how AI systems learn, simulate, and reason about dynamic environments [7, 22, 41].”

“Early efforts [28, 53, 70] were limited to short, single-view videos, while recent methods have achieved the generation of long sequences and high-resolution driving videos [15, 16, 21, 23, 40, 67].”

“There have been several adaptations of VLMs in autonomous vehicles (AVs) [8], [9], propelled by the availability of large-scale datasets developed for autonomous driving in structured road networks governed by clear traffic rules [9], [10].”

“This improved spatiotemporal representation significantly benefits complex tasks such as object grasping and dexterous manipulation. cations in autonomous driving [15, 16, 39] and robotic manipulation [24, 32, 42].”

“Recent advances have demonstrated their effectiveness in downstream applications, such as robotics [2, 6–10] and autonomous driving [11–14].”

“…et al., 2022) and large-scale video datasets, contemporary video world models achieve substantial improvements in both fidelity and diversity of synthesized visual environments through training action-conditioned video generation models (Hu et al., 2023; Guo et al., 2025b; Bar et al., 2025).”

“…al., 2023; Feng et al., 2023; Mattes et al., 2023; Sun et al., 2024; Georgiev et al., 2024), Generative Mdoel (Chen et al., 2022; Wang et al., 2023; Hu et al., 2023; Zhang et al., 2023; Rigter et al., 2023; Alonso et al., 2024; Ding et al., 2024; Bruce et al., 2024; Du et al., 2024; Zhen et al.,…”

“One line of emerging works is derived from the general family of transformer-based language models [17, 21, 33, 39], where they leverage visual tokenizers and text tokenizers to obtain the discrete token of each video frame.”

“Pixel-space methods cast prediction as conditional video generation, from latent-action world models [52, 32] to recent large-scale video generators [11, 1, 67]; these models are general but spend substantial capacity rendering appearance, lighting, and camera motion.”

“Generative world models can be broadly categorized as discrete [10, 34] or continuous [8, 12, 71].”

“Although these methods are capable of generating driving scenarios, driving world models need to simulate reasonable scenarios based on motion conditions [17, 19, 22, 51].”

“A prominent line of research leverages this capability to generate realistic future scenarios in video form [6, 8, 10, 20, 36, 37], supporting data augmentation and simulation-based validation.”

“Recent advances have led to the development of numerous world models of RGB videos that have demonstrated impressive performance in applications such as autonomous driving [36, 15, 42], robotic navigation [4], and other embodied tasks [7, 20, 1, 3].”

“Other models [6, 7, 17, 28], like Dreamforge, produce videos based on predefined traffic dynamics, which restricts interactivity and user-defined scene generation.”

“It leverages historical data to forecast future events, with far-reaching implications for diverse fields such as meteo-rology [3, 6, 23, 27], robotics [10, 13], and autonomous vehicles [21].”

“GAIA-1 [22] adopts discrete sequence modeling for controllable driving video synthesis, while GAIA-2 [47] advances to controllable multi-camera latent diffusion with broader geographic diversity. and tighter alignment to structured priors, an industrial path toward world models and control priors.”

“Building upon this, GAIA-1 [16] and GAIA-2 [24] introduced large-scale multimodal world models capable of performing tool-augmented reasoning and web-based information synthesis, moving beyond static simulation toward interactive reasoning .”

“GAIA-1 [247] represents an interactive autonomous driving world model that casts world modeling as an unsupervised sequence modeling problem, mapping multimodal inputs into discrete tokens and predicting the next token in the sequence to simulate future scenarios.”

“For instance, GAIA-1 (Hu et al., 2023) synthesizes realistic driving scenarios, while DriveDreamer (Wang et al., 2024e) learns policies from real-world data.”

“Most existing world modeling work Hu et al. (2023) has focused on autonomous driving, a domain where generating realistic and diverse data—particularly for rare corner cases—is both critical and difficult.”

“Recent research has increasingly focused on predicting raw RGB values for future video frames [18, 24, 25, 28, 30, 45, 64, 80, 83, 84].”

“Prior art, such as text-based scene generation [15], often lacks practicality for tasks like inserting or swapping objects within a scene.”

“The recent progress of LLMs for AV [54], [55], [16], [56], [15], [57], [58] further enhances the significance of semantic signals, fostering research on integration of LLMs with existing AV modules [59].”

“More recent approaches jointly model environment evolution and driving policies [6, 18, 28–30, 32–35, 60–65].”

“In the realm of decision-making and planning, world models such as Dreamer [Hafner et al., 2024] and GAIA-1 [Hu et al., 2023] demonstrate that predicting future latent states, rather than raw inputs, is crucial for robust long-horizon reasoning.”

“Existing solutions often produce lower-resolution outputs with reduced temporal consistency, are typically restricted to single modalities ( e.g. , camera [15, 12, 26] or LiDAR [21, 70, 65, 1]), and require substantial computational resources.”

“Generative world models, such as video generation models (VGMs) (Agarwal et al., 2025b; OpenAI, 2024; Wan et al., 2025; Hu et al., 2023; Blattmann et al., 2023; Yang et al., 2025b; Guo* et al., 2023; 2025; Chen* et al., 2025) that are trained on internet-scale video data, encode strong priors over…”

“Representative examples include DriveDreamer [27], DriveDreamer-2 [28], Panacea [29], VISTA [30], GAIA-1 [31], GAIA-2 [32], MagicDrive-V2 [33], MiLA [34], CoGen [35], Genesis [36], and OmniNWM [37].”

“Examples include GAIA-1 [15], DriveDreamer [44, 60], Vista [13] for controllable rollouts, DriveDreamer4D [59] for 4D scene generation, UniDrive-WM, UniWM, and DriveVA [10,31,49] for unified planning and generation, and CausalDrive [52] for interactive causal simulation.”

“With advances in video generation and the introduction of the Joint Embedding Predictive Architecture by Assran et al. [2], the scope and applications of world models have broadened considerably [24, 25, 44, 46, 93].”

“Recently, two major paradigms have gained attention in end-to-end autonomous driving: Vision-Language-Action (VLA) models [3, 16, 20, 30, 33, 42, 76, 84, 86] and World Models [22, 28, 41, 44, 59–61].”

“Given the safety risks of conducting online RL on real vehicles [24] and the high computational cost of world models [1, 19], we adopt a non-reactive pseudo-closed-loop simulation [14] based on real-world datasets.”

“Foundation models like UniSim [93] and GAIA-1 [38] leverage internet-scale data to learn high-level physics for zero-shot generalization.”

“Occupancy world model, as a specialized type of world model, has gained significant attention for its expressiveness of the 3D geometry.”

“…has also been increasingly explored in other domains such as game engines [Menapace et al., 2021, Valevski et al., 2024, Decart et al., 2024], autonomous driving [Hu et al., 2023, Wang et al., 2023a, Gao et al., 2024], robot manipulation [Yang et al., 2023, Wu et al., 2022, Bar et al., 2024].”

“[28] LLaMA-7b [193] Custom”

“GAIA-1 [28] maps multi-modal driving signals into discrete tokens, and processes them using an autoregressive transformer.”

“World Models [11] have emerged in recent years as a promising paradigm to enable a wide range of downstream tasks such as video prediction [50, 13], physical reasoning [7], and model-based RL [12].”

“As one of the earliest proposed world models, GAIA-1 [5] utilizes Transformers as the core component for world modeling.”

“The diffusion-based batch decoder addresses temporal consistency issues associated with single-step decoders [17] and mitigates compression losses by conditioning on prior rasterized L-OGMs, at the cost of real-time feasibility.”

“Recently, GAIA-1 [49] and Sora [11] leverage the strengths of both diffusion models and large language models to achieve more creative, general, and scalable video synthesis.”

“In the context of autonomous driving, world models like GAIA-1 [203] and DriveDreamer [204] also function as advanced scene interpreters, understanding current driving scenarios from text and video to generate realistic predictive scenarios for subsequent decision-making.”

“Recently, inspired by the success of video generation [25], several cutting-edge approaches [8, 17, 35, 38] aim to build driving world models in an explicit way.”

“They have seen a recent resurgence in variants trained large-scale with next-token prediction [33, 82].”

“GAIA-1 [23], developed by Wayve, is a generative AI model capable of producing realistic driving videos using video, text, and action inputs.”

“A generative world model for autonomy was introduced by Hu et al. [8], capable of creating realistic driving scenarios from video, text, and action inputs, offering detailed control over vehicle behavior and scene features.”

“In practice, GAIA-1 (Hu et al., 2023), UniSim (Yang et al., 2024), and Genie (Bruce et al., 2024) show that the learned world model can enable the control policy to generalize to the real world when trained purely in simulation and bridge the sim-to-real gap.”

“…[54], VIMA-Bench [92], VirtualHome [178], AI2-THOR [102], ProcTHOR [42], ThreeDWorld [53] Generative model, world model Evaluate the embodied AI models’ abilities with interactive models representing the environments GAIA-1 [77], UniSim [238], LWM [131], Genesis [9], RoboGen [218] Table 3.”

“…into interactive world models capable of simulating complex visual environments [Hong et al. 2025; Team et al. 2026], offering significant potential for long-horizon applications in autonomous driving [Gao et al. 2024; Hu et al. 2023] and embodied intelligence [Wu et al. 2026; Yang et al. 2023].”

“…moves toward world-level scene simulation (Sun et al., 2025; Huang et al., 2025a; Chen et al., 2025; HY-World et al., 2026; Dai et al., 2025; Wei et al., 2026), 3D consistency becomes essential for applications such as autonomous driving (Hu et al., 2023; Wang et al., 2024) and embodied AI.”

“GAIA-1 [104] and DriveDreamer [6] unify different tasks like scene perception and planning within a shared generative framework.”

“These existing approaches face significant computational challenges in real-time settings, especially with large models like diffusion-based video generation models [5, 16, 35, 73].”

“Currently, most driving world models Hu et al. (2023a); Russell et al. (2025) are generative world models that explicitly generate pixel-level details for predicted future frames.”

“These approaches differ in their sensor assumptions: multi-view methods [18, 34, 44] exploit surround-camera geometry for 3D-consistent scene generation, whereas single-view models [4, 10, 16, 28, 42, 48] operate purely from monocular ego video, offering a critical scalability advantage, as…”

“GAIA-1 [20] conditions on video and action inputs to generate full scenarios.”

“Using this model, large-scale learning from diverse datasets could create AI navigation systems that can understand, predict, and adapt to real-world complexities [22].”