GAIA-1: A Generative World Model for Autonomous Driving

“Its training objective is to predict the next image token in the sequence conditioned on all past tokens, using causal masking in the attention matrix of the transformer blocks [35].”

“The discrete autoencoder is a fully convolutional 2D U-Net [31].”

“Because these text sources are imperfect, to improve the alignment between generated futures and the text prompt, we employ classifier-free guidance [45, 46] at inference time.”

“Large language models have shown clear benefits in scaling model size and data [90, 24, 26, 20, 21, 22, 23].”

“The advantage of such a perspective is that world models can benefit from scaling properties of high-capacity sequence model architectures applied to large-scale unsupervised training [26].”

“Following the recent advances in image [36, 37] and video generation [16, 18] we use denoising video diffusion models for the GAIA-1 decoder.”

“Most language models [25, 26, 24, 21, 27, 22] use byte-pair encoding (or equivalent) as a trade-off between character-level and word-level tokenization.”

“The image reconstruction loss is a weighted sum of L 1 , L 2 , perceptual loss L perceptual [32], and GAN loss L GAN [33].”

“Large language models have shown clear benefits in scaling model size and data [90, 24, 26, 20, 21, 22, 23].”

“The models used to fit the power-law ranged from 10,000x to 10x smaller models in terms of parameters (0.65M to 650M), as visualized in Figure 8b. Similarly to [49], the compute was estimated as a function of the parameter count.”

“Language conditioning lets them learn from web-scale video (Brooks et al., 2024; Wan et al., 2025), and the resulting models serve as interactive simulators for games (Bruce et al., 2024), driving (Hu et al., 2023), and robotics (Yang et al., 2024; Agarwal et al., 2025; Ali et al., 2025).”

“Recent advances in diffusion have increasingly pushed generative video models toward the paradigm of video world modeling [4, 5, 21], where the goal is to capture not only visual appearance but also the underlying dynamics and interactions of the physical world.”

“…pixel-level fidelity: Genie [23] learns controllable interactive environments from unlabeled video, GameNGen [42] and DIAMOND [43] simulate game engines with diffusion models, GAIA-1 [44] generates driving scenarios, and Cosmos [28] provides a large-scale foundation world model for physical AI.”

“With zero-based frame indices, the four resulting reconstruction windows are [0 , 3] , [4 , 7] , [8 , 11] , and [12 , 15] .”

“A world model, as the term is used throughout this article, is a neural network that learns a compressed dynamic representation of the driving environment and can synthesize future sensor readings on demand [16], [18].”

“Advanced by large-scale egocentric datasets such as Ego4D [20] and Ego-Exo4D [21], the field has expanded from passive video understanding to agent-centric embodied world models (e.g., GAIA-1 [22], UniSim [23]).”

“Methods such as MagicDrive, Panacea, Drive-Dreamer, and VISTA condition generation on semantic lay-outs, depth, camera geometry, trajectories, or 3D bounding boxes [2, 9–12, 18, 21, 32, 34–36, 41, 43, 45].”

“…video modeling have pushed this paradigm into embodied manipulation, where a model is expected to predict action-conditioned futures for robotic interaction (Hu et al., 2023; Bruce et al., 2024; Seo et al., 2023; Zhou et al., 2024; Zhu et al., 2025; Guo et al., 2025; NVIDIA et al., 2025).”

“Video world models learn visual dynamics from video and simulate scene evolution through generated observations [1, 2, 7, 8, 10, 22, 23, 25, 30, 31, 37].”

“Visual world models predict the future in pixel space (Hu et al., 2023; Wang et al., 2024b; Gao et al., 2024; NVIDIA et al., 2025), which can introduce substantial representational redundancy and make geometric consistency difficult to maintain.”

“Early work cloned human driving demonstrations end-to-end [8, 9], and the paradigm has since been scaled to video-and language-conditioned foundation models trained on fleets of real-world driving data [10].”

“…et al., 2024, Parker-Holder et al., 2024], robotics simulators [Yang et al., 2024, Zhu et al., 2025, Quevedo et al., 2025], driving environments [Hu et al., 2023, Russell et al., 2025, Wang et al., 2023a, Gao et al., 2024, Hassan et al., 2025], games from user input [Valevski et al., 2025, Che…”

“…for interactive simulation across embod-ied AI (NVIDIA et al., 2025; Yang et al., 2024; Jang et al., 2025; Gao et al., 2026a; Huang et al., 2026a), games (Bruce et al., 2024; Alonso et al., 2024; Kanervisto et al., 2025), autonomous driving (Hu et al., 2023; Gao et al., 2024b; Yang et al., 2025).”

“Applications span games [21], robotics [34], autonomous driving [14] and interactive environments [3].”

“Such consistency is essential for interactive world modeling [5, 6], robotic planning [7, 8], and autonomous driving simulation [9, 10], where generated videos must represent temporally coherent environments.”

“One line of work predicts future observations with large-scale video generation models [25, 26, 27, 28], producing visually realistic videos from text, image, or action-conditioned inputs.”

“A separate line learns interactive dynamics directly from video, including Genie [40], UniSim [41], GAIA-1 [42], WorldDreamer [43], and Ctrl-World [44].”

“Generative world models such as GAIA-1 (Hu et al., 2023), Navigation World Models (Bar et al., 2025), and Cos-mos (NVIDIA, 2025; 2026) predict future visual observations from current scenes and motion or action conditions.”

“Reliable autonomous driving requires three core capabilities: scene understanding, future prediction, and trajectory planning (Hu et al., 2023a;b; Wang et al., 2024a).”

“…TrafficBots for data-driven traffic simulation and motion prediction [14], GAIA-1 and GAIA-2 for controllable generative driving world models [16], [17], Copilot4D for tokenized discrete-diffusion world modeling [18], BEVWorld for multimodal bird’s-eye-view latent simulation [19], and…”

“Recent video-based world models have progressed from action-conditioned future generation (Hu et al., 2023) toward continuously controllable environments that support interactive rollout through user actions or camera trajectories (Bruce et al., 2024; Sun et al., 2026; HunyuanWorld, 2025; Chen et…”

“Such consistency is essential when video generation models serve as world simulators for robotics (Shan et al., 2026; Zhang et al., 2026), autonomous driving (Hu et al., 2023; Wang et al., 2024; Gao et al., 2024), and interactive games (Bruce et al., 2024; Sun et al., 2025; Team et al., 2026).”

“Action-conditioned video generation offers a complementary, pixel-space route to the same capability [43, 53].”

“Approaches include learning dynamics [122, 123], predictive representations [124], or generative models of environments [125–128].”

“Large video generators such as Cos-mos [6], [11] and Wan [12], and driving-specific models such as GAIA-1 [13] and Vista [14], are used as world models by rolling out future frames, from which geometry can then be reconstructed.”

“GAIA-1) synthesises driving video [50], and software engineering has studied learned or lower-fidelity simulators as test oracles [51, 52].”

“GAIA-1 generates driving scenes conditioned on video, text, and actions, while DriveDreamer incorporates structured traffic constraints into diffusion modeling for video generation and action prediction [18, 19].”

“Some existing works focus on generative driving world models [8, 13, 14, 24, 40,43], which leverage the current frame or historical observations and introduce conditions such as trajectories, velocities, or text to improve controllability of Fig.”

“This capability has been widely adopted in autonomous driving simulation (Gao et al., 2024; Hu et al., 2023), robotic manipulation (Wu et al., 2024), and virtual content creation (Valevski et al., 2025; Wan et al., 2025).”

“Wayve has released a research paper titled "GAIA-1: A Generative World Model for Autonomous Driving" [3] that explores the challenges of building autonomous driving systems, while focusing primarily on the problem of effectively predicting outcomes in response to the vehicle’s actions in an…”

“Following LingBot-World, the video model is optimized using a flow-matching objective.”

“World models (WMs) are used across environment simulation (Ha & Schmidhuber, 2018; Alonso et al., 2024), reinforcement learning and planning (Hafner et al., 2019; Schrittwieser et al., 2020; Hafner et al., 2025), and synthetic data generation (Hu et al., 2023; Jang et al., 2025).”

“End-to-end driving systems learn such decisions directly from sensor observations [42; 21; 24; 34], driving world models generate future scenes [55; 19; 56; 16], increasingly across views and with geometric conditioning [15; 26], and recent world-action models predict future observations and ego…”

“World models have also gained attention in E2EAD, mainly for aligning predicted and ground-truth future representations (Li et al., 2024a; Li & Cui, 2024) or generating future driving videos (Zhang et al., 2025; Hu et al., 2023a).”

“Early studies (Hu et al., 2023; Yang et al., 2024) trained video generation models on driving videos to simulate driving scenarios.”

“Later pixel world models ask that kernel to render: diffusion world models (Alonso et al., 2024), interactive environments (Bruce et al., 2024), and driving (Hu et al., 2023).”

“Early representative systems such as GAIA-1 [45] formulated driving video generation as autoregressive prediction over discrete tokens conditioned on text and ego actions, and the subsequent model GAIA-2 [46] adopted a latent diffusion formulation with structured control over ego motion,…”

“Video world models predict future frames from actions (Hu et al., 2023; Yang et al., 2024; NVIDIA et al., 2025b) or from latent actions inferred from video (Bruce et al., 2024), while VPP (Hu et al., 2025) and UVA (Li et al., 2025) build robot policies on video prediction models.”

“Furthermore, the growing use of world models [6], [7], [8], [9] enables agents to build an internal representation of their environment and simulate future outcomes, allowing for proactive, long-term planning.”

“World models (WMs) can enrich AD solutions with this critical capability by rolling the scene forward and exposing predicted futures to the planner [5, 6].”

“Persistent memory, fully causal conditioning and policy-coupled dynamics remain directions for extending the observation model.”

“Generative driving world models predict future visual observations [8]–[11], whereas latent world models reason in compact feature spaces that integrate efficiently with downstream planning.”

“World models [3,11,24] provide agents with an “imagination space” for planning and decision-making, and have shown great potential in applications such as autonomous driving [16,40] and robotics [9,44].”

“The same recipe has scaled to generative interactive environments [Bruce et al., 2024], video models used as general-purpose simulators [Brooks et al., 2024, Assran et al., 2025], and driving [Hu et al., 2023].”

“In sequential decision making, generalist agents and vision-language-action (VLA) models further push toward large-scale multi-task control [1, 2, 14, 17, 34, 49].”

“Prior driving world models address autoregressive prediction, controllable multi-view diffusion, and long-horizon extrapolation [10, 13, 19, 24, 35, 36, 38, 43, 53].”

“Driving applications include visual future generation [24]–[26], occupancy forecasting [27], 4D scene-representation pretraining [28], and joint future image–depth prediction [29].”

“The same supply-chain concern becomes especially relevant for foundation-scale world models such as Ge-nie [6], GAIA-1 [7], and V-JEPA [8], where retraining from scratch may be infeasible for many users.”

“End-to-end systems commonly construct bird’s-eye-view features and occupancy predictions [3], [4], world models predict how the scene may evolve [5]–[7], and Vision-Language-Action models introduce semantic and latent reasoning [8]–[12].”

“GAIA-1 combines video, text, and action inputs to generate controllable driving scenarios [57], while DriveDreamer conditions future-video generation on structured traffic information and driving actions [58].”

“Generative world models such as GAIA-1 [8], DriveDreamer [4], and Vista [7] synthesize future camera observations, providing realistic and controllable simulations for data generation and policy evaluation.”

“DriveDreamer learns controllable real-world driving-video generation, GAIA-1 models video, text, and actions autoregressively, and Vista emphasizes high-fidelity generation with versatile control [30, 31, 32].”

“These approaches have matched or surpassed the state of the art across diverse single-agent domains, including robotics [41], autonomous driving [11, 18], and nuclear fusion control [7].”

“The choices impose different trade-offs: early fusion preserves low-level complementarity but requires tight alignment, shared spatial fusion offers a common geometric state but remains calibration-sensitive, and late fusion is modular but delays cross-modal correction.”

“In physical AI systems such as autonomous driving and embodied intelligence, large language models (LLMs) are increasingly used as foundation models to support perception, reasoning, decision-making, and the prediction of action-conditioned environment dynamics [1–5].”

“Early works such as GAIA-1 [48] focused on neural world models for driving scenes, while Ge-nie [15] inferred latent actions without labels from internet videos and generated interactive 2D game worlds.”

“GAIA-1 (Hu et al., 2023a) and Vista (Gao et al., 2024b) scale generative driving world models with high fidelity and versatile action control.”

“Using this model, large-scale learning from diverse datasets could create AI-based navigation systems that can understand, predict, and adapt to real-world complexities [29].”

“GAIA-1 [12] introduces an autoregressive framework for driving video generation.”

“Conversely, generative 2D image synthesis—particularly latent diffusion models—has recently achieved impressive photorealism for scene synthesis and controllable simulation [15,25,30,37,41,44], but these methods are fundamentally 2D.”

“…• interactive game-world simulation [25, 35, 50, 54, 154, 192]; • robotics dynamics prediction [6, 57, 59–61, 76, 173]; • autonomous driving trajectory prediction [15, 47, 67, 97, 114, 118, 133, 166, 179, 181, 185, 204, 211]; • model-based reinforcement learning [4, 57–59, 76, 79, 116, 136, 205].”

“Representative systems include generative world models such as GAIA-1 [10], closed-loop generative simulation platforms such as DriveArena [11], and reliability-oriented driving world models such as ReSim [12].”

“Genie, GAIA-1/2, and Cosmos have demonstrated the foundational capabilities of video world models (Bruce et al. 2024; Hu et al. 2023; Russell et al. 2025; NVIDIA et al. 2025).”

“Video-diffusion driving world models generate realistic sensor streams and have become increasingly popular for simulation and planning [1, 2, 3].”

“GAIA-1 [17] performs action-conditioned world-model video generation for autonomous driving, and Genie [4] learns a latent action space enabling unsupervised and action-controllable interactive environments.”

“Recent work has explored generative world models for AV [Hu et al., 2023, Kim et al., 2021, Wayve, 2023].”

“In the autonomous driving domain, generative models have recently been used to synthesize realistic driving scenarios for evaluation and failure mode discovery [17, 27].”

“Generating videos with precise object control is essential for applications such as simulation and synthetic data generation for autonomous driving [13, 32, 11], robot learning [6, 19], and professional video editing [35, 31], where multiple objects must follow prescribed motion patterns while…”

“The same principle has been applied to physical robot learning and autonomous driving, where learned roll-outs support action-conditioned prediction in embodied or driving environments [11, 36].”

“Some studies, such as GAIA-1 Hu et al. (2023) and GUMP Hu et al. (2024), propose generative world models for scenario generation of autonomous driving, where the dynamic information is encoded into a sequence of tokens via an auto-regressive Transformer.”

“Beyond BEV-anchored models, GAIA-1 [19] generates future driving scenes autoregressively in a tokenized video latent space, and DriveWM [20] forecasts multiview images and uses the predicted futures to guide planning.”

“Some successful world model-based systems [5], [6], which learn structured representations of dynamic environments, enable the generation of long, temporally consistent video sequences through autoregressive next-token prediction.”

“WorldDreamer and Genie [25], [26] formulate world modeling as a state-action sequence modeling task.”

“Generative world models such as GAIA-1 [10] and DriveDreamer [17] enable scalable simulation, but tend to model reciprocal, cooperative agent behaviors.”

“In video synthesis, GAIA-1 [14] and DriveArena [37] pioneer autoregressive modeling, while Epona [38] and MagicDrive [6] introduce DiT architectures and BEV constraints to improve temporal consistency.”

“Latent World Models (Ha and Schmidhuber, 2018; Hafner et al., 2023; Hu et al., 2023): These serve as substrate beneath the semantic layer.”

“Frameworks like GAIA-1 [46], MILE [47], and recent generative world models[48, 49] utilize reconstruction-guided Vector Quantization[50] to discretize scenes for autoregressive prediction.”

“Direct end-to-end video generation models [14, 15, 17, 37, 48, 64] can produce visually compelling sequences from coarse prompts, yet their implicit mappings often result in inconsistent geometry across views and time, making fine-grained control difficult.”

“Generative world models, such as GAIA-1 [20] and DriveDreamer [21], can synthesize future driving scenarios for training and evaluation.”

“One line of research explores high-fidelity scene generation across various perceptual modalities, including images [17, 59, 13, 15] and 3D representations [53, 55, 60], in order to construct the realistic simulator.”

“One branch generates future scenes or occupancy states, including GAIA-1 [12], Copilot4D [13], DriveDreamer [14], DriveDreamer-2 [15], GenAD [16], and Driving in the Occu-pancy World [17].”

“This supports predictive look-ahead, downstream evaluation, and planning beyond purely reactive control policies [18]–[20].”

“Existing synthetic data methods can be divided into two categories: generation-based approaches [19, 15, 33, 10, 46, 14, 45, 13] and reconstruction-based approaches [30, 16, 41].”

“This capability underpins diverse applications including virtual reality, autonomous driving simulation, and cinematic production (Mei et al. 2022; Hu et al. 2023; Zuo et al. 2024), where precise camera control is essential for creating immersive visual experiences.”

“In highly dynamic scenarios such as autonomous driving, generative world models such as GAIA-1 use video, text, and action inputs to model complex road environments, showing the potential of world models in future-scene generation, behavior prediction, and controllable simulation [30].”

“Subsequently, large-scale world models with implications for being deployed in real-world physical environments include such vector quantization approaches (Hu et al., 2023; 2024).”

“…models generate future states conditioned on actions, language instructions, or multimodal observations (Yang et al., 2023; Bruce et al., 2024; Hu et al., 2023), especially in autonomous driving and robotics, where world models serve both as implicit scene-understanding modules and as…”

“In the field of autonomous driving, generated content primarily focuses on realistic visual imagery [13, 14, 17, 19, 48, 74], road topology in the BEV space [9, 21, 50], and scene-level occupancy representations [61, 68, 76] that capture the spatial layout and dynamic evolution of surrounding agents…”