GAIA-1: A Generative World Model for Autonomous Driving

“Its training objective is to predict the next image token in the sequence conditioned on all past tokens, using causal masking in the attention matrix of the transformer blocks [35].”

“The discrete autoencoder is a fully convolutional 2D U-Net [31].”

“Because these text sources are imperfect, to improve the alignment between generated futures and the text prompt, we employ classifier-free guidance [45, 46] at inference time.”

“Large language models have shown clear benefits in scaling model size and data [90, 24, 26, 20, 21, 22, 23].”

“The advantage of such a perspective is that world models can benefit from scaling properties of high-capacity sequence model architectures applied to large-scale unsupervised training [26].”

“Following the recent advances in image [36, 37] and video generation [16, 18] we use denoising video diffusion models for the GAIA-1 decoder.”

“Most language models [25, 26, 24, 21, 27, 22] use byte-pair encoding (or equivalent) as a trade-off between character-level and word-level tokenization.”

“The image reconstruction loss is a weighted sum of L 1 , L 2 , perceptual loss L perceptual [32], and GAN loss L GAN [33].”

“Large language models have shown clear benefits in scaling model size and data [90, 24, 26, 20, 21, 22, 23].”

“The models used to fit the power-law ranged from 10,000x to 10x smaller models in terms of parameters (0.65M to 650M), as visualized in Figure 8b. Similarly to [49], the compute was estimated as a function of the parameter count.”

“Recent advances in generative models such as diffusion models have provided a new way of modeling the world – learning a video generator that can predict the future frames conditioned on action commands (e.g., Yang et al., 2023; Hu et al., 2023).”

“These methods include generative approaches like diffusion-based models [8, 27, 34, 38, 39, 99], autoregressive models [40, 42, 52, 86], and masked video generation [95, 96].”

“GAIA-1 [11] and ADriver-I [15] integrate large language models into video generation; a concurrent work DriveDreamer-2 [47] proposes a traffic simulation pipeline employing only text prompts as input, which can be utilized to generate diverse traffic conditions for driving video generation,…”

“Another important line of work formulates world models autoregressively by discretizing visual inputs into sequences of tokens [45, 25, 69, 1].”

“Notably, although simulated data has been adopted for world models in synthetic environments [2, 10, 36, 37], such a source is overlooked in driving world models that operate in real scenarios [15, 14, 16, 38].”

“…generative world-model view more explicit: GAIA-1 casts driving world modeling as multimodal sequence modeling over video, text, and action tokens (Hu et al., 2023), and DriveDreamer uses diffusion-based modeling with structural constraints to capture complex traffic evolution from real driving…”

“Some efforts has been put into the area of generating the driving scenarios using diffusion models [Li et al., 2023, Gao et al., 2023, Wang et al., 2023a, Hu et al., 2023, Zhong et al., 2023].”

“GAIA-1 [69] as a generative world model, seamlessly integrates video, action, and text inputs, enabling fine-grained control and comprehensive understanding of complex environments.”

“Recent work further demonstrates that task-agnostic generative world models trained on large-scale, real-world video can produce realistic and diverse simulations of physical environments [22, 57, 8, 5], highlighting the potential of scaling predictive modeling beyond task-specific datasets.”

“While traditional metrics like average displacement error (ADE) and final displacement error (FDE) have been widely used [20, 22, 28], recent research [34] has highlighted their limitations in comprehensively assessing autonomous driving systems.”

“State-of-the-art approaches often leverage generative models to build these world models (Jia et al., 2023; Hu et al., 2023; Yang et al., 2023; Bruce et al., 2024).”

“Efforts have been made to overcome the scarcity of driving scenario datasets using LLMs. GAIA-1 [12], for example, generates data on traf-fic scenarios, environment elements, and potential risks by incorporating video, text, and action inputs.”

“IV-A, followed by the evaluation of sensor fusion strategies in Sec.”

“Autoregressive models such as GAIA-1/2 [7, 49] unify frames, text, and actions into a single sequence for multi-condition control and long-horizon prediction, while InfinityDrive [50] and DrivingWorld [51] extend to longer and continuous videos.”

“…like autonomous driving, generative models are increasingly employed to simulate complex traffic scenarios, as evidenced by approaches such as (Gao et al., 2023, 2024; Hu et al., 2023; Ni et al., 2024, 2025; Ren et al., 2025; Russell et al., 2025; Wang et al., 2024; Zhao et al., 2024, 2025,).”

“Contemporary research is increasingly validating the capabilities of video generation in this realm, including applications in autonomous driving [25,58,60,71, 78,79], autonomous agents [5,10,18,19,64,72,82], and even in general world [4, 13].”

“Video world models have been applied to numerous domains including robotics [44, 57, 39, 38, 60, 18], video games [14, 46, 53, 3, 26, 56], self-driving [1, 29, 32], and physical simulation [37, 64].”

“Some works [21, 28, 78] utilize diffusion models to generate future driving scenes based on historical information, but they lack the ability to control scenes through layout.”

“In autonomous driving, models such as DriveDreamer (Wang et al., 2024a), MagicDrive (Gao et al., 2024b), Drive-WM (Wang et al., 2024b), and GAIA-1 (Hu et al., 2023) utilize inputs like text, bounding boxes, Bird’s Eye View (BEV) maps, and driver actions tocontrol videogenerationfor complexdriving scenarios.”

“To overcome these constraints, recent works have proposed using diffusion generative models to generate driving simulation environments from sensor inputs [17, 23, 74] or post-perception data [10, 56].”

“Other works (Alonso et al., 2024; Yang et al., 2024a; Rigter et al., 2024; Hu et al., 2023; Zhang et al., 2024) utilise diffusion models to train action-conditioned world models.”

“This method is applied in ITS tasks like tra ffi c scenario generation, where the goal is to maintain realistic outputs while allowing for diverse possibilities [38].”

“…2022; Ha and Schmidhuber, 2018; Fung et al., 2025), showing strong performance in planning across low-level, continuous control tasks such as robotic control (Oquab et al., 2023; Yang et al., 2024; Assran et al., 2025; Pan et al., 2025) and autonomous driving (Hu et al., 2023; Wang et al., 2024c).”

“World models for autonomous driving [23,24,56,59] have drawn extensive attention from both the industry and academia in recent years.”

“GAIA-1 [28] introduced a generative world model for autonomous driving, capable of simulating realistic driving videos from inputs like images, texts, and actions.”

“GAIA-1 [Hu et al. , 2023] similarly leverages real-world driving data to develop a world model that generates life-like driving behavior by predicting outcomes in the video space.”

“While some methods [85,86] leverage prompts like text to control the content of the generated driving data, they can not guarantee the accuracy of the spatial information.”

“Furthermore, the recent emergence of research related to the world model is an advanced form of scenario understanding [27], [52], [53], [79], [80], which is capable of understanding the world and even generating predictions of future world content.”

“Their applications span immersive simulation [13, 90, 175, 212, 290, 342], end-to-end planning [69, 92, 265, 267], and feature learning for downstream tasks [29, 30, 138, 170, 188, 298, 319].”

“Other works focus on augmenting driving datasets with various language descriptions [66, 88], or constructing environments for novel driving scenarios [24, 31, 85, 100, 102].”

“Artistic painting and text-guided image editing [21], MRI reconstruction [32], self-driving cars [33], and video production from text [21] are among the successful applications of DMs in the domain of image and video generation.”

“In contrast, World4RL achieves competitive policy refinement without frequent online real-robot interaction.”

“…features obtained by self-supervised learning is more contextually meaningful [5], which can be used as an auxiliary supervision to further improve the generation quality [19] However, integrating existing mask reconstruction for image generation into driving world models is not straightforward.”

“Ensuring physics correctness in video generation is not only crucial to increase user satisfaction, but also for reliable world modeling [34] and downstream applications like robotics [66] or autonomous driving [25].”

“…capabilities has led to promising applications in diverse domains, including game simulation like Genie [9], GameNGen [10], Oasis [44], Gamegen-x [45], The Matrix [37], Mineworld [46], GameFactory [47] and so on [43], autonomous driving [48], robotic manipulation [35, 49], and navigation [50].”

“When it comes to urban scenes which is the most important open environment, existing works mostly focus on the generative world model for autonomous driving capable of learning scene dynamics and understanding the geometry of the physical world [10, 11, 28].”

“This approach opened up avenues for future model-based meth-ods to capitalize on advances in generative modelling (Ville-gas et al., 2022; Achiam et al., 2023), and has already been adopted beyond its original domain (comma.ai, 2023; Hu et al., 2023).”

“For instance, GAIA-1 (Hu et al. 2023a) creates realistic driving scenarios from multi-modal inputs and allows detailed control over vehicle behavior, while Drive-Dreamer4D (Zhao et al. 2025) generates novel trajectory videos from real driving data to enhance 4D reconstruction.”

“The development of world models has seen significant advancements in recent years, with notable progress reported in [38, 39, 40, 41, 42, 43, 44].”

“Current large world models such as GAIA (Hu et al., 2023), LWM (Liu et al., 2024) learn to encode the entire images into a latent space based on a reconstruction loss which can capture many irrelevant details for decision-making.”

“…approaches build from driving videos towards ‘neural world simulators’ or so-called ‘world-models’ by leveraging pre-trained diffusion networks, as VISTA [31], GEM [36], and InfinityDrive [35] do, or by exploiting discrete-based auto-regressive models, as GAIA-1 [44] and DrivingWorld [46] do.”

“Existing meth-ods in vision-based [39], [16] and LiDAR-based [41], [38] approaches, while enhancing vehicle navigation and environmental understanding, are computationally intensive and often lack transparency in intermediate reasoning processes.”

“Robotics: There has been an influx of training general purpose action-conditioned video-generation models (Hu et al., 2023; Russell et al., 2025; Yang et al., 2024; Bruce et al., 2024; Agarwal et al., 2025; Assran et al., 2025).”

“These models can generate realistic scenarios based on specific instructions, demonstrating coherent scene generation and prolonged temporal consistency [24], [25].”

“Domain-specific world models for gaming (Micheli et al., 2022; Alonso et al., 2024; Bruce et al., 2024), autonomous driving (Wang et al., 2023b; Hu et al., 2023), and robotics (Yang et al., 2024a; NVIDIA, 2025) achieve strong within-domain performance but rely on restricted action spaces and…”

“…for constrained settings like Atari games [24], their ability to model state transitions has been extended to real-world scenarios through joint learning of policies and world models, improving sample efficiency in simulated robotics [53], real-world robots [63] and autonomous driving [29, 61].”

“For instance, GAIA-1 [66] proposes a 2D generative world model for autonomous driving, which utilizes video, text, and action as conditional inputs to generate realistic future driving scenes.”

“On the other hand, generation-based methods [10]– [15] can learn from in-the-wild driving data and synthesize action-conditioned dynamic driving videos via image [16] or video diffusion models [17].”

“Recent progress in controllable video generation [17, 21] has enabled action-conditioned world models, suggesting their potential as learned simulators.”

“…while recent approaches use transformer and diffusion backbones to roll out realistic videos conditioned on actions, text, or camera trajectories [1, 2, 6, 9, 12, 20, 36, 40, 45, 50, 70, 88, 103, 110, 115], and further extend temporal horizons with memories or long-sequence models [52, 73, 101].”

“Novel View Synthesis for Urban Scene With the emergence of NeRF [35] and 3D Gaussian Splatting (3DGS) [25], many methods have adopted these representations across a wide range of tasks, including robotic mapping and localization [9, 11, 12, 14], VR [13, 18, 29], and autonomous driving [5, 36].”

“…model-based reinforcement learning and trajectory prediction (Yang et al., 2023; Zhou et al., 2024); in autonomous driving (Wang et al., 2023; Hu et al., 2023), they facilitate path planning; and in gaming (Hafner et al., 2019; Bruce et al., 2024; Ha and Schmidhuber, 2018a,c), they power…”

“…generation (Blattmann et al., 2023a), researchers have introduced action-driven video generation that could generate future frames conditioned on actions (Yang et al., 2024; Wang et al., 2024a; Hu et al., 2023; Wu et al., 2024), mimicking the slow learning paradigm for building the world model.”

“Pioneering work, such as GAIA-1 [119], encodes past images, text, and actions to predict the next image token, emphasizing a deep understanding of driving data.”

“3(a), DWM-based simulators synthesize dynamic driving environments directly from raw observations [27, 43, 67, 158, 167] and structural conditions [24, 25, 119, 122, 178], in contrast to conventional engine-based simulators that necessitate laborious manual asset creation, which suffer from high…”

“Furthermore, the promising results of large-scale generative models like [4, 32] have inspired researchers to use these models as real-world simulators, significantly influencing the field of driving simulation [20, 50, 58].”

“Generative image models play a crucial role in diverse applications, notably in medical image generation and self-driving systems [Guibas et al., 2017, Kazerouni et al., 2022, Hu et al., 2023].”

“To address the challenges of predicting future changes in driving scenarios, Wayve proposed a generative world model, GAIA-1 [165].”

“These AI systems must first build an internal representation of an environment, and then use it to simulate future events within that environment [40], either in pixel representations [3, 21, 50] or abstract latent future simulations [16–18].”

“This long-form generation is essential for powering advanced applications such as continuous world simulation [2, 12, 17, 35, 45], cinematic long takes [11, 32], and synthetic data generation [18, 33].”

“Additionally, this data could be transformational in robotics for tasks like imitation learning [1, 23, 58] or training in realistic simulation environments [6, 35, 89].”

“…temporal information from long video and text inputs, which is crucial for various applications including robotics (Black et al., 2024; Prasad et al., 2024; Cheang et al., 2024), autonomous driving (Hu et al., 2023; Wang et al., 2024c; Gao et al., 2024), and health-care (Liu et al., 2024b).”

“For instance, Hu et al. [24] uses video, text, and action inputs to generate realistic driving scenarios with fine-grained scene features.”

“Early work on learned world models [48, 74, 102] has primarily focused on small-scale tasks with constrained environments and limited action spaces.”

“Most approaches adopt generative models, such as autoregressive Transformers (e.g., GAIA-1 [30]) or diffusion models (e.g., DriveDreamer [11], Drive-WM [31]), to synthesize future visual representations.”

“The third strategy uses world model as a generative model to predict future video sequences [11, 21, 32, 55, 56] by utilizing video generation techniques [17, 24, 54], which are subsequently used for task-specific purposes.”

“It explores additional training techniques to optimize vision transformer performance.”

“In the context of simulation, generative models have shown remarkable potential for creating realistic scenarios, which are crucial for training and evaluating safety-critical embodied agents like autonomous vehicles [11, 44].”

“…cloning objectives in video games (Baker et al., 2022; Raad et al., 2024), robotics (Brohan et al., 2022; 2023; Padalkar et al., 2023; Bousmalis et al., 2023), or multiple domains (Reed et al., 2022), as well as world modeling objectives (Hu et al., 2023; Yang et al., 2023; Bruce et al., 2024).”

“Beyond video games and simulation, world models have also been applied to real-world video for autonomous driving (Hu et al., 2023; Russell et al., 2025).”

“Recent innovations have explored hybrid architectures of diffusion models and next-token prediction mechanisms, aiming to preserve visual quality while improving temporal coherence [51, 98–102, 108 ], as (”

“Other approaches like GAIA-1 [29], MUVO [3], ADriver-I [33], DriveDreamer-2 [76], DriveWorld [45], BEVWorld [74], WorldGPT [16] and HERMES [78] rely on text prompts or discrete control signals.”

“Existing methods can be broadly categorized by their output modality: some focus on visual scene generation [10, 11, 13, 48], others emphasize future trajectory planning [9, 26, 28], while several works address 3D scene reconstruction in the form of occupancy [36, 40], Li-DAR [43, 58], or point…”

“Morespecifically, in the first approach, GAIA-1 [24] utilizes a WM to generate realistic driving scenarios given videos, texts, and actions as inputs.”

“For instance, (Hu et al., 2023)”

“The advent of large-scale video generation models has further highlighted this potential, demonstrating their ability to serve as physical simulators [32, 43], driving scene simulators [20, 28, 62], and game engines [1, 2, 57].”

“Specifically, GAIA-1 [18] formulates world modeling as a sequence task using an autoregressive transformer, while ADriver-I [23] integrates multi-modal LLMs and diffusion for control and frame generation.”

“Researchers have delved into the development of intricately designed driver agents that could perceive environmental stimuli[5], [6], [7], comprehend the situation[8], fetch their memories[9], [10] and deduce subsequent driving actions[11] that mirrors human decision-making.”

“Videos capture how the physical world evolves and how agents act within it, recording the dynamics of physical interactions at scale [24, 25, 26, 50, 9, 1, 2].”

“One line of work applies video generation models [9, 11, 13, 26, 37, 38, 49] to construct pixel-level driving world representations, while 3D world models based on occupancy and point clouds [7, 48, 50, 55] enforce geometric constraints in 3D space, with subsequent works [17, 18] unifying 2D and 3D…”

“…to improve the outputs’ quality (Kang et al., 2024; Li et al., 2025), but also a essential pre-requisit for them to serve as reliable world models (LeCun, 2022; Ha & Schmidhuber, 2018) with applications in robotics (Yang et al., 2023) and autonomous driving (Hu et al., 2023; Zhao et al., 2025).”

“As a pioneering 2D image-based world model, GAIA-1 [11] employs the video diffusion model to forecast future driving videos by taking video, text, and action as inputs.”

“Additionally, due to the capability of world models for comprehending driving environments, some works [203]–[205] explored world models to generate high-quality driving videos.”

“In many applications, from immersive game environments and realistic cinematic worlds [55], to training world models for games [10, 29, 67], autonomous driving [1, 14, 27, 48] and robotic control [4, 35, 64], generated videos serve as data for perception, reasoning, and action [60].”

“However, current methods that fine-tune on numerical camera parameters lack 3D geometry awareness, compromising their action controllability and consistency ([7], [1], [16]).”

“Given the surge in foundation models [21, 60], world models [45, 80], and large-scale pre-training approaches [43,78], we see potential for the application of self-and unsupervised training paradigms in the field of anomaly detection as well.”

“GAIA-1 (‘Generative AI for Autonomy’) [240], , shown in Fig.”

“Besides general video generation, driving-focused generative models use sensor data such as lidar point clouds (Zheng et al., 2024a; Yang et al., 2024d) or images (Gao et al., 2024; Hassan et al., 2024; Hu et al., 2023; Wang et al., 2024c;f; Yang et al., 2024b; Zhao et al., 2024).”

“Beyond these approaches, video diffusion models have also been explored as simulators for training embodied agents in driving [17, 41], game environments [3, 8, 44], or text-generated worlds [6].”

“Genie [8] leverages a video tokenizer and a Latent Action Model for dynamic next-frame generation, whereas GAIA-1 [46], GAIA-2 [66] tackle autonomous driving by autoregressively predicting image tokens from multi-modal inputs.”

“GAIA-1 [21], trained on a large corpus of driving data, is able to generate long and diverse driving scenes at a high frame rate; however, its low resolution limits its usefulness.”