GAIA-1: A Generative World Model for Autonomous Driving

“Its training objective is to predict the next image token in the sequence conditioned on all past tokens, using causal masking in the attention matrix of the transformer blocks [35].”

“The discrete autoencoder is a fully convolutional 2D U-Net [31].”

“Because these text sources are imperfect, to improve the alignment between generated futures and the text prompt, we employ classifier-free guidance [45, 46] at inference time.”

“Large language models have shown clear benefits in scaling model size and data [90, 24, 26, 20, 21, 22, 23].”

“The advantage of such a perspective is that world models can benefit from scaling properties of high-capacity sequence model architectures applied to large-scale unsupervised training [26].”

“Following the recent advances in image [36, 37] and video generation [16, 18] we use denoising video diffusion models for the GAIA-1 decoder.”

“Most language models [25, 26, 24, 21, 27, 22] use byte-pair encoding (or equivalent) as a trade-off between character-level and word-level tokenization.”

“The image reconstruction loss is a weighted sum of L 1 , L 2 , perceptual loss L perceptual [32], and GAN loss L GAN [33].”

“Large language models have shown clear benefits in scaling model size and data [90, 24, 26, 20, 21, 22, 23].”

“The models used to fit the power-law ranged from 10,000x to 10x smaller models in terms of parameters (0.65M to 650M), as visualized in Figure 8b. Similarly to [49], the compute was estimated as a function of the parameter count.”

“For example, when attempting to auto-regressively generate a video (as opposed to text [6] or vector-quantized latents [33]) past the training horizon, slight errors in frame-to-frame predictions accumulate and the model diverges.”

“Looking forward, we believe recent advances in video/image generation and world models [13, 20, 27, 62, 66] present promising opportunities to enhance generalization capabilities through improved visual reasoning and predictive modeling. dreamer: Learning to follow instructions via…”

“Recent works [19, 51, 52] argue that language captions are an important medium to connect human knowledge with the driving objective, helping to inform decisions and actions.”

“Both families have been extended to text-to-video [36, 37, 65, 73] using paired data.”

“One example is the GAIA-1 model which generates predictions of driving video conditioned on arbitrary combinations of video, action, and text [191].”

“Teaching the AI system to generate various video content plays a vital role in simulating the real-world dynamics (Hu et al., 2023; Brooks et al., 2024) and interacting with humans (Bruce et al., 2024; Valevski et al., 2024).”

“Auto-regressive Diffusion Models .”

“(c ) Vision-to-text models accept video or image as input and produce text as output, e.g. GAIA-1 [60].”

“This progress has been propelled by the utilization of cutting-edge diffusion models [5,17, 18, 39,40,46], exemplified by notable contributions such as DriveDreamer [59], Drive-WM [62], Magic-Drive [7], Panacea [64], and the integration of large language models like GAIA-1 [23], ADriver-I [25].”

“For driving scenario generation, most prior works are built upon diffusion models, with the exception of GAIA-1 [13] which incorporates a progressive next-token predictor and an additional diffusion image decoder.”

“GAIA-1 [124] leverages video, text, and action inputs to generate traffic scenarios, environmental elements, and potential risks.”

“Inspired by this success, we propose adopting a world model [15, 19, 38–40] This ICCV paper is the Open Access version, provided by the Computer Vision Foundation.”

“GAIA-1 [12] is a generative world model that utilizes video, text, and action inputs to create realistic driving scenarios.”

“Consequently, world models show great promise in fields such as game engines [9,12,42], autonomous driving [17], and spatial intelligence [2,3,29,46].”

“Fortunately, there are some prelim-inary attempts on developing a video predictive model in the driving domain [4, 15, 19, 23, 25, 33, 38, 45, 47].”

“These foundation models enable many downstream tasks, ranging from image editing [4, 17] to synthetic data generation [20], to video and 3D generations [34, 46].”

“Recent works on long video generation [4, 8, 12, 21, 34, 40, 54] have facilitated research in this field.”

“This concept has been extensively explored in the construction of world models [11] for agent learning [10, 12, 13, 18].”

“Our analysis effectively applies to datasets and architectures that follow the approach of exploring algorithmic and universal circuits as represented by our simple MCNs and Transformers.”

“…to learn the environmental dynamics for downstream tasks by anticipating the future state s ( t +1) based on the current state s ( t ) and action a ( t ) at the t th step, which have been applied to a variety of tasks including autonomous driving [10, 21, 22, 57] and game agent [14–18, 49, 66].”

“They have emerged as a cornerstone in a range of domains, including interactive video generation [10, 11], autonomous driving [35, 69, 67, 26], and robotics [24, 71, 72].”

“…separate perception-planning pipelines, which require extensive annotations and explicit supervision, generative driving world models [11, 15, 17, 23, 25, 56, 57, 66] integrate visual scene understanding with future prediction in a self-supervised manner, offering a new solution toward…”

“GAIA-1 [66] Autoregressive Transformer Crafted a system to generate realistic driving scenes using video, text, and action inputs, providing fine-grained control over the generated environment.”

“GAIA 72,73 generates driving scenarios from real world data.”

“Genie [6] proposes a foundation model for playable world based on video generation.”

“Image generation methods can generate realistic scene images based on different control signals, such as HD maps [26, 39, 59], sketch layout [71], bounding boxes [26, 39, 63], text [26, 33, 39, 63] and driving actions [33, 63].”

“Prominent works like GAIA-1 (Hu et al., 2023), DrivingGPT (Chen et al., 2024), and Doe-1 (Zheng et al., 2024a) generate future images or multi-modal tokens to create realistic scenarios, while Copilot4D (Zhang et al., 2023) specifically predicts future discrete visual tokens.”

“Additionally, due to the capability of world models for comprehending driving environments, some works [203]–[205] explored world models to generate high-quality driving videos.”

“Image-based video generation encompasses driving video approaches us-ing diffusion models like DriveDreamer [37, 45], Vista [6], and Drive-WM [38], as well as driving video generation methods based on autoregressive models such as Drive-World [28] and GAIA[10].”

“The primary ob-jective of pre-training is to minimize the dependence on precise 3D annotations.”

“…beneficial for gaming agents (Kim et al., 2020; Alonso et al., 2024; Pearce et al., 2024; He et al., 2025), autonomous vehicles (Kim et al., 2021; Hu et al., 2023; Yang et al., 2024a; Gao et al., 2024; Wang et al., 2024; Bar et al., 2025; Hassan et al., 2025), and real robots (Seo et al., 2023;…”

“In autonomous driving, world models have been used to infer future states of the environment for safe and robust driving [19, 30, 41].”

“Driving world models [19, 27, 31, 60, 63, 74] have gained significant attention as model-based searching and planning are widely considered essential paths toward human-level physical intelligence [34].”

“GAIA-1 [19] proposes a generative world model that takes video, text, and action as inputs to generate realistic driving scenarios.”

“…[58], VIMA-Bench [91], Vir-tualHome [176], AI2-THOR [99], ProcTHOR [41], ThreeDWorld [57] Generative model, world model Evaluate the embodied AI models’ abilities with interactive models representing the environments GAIA-1 [79], UniSim [227], LWM [136], Genesis [11], RoboGen [212] Table 3.”

“End-to-end autonomous driving has traditionally relied on regression-based approaches that predict a single trajectory to mimic expert behavior (Jiang et al. 2023; Hu et al. 2023; Wang et al. 2023) .”

“GAIA-1 [28] and DriveDreamer [63] further extend to action-conditional diffusion models, enhancing the controllability and realism of generated videos.”

“MagicDrive-V2 generates high-resolution and long videos with multi-view and control supports, significantly exceeding the limitation of previous works [12, 13, 16, 36].”

“Unlike world model approaches [11, 15] that rely on diffusion models for both image generation and vehicle movement prediction, our Traffic Manager utilizes explicit traffic flow generation algorithms [35].”

“LongVideoGAN [49], NUWA-XL [50], and GAIA-1 [51] utilized GAN-based methods, diffusion-over-diffusion techniques, or world models but were limited to specific domains.”

“…the availability of large-scale video datasets, have led to significant progress in generating realistic visual environments conditioned on text descriptions (OpenAI, 2024; Yang et al., 2024; Polyak et al., 2024; Google, 2025) or agent actions (Hu et al., 2023; Guo et al., 2025; Bar et al., 2025).”

“Similarly, many works generate videos in [17, 56, 11, 55] from text prompts, BEV maps, bounding boxes, and driver actions to simulate autonomous driving scenarios.”

“In autonomous driving, world models [13, 21, 49, 52, 60, 69] utilize predictive techniques to interpret driving environments.”

“With a generative simulator, recent work has shown that different driving conditions (e.g., sunny, foggy, snowy, rainy, at night) can be introduced into the simulator (Hu et al., 2023).”

“Various models represents the scene in different spaces, they can be divided into 2D image representation (Hu et al. 2023a; Li, Zhang, and Ye 2023; Zhang et al. 2024; Wang et al. 2023b, 2024c), 3D point clouds representation (Khurana et al. 2022, 2023; Zhang et al. 2023b) and 3D occupancy…”

“The increased availability of open-source robotic datasets has enabled training large-scale robotic foundation models that leverage data across many embodiments [4, 5, 53–56].”

“In addition to transmission rate optimization for V2X Communications, 6G offers flexible channel management solutions via Multiple-input Multiple-output (MIMO) settings, where beamforming management plays a crucial role [126].”

“Numerous approaches [31,32,49,50] have been introduced to assess the efficacy of world models in autonomous driving scenarios.”

“In video-level control, a control signal is given at the start, and the model generates a responsive video sequence; notable examples include UniSim [45], Pan-dora [44], GameGen-X [5], MicroVGG [31], and GAIA-1 [19].”

“Based on strong image pretrained models, image-based world models [15, 42] can generate high-quality driving-scene images with conditions of 3D bounding boxes.”

“As such, there are attempts that use the autoregressive model for video generation (Kondratyuk et al., 2023; Hu et al., 2023).”

“World models for autonomous driving represent the world using sensor obser-2 vations, such as lidar-generated point clouds [6, 76, 79, 85], with limited datasets often constraining their scale, or images [16, 30, 39, 41, 61, 65, 71, 84].”

“Delphi [33] and GAIA [19] further advance this frontier by conditioning generation on trajectory and semantic inputs, enabling goal-directed synthesis.”

“Current applications of world models in autonomous driving mainly include driving scene generation [8, 9, 14, 42, 49], planning [5, 6, 13, 45, 54], and representation learning [7, 33, 51].”

“For example, many studies (Ma et al. 2024a; Wang et al. 2023b; Hu et al. 2023a) aimed to generate high-fidelity driving videos through world models to provide additional data for downstream training.”

“An exciting possibility is to combine large-scale self-play training with data-driven simulation of the associated perceptual inputs (e.g., camera images) (Ost et al., 2021; Yang et al., 2023; Hu et al., 2023).”

“GAIA-1 [22] is a generative world model capable of multimodal video generation that leverages video, language and actions to synthesize driving scenarios which can comply with given language instructions.”

“Many works on world models build on video generation techniques (Bruce et al., 2024; Wu et al., 2024; Hu et al., 2023; Yang et al., 2024a; Xiang et al., 2024).”

“These developments have sparked efforts to construct world models applicable to real-world scenarios, particularly in autonomous driving [4, 23, 49].”

“Transformer-based models have also been applied to robotic manipulation ( 42 ) and large-scale autonomous driving data ( 43 ).”

“World models [2, 23, 25, 31, 47, 70, 74, 92] are causal generative models designed to predict how world states evolve in response to actions, enabling interactive simulation of complex environments.”

“Future work could explore the use of co-training data produced by video generation models and world models [66]–[68] as a way to bridge this gap.”

“Recent advances integrate multi-sensor data to refine these representations, such as generative approaches [14], [15] that simulate the physical world for training that unify heterogeneous sensor inputs into consistent top-down perspectives [16], [17].”

“GAIA-1 [17] investigated real-world driving planners based on autoregressive models, but GAIA-1 had large parameters and computational demands, which increased as the number of condition frames grew.”

“Our model also opens up new possibilities for constructing generative world models [16], [17], [18] by simulating Earth’s visuals from an innovative overhead (bird’s-eye view) perspective.”

“GAIA-1[43] is developed by Wayve to generate realistic driving videos.”

“[20] build a world-model from a large-scale real-world automotive dataset.”

“GAIA-1 [239] pioneered the use of generative world models for AD, by demonstrating the ability to generate diverse traffic scenarios with multiple interacting agents.”

“Since the perplexities are influenced by the sampling strategies (Hu et al., 2023), we report the perplexities under various K values.”

“Current LiDAR generation methods fall into two broad categories, each of which suffers from specific challenges.”

“Other studies [11] incorporate causal mechanisms to generate longer sequences or extend video-generation models into world modeling by forecasting future states [17, 5, 48].”

“In the more specific domain of driving video generation, several works have conditioned the video generation on the HD maps and the car bounding boxes [17, 19, 25, 57, 59, 69], and can generate a local occupancy map alongside [20, 39], turning them into a “world model” for planning.”

“First, world-model simulators [20, 22, 29, 38, 77, 86] synthesize downstream data or serve as closed-loop environments to guide policy learning, which is indirect and does not transmit the model’s physical understanding into the planner’s state.”

“• Each frame is encoded with a frozen DINOv2 ViT-B/14 with registers, the features from the intermediate layers [2 , 5 , 8 , 11] (zero-indexed) are concatenated as a 4 · 768 vector, and projected to the first 1152 principal components.”

“Recently, discrete tokenization has enabled a shift towards integrating numerous vision tasks into unified multimodal and multitask models such as Gato [74], OFA [93], Pix2Seq [17, 18], Uni-fiedIO [61, 60], 4M [65], and more [51, 83, 5, 41, 26, 44, 4, 97, 87, 108, 54, 105].”

“GAIA-1 [16] first introduced a learned simulator based on an autoregressive model.”

“Recent advancements [9–12], particularly those utilizing large-scale online video data for learning a video generator, demonstrate significant potential in comprehending complex physical dynamics of the real world.”

“Inspired by breakthroughs in large language models, Autoregressive (AR) approaches tokenize images/videos and learn causal next-token dynamics, which makes online rollouts and tight policy coupling natural (Hu et al., 2023; Bruce et al., 2024).”

“(c ) Vision-to-text models accept video or image as input and produce text as output, e.g. GAIA-1 [60].”

“…and planning [1, 30], representation-centric predictive models that learn visual or video abstractions without directly generating pixels [31, 32, 33], and generative simulators that roll out observations under actions or conditions [2, 3, 6, 7, 8, 9, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44].”

“Hu et al. (2023a) introduces GAIA-1, a generative world model for autonomous driving that uses video, text, and action inputs to create realistic driving scenarios.”

“GAIA-1 [282] pioneered treating world modeling as sequence prediction in autonomous driving, integrating video, text, and action inputs through an autoregressive transformer architecture to generate realistic driving scenarios.”

“Autoregressive transformer architectures have also been employed [105, 289], including on large-scale internet (video-game) data [40].”

“Although previous diffusion-based methods(Wang et al., 2023a; Lu et al., 2023b; Hu et al., 2023; Wang et al., 2024; Yang et al., 2024) are capable of controlling the camera motion in a coarse trajectory, their control precision is far from enough for safety-critical simulation purposes.”

“GAIA-1 [22] is a generative world model capable of multimodal video generation that leverages video, language and actions to synthesize driving scenarios which can comply with given language instructions.”

“They have been widely explored in autonomous driving [14, 26, 27, 81, 99, 103], game agents [1, 18–22, 67, 90], and robotic manipulation [23, 68, 84].”

“…are processed according to the specifications in Section III-C, and based on this, the voxels and trajectories are predicted, ultimately outputting the predicted occupancy and self-planning. evolving, world models are also applied for predicting future scenarios and decisions making [39].”

“More recently, large-scale world models have flourished (Hu et al., 2023; Yang et al., 2023; Brooks et al., 2024; Bruce et al., 2024; Parker-Holder et al., 2024; Bartoccioni et al., 2025; Agarwal et al., 2025; Bar et al., 2025).”

“This phenomenon suggests that our model still harbors considerable potential for enhancement, as indicated by the scaling law [31,37].”

“Tesla is a pioneer in the investigation of real-time occupancy networks with high performance and low latency at 10 ms on embedded FSD computers.”

“…manipulation (Du et al., 2023; Ko et al., 2023; Du et al., 2024; Yang et al., 2024a; Zhen et al., 2025), autonomous driving (Gao et al., 2024; Hu et al., 2023), and navigation (Bar et al., 2025b; Wang et al., 2023; Koh et al., 2021a), with extensions to broader embodied tasks (Lu et al.,…”

“Transformer GAIA-1 [101] Generative world model predicting driving scenarios for autonomous driving.”