GAIA-1: A Generative World Model for Autonomous Driving

“Its training objective is to predict the next image token in the sequence conditioned on all past tokens, using causal masking in the attention matrix of the transformer blocks [35].”

“The discrete autoencoder is a fully convolutional 2D U-Net [31].”

“Because these text sources are imperfect, to improve the alignment between generated futures and the text prompt, we employ classifier-free guidance [45, 46] at inference time.”

“Large language models have shown clear benefits in scaling model size and data [90, 24, 26, 20, 21, 22, 23].”

“The advantage of such a perspective is that world models can benefit from scaling properties of high-capacity sequence model architectures applied to large-scale unsupervised training [26].”

“Following the recent advances in image [36, 37] and video generation [16, 18] we use denoising video diffusion models for the GAIA-1 decoder.”

“Most language models [25, 26, 24, 21, 27, 22] use byte-pair encoding (or equivalent) as a trade-off between character-level and word-level tokenization.”

“The image reconstruction loss is a weighted sum of L 1 , L 2 , perceptual loss L perceptual [32], and GAN loss L GAN [33].”

“Large language models have shown clear benefits in scaling model size and data [90, 24, 26, 20, 21, 22, 23].”

“The models used to fit the power-law ranged from 10,000x to 10x smaller models in terms of parameters (0.65M to 650M), as visualized in Figure 8b. Similarly to [49], the compute was estimated as a function of the parameter count.”

“To achieve this, the development of a robust model of the world is imperative [1] and huge efforts have been made in the past to build such predictive world models for autonomous driving [2, 3, 4, 5, 6].”

“The quantized image features are encouraged to match the image features of a pre-trained DINO [30] model with a cosine similarity loss.”

“Large language models have shown clear benefits in scaling model size and data [90, 24, 26, 20, 21, 22, 23].”

“The encoder E θ quantizes the image features using nearest neighbor look-up from a learnable embedding table [28], resulting in image tokens z t = E θ ( x t ) .”

“They can be grouped in four categories: VAE-based (variational autoencoder [50], GAN-based (generative adversarial network [51]), diffusion-based [52], and autoregressive-based [53].”

“Large language models have shown clear benefits in scaling model size and data [90, 24, 26, 20, 21, 22, 23].”

“Following the recent advances in image [36, 37] and video generation [16, 18] we use denoising video diffusion models for the GAIA-1 decoder.”

“The image reconstruction loss is a weighted sum of L 1 , L 2 , perceptual loss L perceptual [32], and GAN loss L GAN [33].”

“We use the DDIM sampler [48] with 50 diffusion steps.”

“Transferring the scaling principles from large language models to the visual domain holds the potential for delivering consistent and expected performance improvements [92, 93, 43, 16, 94].”

“GAN-based methods produce more realistic videos [59, 60, 61, 62, 63, 64] but are known to suffer from training instability and a lack of generation diversity [65].”

“They can be grouped in four categories: VAE-based (variational autoencoder [50], GAN-based (generative adversarial network [51]), diffusion-based [52], and autoregressive-based [53].”

“Empirically we observed that maximization-based sampling (i.e. argmax) generates futures that get stuck in a repetitive loop, similarly to language models [44].”

“We used the FlashAttention v2 implementation [41] in the transformer module, as it offered significant advantages in terms of both memory utilization and inference speed.”

“World models have been successfully applied to control tasks in both simulation [9, 10, 11, 12, 13] and to real-world robotics tasks [14, 15].”

“To achieve this, the development of a robust model of the world is imperative [1] and huge efforts have been made in the past to build such predictive world models for autonomous driving [2, 3, 4, 5, 6].”

“Because these text sources are imperfect, to improve the alignment between generated futures and the text prompt, we employ classifier-free guidance [45, 46] at inference time.”

“GAN-based methods produce more realistic videos [59, 60, 61, 62, 63, 64] but are known to suffer from training instability and a lack of generation diversity [65].”

“Subsequently, we reframe the challenge of predicting the future into predicting the next token in the sequence [10, 19].”

“Subsequently, we reframe the challenge of predicting the future into predicting the next token in the sequence [10, 19].”

“To optimize distributed training, we used the Deepspeed ZeRO-2 training strategy [42] with activation checkpointing.”

“They can operate either at the pixel level [38, 17, 66, 67, 68, 69, 16] or in the latent space of a pre-trained image tokenizer [70, 18, 71].”

“To achieve this, the development of a robust model of the world is imperative [1] and huge efforts have been made in the past to build such predictive world models for autonomous driving [2, 3, 4, 5, 6].”

“Additionally, world models can be a solution to the sample efficiency issues of RL algorithms by acting as a simulator of the environment [7, 85, 86, 62, 13, 15, 87], although this assumes the world model is an accurate model of the environment.”

“To achieve this, the development of a robust model of the world is imperative [1] and huge efforts have been made in the past to build such predictive world models for autonomous driving [2, 3, 4, 5, 6].”

“We adopted the linear projection of the embedding and L 2 normalization from [34] as we found this helped increase vocabulary usage.”

“They can operate either at the pixel level [38, 17, 66, 67, 68, 69, 16] or in the latent space of a pre-trained image tokenizer [70, 18, 71].”

“Inspired by [29], we guide the compression towards meaningful representations by regressing to the latent features of a pre-trained DINO model [30], a self-supervised image model that is known to contain semantic information.”

“Additionally, world models can be a solution to the sample efficiency issues of RL algorithms by acting as a simulator of the environment [7, 85, 86, 62, 13, 15, 87], although this assumes the world model is an accurate model of the environment.”

“Latent-variable video models (VAE-based) try to infer the underlying latent process that generated the videos [54, 55, 56, 57, 58].”

“GAN-based methods produce more realistic videos [59, 60, 61, 62, 63, 64] but are known to suffer from training instability and a lack of generation diversity [65].”

“[91] showed that the quality of the training data plays a critical role in the performance of language models in downstream tasks.”

“Meanwhile, progress in generative image and video generation has harnessed the power of self-supervised learning to learn from large quantities of real-world data to generate remarkably realistic video samples [16, 17, 18].”

“They can operate either at the pixel level [38, 17, 66, 67, 68, 69, 16] or in the latent space of a pre-trained image tokenizer [70, 18, 71].”

“We choose to train a single video diffusion model for all tasks as it has been shown that multi-task training improves performance on individual tasks [17].”

“Latent-variable video models (VAE-based) try to infer the underlying latent process that generated the videos [54, 55, 56, 57, 58].”

“Likewise, they can operate at the pixel level [72, 73], or in a discrete learned token space [74, 75, 76, 77].”

“We follow [38] and use a 3D U-Net with factorized spatial and temporal attention layers.”

“Large language models have shown clear benefits in scaling model size and data [90, 24, 26, 20, 21, 22, 23].”

“To achieve this, the development of a robust model of the world is imperative [1] and huge efforts have been made in the past to build such predictive world models for autonomous driving [2, 3, 4, 5, 6].”

“World models have been successfully applied to control tasks in both simulation [9, 10, 11, 12, 13] and to real-world robotics tasks [14, 15].”

“Additionally, world models can be a solution to the sample efficiency issues of RL algorithms by acting as a simulator of the environment [7, 85, 86, 62, 13, 15, 87], although this assumes the world model is an accurate model of the environment.”

“World modeling has been used as a pre-training task to learn a compact and general representation in a self-supervised way [82, 83].”

“Likewise, they can operate at the pixel level [72, 73], or in a discrete learned token space [74, 75, 76, 77].”

“GAN-based methods produce more realistic videos [59, 60, 61, 62, 63, 64] but are known to suffer from training instability and a lack of generation diversity [65].”

“A known limitation is the slow generation speed, but this issue could be alleviated by future research on parallel sampling [78, 79, 80], reducing the number of latent variables [81], and improvements in hardware accelerators.”

“They can operate either at the pixel level [38, 17, 66, 67, 68, 69, 16] or in the latent space of a pre-trained image tokenizer [70, 18, 71].”

“Because these text sources are imperfect, to improve the alignment between generated futures and the text prompt, we employ classifier-free guidance [45, 46] at inference time.”

“Likewise, they can operate at the pixel level [72, 73], or in a discrete learned token space [74, 75, 76, 77].”

“GAN-based methods produce more realistic videos [59, 60, 61, 62, 63, 64] but are known to suffer from training instability and a lack of generation diversity [65].”

“They can operate either at the pixel level [38, 17, 66, 67, 68, 69, 16] or in the latent space of a pre-trained image tokenizer [70, 18, 71].”

“World models have been successfully applied to control tasks in both simulation [9, 10, 11, 12, 13] and to real-world robotics tasks [14, 15].”

“Transferring the scaling principles from large language models to the visual domain holds the potential for delivering consistent and expected performance improvements [92, 93, 43, 16, 94].”

“Latent-variable video models (VAE-based) try to infer the underlying latent process that generated the videos [54, 55, 56, 57, 58].”

“Latent-variable video models (VAE-based) try to infer the underlying latent process that generated the videos [54, 55, 56, 57, 58].”

“Likewise, they can operate at the pixel level [72, 73], or in a discrete learned token space [74, 75, 76, 77].”

“A known limitation is the slow generation speed, but this issue could be alleviated by future research on parallel sampling [78, 79, 80], reducing the number of latent variables [81], and improvements in hardware accelerators.”

“Latent-variable video models (VAE-based) try to infer the underlying latent process that generated the videos [54, 55, 56, 57, 58].”

“They can operate either at the pixel level [38, 17, 66, 67, 68, 69, 16] or in the latent space of a pre-trained image tokenizer [70, 18, 71].”

“To achieve this, the development of a robust model of the world is imperative [1] and huge efforts have been made in the past to build such predictive world models for autonomous driving [2, 3, 4, 5, 6].”

“World models have been successfully applied to control tasks in both simulation [9, 10, 11, 12, 13] and to real-world robotics tasks [14, 15].”

“GAN-based methods produce more realistic videos [59, 60, 61, 62, 63, 64] but are known to suffer from training instability and a lack of generation diversity [65].”

“They have proven to be highly effective in game environments or board games [9, 84].”

“A recent line of work suggests casting world modeling as a single sequence model, treating states, actions and rewards as simply a stream of data [10, 19, 14, 88, 12, 89].”

“A recent line of work suggests casting world modeling as a single sequence model, treating states, actions and rewards as simply a stream of data [10, 19, 14, 88, 12, 89].”

“Transferring the scaling principles from large language models to the visual domain holds the potential for delivering consistent and expected performance improvements [92, 93, 43, 16, 94].”

“A known limitation is the slow generation speed, but this issue could be alleviated by future research on parallel sampling [78, 79, 80], reducing the number of latent variables [81], and improvements in hardware accelerators.”

“They can operate either at the pixel level [38, 17, 66, 67, 68, 69, 16] or in the latent space of a pre-trained image tokenizer [70, 18, 71].”

“We used a cosine β -noise schedule [43].”

“Likewise, they can operate at the pixel level [72, 73], or in a discrete learned token space [74, 75, 76, 77].”

“They can be grouped in four categories: VAE-based (variational autoencoder [50], GAN-based (generative adversarial network [51]), diffusion-based [52], and autoregressive-based [53].”

“A recent line of work suggests casting world modeling as a single sequence model, treating states, actions and rewards as simply a stream of data [10, 19, 14, 88, 12, 89].”

“World modeling has been used as a pre-training task to learn a compact and general representation in a self-supervised way [82, 83].”

“Additionally, world models can be a solution to the sample efficiency issues of RL algorithms by acting as a simulator of the environment [7, 85, 86, 62, 13, 15, 87], although this assumes the world model is an accurate model of the environment.”

“Likewise, they can operate at the pixel level [72, 73], or in a discrete learned token space [74, 75, 76, 77].”

“A known limitation is the slow generation speed, but this issue could be alleviated by future research on parallel sampling [78, 79, 80], reducing the number of latent variables [81], and improvements in hardware accelerators.”

“Large language models have shown clear benefits in scaling model size and data [90, 24, 26, 20, 21, 22, 23].”

“For example, when attempting to auto-regressively generate a video (as opposed to text [6] or vector-quantized latents [33]) past the training horizon, slight errors in frame-to-frame predictions accumulate and the model diverges.”

“Looking forward, we believe recent advances in video/image generation and world models [13, 20, 27, 62, 66] present promising opportunities to enhance generalization capabilities through improved visual reasoning and predictive modeling. dreamer: Learning to follow instructions via…”

“Recent works [19, 51, 52] argue that language captions are an important medium to connect human knowledge with the driving objective, helping to inform decisions and actions.”

“Both families have been extended to text-to-video [36, 37, 65, 73] using paired data.”

“One example is the GAIA-1 model which generates predictions of driving video conditioned on arbitrary combinations of video, action, and text [191].”