The DOI matches its DataCite record; the references it cites are not checked yet.
Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
Open accessAbstract
We present Stable Video Diffusion - a latent video diffusion model for high-resolution, state-of-the-art text-to-video and image-to-video generation. Recently, latent diffusion models trained for 2D image synthesis have been turned into generative video models by inserting temporal layers and finetuning them on small, high-quality video datasets. However, training methods in the literature vary widely, and the field has yet to agree on a unified strategy for curating video data. In this paper, we identify and evaluate three different stages for successful training of video LDMs: text-to-image pretraining, video pretraining, and high-quality video finetuning. Furthermore, we demonstrate the necessity of a well-curated pretraining dataset for generating high-quality videos and present a systematic curation process to train a strong base model, including captioning and filtering strategies. We then explore the impact of finetuning our base model on high-quality data and train a text-to-video model that is competitive with closed-source video generation. We also show that our base model provides a powerful motion representation for downstream tasks such as image-to-video generation and adaptability to camera motion-specific LoRA modules. Finally, we demonstrate that our model provides a strong multi-view 3D-prior and can serve as a base to finetune a multi-view diffusion model that jointly generates multiple views of objects in a feedforward fashion, outperforming image-based methods at a fraction of their compute budget. We release code and model weights at https://github.com/Stability-AI/generative-models .
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The majority of video generative models adopt the diffusion model framework to gradually transform noise into video sequences.
“The majority of video generative models adopt the diffusion model framework (Blattmann et al., 2023; Ge et al., 2023; Lin et al., 2024; Ma et al., 2024; Yang et al., 2024) to gradually transform noise into video sequences.”
From Cosmos World Foundation Model Platform for Physical AI (2025) · cites this paper as Blattmann et al., 2023
Most existing work on video generative models focuses on text-to-video tasks, which generate videos based on text prompt inputs.
“Most existing work on video generative models focuses on text-to-video tasks, which generate videos based on text prompt inputs (Blattmann et al., 2023; Ge et al., 2023; Girdhar et al., 2024; Lin et al., 2024; Ma et al., 2024; Yang et al., 2024).”
From Cosmos World Foundation Model Platform for Physical AI (2025) · cites this paper as Blattmann et al., 2023
The effectiveness of world models has been demonstrated in computer games, real-world robots, and autonomous driving.
“The effectiveness of world models has been demonstrated in domains such as computer games (Alonso et al., 2024; Bruce et al., 2024; Hafner et al., 2021; Kim et al., 2020; Valevski et al., 2024), real-world robots (Wu et al., 2023; Yang et al., 2023), and autonomous driving (Blattmann et al., 2023; Hu et al., 2023; Kim et al., 2021; Zhao et al., 2024).”
From Cosmos World Foundation Model Platform for Physical AI (2025) · cites this paper as Blattmann et al., 2023
Video generative models have the potential to revolutionize autonomous driving simulation by enabling the generation of realistic driving videos conditioned on diverse input modalities.
“Video generative models have the potential to revolutionize autonomous driving simulation by enabling the generation of realistic driving videos conditioned on diverse input modalities, such as text, images, trajectories, 3D data, or maps (Blattmann et al., 2023; Gao et al., 2024,,; Hu et al., 2023; Jia et al., 2023; Kim et al., 2021; Lu et al., 2025; Wang et al., 2023, 2024; Yang et al., 2024).”
From Cosmos World Foundation Model Platform for Physical AI (2025) · cites this paper as Blattmann et al., 2023
Existing autonomous driving video generation approaches have been limited by constraints in data scale, resolution, and the number of camera views, restricting their effectiveness as comprehensive driving world simulators.
“Despite their potential, existing approaches have been limited by constraints in data scale (Gao et al., 2024,; Jia et al., 2023; Lu et al., 2025; Wang et al., 2023, 2024), resolution (Hu et al., 2023; Yang et al., 2024), and the number of camera views (Blattmann et al., 2023; Gao et al., 2024), restricting their effectiveness as comprehensive driving world simulators.”
From Cosmos World Foundation Model Platform for Physical AI (2025) · cites this paper as Blattmann et al., 2023
Existing video foundation models such as SoRA, Pika, Luma, Hailuo, and Kling primarily generate characters with limited speech capabilities, simplified mouth movements, and emotional expressions detached from meaningful dialogue, lacking control over spoken content.
“Despite significant advancements in visually compelling content and dynamic environments, models such as SoRA, Pika, Luma, Hailuo, and Kling [2, 3, 5-7, 19, 27, 42] primarily generate characters with limited speech capabilities. Typically, these models exhibit simplified mouth movements and emotional expressions detached from meaningful dialogue, lacking control over actual spoken content.”
From MoCha: Towards Movie-Grade Talking Character Synthesis (2025) · cites this paper as [2]
Citations
The citation lists haven’t been fetched yet
Cited by 105
Metadata
- Authors
- Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., Jampani, V., & Rombach, R.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2311.15127
- OpenAlex
- W4389115686
- Cited by
- 105
- References
- 0
- Citations by year
- 2026: 7 · 2025: 52 · 2024: 44 · 2023: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Keywords
- text-to-video generation, image-to-video generation, data curation, captioning, video filtering, motion representation, LoRa, high-resolution video generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
