CoChat
Sign Up
PreprintCited by 82Generative Adversarial Networks and Image SynthesisMultimodal Machine Learning ApplicationsDigital Media Forensic Detection

MagicVideo: Efficient Video Generation With Latent Diffusion Models

Open access
Zhou, Daquan, Wang, Weimin, Yan, Hanshu, Lv, Weiwei, Zhu, Yizhe, Feng, Jiashi·2022·arXiv
DOI 10.48550/arxiv.2211.11018arXiv 2211.11018
Open this paper in CoChat — free accountOpen PDF

Abstract

We present an efficient text-to-video generation framework based on latent diffusion models, termed MagicVideo. MagicVideo can generate smooth video clips that are concordant with the given text descriptions. Due to a novel and efficient 3D U-Net design and modeling video distributions in a low-dimensional space, MagicVideo can synthesize video clips with 256x256 spatial resolution on a single GPU card, which takes around 64x fewer computations than the Video Diffusion Models (VDM) in terms of FLOPs. In specific, unlike existing works that directly train video models in the RGB space, we use a pre-trained VAE to map video clips into a low-dimensional latent space and learn the distribution of videos' latent codes via a diffusion model. Besides, we introduce two new designs to adapt the U-Net denoiser trained on image tasks to video data: a frame-wise lightweight adaptor for the image-to-video distribution adjustment and a directed temporal attention module to capture temporal dependencies across frames. Thus, we can exploit the informative weights of convolution operators from a text-to-image model for accelerating video training. To ameliorate the pixel dithering in the generated videos, we also propose a novel VideoVAE auto-encoder for better RGB reconstruction. We conduct extensive experiments and demonstrate that MagicVideo can generate high-quality video clips with either realistic or imaginary content. Refer to \url{https://magicvideo.github.io/#} for more examples.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2211.11018

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Spatial and temporal coherence across frames can be trained independently on video data that can even be unlabeled.

    “Meanwhile, the spatial and temporal coherence across frames can be trained independently on video data that can even be unlabeled (Zhou et al., 2022).”

    From Video Diffusion Models: A Survey (2024) · cites this paper as Zhou et al., 2022

    Section 3.1.2 describes separate, parallel spatial and temporal attention operations, and Section 3.1 describes unsupervised pretraining on video-only data. The paper does not say spatial and temporal coherence are trained independently: the branches are combined in one model trained with a shared reconstruction objective. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • MagicVideo's VAE trained on video data can reduce motion artifacts compared to VAEs used in text-to-image models.

    “This VAE is trained on video data and can thereby reduce motion artifacts compared to VAEs used in text-to-image models.”

    From Video Diffusion Models: A Survey (2024) · cites this paper as Zhou et al.'s (2022)

    Section 3.3 (page not provided) says MagicVideo’s VideoVAE decoder alleviates pixel-dithering artifacts compared with reconstructing frames using a conventional VAE decoder. The Introduction attributes improved motion consistency to directed self-attention, not to the VideoVAE, and the cited passages do not establish that training the VideoVAE on video data is what reduces those artifacts. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Recent video diffusion approaches often rely on a pre-trained image generation model such as Stable Diffusion.

    “In contrast to models that are trained from scratch (e.g. Ho et al. 2022c, Singer et al. 2022, Ho et al. 2022a), recent video diffusion approaches (e.g. Zhou et al. 2022, Khachatryan et al. 2023, Blattmann et al. 2023b) often rely on a pre-trained image generation model such as Stable Diffusion (Rombach et al. 2022).”

    From Video Diffusion Models: A Survey (2024) · cites this paper as Zhou et al. 2022

  • MagicVideo has the best FID score on UCF-101 and the best FVD score on MSR-VTT.

    “MagicVideo (Zhou et al., 2022) has the best FID score on UCF-101 and the best FVD score on MSR-VTT, although our comparison includes only few other datasets competing in those categories.”

    From Video Diffusion Models: A Survey (2024) · cites this paper as Zhou et al., 2022

  • Both computational demands and visual fidelity of video diffusion models have significantly improved through architectural advancements such as moving the denoising process to the latent space of a variational auto-encoder and using upsampling techniques such as CDMs.

    “Both aspects have significantly been improved through architectural advancements, such as moving the denoising process to the latent space of a variational auto-encoder (He et al., 2022b; Zhou et al., 2022; Chen et al., 2023a; 2024; Blattmann et al., 2023a; Zhang et al., 2023a) and using upsampling techniques such as CDMs (Ho et al. 2022a; Wang et al. 2023c, see section 4.3).”

    From Video Diffusion Models: A Survey (2024) · cites this paper as Zhou et al., 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Cited by 82

Research works printed but not linked: 39

  1. [1][1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. · arXiv
  2. [2][2] Mohammad Babaeizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan. FitVid: Overfitting in Pixel-Level Video Pre- diction, June 2021. arXiv:2106.13195. · arXiv
  3. [3][3] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and im- age encoder for end-to-end retrieval. In ICCV, pages 1728–1738, 2021.
  4. [4][4] Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misogyny, pornog- raphy, and malignant stereotypes. arXiv preprint arXiv:2110.01963, 2021. · arXiv
  5. [5][5] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinet- ics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017.
  6. [6][6] Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial Video Generation on Complex Datasets, Sept. 2019. arXiv:1907.06571. · arXiv
  7. [7][7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. · arXiv
  8. [9][9] William Harvey, Saeid Naderiparizi, Vaden Mas- rani, Christian Weilbach, and Frank Wood. Flex- ible Diffusion Modeling of Long Videos. Tech- nical Report arXiv:2205.11495, arXiv, May 2022. arXiv:2205.11495. · arXiv
  9. [10][10] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aber- man, Yael Pritch, and Daniel Cohen-Or. Prompt-to- Prompt Image Editing with Cross Attention Control, Aug. 2022. arXiv:2208.01626. · arXiv
  10. [11][11] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen Video: High Def- inition Video Generation with Diffusion Models, Oct. 2022. arXiv:2210.02303. · arXiv
  11. [12][12] Jonathan Ho, Ajay Jain, and Pieter Abbeel. De- noising Diffusion Probabilistic Models, Dec. 2020. arXiv:2006.11239. · arXiv
  12. [13][13] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458, 2022. · arXiv
  13. [17][17] Nal Kalchbrenner, Aäron Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video Pixel Networks. In Proceed- ings of the 34th International Conference on Machine Learning, pages 1771–1779. PMLR, July 2017. ISSN: 2640-3498.
  14. [18][18] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-Based Real Image Editing with Diffusion Models, Oct. 2022. arXiv:2210.09276. · arXiv
  15. [20][20] Jun Hao Liew, Hanshu Yan, Daquan Zhou, and Jiashi Feng. MagicMix: Semantic Mixing with Diffusion Models, Oct. 2022. arXiv:2210.16056. · arXiv
  16. [21][21] Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Ger- ard de Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Frozen clip models are efficient video learners. arXiv preprint arXiv:2208.03550, 2022. · arXiv
  17. [23][23] Michael Mathieu, Camille Couprie, and Yann Le- Cun. Deep multi-scale video prediction beyond mean square error, Feb. 2016. arXiv:1511.05440. · arXiv
  18. [24][24] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Er- mon. SDEdit: Guided Image Synthesis and Edit- ing with Stochastic Differential Equations. Tech- nical Report arXiv:2108.01073, arXiv, Jan. 2022. arXiv:2108.01073. · arXiv
  19. [25][25] Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evalu- ation. In CVPR, 2022.
  20. [26][26] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d dif- fusion. arXiv preprint arXiv:2209.14988, 2022. · arXiv
  21. [27][27] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natu- ral language supervision. In ICML, pages 8748–8763. PMLR, 2021.
  22. [30][30] MarcAurelio Ranzato, Arthur Szlam, Joan Bruna, Michael Mathieu, Ronan Collobert, and Sumit Chopra. Video (language) modeling: a baseline for generative models of natural videos, May 2016. arXiv:1412.6604. · arXiv
  23. [2]2. Advances in neural information processing systems, 32, 2019.
  24. [32][32] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approxi- mate inference in deep generative models. In Interna- tional conference on machine learning, pages 1278– 1286. PMLR, 2014.
  25. [33][33] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- Resolution Image Synthesis with Latent Diffusion Models, Apr. 2022. arXiv:2112.10752. · arXiv
  26. [34][34] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion mod- els. In CVPR, pages 10684–10695, 2022.
  27. [35][35] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealis- tic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487, 2022. · arXiv
  28. [37][37] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 2022.
  29. [39][39] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-A-Video: Text-to-Video Generation without Text-Video Data, Sept. 2022. arXiv:2209.14792. · arXiv
  30. [41][41] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. · arXiv
  31. [42][42] Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N Metaxas, and Sergey Tulyakov. A good image generator is what you need for high- resolution video synthesis. ICLR, 2021.
  32. [43][43] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 6450–6459, 2018.
  33. [44][44] Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. MoCoGAN: Decomposing Mo- tion and Content for Video Generation, Dec. 2017. arXiv:1707.04993. · arXiv
  34. [46][46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  35. [47][47] Carl Vondrick, Hamed Pirsiavash, and Antonio Tor- ralba. Generating Videos with Scene Dynamics, Oct. 2016. arXiv:1609.02612. · arXiv
  36. [48][48] Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. Nüwa: Visual syn- thesis pre-training for neural visual world creation. In Computer Vision–ECCV 2022: 17th European Con- ference, Tel Aviv, Israel, October 23–27, 2022, Pro- ceedings, Part XVI, pages 720–736. Springer, 2022.
  37. [49][49] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, pages 5288–5296, 2016.
  38. [50][50] Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yu- chong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Advancing high-resolution video- language representation with large-scale video tran- scriptions. In CVPR, pages 5036–5045, 2022.
  39. [51][51] Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Animashree Anandkumar, Jiashi Feng, and Jose M Al- varez. Understanding the robustness in vision trans- formers. In International Conference on Machine Learning, pages 27378–27394. PMLR, 2022.

Authors
Zhou, D., Wang, W., Yan, H., Lv, W., Zhu, Y., & Feng, J.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2211.11018
arXiv
2211.11018
OpenAlex
W4309800375
Cited by
82
References
0
Citations by year
2026: 1 · 2025: 20 · 2024: 49 · 2023: 12
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
Keywords
text-to-video generation, latent diffusion models, 3D U-Net, video diffusion models, temporal attention, computational efficiency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review