CoChat
Sign Up
PreprintCited by 57Generative Adversarial Networks and Image SynthesisMultimodal Machine Learning ApplicationsDigital Media Forensic Detection

Lumiere: A Space-Time Diffusion Model for Video Generation

Open access
Bar-Tal, Omer, Chefer, Hila, Tov, Omer, Herrmann, Charles, Paiss, Roni, Zada, Shiran, Ephrat, Ariel, Hur, Junhwa +9 more·2024·arXiv
DOI 10.48550/arxiv.2401.12945arXiv 2401.12945
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce Lumiere -- a text-to-video diffusion model designed for synthesizing videos that portray realistic, diverse and coherent motion -- a pivotal challenge in video synthesis. To this end, we introduce a Space-Time U-Net architecture that generates the entire temporal duration of the video at once, through a single pass in the model. This is in contrast to existing video models which synthesize distant keyframes followed by temporal super-resolution -- an approach that inherently makes global temporal consistency difficult to achieve. By deploying both spatial and (importantly) temporal down- and up-sampling and leveraging a pre-trained text-to-image diffusion model, our model learns to directly generate a full-frame-rate, low-resolution video by processing it in multiple space-time scales. We demonstrate state-of-the-art text-to-video generation results, and show that our design easily facilitates a wide range of content creation tasks and video editing applications, including image-to-video, video inpainting, and stylized generation.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2401.12945

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The inevitable resolution downsampling operation in most video generation models with latent encoding is detrimental to the video inpainting task.

    “Furthermore, we note that the inevitable resolution downsampling operation in most video generation models with latent encoding [4; 2; 42; 8] is detrimental to the video inpainting task.”

    From SVG: 3D Stereoscopic Video Generation via Denoising Frame Matrix (2024) · cites this paper as [2]

    The citation is attached to the description of video generation models with latent encoding. The Introduction (p. 3) explicitly says Lumiere’s pretrained model works in pixel space, not latent space. Although Lumiere uses spatial and temporal downsampling, it does not support identifying Lumiere as a latent-encoding model. (LLM, conf=0.88)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 51

  1. Bar-Tal, O., Yariv, L., Lipman, Y., and Dekel, T. MultiDiffusion: Fusing diffusion paths for controlled image generation. In ICML , 2023.
  2. Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 , 2023a. · arXiv
  3. Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S., and Kreis, K. Align your latents: Highresolution video synthesis with latent diffusion models. In CVPR , 2023b.
  4. Carreira, J. and Zisserman, A. Quo vadis, action recognition? A new model and the kinetics dataset. In CVPR , pp. 6299-6308, 2017.
  5. Chen, T. On the importance of noise scheduling for diffusion models. arXiv preprint arXiv:2301.10972 , 2023. · arXiv
  6. Chong, M. J. and Forsyth, D. Effectively unbiased FID and Inception Score and where to find them. In CVPR , pp. 6070-6079, 2020.
  7. C ¸ ic ¸ek, ¨ O., Abdulkadir, A., Lienkamp, S. S., Brox, T., and Ronneberger, O. 3d u-net: learning dense volumetric segmentation from sparse annotation. In MICCAI , pp. 424-432. Springer, 2016.
  8. Croitoru, F.-A., Hondru, V., Ionescu, R. T., and Shah, M. Diffusion models in vision: A survey. IEEE T. Pattern Anal. Mach. Intell. , 2023a.
  9. Croitoru, F.-A., Hondru, V., Ionescu, R. T., and Shah, M. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023b.
  10. Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. NeurIPS , 2021.
  11. Gal, R., Vinker, Y., Alaluf, Y., Bermano, A. H., CohenOr, D., Shamir, A., and Chechik, G. Breathing life into sketches using text-to-video priors. arXiv preprint arXiv:2311.13608 , 2023. · arXiv
  12. Ge, S., Nah, S., Liu, G., Poon, T., Tao, A., Catanzaro, B., Jacobs, D., Huang, J.-B., Liu, M.-Y., and Balaji, Y. Preserve your own correlation: A noise prior for video diffusion models. In ICCV , pp. 22930-22941, 2023.
  13. Girdhar, R., Singh, M., Brown, A., Duval, Q., Azadi, S., Rambhatla, S. S., Shah, A., Yin, X., Parikh, D., and Misra, I. Emu Video: Factorizing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:2311.10709 , 2023. · arXiv
  14. Gu, J., Zhai, S., Zhang, Y., Susskind, J., and Jaitly, N. Matryoshka diffusion models. arXiv preprint arXiv:2310.15111 , 2023. · arXiv
  15. Guo, Y., Yang, C., Rao, A., Wang, Y., Qiao, Y., Lin, D., and Dai, B. AnimateDiff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725 , 2023. · arXiv
  16. Gupta, A., Yu, L., Sohn, K., Gu, X., Hahn, M., Fei-Fei, L., Essa, I., Jiang, L., and Lezama, J. Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.06662 , 2023. · arXiv
  17. He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pp. 1026-1034, 2015.
  18. Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. NeurIPS , 33:6840-6851, 2020.
  19. Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 , 2022a. · arXiv
  20. Hong, W., Ding, M., Zheng, W., Liu, X., and Tang, J. CogVideo: Large-scale pretraining for text-tovideo generation via transformers. arXiv preprint arXiv:2205.15868 , 2022. · arXiv
  21. Hoogeboom, E., Heek, J., and Salimans, T. Simple diffusion: End-to-end diffusion for high resolution images. In ICML , 2023.
  22. Kolkin, N., Salavon, J., and Shakhnarovich, G. Style transfer by relaxed optimal transport and self-similarity. In CVPR , pp. 10051-10060, 2019.
  23. Kondratyuk, D., Yu, L., Gu, X., Lezama, J., Huang, J., Hornung, R., Adam, H., Akbari, H., Alon, Y., Birodkar, V., et al. VideoPoet: A large language model for zeroshot video generation. arXiv preprint arXiv:2312.14125 , 2023. · arXiv
  24. Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.-Y ., and Ermon, S. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR , 2022.
  25. Nichol, A. Q. and Dhariwal, P. Improved denoising diffusion probabilistic models. In ICML , pp. 8162-8171, 2021.
  26. Parmar, G., Zhang, R., and Zhu, J.-Y. On aliased resizing and surprising subtleties in gan evaluation. In CVPR , 2022.
  27. Pinkney, J. N. and Adler, D. Resolution dependent GAN interpolation for controllable image synthesis between domains. In Machine Learning for Creativity and Design NeurIPS 2020 Workshop , 2020.
  28. Po, R., Yifan, W., Golyanik, V., Aberman, K., Barron, J. T., Bermano, A. H., Chan, E. R., Dekel, T., Holynski, A., Kanazawa, A., et al. State of the art on diffusion models for visual computing. arXiv preprint arXiv:2310.07204 , 2023. · arXiv
  29. Poole, B., Jain, A., Barron, J. T., and Mildenhall, B. DreamFusion: Text-to-3D using 2D diffusion. In ICLR , 2023.
  30. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125 , 2022. · arXiv
  31. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In CVPR , pp. 10684-10695, 2022.
  32. Ronneberger, O., Fischer, P., and Brox, T. U-Net: Convolutional networks for biomedical image segmentation. In MICCAI , pp. 234-241. Springer, 2015.
  33. Saharia, C., Chan, W., Chang, H., Lee, C., Ho, J., Salimans, T., Fleet, D., and Norouzi, M. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings , pp. 1-10, 2022a.
  34. Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS , 35:36479-36494, 2022b.
  35. Saito, M., Saito, S., Koyama, M., and Kobayashi, S. Train sparsely, generate densely: Memory-efficient unsupervised training of high-resolution temporal GAN. Int. J. Comput. Vision , 128(10-11):2586-2606, 2020.
  36. Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training GANs. NIPS , 29, 2016.
  37. Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., et al. Make-aVideo: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 , 2022. · arXiv
  38. Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML , pp. 2256-2265, 2015.
  39. Sohn, K., Ruiz, N., Lee, K., Chin, D. C., Blok, I., Chang, H., Barber, J., Jiang, L., Entis, G., Li, Y ., et al. StyleDrop: Text-to-image generation in any style. arXiv preprint arXiv:2306.00983 , 2023. · arXiv
  40. Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020. · arXiv
  41. Soomro, K., Zamir, A. R., and Shah, M. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 , 2012. · arXiv
  42. Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., and Paluri, M. A closer look at spatiotemporal convolutions for action recognition. In CVPR , pp. 6450-6459, 2018.
  43. Unterthiner, T., Van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., and Gelly, S. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717 , 2018. · arXiv
  44. Villegas, R., Babaeizadeh, M., Kindermans, P.-J., Moraldo, H., Zhang, H., Saffar, M. T., Castro, S., Kunze, J., and Erhan, D. Phenaki: Variable length video generation from open domain textual description. In ICLR , 2023.
  45. Wang, J., Yuan, H., Chen, D., Zhang, Y., Wang, X., and Zhang, S. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06571 , 2023a. · arXiv
  46. Wang, X., Yuan, H., Zhang, S., Chen, D., Wang, J., Zhang, Y., Shen, Y., Zhao, D., and Zhou, J. Videocomposer: Compositional video synthesis with motion controllability. arXiv preprint arXiv:2306.02018 , 2023b. · arXiv
  47. Wu, C., Liang, J., Ji, L., Yang, F., Fang, Y., Jiang, D., and Duan, N. N¨ uwa: Visual synthesis pre-training for neural visual world creation. In ECCV , pp. 720-736. Springer, 2022.
  48. Zhang, D. J., Wu, J. Z., Liu, J.-W., Zhao, R., Ran, L., Gu, Y., Gao, D., and Shou, M. Z. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. arXiv preprint arXiv:2309.15818 , 2023a. · arXiv
  49. Zhang, L., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 3836-3847, 2023b.
  50. Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR , pp. 586-595, 2018.
  51. Zhou, D., Wang, W., Yan, H., Lv, W., Zhu, Y., and Feng, J. MagicVideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018 , 2022. · arXiv

Cited by

Sorted by citations

Related

Authors
Bar-Tal, O., Chefer, H., Tov, O., Herrmann, C., Paiss, R., Zada, S., Ephrat, A., Hur, J., Liu, G., Raj, A., Li, Y., Rubinstein, M., Michaeli, T., Wang, O., Sun, D., Dekel, T., & Mosseri, I.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2401.12945
arXiv
2401.12945
OpenAlex
W4391212809
Cited by
17
References
0
Citations by year
2025: 3 · 2024: 14
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
Keywords
text-to-video generation, video diffusion models, video generation, temporal coherence, motion generation, image-to-video generation, video inpainting, video editing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review