CoChat
Sign Up
PreprintCited by 16Generative Adversarial Networks and Image SynthesisAdvanced Image Processing TechniquesMultimodal Machine Learning Applications

The DOI matches its DataCite record.

LaMD: Latent Motion Diffusion for Image-Conditional Video Generation

Open access
Hu, Yaosi, Chen, Zhenzhong, Luo, Chong·2023·arXiv
DOI 10.48550/arxiv.2304.11603arXiv 2304.11603
Open this paper in CoChat — free accountOpen PDF

Abstract

The video generation field has witnessed rapid improvements with the introduction of recent diffusion models. While these models have successfully enhanced appearance quality, they still face challenges in generating coherent and natural movements while efficiently sampling videos. In this paper, we propose to condense video generation into a problem of motion generation, to improve the expressiveness of motion and make video generation more manageable. This can be achieved by breaking down the video generation process into latent motion generation and video reconstruction. Specifically, we present a latent motion diffusion (LaMD) framework, which consists of a motion-decomposed video autoencoder and a diffusion-based motion generator, to implement this idea. Through careful design, the motion-decomposed video autoencoder can compress patterns in movement into a concise latent motion representation. Consequently, the diffusion-based motion generator is able to efficiently generate realistic motion on a continuous latent space under multi-modal conditions, at a cost that is similar to that of image diffusion models. Results show that LaMD generates high-quality videos on various benchmark datasets, including BAIR, Landscape, NATOPS, MUG and CATER-GEN, that encompass a variety of stochastic dynamics and highly controllable movements on multiple image-conditional video generation tasks, while significantly decreases sampling time.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2304.11603

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 80

  1. Alemi AA, Fischer I, Dillon JV, Murphy K (2017) Deep variational information bottleneck. In: ICLR 4
  2. Babaeizadeh M, Finn C, Erhan D, Campbell R, Levine S (2018) Stochastic variational video prediction. In: ICLR 3
  3. Babaeizadeh M, Saffar MT, Nair S, Levine S, Finn C, Erhan D (2021) Fitvid: Overfitting in pixel-level video prediction. arXiv preprint arXiv:210613195 7, 11
  4. Blattmann A, Milbich T, Dorkenwald M, Ommer B (2021) Understanding object dynamics for interactive image-to-video synthesis. In: CVPR, pp 5171-5181 3
  5. Blattmann A, Dockhorn T, Kulal S, Mendelevitch D, Kilian M, Lorenz D, Levi Y, English Z, Voleti V, Letts A, et al (2023a) Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:231115127 3
  6. Blattmann A, Rombach R, Ling H, Dockhorn T, Kim SW, Fidler S, Kreis K (2023b) Align your latents: High-resolution video synthesis with latent diffusion models. In: CVPR, pp 22563-22575 3
  7. Chang H, Zhang H, Jiang L, Liu C, Freeman WT (2022) Maskgit: Masked generative image transformer. In: CVPR, pp 11315-11325
  8. Chen H, Xia M, He Y, Zhang Y, Cun X, Yang S, Xing J, Liu Y, Chen Q, Wang X, et al (2023) Videocrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:231019512 3
  9. Chen M, Radford A, Child R, Wu J, Jun H, Luan D, Sutskever I (2020) Generative pretraining from pixels. In: ICML, PMLR, pp 16911703 3
  10. C ¸ ic ¸ek ¨ O, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O (2016) 3d u-net: learning dense volumetric segmentation from sparse annotation. In: MICCAI, Springer, pp 424-432 4
  11. Clark A, Donahue J, Simonyan K (2019) Adversarial video generation on complex datasets. arXiv preprint arXiv:190706571 11
  12. Dhariwal P, Nichol A (2021) Diffusion models beat gans on image synthesis. In: NeurIPS, vol 34, pp 8780-8794 2
  13. Dorkenwald M, Milbich T, Blattmann A, Rombach R, Derpanis KG, Ommer B (2021) Stochastic image-to-video synthesis using cinns. In: CVPR, pp 3741-3752 1, 3, 7, 10, 11
  14. Dosovitskiy A, Brox T (2016a) Generating images with perceptual similarity metrics based on deep networks. In: NeurIPS, p 658-666 5
  15. Dosovitskiy A, Brox T (2016b) Generating images with perceptual similarity metrics based on deep networks. In: NeurIPS, vol 29, pp 658-666 7
  16. Ebert F, Finn C, Lee AX, Levine S (2017) Self-supervised visual planning with temporal skip connections. In: CoRL, pp 344-356 7
  17. Endo Y, Kanamori Y, Kuriyama S (2019) Animating landscape: selfsupervised learning of decoupled motion and appearance for singleimage video synthesis. ACM TOG 38(6):1-19 11
  18. Esser P, Rombach R, Ommer B (2021) Taming transformers for highresolution image synthesis. In: CVPR, pp 12873-12883 2, 3, 5, 6, 9
  19. Fu TJ, Yu L, Zhang N, Fu CY, Su JC, Wang WY, Bell S (2023) Tell me what happened: Unifying text-guided video completion via multimodal masked video generation. In: CVPR, pp 10681-10692 3, 11
  20. Gal R, Vinker Y, Alaluf Y, Bermano AH, Cohen-Or D, Shamir A, Chechik G (2024) Breathing life into sketches using text-to-video priors. In: CVPR 3
  21. Ge S, Hayes T, Yang H, Yin X, Pang G, Jacobs D, Huang JB, Parikh D (2022) Long video generation with time-agnostic vqgan and timesensitive transformer. In: ECCV, pp 102-118 3
  22. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2020) Generative adversarial networks. Communications of the ACM 63(11):139-144 3
  23. Guo Y, Yang C, Rao A, Liang Z, Wang Y, Qiao Y, Agrawala M, Lin D, Dai B (2024) AnimateDiff: Animate your personalized text-toimage diffusion models without specific tuning. In: ICLR 3
  24. Gupta A, Yu L, Sohn K, Gu X, Hahn M, Li FF, Essa I, Jiang L, Lezama J (2024) Photorealistic video generation with diffusion models. In: Leonardis A, Ricci E, Roth S, Russakovsky O, Sattler T, Varol G (eds) ECCV, pp 393-411 3
  25. Han L, Ren J, Lee HY, Barbieri F, Olszewski K, Minaee S, Metaxas D, Tulyakov S (2022a) Show me what and tell me how: Video synthesis via multimodal conditioning. In: CVPR, pp 3615-3625 1
  26. Han L, Ren J, Lee HY, Barbieri F, Olszewski K, Minaee S, Metaxas D, Tulyakov S (2022b) Show me what and tell me how: Video synthesis via multimodal conditioning. In: CVPR, pp 3615-3625 3
  27. Han L, Li Y, Zhang H, Milanfar P, Metaxas D, Yang F (2023) SVDiff: Compact parameter space for diffusion fine-tuning. In: ICCV, pp 7323-7334 3
  28. He Y, Yang T, Zhang Y, Shan Y, Chen Q (2022) Latent video diffusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:221113221 2, 3, 11, 14
  29. Heusel M, Ramsauer H, Unterthiner T, Nessler B, Hochreiter S (2017) Gans trained by a two time-scale update rule converge to a local nash equilibrium. In: NeurIPS, vol 30 7
  30. Ho J, Jain A, Abbeel P (2020) Denoising diffusion probabilistic models. In: NeurIPS, vol 33, pp 6840-6851 2, 3, 6
  31. Ho J, Chan W, Saharia C, Whang J, Gao R, Gritsenko A, Kingma DP, Poole B, Norouzi M, Fleet DJ, et al (2022a) Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:221002303 3
  32. Ho J, Salimans T, Gritsenko AA, Chan W, Norouzi M, Fleet DJ (2022b) Video diffusion models. In: NeurIPS 1, 3, 11, 14
  33. Hong W, Ding M, Zheng W, Liu X, Tang J (2023) Cogvideo: Largescale pretraining for text-to-video generation via transformers. In: ICLR 3
  34. Hu L (2024) Animate anyone: Consistent and controllable image-tovideo synthesis for character animation. In: CVPR, pp 8153-8163 3
  35. Hu Y, Luo C, Chen Z (2022) Make it move: controllable image-tovideo generation with text descriptions. In: CVPR, pp 18219-18228 7, 12
  36. Hu Y, Luo C, Chen Z (2023) A benchmark for controllable text-imageto-video generation. IEEE TMM pp 1-14 8, 11, 12
  37. Huang Z, He Y, Yu J, Zhang F, Si C, Jiang Y, Zhang Y, Wu T, Jin Q, Chanpaisit N, Wang Y, Chen X, Wang L, Lin D, Qiao Y, Liu Z (2024) Vbench: Comprehensive benchmark suite for video generative models. In: CVPR, pp 21807-21818 7
  38. Kingma DP, Welling M (2014) Auto-encoding variational bayes. In: ICLR 3
  39. Lee D, Kim C, Kim S, Cho M, Han WS (2022) Autoregressive image generation using residual quantization. In: CVPR, pp 11523-11532 3
  40. Li Y, Fang C, Yang J, Wang Z, Lu X, Yang MH (2018) Flow-grounded spatial-temporal video prediction from still images. In: ECCV, p 609-625 1
  41. Logacheva E, Suvorov R, Khomenko O, Mashikhin A, Lempitsky V (2020) Deeplandscape: Adversarial modeling of landscape videos. In: ECCV, Springer, pp 256-272 11
  42. Menapace W, Lathuili` ere S, Tulyakov S, Siarohin A, Ricci E (2021) Playable video generation. In: CVPR, pp 10061-10070 3
  43. Ni H, Shi C, Li K, Huang SX, Min MR (2023) Conditional image-tovideo generation with latent flow diffusion models. In: CVPR, pp 18444-18455 4, 7, 8, 10, 11, 12, 14
  44. Ni H, Egger B, Lohit S, Cherian A, Wang Y, Koike-Akino T, Huang SX, Marks TK (2024) Ti2v-zero: Zero-shot image conditioning for text-to-video diffusion models. In: CVPR, pp 9015-9025 3, 7, 8, 11, 12
  45. Nichol AQ, Dhariwal P (2021) Improved denoising diffusion probabilistic models. In: ICML, pp 8162-8171 6
  46. van den Oord A, Vinyals O, kavukcuoglu k (2017) Neural discrete representation learning. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R (eds) NeurIPS, vol 30 3
  47. Ramesh A, Pavlov M, Goh G, Gray S, Voss C, Radford A, Chen M, Sutskever I (2021a) Zero-shot text-to-image generation. In: ICML, pp 8821-8831 2
  48. Ramesh A, Pavlov M, Goh G, Gray S, Voss C, Radford A, Chen M, Sutskever I (2021b) Zero-shot text-to-image generation. In: ICML, vol 139, pp 8821-8831 3
  49. Ramesh A, Dhariwal P, Nichol A, Chu C, Chen M (2022) Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:220406125 1(2):3 3
  50. Razavi A, van den Oord A, Vinyals O (2019) Generating diverse highfidelity images with vq-vae-2. In: NeurIPS 3
  51. Rombach R, Blattmann A, Lorenz D, Esser P, Ommer B (2022) Highresolution image synthesis with latent diffusion models. In: CVPR, pp 10684-10695 2, 3, 6, 7, 11, 12, 14
  52. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: MICCAI, Springer, pp 234-241 4
  53. Saito M, Saito S, Koyama M, Kobayashi S (2020) Train sparsely, generate densely: Memory-efficient unsupervised training of highresolution temporal gan. IJCV 128(10-11):2586-2606 3
  54. Shen S, Zhao W, Meng Z, Li W, Zhu Z, Zhou J, Lu J (2023) Difftalk: Crafting diffusion models for generalized talking head synthesis. arXiv preprint arXiv:230103786 2
  55. Sheng L, Pan J, Guo J, Shao J, Loy CC (2020) High-quality video generation from static structural annotations. IJCV 128:2552-2569 3
  56. Shi X, Huang Z, Wang FY, Bian W, Li D, Zhang Y, Zhang M, Cheung KC, See S, Qin H, Dai J, Li H (2024) Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling. In: ACM SIGGRAPH, New York, NY, USA, SIGGRAPH '24, DOI 10.1145/3641519.3657497 4 · DOI
  57. Shrivastava G, Shrivastava A (2021) Diverse video generation using a gaussian process trigger. In: ICLR 3
  58. Singer U, Polyak A, Hayes T, Yin X, An J, Zhang S, Hu Q, Yang H, Ashual O, Gafni O, Parikh D, Gupta S, Taigman Y (2023) Makea-video: Text-to-video generation without text-video data. In: ICLR 3
  59. Sohl-Dickstein J, Weiss E, Maheswaranathan N, Ganguli S (2015) Deep unsupervised learning using nonequilibrium thermodynamics. In: ICML, pp 2256-2265 6
  60. Song Y, Demirdjian D, Davis R (2011) Tracking body and hands for gesture recognition: Natops aircraft handling signals database. In: IEEE FG, pp 500-506 7
  61. Sun M, Wang W, Zhu X, Liu J (2023) Moso: Decomposing motion, scene and object for video prediction. In: CVPR, pp 18727-18737 11
  62. Tulyakov S, Liu MY, Yang X, Kautz J (2018) Mocogan: Decomposing motion and content for video generation. In: CVPR, pp 1526-1535 3
  63. Unterthiner T, van Steenkiste S, Kurach K, Marinier R, Michalski M, Gelly S (2018) Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:181201717 7
  64. Villegas R, Babaeizadeh M, Kindermans PJ, Moraldo H, Zhang H, Saffar MT, Castro S, Kunze J, Erhan D (2023) Phenaki: Variable length video generation from open domain textual descriptions. In: ICLR 3
  65. Walker J, Razavi A, Oord Avd (2021) Predicting video with vqvae. arXiv preprint arXiv:210301950 3
  66. Wallace GK (1991) The jpeg still picture compression standard. Communications of the ACM 34(4):30-44 3
  67. Wang Y, Bilinski P, Bremond F, Dantcheva A (2020) G3an: Disentangling appearance and motion for video generation. In: CVPR, pp 5264-5273 3
  68. Weissenborn D, T¨ ackstr¨ om O, Uszkoreit J (2019) Scaling autoregressive video models. In: ICLR 7, 11
  69. Wu C, Liang J, Ji L, Yang F, Fang Y, Jiang D, Duan N (2022) N¨ uwa: Visual synthesis pre-training for neural visual world creation. In: ECCV, Springer, pp 720-736 3, 11
  70. Xing J, Xia M, Zhang Y, Chen H, Yu W, Liu H, Liu G, Wang X, Shan Y, Wong TT (2024) Dynamicrafter: Animating open-domain images with video diffusion priors. In: Leonardis A, Ricci E, Roth S, Russakovsky O, Sattler T, Varol G (eds) ECCV, pp 399-417 3, 10, 11, 12
  71. Xiong W, Luo W, Ma L, Liu W, Luo J (2018) Learning to generate time-lapse videos using multi-stage dynamic generative adversarial networks. In: CVPR, pp 2364-2373 7, 11
  72. Xu X, Wang Y, Wang L, Yu B, Jia J (2023) Conditional temporal variational autoencoder for action video prediction. IJCV pp 1-24 3
  73. Yan W, Zhang Y, Abbeel P, Srinivas A (2021) Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:210410157 1, 3, 7, 11
  74. Yu J, Li X, Koh JY, Zhang H, Pang R, Qin J, Ku A, Xu Y, Baldridge J, Wu Y (2022) Vector-quantized image modeling with improved VQGAN. In: ICLR 3
  75. Zeng Y, Wei G, Zheng J, Zou J, Wei Y, Zhang Y, Li H (2024) Make pixels dance: High-dynamic video generation. In: CVPR, pp 88508860 3
  76. Zhang J, Xu C, Liu L, Wang M, Wu X, Liu Y, Jiang Y (2020) Dtvnet: Dynamic time-lapse video generation via single still image. In: ECCV, Springer, pp 300-315 3, 7, 11
  77. Zhang S, Wang J, Zhang Y, Zhao K, Yuan H, Qin Z, Wang X, Zhao D, Zhou J (2023) I2VGen-XL: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint arXiv:231104145 3
  78. Zheng C, Vuong LT, Cai J, Phung D (2022) MoVQ: Modulating quantized vectors for high-fidelity image generation. In: NeurIPS 3
  79. Zhou D, Wang W, Yan H, Lv W, Zhu Y, Feng J (2022) Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:221111018 2
  80. Zhu J, Yang H, He H, Wang W, Tuo Z, Cheng WH, Gao L, Song J, Fu J (2023) MovieFactory: Automatic movie creation from text using large generative models for language and images. In: ACM MM, p 9313-9319 3

Cited by

Sorted by citations

Metadata

Authors
Hu, Y., Chen, Z., & Luo, C.
Affiliations
  • Wuhan University (CN)
  • Microsoft Research Asia (China) (CN)
Corresponding author
Chen, Zhenzhong
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Volume
133
Issue
7
Pages
4384-4400
ISSN
0920-5691 (ISSN-L)
DOI
10.48550/arxiv.2304.11603
arXiv
2304.11603
OpenAlex
W4408108503
Cited by
16
References
0
Citations by year
2026: 8 · 2025: 3 · 2024: 5
FWCI
3.89
Type
journal article
Language
en
Open access
Licence
Locations
  • International Journal of Computer Vision (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
National Natural Science Foundation of China (62036005)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
motion generation, video reconstruction, diffusion models, motion representation, sampling efficiency, controllable video generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review