CoChat
Sign Up
PreprintCited by 7Generative Adversarial Networks and Image SynthesisMultimodal Machine Learning ApplicationsVideo Analysis and Summarization

The DOI matches its DataCite record.

VideoTetris: Towards Compositional Text-to-Video Generation

Open access
Tian, Ye, Yang, Ling, Yang, Haotian, Gao, Yuan, Deng, Yufan, Chen, Jingmin, Wang, Xintao, Yu, Zhaochen, Tao, Xin, Wan, Pengfei, Zhang, Di, Cui, Bin·2024·arXiv
DOI 10.48550/arxiv.2406.04277arXiv 2406.04277
Open this paper in CoChat — free accountOpen PDF

Abstract

Diffusion models have demonstrated great success in text-to-video (T2V) generation. However, existing methods may face challenges when handling complex (long) video generation scenarios that involve multiple objects or dynamic changes in object numbers. To address these limitations, we propose VideoTetris, a novel framework that enables compositional T2V generation. Specifically, we propose spatio-temporal compositional diffusion to precisely follow complex textual semantics by manipulating and composing the attention maps of denoising networks spatially and temporally. Moreover, we propose an enhanced video data preprocessing to enhance the training data regarding motion dynamics and prompt understanding, equipped with a new reference frame attention mechanism to improve the consistency of auto-regressive video generation. Extensive experiments demonstrate that our VideoTetris achieves impressive qualitative and quantitative results in compositional T2V generation. Code is available at: https://github.com/YangLing0818/VideoTetris

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2406.04277

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 51

  1. J. Ho, A. Jain, and P. Abbeel, "Denoising diffusion probabilistic models," Advances in neural information processing systems, vol. 33, pp. 6840-6851, 2020.
  2. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, "Score-based generative modeling through stochastic differential equations," arXiv preprint arXiv:2011.13456, 2020. · arXiv
  3. J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, "Deep unsupervised learning using nonequilibrium thermodynamics," in International conference on machine learning, pp. 2256-2265, PMLR, 2015.
  4. H. Chen, Y. Zhang, X. Cun, M. Xia, X. Wang, C. Weng, and Y. Shan, "Videocrafter2: Overcoming data limitations for high-quality video diffusion models," arXiv preprint arXiv:2401.09047, 2024. · arXiv
  5. Y. Guo, C. Yang, A. Rao, Z. Liang, Y. Wang, Y. Qiao, M. Agrawala, D. Lin, and B. Dai, "Animatediff: An- imate your personalized text-to-image diffusion models without specific tuning," International Conference on Learning Representations, 2024.
  6. X. Ma, Y. Wang, G. Jia, X. Chen, Z. Liu, Y.-F. Li, C. Chen, and Y. Qiao, "Latte: Latent diffusion transformer for video generation," arXiv preprint arXiv:2401.03048, 2024. · arXiv
  7. J. Wang, H. Yuan, D. Chen, Y. Zhang, X. Wang, and S. Zhang, "Modelscope text-to-video technical report," arXiv preprint arXiv:2308.06571, 2023. · arXiv
  8. G. Oh, J. Jeong, S. Kim, W. Byeon, J. Kim, S. Kim, H. Kwon, and S. Kim, "Mtvg: Multi-text video generation with text-to-video models," arXiv preprint arXiv:2312.04086, 2023. · arXiv
  9. S. Zhuang, K. Li, X. Chen, Y. Wang, Z. Liu, Y. Qiao, and Y. Wang, "Vlogger: Make your dream a vlog," arXiv preprint arXiv:2401.09414, 2024. · arXiv
  10. H. Qiu, M. Xia, Y. Zhang, Y. He, X. Wang, Y. Shan, and Z. Liu, "Freenoise: Tuning-free longer video diffusion via noise rescheduling," arXiv preprint arXiv:2310.15169, 2023. · arXiv
  11. R. Henschel, L. Khachatryan, D. Hayrapetyan, H. Poghosyan, V. Tadevosyan, Z. Wang, S. Navasardyan, and H. Shi, "Streamingt2v: Consistent, dynamic, and extendable long video generation from text," arXiv preprint arXiv:2403.14773, 2024. · arXiv
  12. L. Yang, Z. Zhang, Y. Song, S. Hong, R. Xu, Y. Zhao, W. Zhang, B. Cui, and M.-H. Yang, "Diffusion models: A comprehensive survey of methods and applications," ACM Computing Surveys, vol. 56, no. 4, pp. 1-39, 2023.
  13. L. Yang, Z. Zhang, Z. Yu, J. Liu, M. Xu, S. Ermon, and B. CUI, "Cross-modal contextualized diffu- sion models for text-guided visual generation and editing," in International Conference on Learning Representations, 2024.
  14. Y. Wang, Y. He, Y. Li, K. Li, J. Yu, X. Ma, X. Li, G. Chen, X. Chen, Y. Wang, P. Luo, Z. Liu, Y. Wang, L. Wang, and Y. Qiao, "Internvid: A large-scale video-text dataset for multimodal understanding and generation," in The Twelfth International Conference on Learning Representations, 2024.
  15. T.-S. Chen, A. Siarohin, W. Menapace, E. Deyneka, H.-w. Chao, B. E. Jeon, Y. Fang, H.-Y. Lee, J. Ren, M.-H. Yang, et al., "Panda-70m: Captioning 70m videos with multiple cross-modality teachers," arXiv preprint arXiv:2402.19479, 2024. · arXiv
  16. J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet, "Video diffusion models," Advances in Neural Information Processing Systems, vol. 35, pp. 8633-8646, 2022.
  17. O. Bar-Tal, H. Chefer, O. Tov, C. Herrmann, R. Paiss, S. Zada, A. Ephrat, J. Hur, Y. Li, T. Michaeli, et al., "Lumiere: A space-time diffusion model for video generation," arXiv preprint arXiv:2401.12945, 2024. · arXiv
  18. A. Gupta, L. Yu, K. Sohn, X. Gu, M. Hahn, L. Fei-Fei, I. Essa, L. Jiang, and J. Lezama, "Photorealistic video generation with diffusion models," arXiv preprint arXiv:2312.06662, 2023. · arXiv
  19. F.-Y. Wang, W. Chen, G. Song, H.-J. Ye, Y. Liu, and H. Li, "Gen-l-video: Multi-text to long video generation via temporal co-denoising," arXiv preprint arXiv:2305.18264, 2023. · arXiv
  20. Y. Guo, C. Yang, A. Rao, M. Agrawala, D. Lin, and B. Dai, "Sparsectrl: Adding sparse controls to text-to-video diffusion models," arXiv preprint arXiv:2311.16933, 2023. · arXiv
  21. L. Zhang, A. Rao, and M. Agrawala, "Adding conditional control to text-to-image diffusion models," in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023.
  22. L. Yang, Z. Yu, C. Meng, M. Xu, S. Ermon, and B. Cui, "Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal llms," in International Conference on Machine Learning, 2024.
  23. Y. Li, H. Liu, Q. Wu, F. Mu, J. Yang, J. Gao, C. Li, and Y. J. Lee, "Gligen: Open-set grounded text-to-image generation," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22511-22521, 2023.
  24. J. Xie, Y. Li, Y. Huang, H. Liu, W. Zhang, Y. Zheng, and M. Z. Shou, "Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion," in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7452-7461, 2023.
  25. L. Lian, B. Li, A. Yala, and T. Darrell, "Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models," arXiv preprint arXiv:2305.13655, 2023. · arXiv
  26. L. Qu, S. Wu, H. Fei, L. Nie, and T.-S. Chua, "Layoutllm-t2i: Eliciting layout guidance from llm for text-to-image generation," in Proceedings of the 31st ACM International Conference on Multimedia, pp. 643-654, 2023.
  27. L. Lian, B. Shi, A. Yala, T. Darrell, and B. Li, "Llm-grounded video diffusion models," in The Twelfth International Conference on Learning Representations, 2023.
  28. H. Lin, A. Zala, J. Cho, and M. Bansal, "Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning," arXiv preprint arXiv:2309.15091, 2023. · arXiv
  29. N. Liu, S. Li, Y. Du, A. Torralba, and J. B. Tenenbaum, "Compositional visual generation with composable diffusion models," in European Conference on Computer Vision, pp. 423-439, Springer, 2022.
  30. R. Wang, Z. Chen, C. Chen, J. Ma, H. Lu, and X. Lin, "Compositional text-to-image synthesis with attention map control of diffusion models," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 5544-5552, 2024.
  31. H. Chefer, Y. Alaluf, Y. Vinker, L. Wolf, and D. Cohen-Or, "Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models," ACM Transactions on Graphics (TOG), vol. 42, no. 4, pp. 1-10, 2023.
  32. W. Feng, X. He, T.-J. Fu, V. Jampani, A. Akula, P. Narayana, S. Basu, X. E. Wang, and W. Y. Wang, "Training-free structured diffusion guidance for compositional text-to-image synthesis," arXiv preprint arXiv:2212.05032, 2022. · arXiv
  33. J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al., "Chain-of-thought prompting elicits reasoning in large language models," Advances in neural information processing systems, vol. 35, pp. 24824-24837, 2022.
  34. A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. English, V. Voleti, A. Letts, et al., "Stable video diffusion: Scaling latent video diffusion models to large datasets," arXiv preprint arXiv:2311.15127, 2023. · arXiv
  35. G. Farnebäck, "Two-frame motion estimation based on polynomial expansion," in Image Analysis: 13th Scandinavian Conference, SCIA 2003 Halmstad, Sweden, June 29-July 2, 2003 Proceedings 13, pp. 363- 370, Springer, 2003.
  36. Z. Teed and J. Deng, "Raft: Recurrent all-pairs field transforms for optical flow," in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 402-419, Springer, 2020.
  37. E. Segalis, D. Valevski, D. Lumen, Y. Matias, and Y. Leviathan, "A picture is worth a thousand words: Principled recaptioning improves image generation," arXiv preprint arXiv:2310.16656, 2023. · arXiv
  38. W. Zheng, J. Teng, Z. Yang, W. Wang, J. Chen, X. Gu, Y. Dong, M. Ding, and J. Tang, "Cogview3: Finer and faster text-to-image generation via relay diffusion," arXiv preprint arXiv:2403.05121, 2024. · arXiv
  39. H. Ye, J. Zhang, S. Liu, X. Han, and W. Yang, "Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models," arXiv preprint arXiv:2308.06721, 2023. · arXiv
  40. Q. Wang, X. Bai, H. Wang, Z. Qin, and A. Chen, "Instantid: Zero-shot identity-preserving generation in seconds," arXiv preprint arXiv:2401.07519, 2024. · arXiv
  41. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., "Learning transferable visual models from natural language supervision," in International conference on machine learning, pp. 8748-8763, PMLR, 2021.
  42. T. Unterthiner, S. Van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly, "Towards accurate generative models of video: A new metric & challenges," arXiv preprint arXiv:1812.01717, 2018. · arXiv
  43. K. Huang, K. Sun, E. Xie, Z. Li, and X. Liu, "T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation," Advances in Neural Information Processing Systems, vol. 36, pp. 78723-78747, 2023.
  44. J. Li, D. Li, C. Xiong, and S. Hoi, "Blip: Bootstrapping language-image pre-training for unified vision- language understanding and generation," in International conference on machine learning, pp. 12888- 12900, PMLR, 2022.
  45. X. Zhou, V. Koltun, and P. Krähenbühl, "Simple multi-dataset detection," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7571-7580, 2022.
  46. C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al., "Laion-5b: An open large-scale dataset for training next generation image-text models," Advances in Neural Information Processing Systems, vol. 35, pp. 25278-25294, 2022.
  47. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al., "Gpt-4 technical report," arXiv preprint arXiv:2303.08774, 2023. · arXiv
  48. H. Liu, W. Yan, M. Zaharia, and P. Abbeel, "World model on million-length video and language with ringattention," arXiv preprint arXiv:2402.08268, 2024. · arXiv
  49. H. Zhang, X. Li, and L. Bing, "Video-llama: An instruction-tuned audio-visual language model for video understanding," arXiv preprint arXiv:2306.02858, 2023. · arXiv
  50. M. Maaz, H. Rasheed, S. Khan, and F. S. Khan, "Video-chatgpt: Towards detailed video understanding via large vision and language models," arXiv preprint arXiv:2306.05424, 2023. · arXiv
  51. B. Lin, B. Zhu, Y. Ye, M. Ning, P. Jin, and L. Yuan, "Video-llava: Learning united visual representation by alignment before projection," arXiv preprint arXiv:2311.10122, 2023. · arXiv

Cited by

Sorted by citations

Metadata

Authors
Tian, Y., Yang, L., Yang, H., Gao, Y., Deng, Y., Chen, J., Wang, X., Yu, Z., Tao, X., Wan, P., Zhang, D., & Cui, B.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Pages
29489-29513
ISSN
1049-5258 (ISSN-L)
DOI
10.48550/arxiv.2406.04277
arXiv
2406.04277
OpenAlex
W4415798416
Cited by
7
References
0
Citations by year
2026: 4 · 2025: 3
FWCI
1.45
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
Keywords
text-to-video generation, video editing, video summarization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review