GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions
Open accessAbstract
Generating videos from text is a challenging task due to its high computational requirements for training and infinite possible answers for evaluation. Existing works typically experiment on simple or small datasets, where the generalization ability is quite limited. In this work, we propose GODIVA, an open-domain text-to-video pretrained model that can generate videos from text in an auto-regressive manner using a three-dimensional sparse attention mechanism. We pretrain our model on Howto100M, a large-scale text-video dataset that contains more than 136 million text-video pairs. Experiments show that GODIVA not only can be fine-tuned on downstream video generation tasks, but also has a good zero-shot capability on unseen texts. We also propose a new metric called Relative Matching (RM) to automatically evaluate the video generation quality. Several challenges are listed and discussed as future work.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · see citing works
Backpropagation Applied to Handwritten Zip Code Recognition
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891 · see citing works
Large Scale GAN Training for High Fidelity Natural Image Synthesis
Brock, A., Donahue, J., & Simonyan, K. (2018) · arXiv · doi:10.48550/arxiv.1809.11096 · Cited by 3,717 · see citing works
The Kinetics Human Action Video DatasetPreprint
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.06950 · Cited by 3,711 · see citing works
Self-Attention Generative Adversarial Networks
Zhang, H., Goodfellow, I., Metaxas, D., & Odena, A. (2018) · arXiv · doi:10.48550/arxiv.1805.08318 · Cited by 3,353 · see citing works
Neural Discrete Representation Learning
Oord, A. V. D., Vinyals, O., & Kavukcuoglu, K. (2017) · arXiv · doi:10.48550/arxiv.1711.00937 · Cited by 2,729 · see citing works
Zero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417 · see citing works
MSR-VTT: A Large Video Description Dataset for Bridging Video and Language
Xu, J., Mei, T., Yao, T., & Rui, Y. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.571 · Cited by 1,810 · see citing works
MoCoGAN: Decomposing Motion and Content for Video Generation
Tulyakov, S., Liu, M.-Y., Yang, X., & Kautz, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00165 · Cited by 1,076 · see citing works
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
Miech, A., Zhukov, D., Alayrac, J.-B., Tapaswi, M., Laptev, I., & Sivic, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00272 · Cited by 986 · see citing works
Generating Videos with Scene DynamicsPreprint
Vondrick, C., Pirsiavash, H., & Torralba, A. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.02612 · Cited by 976 · see citing works
Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
Wen, T.-H., Gašić, M., Mrkšić, N., Su, P.-H., Vandyke, D., & Young, S. J. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1199 · Cited by 876 · see citing works
Creativity and artificial intelligence
Boden, M. A. (1998) · Artificial Intelligence · doi:10.1016/s0004-3702(98)00055-1 · Cited by 846 · see citing works
Deep Predictive Coding Networks for Video Prediction and Unsupervised LearningPreprint
Lotter, W. E., Kreiman, G., & Cox, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08104 · Cited by 805 · see citing works
Unsupervised Learning for Physical Interaction through Video Prediction
Finn, C., Goodfellow, I., & Levine, S. (2016) · arXiv · doi:10.48550/arxiv.1605.07157 · Cited by 759 · see citing works
PredRNN: recurrent neural networks for predictive learning using spatiotemporal LSTMs
Wang, Y., Long, M., Wang, J., Gao, Z., & Yu, P. S. (2017) · neural information processing systems · Cited by 734 · see citing works
Generating diverse high-fidelity images with vq-vae-2
Razavi, A., Oord, A. V. D., & Vinyals, O. (2019) · arXiv · doi:10.48550/arxiv.1906.00446 · Cited by 709 · see citing works
Stochastic Video Generation with a Learned PriorPreprint
Denton, E., & Fergus, R. (2018) · International Conference on Machine Learning · doi:10.48550/arxiv.1802.07687 · Cited by 599 · see citing works
Temporal Generative Adversarial Nets with Singular Value Clipping
Saito, M., Matsumoto, E., & Saito, S. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.308 · Cited by 412 · see citing works
Stochastic Variational Video PredictionPreprint
Babaeizadeh, M., Finn, C., Erhan, D., Campbell, R. H., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.11252 · Cited by 351 · see citing works
Neural Paraphrase Generation with Stacked Residual LSTM NetworksPreprint
Prakash, A., Hasan, S. A., Lee, K., Datla, V. V., Qadir, A., Liu, J., & Farri, O. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.03098 · Cited by 245 · see citing works
ContextVP: Fully Context-Aware Video Prediction
Byeon, W., Wang, Q., Srivastava, R. K., & Koumoutsakos, P. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01270-0_46 · Cited by 145 · see citing works
Pan, Y., Qiu, Z., Yao, T., Li, H., & Mei, T. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3127905 · Cited by 124 · see citing works
Conditional GAN with Discriminative Filter Generation for Text-to-Video Synthesis
Balaji, Y., Min, M. R., Bai, B., Chellappa, R., & Graf, H. P. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/276 · Cited by 123 · see citing works
Saito, M., Saito, S., Koyama, M., & Kobayashi, S. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-020-01333-y · Cited by 116 · see citing works
Rakhimov, R., Volkhonskiy, D., Artemov, A., Zorin, D., & Burnaev, E. V. (2021) · International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP) · doi:10.5220/0010241801010112 · Cited by 85 · see citing works
Mittal, G., Marwah, T., & Balasubramanian, V. N. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123309 · Cited by 58 · see citing works
IRC-GAN: Introspective Recurrent Convolutional GAN for Text-to-video Generation
Deng, K., Fei, T., Huang, X., & Peng, Y. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/307 · Cited by 49 · see citing works
Predicting Video with VQVAEPreprint
Walker, J., Razavi, A. S., & van den Oord, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.01950 · Cited by 29 · see citing works
VideoGen: Generative Modeling of Videos using VQ-VAE and Transformers
Zhang, Y., Yan, W., Abbeel, P., & Srinivas, A. (2021) · Cited by 2 · see citing works
Flagged first, then by citations
1 of the 31 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- [11][11] Yitong Li, Martin Min, Dinghan Shen, David Carlson, and Lawrence Carin. Video generation from text. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018. -
- [14][14] Gaurav Mittal, Tanya Marwah, and Vineeth N. Balasubramanian. Sync-draw: Automatic video generation using deep recurrent attentive architectures. In Proceedings of the 25th ACM International Conference on Multimedia , pages 1096-1104, 2017. -
- [17][17] Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. MirrorGAN: Learning Text-toimage Generation by Redescription. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1505-1514, 2019. -
- [19][19] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog , 1(8), 2019. -
Cited by
Sorted by citations
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models
Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S., & Kreis, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02161 · Cited by 724 · see citing works
VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models
Chen, H., Zhang, Y., Cun, X., Xia, M., Wang, X., Weng, C., & Shan, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00698 · Cited by 257 · see citing works
NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion
Wu, C., Liang, J., Ji, L., Yang, F., Fang, Y., Jiang, D., & Duan, N. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19787-1_41 · Cited by 167 · see citing works
Self-citationVideo-P2P: Video Editing with Cross-Attention Control
Liu, S., Zhang, Y., Li, W., Lin, Z., & Jia, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00821 · Cited by 158 · see citing works
Follow Your Pose: Pose-Guided Text-to-Video Generation Using Pose-Free Videos
Ma, Y., He, Y., Cun, X., Wang, X., Chen, S., Li, X., & Chen, Q. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28206 · Cited by 153 · see citing works
Conditional Image-to-Video Generation with Latent Flow Diffusion Models
Ni, H., Shi, C., Li, K., Huang, X., & Min, M. R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01769 · Cited by 127 · see citing works
Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers
Chen, T.-S., Siarohin, A., Menapace, W., Deyneka, E., Chao, H.-W., Jeon, B. E., Fang, Y., Lee, H.-Y., Ren, J. F., Yang, M.-H., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01265 · Cited by 102 · see citing works
Phenaki: Variable Length Video Generation From Open Domain Textual Description
Villegas, R., Babaeizadeh, M., Kindermans, P.-J., Moraldo, H., Zhang, H., Saffar, M. T., Castro, S., Kunze, J., & Erhan, D. (2022) · arXiv · doi:10.48550/arxiv.2210.02399 · Cited by 92 · see citing works
“We compare C-ViViT against per-frame image based encoder-decoders that have been used as video quantizers for conditional video generation [57, 54, 18, 54, 18, 52]: a ViT [58] and a convolutional VQ-GAN[12].”
MethodHumanSD: A Native Skeleton-Guided Diffusion Model for Human Image Generation
Ju, X., Zeng, A., Zhao, C., Wang, J., Zhang, L., & Xu, Q. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01465 · Cited by 84 · see citing works
Valle-Pérez, G., Henter, G. E., Beskow, J., Holzapfel, A., Oudeyer, P., & Alexanderson, S. (2021) · ACM Transactions on Graphics · doi:10.1145/3478513.3480570 · Cited by 78 · see citing works
Make It Move: Controllable Image-to-Video Generation with Text Descriptions
Hu, Y., Luo, C., & Chen, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01768 · Cited by 62 · see citing works
NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation
Yin, S., Wu, C., Yang, H., Wang, J., Wang, X., Ni, M., Yang, Z., Li, L., Liu, S., Yang, F., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.73 · Cited by 68 · see citing works
Self-citationCelebV-Text: A Large-Scale Facial Text-Video Dataset
Yu, J., Zhu, H., Jiang, L., Loy, C. C., Cai, W., & Wu, W. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01422 · Cited by 53 · see citing works
A Survey of AI Text-to-Image and AI Text-to-Video Generators
Singh, A. (2023) · International Conference on Artificial Intelligence, Robotics and Control (AIRC) · doi:10.1109/airc57904.2023.10303174 · Cited by 53 · see citing works
Make-A-Story: Visual Memory Conditioned Consistent Story Generation
Rahman, T., Lee, H. Y., Ren, J., Tulyakov, S., Mahajan, S., & Sigal, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00246 · Cited by 54 · see citing works
SimDA: Simple Diffusion Adapter for Efficient Video Generation
Xing, Z., Dai, Q., Hu, H., Wu, Z., & Jiang, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00748 · Cited by 59 · see citing works
Make Pixels Dance: High-Dynamic Video Generation
Zeng, Y., Wei, G., Zheng, J., Zou, J., Yang, W., Zhang, Y., & Li, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00845 · Cited by 59 · see citing works
VideoXum: Cross-Modal Visual and Textural Summarization of Videos
Lin, J., Hua, H., Chen, M., Li, Y., Hsiao, J.-H., Ho, C., & Luo, J. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3335875 · Cited by 48 · see citing works
SGDM: An Adaptive Style-Guided Diffusion Model for Personalized Text to Image Generation
Xu, Y., Xu, X., Gao, H., & Xiao, F. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3399075 · Cited by 43 · see citing works
Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation
Yariv, G., Gat, I., Benaim, S., Wolf, L., Schwartz, I., & Adi, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i7.28486 · Cited by 46 · see citing works
AVID: Any-Length Video Inpainting with Diffusion Model
Zhang, Z., Wu, B., Wang, X., Luo, Y., Zhang, L., Zhao, Y., Vajda, P., Metaxas, D., & Yu, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00684 · Cited by 43 · see citing works
Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis
Menapace, W., Siarohin, A., Skorokhodov, I., Deyneka, E., Chen, T.-S., Kag, A., Fang, Y., Stoliar, A., Ricci, E., Ren, J., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00672 · Cited by 42 · see citing works
Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioning
Han, L., Ren, J., Lee, H.-Y., Barbieri, F., Olszewski, K., Minaee, S., Metaxas, D., & Tulyakov, S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00360 · Cited by 38 · see citing works
FreeInit: Bridging Initialization Gap in Video Diffusion Models
Wu, T., Si, C., Jiang, Y., Huang, Z., & Liu, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72646-0_22 · Cited by 31 · see citing works
Vlogger: Make Your Dream A Vlog
Zhuang, S., Li, K., Chen, X., Wang, Y., Liu, Z., Qiao, Y., & Wang, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00841 · Cited by 32 · see citing works
Breathing Life Into Sketches Using Text-to-Video Priors
Gal, R., Vinker, Y., Alaluf, Y., Bermano, A. H., Cohen‐Or, D., Shamir, A., & Chechik, G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00414 · Cited by 35 · see citing works
On the Content Bias in Fréchet Video Distance
Ge, S., Mahapatra, A., Parmar, G., Zhu, J.-Y., & Huang, J.-B. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00695 · Cited by 31 · see citing works
FitVid: Overfitting in Pixel-Level Video PredictionPreprint
Babaeizadeh, M., Saffar, M., Nair, S., Levine, S., Finn, C., & Erhan, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.13195 · Cited by 39 · see citing works
TA2V: Text-Audio Guided Video Generation
Zhao, M., Wang, W., Chen, T., Zhang, R., & Li, R. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3362149 · Cited by 26 · see citing works
ART•V: Auto-Regressive Text-to-Video Generation with Diffusion Models
Weng, W., Feng, R., Wang, Y., Dai, Q., Wang, C., Yin, D., Zhao, Z., Qiu, K., Bao, J., Yuan, Y., et al. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00735 · Cited by 26 · see citing works
Complementarity-Aware Space Learning for Video-Text Retrieval
Zhu, J., Zeng, P., Gao, L., Li, G., Liao, D., & Song, J. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3235523 · Cited by 25 · see citing works
Hierarchical Spatio-temporal Decoupling for Text-to- Video Generation
Qing, Z., Zhang, S., Wang, J., Wang, X., Wei, Y., Zhang, Y., Gao, C., & Sang, N. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00634 · Cited by 28 · see citing works
Tell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generation
Fu, T.-J., Yu, L., Zhang, N., Fu, C.-Y., Su, J.-C., Wang, W. Y., & Bell, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01029 · Cited by 24 · see citing works
InstructVideo: Instructing Video Diffusion Models with Human Feedback
Yuan, H., Zhang, S., Wang, X., Wei, Y., Feng, T., Pan, Y., Zhang, Y., Liu, Z., Albanie, S., & Ni, D. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00618 · Cited by 24 · see citing works
TI2V-Zero: Zero-Shot Image Conditioning for Text-to-Video Diffusion Models
Ni, H., Egger, B., Lohit, S., Cherian, A., Wang, Y., Koike–Akino, T., Huang, X., & Marks, T. K. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00861 · Cited by 20 · see citing works
Foundation Models for Video Understanding: A SurveyPreprint
Madan, N., Møgelmose, A., Modi, R., Rawat, Y. S., & Moeslund, T. B. (2024) · doi:10.36227/techrxiv.171769139.99464428/v1 · Cited by 18 · see citing works
MOSO: Decomposing MOtion, Scene and Object for Video Prediction
Sun, M., Wang, N., Zhu, X., & Liu, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01796 · Cited by 17 · see citing works
Mirasol3B: A Multimodal Autoregressive Model for Time-Aligned and Contextual Modalities
Piergiovanni, A., Noble, I., Kim, D., Ryoo, M. S., Gomes, V., & Angelova, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02531 · Cited by 17 · see citing works
A Benchmark for Controllable Text -Image-to-Video Generation
Hu, Y., Luo, C., & Chen, Z. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3284989 · Cited by 17 · see citing works
MagicTime: Time-Lapse Video Generation Models as Metamorphic Simulators
Yuan, S., Huang, J., Shi, Y., Xu, Y., Zhu, R., Lin, B., Cheng, X., Yuan, L., & Luo, J. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3558507 · Cited by 22 · see citing works
Xie, W., Hu, A., Xie, Q., Chen, J., Wan, R., & Liu, Y. (2025) · Discover Computing · doi:10.1007/s10791-025-09628-9 · Cited by 15 · see citing works
AssistGUI: Task-Oriented PC Graphical User Interface Automation
Gao, D., Ji, L., Bai, Z., Ouyang, M., Li, P., Mao, D., Wu, Q., Zhang, W., Wang, P., Guo, X., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01262 · Cited by 14 · see citing works
Self-citationFastCLIPstyler: Optimisation-free Text-based Image Style Transfer Using Style Representations
Suresh, A. P., Jain, S., Noinongyao, P., Ganguly, A., Watchareeruetai, U., & Samacoïts, A. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00715 · Cited by 13 · see citing works
Chen, H., Wang, X. Y., Zeng, G., Zhang, Y. J., Zhou, Y., Han, F., Wu, Y., & Zhu, W. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3557634 · Cited by 13 · see citing works
Learning Interactive Real-World Simulators
Yang, S., Du, Y., Ghasemipour, K., Tompson, J., Kaelbling, L., Schuurmans, D., & Abbeel, P. (2023) · arXiv · doi:10.48550/arxiv.2310.06114 · Cited by 12 · see citing works
“Internet-scale generative models can synthesize realistic images and videos [44, 3, 45, 46, 47], but have mostly been applied to generative media [48] as opposed to empowering sophisticated agents capable of multi-turn interactions.”
BackgroundGenerative Artificial Intelligence in the Metaverse Era: A Review on Models and Applications
Zhou, H., Chen, X., Li, J., Zhang, Z., Fu, Y., Pérez-Liva, M., Greenbaum, D. S., & Hui, P. (2025) · Research · doi:10.34133/research.0804 · Cited by 17 · see citing works
Sounding Video Generator: A Unified Framework for Text-Guided Sounding Video Generation
Liu, J.-W., Wang, W., Chen, S., Zhu, X., & Liu, J. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3262180 · Cited by 12 · see citing works
3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation
Zhang, S., Zhang, Y., Zheng, Q., Ma, R., Hua, W., Bao, H., Xu, W., & Zou, C. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00969 · Cited by 9 · see citing works
Risk Analysis of AIGC in Market Regulation
Liu, C., Ouyang, X., Bai, R., & Liu, Y. (2023) · International Conference on Financial Innovation, FinTech and Information Technology (FFIT) · doi:10.4108/eai.7-7-2023.2338069 · Cited by 11 · see citing works
Story Visualization by Online Text Augmentation with Context Memory
Ahn, D., Kim, D., Song, G., Kim, S. H., Lee, H., Kang, D., & Choi, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00290 · Cited by 8 · see citing works
ConditionVideo: Training-Free Condition-Guided Video Generation
Peng, B., Chen, X., Wang, Y., Lu, C., & Qiao, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28244 · Cited by 10 · see citing works
Foundation Models for Video Understanding: A SurveyPreprint
Madan, N., Møgelmose, A., Modi, R., Rawat, Y. S., & Moeslund, T. B. (2024) · doi:10.36227/techrxiv.171769139.99464428/v2 · Cited by 8 · see citing works
TIA2V: Video generation conditioned on triple modalities of text–image–audio
Zhao, M., Wang, W., Zhang, R., Jia, H., & Chen, Q. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126278 · Cited by 8 · see citing works
ED-T2V: An Efficient Training Framework for Diffusion-based Text-to-Video Generation
Liu, J., Wang, W., Liu, W., He, Q., & Liu, J. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191565 · Cited by 7 · see citing works
Video Generation from Text Employing Latent Path Construction for Temporal Modeling
Mazaheri, A., & Shah, M. A. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956706 · Cited by 8 · see citing works
Language-Driven Image Style TransferPreprint
Fu, T.-J., Wang, X. E., & Wang, W. Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.00178 · Cited by 7 · see citing works
Eutamene, H. B., Hamidouche, W., Keïta, M., Ahmed, A. T., & Hadid, A. (2025) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2025.110699 · Cited by 8 · see citing works
Pix2Gif: Motion-Guided Diffusion for GIF Generation
Kandala, H., Gao, J., & Yang, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73013-9_3 · Cited by 6 · see citing works
T2VEval: Benchmark dataset and objective evaluation method for T2V-generated videos
Qi, Z., Shi, P. A., Wang, S., Zhang, C., Zhao, F., Ying, Z., Pan, D., Yang, X., He, Z., & Dai, T. (2025) · Displays · doi:10.1016/j.displa.2025.103178 · Cited by 7 · see citing works
Foundation Models for Speech, Images, Videos, and Control
Paaß, G., & Giesselbach, S. (2023) · Artificial intelligence: foundations, theory, and algorithms/Artificial intelligence: Foundations, theory, and algorithms · doi:10.1007/978-3-031-23190-2_7 · Cited by 5 · see citing works
Learning 3D Photography Videos via Self-supervised Diffusion on Single Images
Wang, X., Wu, C., Yin, S., Ni, M., Wang, J., Li, L., Yang, Z., Yang, F., Wang, L., Liu, Z., et al. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/167 · Cited by 4 · see citing works
Self-citationVideo Abnormal Action Recognition Based on Multimodal Heterogeneous Transfer Learning
Huang, H., Zheng, Y., & Hu, Z.-Y. (2024) · Advances in Multimedia · doi:10.1155/2024/4187991 · Cited by 4 · see citing works
ImproveYourVideos: Architectural Improvements for Text-to-Video Generation Pipeline
Arkhipkin, V., Shaheen, Z., Vasilev, V., Dakhova, E., Sobolev, K., Кузнецов, А. В., & Dimitrov, D. (2024) · IEEE Access · doi:10.1109/access.2024.3522510 · Cited by 5 · see citing works
DTR: Dynamic Tree-Ring Watermarking Framework for Diffusion-Based Video Generation
Zeng, S., Yang, L., Yang, J., Wang, Y., & Gao, T. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888152 · Cited by 3 · see citing works
Wang, X., Wang, W., Su, S., Lu, M., Zhang, L., & Lu, X. (2025) · Image and Vision Computing · doi:10.1016/j.imavis.2025.105545 · Cited by 3 · see citing works
Video is Worth a Thousand Images: Exploring the Latest Trends in Long Video Generation
Waseem, F., & Shahzad, M. S. (2025) · ACM Computing Surveys · doi:10.1145/3771724 · Cited by 2 · see citing works
DiST-GAN: Distillation-based Semantic Transfer for Text-Guided Face Generation
Yang, G., Fu, F., Fei, N., Wu, H., Ma, R., & Lu, Z. (2023) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme55011.2023.00149 · Cited by 2 · see citing works
TiV-ODE: A Neural ODE-based Approach for Controllable Video Generation From Text-Image Pairs
Xu, Y., Li, N., Goel, A., Yao, Z., Guo, Z., Kasaei, H., Kasaei, M., & Li, Z. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610149 · Cited by 2 · see citing works
Fine-grained Controllable Video Generation via Object Appearance and Context
Huang, H., Su, Y.-C., Sun, D., Jiang, L., Jia, X., Zhu, Y., & Yan, S. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00364 · Cited by 7 · see citing works
Autolume-Live: Turning GANs into a Live VJing tool
Kraasch, J., & Pasquier, P. (2022) · doi:10.24840/xcoax_2022_45 · Cited by 2 · see citing works
Self-labeling video prediction
Zhang, W., Wang, Y., & Yang, X. (2023) · Displays · doi:10.1016/j.displa.2023.102400 · Cited by 2 · see citing works
Language-agnostic Semantic Consistent Text-to-Image Generation
Jung, S., Choi, W. S., Choi, S., & Zhang, B. (2022) · doi:10.18653/v1/2022.mml-1.1 · Cited by 2 · see citing works
NWT: Towards natural audio-to-video generation with representation learningPreprint
Mama, R., Tyndel, M. S., Kadhim, H., Clifford, C., & Thurairatnam, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.04283 · Cited by 2 · see citing works
SpectraSpan: Zero Fine-Tuning Long Video Generation Framework and Its Frequency Domain Optimization
Zhang, W., Wang, F., & Cao, Z. (2025) · International Conference on Multimedia Retrieval · doi:10.1145/3731715.3733435 · Cited by 1 · see citing works
Don’t Forget Your Inverse DDIM for Image Editing
Gomez-Trenado, G., Mesejo, P., Cordón, Ó., & Lathuilière, S. (2025) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2025.3563859 · Cited by 1 · see citing works
Bridging Text and Video Generation
Maragatham, G., Kumar, N., Bhandari, P., Raja, R., & Joel, M. R. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-1504-1.ch007 · Cited by 1 · see citing works
TEXT TO IMAGE GENERATOR USING AI
Athawale, D. (2024) · INTERANTIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT · doi:10.55041/ijsrem31493 · Cited by 1 · see citing works
AcademicVid: Academic PDFs to video generation - Exploratory Literature Survey
Namratha, M., Kumar, M., Nisha, P., Rakshith, R., & Roshan (2024) · International Conference on Emerging Technologies in Computer Science for Interdisciplinary Applications (ICETCS) · doi:10.1109/icetcs61022.2024.10544257 · Cited by 1 · see citing works
Animate Sketch: Text-Driven Sketch Animation Generation
Liu, Z., & Li, X. (2024) · IEEE International Conference on Frontiers Technology of Information and Computer (ICFTIC) · doi:10.1109/icftic64248.2024.10913167 · Cited by 1 · see citing works
Compilation of Text to Video models
Anupama, Y. K., Neha, N., Verma, D., Sankeerthana, S. B., & Jha, M. (2024) · International Conference on IoT, Communication and Automation Technology (ICICAT) · doi:10.1109/icicat62666.2024.10923075 · Cited by 1 · see citing works
Text to video generation via knowledge distillation
Han, H., Li, Z., Fang, F., Luo, F., & Xiao, C. (2024) · Metaverse · doi:10.54517/m.v5i1.2425
VidMagic: Prompt to Image Clips
Ghuikar, M. N., Moon, M. T., Bhagat, M. P., Chafale, M. S., & Sabe, M. G. (2024) · International Journal of Advanced Research in Science Communication and Technology · doi:10.48175/ijarsct-17473
Controlling Attention Map Better for Text-Guided Image Editing Diffusion Models
Xu, S., Sun, L., Liu, G., & Wei, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5615-5_5
EditHuman: Fine-Grained Text-Driven Human Video Editing
Zhang, K., Sun, M., Hu, J., Zhang, K., & Sun, Z. (2024) · IEEE International Joint Conference on Biometrics (IJCB) · doi:10.1109/ijcb62174.2024.10744427
TextureDiffusion: Target Prompt Disentangled Editing for Various Texture Transfer
Su, Z., Zhuang, J., & Yuan, C. Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888836
Chen, G., Liang, X. C., & He, W. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2487877 · Cited by 1 · see citing works
- Authors
- Wu, C., Huang, L., Zhang, Q., Li, B., Ji, L., Yang, F., Sapiro, G., & Duan, N.
- Affiliations
- Microsoft Research Asia (China) (CN)
- Duke University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2104.14806
- OpenAlex
- W3157199281
- Cited by
- 85
- References
- 31
- Citations by year
- 2025: 16 · 2024: 38 · 2023: 20 · 2022: 7 · 2021: 4
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
- Keywords
- text-to-video generation, sparse attention
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
