Phenaki: Variable Length Video Generation From Open Domain Textual Description
Open accessAbstract
We present Phenaki, a model capable of realistic video synthesis, given a sequence of textual prompts. Generating videos from text is particularly challenging due to the computational cost, limited quantities of high quality text-video data and variable length of videos. To address these issues, we introduce a new model for learning video representation which compresses the video to a small representation of discrete tokens. This tokenizer uses causal attention in time, which allows it to work with variable-length videos. To generate video tokens from text we are using a bidirectional masked transformer conditioned on pre-computed text tokens. The generated video tokens are subsequently de-tokenized to create the actual video. To address data issues, we demonstrate how joint training on a large corpus of image-text pairs as well as a smaller number of video-text examples can result in generalization beyond what is available in the video datasets. Compared to the previous video generation methods, Phenaki can generate arbitrary long videos conditioned on a sequence of prompts (i.e. time variable text or a story) in open domain. To the best of our knowledge, this is the first time a paper studies generating videos from time variable prompts. In addition, compared to the per-frame baselines, the proposed video encoder-decoder computes fewer tokens per video but results in better spatio-temporal consistency.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In preliminary experiments, the pretrained models SPARC and Phenaki offer complementary benefits.
“In preliminary experiments, we found that these models offer complementary benefits.”
From Scaling Instructable Agents Across Many Simulated Worlds (2024) · cites this paper as Villegas et al., 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623 · cited 1× · see citing works
“To train our model, we use a combination of L 2 loss, image perceptual loss L IP [20, 61], video perceptual loss L VP by using the I3D network [6] as feature extractor, and adversarial loss L Adv with StyleGAN architecture [21].”
MethodPerceptual Losses for Real-Time Style Transfer and Super-Resolution
Johnson, J., Alahi, A., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_43 · Cited by 11,283 · cited 1× · see citing works
“To train our model, we use a combination of L 2 loss, image perceptual loss L IP [20, 61], video perceptual loss L VP by using the I3D network [6] as feature extractor, and adversarial loss L Adv with StyleGAN architecture [21].”
MethodQuo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570 · cited 1× · see citing works
“To train our model, we use a combination of L 2 loss, image perceptual loss L IP [20, 61], video perceptual loss L VP by using the I3D network [6] as feature extractor, and adversarial loss L Adv with StyleGAN architecture [21].”
MethodAnalyzing and Improving the Image Quality of StyleGAN
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00813 · Cited by 6,332 · cited 3× · see citing works
“The discriminator architecture is the StyleGAN [21] discriminator with blur resample, and channel multiplier of 1 .”
BackgroundMethodViViT: A Video Vision Transformer
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lucic, M., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00676 · Cited by 2,767 · cited 2× · see citing works
“To do so, we introduce C-ViViT , a causal variation of ViViT [1] with additional architectural changes for video generation, which can compress the videos in temporal and spatial dimensions, while staying auto-regressive in time, This capability allows for generating videos of arbitrary length…”
MethodNeural Discrete Representation Learning
Oord, A. V. D., Vinyals, O., & Kavukcuoglu, K. (2017) · arXiv · doi:10.48550/arxiv.1711.00937 · Cited by 2,729 · cited 2× · see citing works
“…discrete latent space, we quantize our encoder outputs into the entries of a learned codebook via the vector quantization (VQ) objective in VQVAEs [45], where sg ( x ) ≡ x , and x sg ( x ) ≡ 0 is the stop-gradient operator, β is the commitment loss weight, and e is a codebook vector from codebook…”
MethodPhotorealistic Text-to-Image Diffusion Models with Deep Language Understanding
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.11487 · Cited by 2,528 · cited 3× · see citing works
“It is now possible to generate realistic high resolution images given a description [34, 35, 32, 38, 59], but generating high quality videos from text remains challenging.”
BackgroundMethodTaming Transformers for High-Resolution Image Synthesis
Esser, P., Rombach, R., & Ommer, B. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01268 · Cited by 2,514 · cited 2× · see citing works
“We compare C-ViViT against per-frame image based encoder-decoders that have been used as video quantizers for conditional video generation [57, 54, 18, 54, 18, 52]: a ViT [58] and a convolutional VQ-GAN[12].”
MethodHierarchical Text-Conditional Image Generation with CLIP Latents
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06125 · Cited by 2,511 · cited 2× · see citing works
“It is now possible to generate realistic high resolution images given a description [34, 35, 32, 38, 59], but generating high quality videos from text remains challenging.”
BackgroundZero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417 · cited 4× · see citing works
“It is now possible to generate realistic high resolution images given a description [34, 35, 32, 38, 59], but generating high quality videos from text remains challenging.”
BackgroundMethodInfluentialUnsupervised Learning of Video Representations using LSTMs
Srivastava, N., Mansimov, E., & Salakhutdinov, R. (2015) · arXiv · doi:10.48550/arxiv.1502.04681 · Cited by 1,861 · cited 1× · see citing works
“These papers include deterministic methods using a combination of recurrent and convolutional networks [36, 42, 13, 50], variational based stochastic meth-ods [2, 10, 46, 3] and more recently by learning a discrete representation [49, 33, 31], auto-regressive models [51, 55, 28, 57], diffusion…”
MethodGLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., & Chen, M. (2021) · arXiv · doi:10.48550/arxiv.2112.10741 · Cited by 1,357 · cited 1× · see citing works
“It is now possible to generate realistic high resolution images given a description [34, 35, 32, 38, 59], but generating high quality videos from text remains challenging.”
BackgroundClassifier-Free Diffusion Guidance
Ho, J., & Salimans, T. (2022) · arXiv · doi:10.48550/arxiv.2207.12598 · Cited by 1,126 · cited 1× · see citing works
“In addition to the MVTM objective, we train using classifier-free guidance by dropping the text condition 10% of the time during training [16, 59] .”
MethodMoCoGAN: Decomposing Motion and Content for Video Generation
Tulyakov, S., Liu, M.-Y., Yang, X., & Kautz, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00165 · Cited by 1,076 · cited 1× · see citing works
“…42, 13, 50], variational based stochastic meth-ods [2, 10, 46, 3] and more recently by learning a discrete representation [49, 33, 31], auto-regressive models [51, 55, 28, 57], diffusion models [47, 14, 56, 19] flow based models [24], and finally adversarial based methods [48, 39, 43, 9, 40, 27].”
BackgroundZhai, X., Kolesnikov, A., Houlsby, N., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01179 · Cited by 891 · cited 2× · see citing works
“For image generation, there are datasets with billions of image-text pairs (such as LAION-5B [41] and JFT4B [60]) while the text-video datasets are substantially smaller e.”
BackgroundUnsupervised Learning for Physical Interaction through Video Prediction
Finn, C., Goodfellow, I., & Levine, S. (2016) · arXiv · doi:10.48550/arxiv.1605.07157 · Cited by 759 · cited 1× · see citing works
“These papers include deterministic methods using a combination of recurrent and convolutional networks [36, 42, 13, 50], variational based stochastic meth-ods [2, 10, 46, 3] and more recently by learning a discrete representation [49, 33, 31], auto-regressive models [51, 55, 28, 57], diffusion…”
MethodStochastic Video Generation with a Learned PriorPreprint
Denton, E., & Fergus, R. (2018) · International Conference on Machine Learning · doi:10.48550/arxiv.1802.07687 · Cited by 599 · cited 2× · see citing works
“…deterministic methods using a combination of recurrent and convolutional networks [36, 42, 13, 50], variational based stochastic meth-ods [2, 10, 46, 3] and more recently by learning a discrete representation [49, 33, 31], auto-regressive models [51, 55, 28, 57], diffusion models [47, 14,…”
BackgroundMethodInfluentialMoments in Time Dataset: One Million Videos for Event Understanding
Monfort, M., Vondrick, C., Oliva, A., Andonian, A., Zhou, B., Ramakrishnan, K., Bargal, S. A., Yan, T., Brown, L., Fan, Q., et al. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2901464 · Cited by 537 · cited 2× · see citing works
“The MiT dataset, in contrast to other publicly available video datasets, is a high quality balanced dataset with high coverage and density of verbs depicting moments of a few seconds [29].”
MethodResultMCVD: Masked Conditional Video Diffusion for Prediction, Generation, and InterpolationPreprint
Voleti, V. S., Jolicoeur-Martineau, A., & Pal, C. (2022) · arXiv.org · doi:10.48550/arxiv.2205.09853 · Cited by 447 · cited 2× · see citing works
“…42, 13, 50], variational based stochastic meth-ods [2, 10, 46, 3] and more recently by learning a discrete representation [49, 33, 31], auto-regressive models [51, 55, 28, 57], diffusion models [47, 14, 56, 19] flow based models [24], and finally adversarial based methods [48, 39, 43, 9, 40, 27].”
BackgroundInfluentialMaskGIT: Masked Generative Image Transformer
Chang, H., Zhang, H., Jiang, L., Liu, C., & Freeman, W. T. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01103 · Cited by 446 · cited 4× · see citing works
“As discussed in MaskGIT [8], the masking schedule γ i and sampling schedule β i have a signifi-cant effect on the samples quality therefore we follow the same strategies.”
MethodInfluentialSelf-citationGenerating Videos with Scene Dynamics
Vondrick, C., Pirsiavash, H., & Torralba, A. (2016) · Maryland Shared Open Access Repository · doi:10.13016/m26gih-tnyz · Cited by 441 · cited 1× · see citing works
“…42, 13, 50], variational based stochastic meth-ods [2, 10, 46, 3] and more recently by learning a discrete representation [49, 33, 31], auto-regressive models [51, 55, 28, 57], diffusion models [47, 14, 56, 19] flow based models [24], and finally adversarial based methods [48, 39, 43, 9, 40, 27].”
BackgroundTemporal Generative Adversarial Nets with Singular Value Clipping
Saito, M., Matsumoto, E., & Saito, S. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.308 · Cited by 412 · cited 1× · see citing works
“…42, 13, 50], variational based stochastic meth-ods [2, 10, 46, 3] and more recently by learning a discrete representation [49, 33, 31], auto-regressive models [51, 55, 28, 57], diffusion models [47, 14, 56, 19] flow based models [24], and finally adversarial based methods [48, 39, 43, 9, 40, 27].”
BackgroundFlexible Diffusion Modeling of Long VideosPreprint
Harvey, W., Naderiparizi, S., Masrani, V., Weilbach, C., & Wood, F. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2205.11495 · Cited by 389 · cited 1× · see citing works
“…42, 13, 50], variational based stochastic meth-ods [2, 10, 46, 3] and more recently by learning a discrete representation [49, 33, 31], auto-regressive models [51, 55, 28, 57], diffusion models [47, 14, 56, 19] flow based models [24], and finally adversarial based methods [48, 39, 43, 9, 40, 27].”
BackgroundSelf-Supervised Visual Planning with Temporal Skip ConnectionsPreprint
Ebert, F., Finn, C., Lee, A. X., & Levine, S. (2017) · Conference on Robot Learning · doi:10.48550/arxiv.1710.05268 · Cited by 373 · cited 2× · see citing works
“Video prediction on BAIR [11].”
MethodScaling Autoregressive Models for Content-Rich Text-to-Image Generation
Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B. K., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.10789 · Cited by 371 · cited 7× · see citing works
“As for computation, training current state-of-the-art image generation models is already pushing the state-of-the-art computational capabilities [59], leaving little to no room for generating videos, particularly videos of variable length.”
BackgroundMethodInfluentialSelf-citationLAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., & Komatsuzaki, A. (2021) · arXiv · doi:10.48550/arxiv.2111.02114 · Cited by 358 · cited 3× · see citing works
“Unless specified otherwise, we train a 1.8B parameter Phenaki model on a corpus of ∼ 15 M text-video pairs at 8 FPS mixed with ∼ 50 M text-images plus ∼ 400 M pairs of LAION-400M [41] (more details in Appendix B.3).”
BackgroundMethodStochastic Variational Video PredictionPreprint
Babaeizadeh, M., Finn, C., Erhan, D., Campbell, R. H., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.11252 · Cited by 351 · cited 1× · see citing works
“…include deterministic methods using a combination of recurrent and convolutional networks [36, 42, 13, 50], variational based stochastic meth-ods [2, 10, 46, 3] and more recently by learning a discrete representation [49, 33, 31], auto-regressive models [51, 55, 28, 57], diffusion models [47,…”
MethodSelf-citationVideo (language) modeling: a baseline for generative models of natural videosPreprint
Ranzato, M., Szlam, A. D., Bruna, J., Mathieu, M., Collobert, R., & Chopra, S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6604 · Cited by 325 · cited 1× · see citing works
“These papers include deterministic methods using a combination of recurrent and convolutional networks [36, 42, 13, 50], variational based stochastic meth-ods [2, 10, 46, 3] and more recently by learning a discrete representation [49, 33, 31], auto-regressive models [51, 55, 28, 57], diffusion…”
MethodHo, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., & Fleet, D. J. (2022) · arXiv · doi:10.48550/arxiv.2204.03458 · Cited by 301 · cited 3× · see citing works
“Method FVD ↓ Video Transformer [51] 170.0 ± 5.00 CogVideo [18] 109.2 DVD-GAN-FP [9] 69.1 ± 0.78 Video VQ-VAE [49] 64.3 ± 2.04 CCVS [28] 55.0 ± 1.00 TrIVD-GAN-FP [27] 25.7 ± 0.66 Transframer [31] 25.4 RaMViD [19] 16.5 Video Diffusion [17] 16.2 ± 0.34 Phenaki (Ours) 36.4 ± 0.19 Table 5.”
MethodInfluentialA Short Note about Kinetics-600
Carreira, J., Noland, E., Banki-Horvath, A., Hillier, C., & Zisserman, A. (2018) · arXiv · doi:10.48550/arxiv.1808.01340 · Cited by 270 · cited 2× · see citing works
“Video prediction on Kinetics-600 [7].”
MethodScaling Up Models and Data with t5x and seqioPreprint
Roberts, A., Chung, H. W., Mishra, G., Levskaya, A., Bradbury, J., Andor, D., Narang, S., Lester, B., Gaffney, C., Mohiuddin, A., et al. (2022) · Journal of machine learning research · doi:10.48550/arxiv.2203.17189 · Cited by 232 · cited 2× · see citing works
“To train MaskGIT, we include a text conditioning in the form of T5X embeddings [37] which are used as input through the use of cross attention with the video tokens.”
MethodVector-quantized Image Modeling with Improved VQGANPreprint
Yu, J., Li, X., Koh, J. Y., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y., Baldridge, J., & Wu, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04627 · Cited by 227 · cited 4× · see citing works
“In addition to the VQ objective, we adopt the factorized and (cid:96) 2 - normalized codes from ViT-VQGAN [58] to improve codebook usage and reconstruction quality.”
MethodInfluentialSelf-citationTowards Accurate Generative Models of Video: A New Metric & Challenges
Unterthiner, T., van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., & Gelly, S. (2018) · arXiv · doi:10.48550/arxiv.1812.01717 · Cited by 204 · cited 1× · see citing works
“As demonstrated in Table 3, we evaluate the video reconstruction quality using FID [15] and FVD [44].”
MethodDiffusion Models for Video Prediction and InfillingPreprint
Hoppe, T., Mehrjou, A., Bauer, S., Nielsen, D., & Dittadi, A. (2022) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2206.07696 · Cited by 188 · cited 3× · see citing works
“Method FVD ↓ Video Transformer [51] 170.0 ± 5.00 CogVideo [18] 109.2 DVD-GAN-FP [9] 69.1 ± 0.78 Video VQ-VAE [49] 64.3 ± 2.04 CCVS [28] 55.0 ± 1.00 TrIVD-GAN-FP [27] 25.7 ± 0.66 Transframer [31] 25.4 RaMViD [19] 16.5 Video Diffusion [17] 16.2 ± 0.34 Phenaki (Ours) 36.4 ± 0.19 Table 5.”
BackgroundInfluentialDiffusion Probabilistic Modeling for Video Generation
Yang, R., Srivastava, P. K., & Mandt, S. (2023) · Entropy · doi:10.3390/e25101469 · Cited by 177 · cited 1× · see citing works
“…42, 13, 50], variational based stochastic meth-ods [2, 10, 46, 3] and more recently by learning a discrete representation [49, 33, 31], auto-regressive models [51, 55, 28, 57], diffusion models [47, 14, 56, 19] flow based models [24], and finally adversarial based methods [48, 39, 43, 9, 40, 27].”
BackgroundNÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion
Wu, C., Liang, J., Ji, L., Yang, F., Fang, Y., Jiang, D., & Duan, N. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19787-1_41 · Cited by 167 · cited 8× · see citing works
“The NUWA [54] paper provided a qualitative evaluation on Kinetics-400.”
BackgroundMethodInfluentialVideoGPT: Video Generation using VQ-VAE and Transformers
Yan, W., Zhang, Y., Abbeel, P., & Srinivas, A. (2021) · arXiv · doi:10.48550/arxiv.2104.10157 · Cited by 156 · cited 2× · see citing works
“Method FVD ↓ DVD-GAN [9] 109.8 VideoGPT [55] 103.3 TrIVD-GAN [27] 103.3 Transframer [31] 100.0 HARP [57] 99.3 CCVS [28] 99.0 Video Transformer [51] 94.0 FitVid [3] 93.6 MCVD [47] 89.5 NUWA [54] 86.9 RaMViD [19] 84.2 Phenaki (Ours) 97.0 Phenaki is not specifically designed for video prediction,…”
BackgroundInfluentialCogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers
Hong, W., Ding, M., Zheng, W., Liu, X., & Tang, J. (2022) · arXiv · doi:10.48550/arxiv.2205.15868 · Cited by 148 · cited 9× · see citing works
“Inspired by the previous work in auto-regressive text to image [34, 59, 38] and text to video [54, 53, 18], Phenaki is designed with two main components (see Figure 2): an encoder-decoder model which compresses videos to discrete embeddings (i.e. tokens) and a transformer model to translate text…”
BackgroundMethodInfluentialConditional GAN with Discriminative Filter Generation for Text-to-Video Synthesis
Balaji, Y., Min, M. R., Bai, B., Chellappa, R., & Graf, H. P. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/276 · Cited by 123 · see citing works
Saito, M., Saito, S., Koyama, M., & Kobayashi, S. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-020-01333-y · Cited by 116 · cited 1× · see citing works
“…42, 13, 50], variational based stochastic meth-ods [2, 10, 46, 3] and more recently by learning a discrete representation [49, 33, 31], auto-regressive models [51, 55, 28, 57], diffusion models [47, 14, 56, 19] flow based models [24], and finally adversarial based methods [48, 39, 43, 9, 40, 27].”
BackgroundLearning Audio-Video Modalities from Image Captions
Nagrani, A., Seo, P. H., Seybold, B., et al. (2022) · European Conference on Computer Vision · doi:10.48550/arxiv.2204.00679 · Cited by 107 · cited 1×
“On the importance of joint text-to-image and text-to-video training While there are some text-video datasets, text-image datasets dominate the internet in terms of quality and quantity [30].”
BackgroundNUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual SynthesisPreprint
Wu, C., Liang, J., Hu, X., et al. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2207.09814 · Cited by 105 · cited 3× · see citing works
“Inspired by the previous work in auto-regressive text to image [34, 59, 38] and text to video [54, 53, 18], Phenaki is designed with two main components (see Figure 2): an encoder-decoder model which compresses videos to discrete embeddings (i.”
MethodInfluentialScaling Autoregressive Video Models
Weissenborn, D., Täckström, O., & Uszkoreit, J. (2019) · arXiv · doi:10.48550/arxiv.1906.02634 · Cited by 93 · cited 3× · see citing works
“Method FVD ↓ Video Transformer [51] 170.0 ± 5.00 CogVideo [18] 109.2 DVD-GAN-FP [9] 69.1 ± 0.78 Video VQ-VAE [49] 64.3 ± 2.04 CCVS [28] 55.0 ± 1.00 TrIVD-GAN-FP [27] 25.7 ± 0.66 Transframer [31] 25.4 RaMViD [19] 16.5 Video Diffusion [17] 16.2 ± 0.34 Phenaki (Ours) 36.4 ± 0.19 Table 5.”
BackgroundInfluentialHigh Fidelity Video Prediction with Large Stochastic Recurrent Neural NetworksPreprint
Villegas, R., Pathak, A., Kannan, H., Erhan, D., Le, Q. V., & Lee, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.01655 · Cited by 87 · cited 2× · see citing works
“These papers include deterministic methods using a combination of recurrent and convolutional networks [36, 42, 13, 50], variational based stochastic methods [2, 10, 46, 3] and more recently by learning a discrete representation [49, 33, 31], auto-regressive models [51, 55, 28, 57], diffusion models [47, 14, 56, 19] f…”
BackgroundMethodSelf-citationGODIVA: Generating Open-DomaIn Videos from nAtural Descriptions
Wu, C., Huang, L., Zhang, Q., Li, B., Ji, L., Yang, F., Sapiro, G., & Duan, N. (2021) · arXiv · doi:10.48550/arxiv.2104.14806 · Cited by 85 · cited 3× · see citing works
“We compare C-ViViT against per-frame image based encoder-decoders that have been used as video quantizers for conditional video generation [57, 54, 18, 54, 18, 52]: a ViT [58] and a convolutional VQ-GAN[12].”
MethodRakhimov, R., Volkhonskiy, D., Artemov, A., Zorin, D., & Burnaev, E. V. (2021) · International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP) · doi:10.5220/0010241801010112 · Cited by 85 · cited 1× · see citing works
“…networks [36, 42, 13, 50], variational based stochastic meth-ods [2, 10, 46, 3] and more recently by learning a discrete representation [49, 33, 31], auto-regressive models [51, 55, 28, 57], diffusion models [47, 14, 56, 19] flow based models [24], and finally adversarial based methods [48,…”
MethodFrozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval
Bain, M., Nagrani, A., Varol, G., & Zisserman, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00175 · Cited by 76 · cited 1× · see citing works
“For image generation, there are datasets with billions of image-text pairs (such as LAION-5B [41] and JFT4B [60]) while the text-video datasets are substantially smaller e.g. WebVid [4] with ∼ 10 M videos, which is not enough given the higher complexity of open domain videos.”
BackgroundTransformation-based Adversarial Video Prediction on Large-Scale DataPreprint
Luc, P., Clark, A., Dieleman, S., Casas, D. D. L., Doron, Y., Cassirer, A., & Simonyan, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04035 · Cited by 53 · cited 3× · see citing works
“Method FVD ↓ Video Transformer [51] 170.0 ± 5.00 CogVideo [18] 109.2 DVD-GAN-FP [9] 69.1 ± 0.78 Video VQ-VAE [49] 64.3 ± 2.04 CCVS [28] 55.0 ± 1.00 TrIVD-GAN-FP [27] 25.7 ± 0.66 Transframer [31] 25.4 RaMViD [19] 16.5 Video Diffusion [17] 16.2 ± 0.34 Phenaki (Ours) 36.4 ± 0.19 Table 5.”
BackgroundInfluentialFitVid: Overfitting in Pixel-Level Video PredictionPreprint
Babaeizadeh, M., Saffar, M., Nair, S., Levine, S., Finn, C., & Erhan, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.13195 · Cited by 39 · cited 4× · see citing works
“…DVD-GAN [9] 109.8 VideoGPT [55] 103.3 TrIVD-GAN [27] 103.3 Transframer [31] 100.0 HARP [57] 99.3 CCVS [28] 99.0 Video Transformer [51] 94.0 FitVid [3] 93.6 MCVD [47] 89.5 NUWA [54] 86.9 RaMViD [19] 84.2 Phenaki (Ours) 97.0 Phenaki is not specifically designed for video prediction, therefore, it…”
BackgroundMethodInfluentialSelf-citationLi, Y., Min, M. R., Shen, D., Carlson, D., & Carin, L. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12233 · Cited by 38 · see citing works
CCVS: Context-aware Controllable Video SynthesisPreprint
Moing, G. L., Ponce, J., & Schmid, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.08037 · Cited by 34 · cited 3× · see citing works
“Method FVD ↓ Video Transformer [51] 170.0 ± 5.00 CogVideo [18] 109.2 DVD-GAN-FP [9] 69.1 ± 0.78 Video VQ-VAE [49] 64.3 ± 2.04 CCVS [28] 55.0 ± 1.00 TrIVD-GAN-FP [27] 25.7 ± 0.66 Transframer [31] 25.4 RaMViD [19] 16.5 Video Diffusion [17] 16.2 ± 0.34 Phenaki (Ours) 36.4 ± 0.19 Table 5.”
BackgroundInfluentialPredicting Video with VQVAEPreprint
Walker, J., Razavi, A. S., & van den Oord, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.01950 · Cited by 29 · cited 3× · see citing works
“Method FVD ↓ Video Transformer [51] 170.0 ± 5.00 CogVideo [18] 109.2 DVD-GAN-FP [9] 69.1 ± 0.78 Video VQ-VAE [49] 64.3 ± 2.04 CCVS [28] 55.0 ± 1.00 TrIVD-GAN-FP [27] 25.7 ± 0.66 Transframer [31] 25.4 RaMViD [19] 16.5 Video Diffusion [17] 16.2 ± 0.34 Phenaki (Ours) 36.4 ± 0.19 Table 5.”
MethodInfluentialTransframer: Arbitrary Frame Prediction with Generative ModelsPreprint
Nash, C., Carreira, J., Walker, J., Barr, I. R., Jaegle, A., Malinowski, M., & Battaglia, P. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.09494 · Cited by 18 · cited 3× · see citing works
“…networks [36, 42, 13, 50], variational based stochastic meth-ods [2, 10, 46, 3] and more recently by learning a discrete representation [49, 33, 31], auto-regressive models [51, 55, 28, 57], diffusion models [47, 14, 56, 19] flow based models [24], and finally adversarial based methods [48, 39,…”
BackgroundMethodInfluentialDulley, I. (2019) · On the Emic Gesture · doi:10.4324/9780429457210-8 · Cited by 15 · cited 1×
“For open domain videos, we test Phenaki on Kinetics600 [7] where the task is to predict 11 frames given 5 frames.”
MethodHARP: Autoregressive Latent Video Prediction with High-Fidelity Image Generator
Seo, Y., Lee, K., Liu, F., James, S., & Abbeel, P. (2022) · Proceedings - International Conference on Image Processing · doi:10.1109/icip46576.2022.9897982 · Cited by 13 · cited 4× · see citing works
“We compare C-ViViT against per-frame image based encoder-decoders that have been used as video quantizers for conditional video generation [57, 54, 18, 54, 18, 52]: a ViT [58] and a convolutional VQ-GAN[12].”
BackgroundMethodInfluential
Flagged first, then by citations
6 of the 62 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- [9][9] Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation on complex datasets. arXiv preprint arXiv:1907.06571 , 2019. - · arXiv
- [15][15] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017. -
- [24][24] Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. Videoflow: A flow-based generative model for video. arXiv preprint arXiv:1903.01434 , 2019. - · arXiv
- [26][26] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR , 2019. -
- [50][50] Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, and Philip S Yu. Predrnn: Recurrent neural networks for predictive learning using spatiotemporal lstms. Advances in neural information processing systems , 30, 2017. -
Cited by
Sorted by citations
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models
Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S., & Kreis, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02161 · Cited by 724 · see citing works
“[90] Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan.”
BackgroundTune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation
Wu, J. Z., Ge, Y., Wang, X., Lei, S. W., Gu, Y., Shi, Y., Hsu, W., Shan, Y., Qie, X., & Shou, M. Z. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00701 · Cited by 622 · see citing works
“To replicate this success in Text-to-Video (T2V) generation, recent works [43, 16, 19, 54, 48] have extended spatial-only T2I generation models to the spatio-temporal domain.”
Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators
Khachatryan, L., Movsisyan, A., Tadevosyan, V., Henschel, R. D., Wang, Z., Navasardyan, S., & Shi, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01462 · Cited by 426 · see citing works
“Phenaki [39] introduces a bidirectional masked transformer with a causal attention mechanism that allows the generation of arbitrary-long videos from text prompt sequences.”
BackgroundBengesi, S., El-Sayed, H., Sarker, M. K., Houkpati, Y., Irungu, J., & Oladunni, T. (2023) · IEEE Access · doi:10.1109/access.2024.3397775 · Cited by 286 · see citing works
VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models
Chen, H., Zhang, Y., Cun, X., Xia, M., Wang, X., Weng, C., & Shan, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00698 · Cited by 257 · see citing works
“Then, since transformers have been successfully applied in various fields, they are also introduced for video synthesis, e.g. , CogVideo [28], VideoGPT [57], NUVA-infinity [53], TATS [19], MAGVIT [58], Phenaki [45].”
MethodGenerative AI-Driven Semantic Communication Networks: Architecture, Technologies, and Applications
Liang, C., Du, H., Sun, Y., Niyato, D. T., Kang, J., Zhao, D., & Imran, M. A. (2024) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2024.3435524 · Cited by 170 · see citing works
“A popular approach is to generate videos from text sequence-to-sequence, like Phenaki [94], where video tokens are predicted from the paired text embeddings using a bidirectional Transformer architecture.”
Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models
Höllein, L., Cao, A., Owens, A., Johnson, J., & Nießner, M. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00727 · Cited by 167 · see citing works
“Text-based generation has been extended to other modalities including audio [35, 17, 28, 70], video [75, 91, 85, 25], and 4D fields [76].”
BackgroundCollaborative Diffusion for Multi-Modal Face Generation and Editing
Huang, Z., Chan, K. C. K., Jiang, Y., & Liu, Z. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00589 · Cited by 155 · see citing works
“Diffusion models [20, 56, 58] have recently become a mainstream approach for image synthesis [9, 11, 38] apart from Generative Adversarial Networks (GANs) [14], and success has also been found in various domains including video generation [17, 19, 55, 63], image restoration [21, 52], semantic segmentation [1, 4, 15],…”
BackgroundYour Diffusion Model is Secretly a Zero-Shot Classifier
Li, A. C., Prabhudesai, M., Duggal, S., Brown, E. V., & Pathak, D. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00210 · Cited by 159 · see citing works
“Diffusion Models: Diffusion models [35, 69] have recently gained significant attention from the research community due to their ability to generate high-fidelity and diverse content like images [66, 54, 24], videos [68, 34, 77], 3D [58, 49], and audio [43, 51] from various input modalities like text.”
Video-P2P: Video Editing with Cross-Attention Control
Liu, S., Zhang, Y., Li, W., Lin, Z., & Jia, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00821 · Cited by 158 · see citing works
Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models
Ge, S., Nah, S., Liu, G., Poon, T., Tao, A., Catanzaro, B., Jacobs, D. W., Huang, J., Liu, M.-Y., & Balaji, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02096 · Cited by 138 · see citing works
“…strategies to leverage knowledge from image data for text-to-video generation, including joint training on the text-image data from scratch [17, 14, 56, 60], first training a text-to-image model and then finetuning partially [18, 4, 61, 29] or entirely [46, 8] on the video dataset, and using CLIP…”
BackgroundMAGVIT: Masked Generative Video Transformer
Yu, L., Cheng, Y., Sohn, K., Lezama, J., Zhang, H., Chang, H., Hauptmann, A. G., Yang, M., Yuan, H., Essa, I. A., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01008 · Cited by 130 · see citing works
“As shown in recent works [31, 50, 64], training such models requires large, and sometimes non-public, datasets of paired texts and images.”
BackgroundSelf-citationPromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions
Chung, J. J. Y., & Adar, E. (2023) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3586183.3606777 · Cited by 101 · see citing works
“Prompts can steer generation of texts [11, 81, 89], images [52, 54, 66, 70], UI designs [45], codes [13], 3D models [41, 63], music [2], and even videos [77, 85].”
BackgroundA Survey of Multimodel Large Language Models
Liang, Z., Xu, Y., Hong, Y., Shang, P., Wang, Q., Fu, Q., & Liu, K. (2024) · International Conference on Computer, Artificial Intelligence and Control Engineering (CAICE) · doi:10.1145/3672758.3672824 · Cited by 100 · see citing works
“Emerging models such as DALL · E3 [1], Stable Diffusion [2], Phenaki [3], Imagen Video [4], among others, can receive text or image input and generate visual information.”
Scaling Robot Learning with Semantically Imagined Experience
Yu, T., Xiao, T. G., Tompson, J., Stone, A., Wang, S., Brohan, A., Singh, J., Tan, C., Dee, M., Peralta, J., et al. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.027 · Cited by 83 · see citing works
Make-Your-Video: Customized Video Generation Using Textual and Structural Guidance
Xing, J., Xia, M., Liu, Y., Zhang, Y., Zhang, Y., He, Y., Liu, H., Chen, H., Cun, X., Wang, X., et al. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3365804 · Cited by 76 · see citing works
Park, J., Oh, C., & Kim, H. Y. (2024) · Technology in Society · doi:10.1016/j.techsoc.2024.102705 · Cited by 71 · see citing works
When Does Sora Show: The Beginning of TAO to Imaginative Intelligence and Scenarios Engineering
Wang, F., Miao, Q., Li, L., Ni, Q., Li, X., Li, J., Fan, L., Tian, Y., & Han, Q. (2024) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2024.124383 · Cited by 64 · see citing works
4D-fy: Text-to-4D Generation Using Hybrid Score Distillation Sampling
Bahmani, S., Skorokhodov, I., Rong, V., Wetzstein, G., Guibas, L. J., Wonka, P., Tulyakov, S., Park, J. J., Tagliasacchi, A., & Lindell, D. B. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00764 · Cited by 63 · see citing works
“Autoregressive models distinguish themselves by their ability to generate videos of varying lengths [66].”
BackgroundCoDeF: Content Deformation Fields for Temporally Consistent Video Processing
Ouyang, H., Wang, Q., Xiao, Y., Bai, Q., Zhang, J., Zheng, K., Zhou, X., Chen, Q., & Chen, Q. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00773 · Cited by 63 · see citing works
“Although text-to-video architectures such as NUWA [62], CogVideo [11], Phenaki [55], Make-A-Video [48], Imagen Video [9], and Gen-1 [7] are capable of generating video frames that semantically correspond to the input text, they may exhibit limitations in terms of precise control over video…”
NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation
Yin, S., Wu, C., Yang, H., Wang, J., Wang, X., Ni, M., Yang, Z., Li, L., Liu, S., Yang, F., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.73 · Cited by 68 · see citing works
A Survey of AI Text-to-Image and AI Text-to-Video Generators
Singh, A. (2023) · International Conference on Artificial Intelligence, Robotics and Control (AIRC) · doi:10.1109/airc57904.2023.10303174 · Cited by 53 · see citing works
CelebV-Text: A Large-Scale Facial Text-Video Dataset
Yu, J., Zhu, H., Jiang, L., Loy, C. C., Cai, W., & Wu, W. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01422 · Cited by 53 · see citing works
“, CogVideo [26], Phenaki [55], Imagen Video [24], and Make-A-Video [51] are not included since their training codes are not”
BackgroundMethodInfluentialDiffusion Models as Masked Autoencoders
Wei, C., Mangalam, K., Huang, P.-Y., Li, Y., Fan, H., Hu, X., Wang, H., Xie, C., Yuille, A., & Feichtenhofer, C. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01492 · Cited by 58 · see citing works
“This process has proven extremely powerful for rich text-conditioned generation of both images [59, 50, 64, 62] and videos [71, 36, 34, 66].”
BackgroundCLIPPO: Image-and-Language Understanding from Pixels Only
Tschannen, M., Mustafa, B., & Houlsby, N. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01059 · Cited by 51 · see citing works
“These broadly use discriminative tasks to learn representations for various downstream modalities; generative approaches to multimodal modelling have been scaled to billions of parameters, generating text [2,9,67,74], images [52,56,75], videos [25,65] or audio [4] from (combinations of) various modalities.”
BackgroundVideoBooth: Diffusion-based Video Generation with Image Prompts
Jiang, Y., Wu, T., Yang, S., Si, C., Lin, D., Qiao, Y., Loy, C. C., & Liu, Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00639 · Cited by 54 · see citing works
“Early explorations [34, 80] on text-to-video models are based on the idea of VQVAE.”
BackgroundMethodVideo Interpolation with Diffusion Models
Jain, S., Watson, D., Tabellion, E., Hołyński, A., Poole, B., & Kontkanen, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00701 · Cited by 44 · see citing works
“This ease of training has led to significant advances in generative modeling, such as compelling results in text-to-image [38, 40, 44] and text-to-video [17, 55] generation, image-conditioned generation tasks [43], and even 3D novel view synthesis [27, 36, 58].”
BackgroundSound to Visual Scene Generation by Audio-to-Visual Latent Alignment
Kim, S.-B., Senocak, A., Ha, H., Owens, A., & Oh, T.-H. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00622 · Cited by 41 · see citing works
“Various tasks have been tack-led in diverse domains, such as text-to-image/video [19,30, 52, 56, 57, 65, 70], speech-to-face/gesture [23, 44], scene graph/layout-to-image [34,75], image/audio-to-caption [4, 36, 40], etc .”
Text2Performer: Text-Driven Human Video Generation
Jiang, Y., Yang, S., Koh, T. L., Wu, W., Loy, C. C., & Liu, Z. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02079 · Cited by 41 · see citing works
“Recently, there are some concurrent works [39, 19, 48, 21] for text-to-video generation.”
BackgroundDiverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation
Yariv, G., Gat, I., Benaim, S., Wolf, L., Schwartz, I., & Adi, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i7.28486 · Cited by 46 · see citing works
Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer
Yatim, D., Fridman, R., Bar-Tal, O., Kasten, Y., & Dekel, T. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00809 · Cited by 47 · see citing works
“Recently, there have been substantial efforts in training large-scale text-to-video models on vast datasets with autoregressive Transformers (e.g., [24, 61, 66]) or Diffusion Models (e.g., [5, 22]).”
BackgroundAVID: Any-Length Video Inpainting with Diffusion Model
Zhang, Z., Wu, B., Wang, X., Luo, Y., Zhang, L., Zhao, Y., Vajda, P., Metaxas, D., & Yu, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00684 · Cited by 43 · see citing works
“The scarcity of high-quality, large-scale video datasets [5, 8, 14, 17, 24–26, 55, 63, 67, 68] further complicates this task.”
BackgroundSnap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis
Menapace, W., Siarohin, A., Skorokhodov, I., Deyneka, E., Chen, T.-S., Kag, A., Fang, Y., Stoliar, A., Ricci, E., Ren, J., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00672 · Cited by 42 · see citing works
“To address the narrow domain limitation, the task of text-to-video generation was proposed [34] and both autoregressive models [23, 34, 61, 63, 64] and GANs [29] emerged.”
Beyond Text-to-Image: Multimodal Prompts to Explore Generative AI
Liu, V. (2023) · Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544549.3577043 · Cited by 38 · see citing works
“Generative models are capable of text-based generation for text [4], images [9, 27], videos [29], and 3D shapes [21].”
BackgroundRetracted: Sora for Scenarios Engineering of Intelligent Vehicles: V&V, C&C, and Beyonds
Li, X., Miao, Q., Li, L., Ni, Q., Fan, L., Wang, Y., Tian, Y., & Wang, F. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3379989 · Cited by 38 · see citing works
“Several LM-based models have been developed, including VideoGPT [12], NUWA [13], CogVideo [14], MaskViT [15], TATS [16], Phenaki [17], and MAGVIT [18].”
MethodRetractedBeing Comes from Not-Being: Open-Vocabulary Text-to-Motion Generation with Wordless Training
Lin, J., Chang, J., Liu, L., Li, G., Lin, L., Tian, Q. C., & Chen, C. W. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02224 · Cited by 38 · see citing works
“Related to ours, recent studies [37, 38] have combined CLIP with diffusion generation models to generate text-consistent 3D meshes [15, 33], while other methods focus on generating static meshes or 2D images for text-video generation, e.g. Make-A-Video [40], Imagen Video [12], and Phenaki [45].”
Vlogger: Make Your Dream A Vlog
Zhuang, S., Li, K., Chen, X., Wang, Y., Liu, Z., Qiao, Y., & Wang, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00841 · Cited by 32 · see citing works
“However, these early works either require extensive training on large well-captioned long video datasets [83] or suffer from noticeable incoherence of shot changes [67].”
InfluentialA Survey of Cross-Modal Visual Content Generation
Nazarieh, F., Feng, Z., Awais, M., Wang, W., & Kittler, J. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3351601 · Cited by 29 · see citing works
“Some work addressed this using a Transformer-based decoder [62], or by conditioning on a sequence of distinct text prompts at each time step [55].”
BackgroundMethodFine-grained Audible Video Description
Shen, X., Li, D., Zhou, J., Qin, Z., He, B., Han, X., Li, A., Dai, Y., Kong, L., Wang, M., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01020 · Cited by 29 · see citing works
Shape-Aware Text-Driven Layered Video Editing
Lee, Y.-C., Jang, J.-Z. G., Chen, Y.-T., Qiu, E., & Huang, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01376 · Cited by 29 · see citing works
“Building upon the success of photorealistic (text-driven) image generation, recent work has shown impressive results on video generation, with a focus on generating long video [5, 11, 49, 60] and videos from free-form text prompts [13, 48, 52].”
BackgroundIntegrating Generative Artificial Intelligence in Intelligent Vehicle Systems
Stappen, L., Dillmann, J., Striegel, S., Vögel, H.-J., Flores-Herr, N., & Schuller, B. W. (2023) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc57777.2023.10422003 · Cited by 28 · see citing works
“The ongoing evolution of generative artificial intelligence systems is characterized by a shift toward multimodal models [26], capable of emulating human cognition by seamlessly processing and integrating multiple modalities.”
BackgroundGAIA-1: A Generative World Model for Autonomous Driving
Hu, A., Russell, L., Yeo, H., Murez, Z., Fedoseev, G., Kendall, A., Shotton, J., & Corrado, G. (2023) · arXiv · doi:10.48550/arxiv.2309.17080 · Cited by 28 · see citing works
“A known limitation is the slow generation speed, but this issue could be alleviated by future research on parallel sampling [78, 79, 80], reducing the number of latent variables [81], and improvements in hardware accelerators.”
BackgroundThe Power of Sound (TPoS): Audio Reactive Video Generation with Stable Diffusion
Jeong, Y., Ryoo, W., Lee, S. H., et al. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00719 · Cited by 27 · see citing works
“Recent text-to-video generation tools, including Make-A-Video [32], Video Diffusion Models [11], Imagen video [10], and Phenaki [36] have shown promising performance in generating videos from textual descriptions.”
BackgroundMethodFeng, Y., Poralla, P., Dash, S., Li, K., Desai, V., & Meikang Qiu, A. (2023) · IEEE International Conference on Big Data Security on Cloud, IEEE International Conference on High Performance and Smart Computing, and IEEE International Conference on Intelligent Data and Security (BigDataSecurity/HPSC/IDS) · doi:10.1109/bigdatasecurity-hpsc-ids58521.2023.00046 · Cited by 26 · see citing works
A Pathway Towards Responsible AI Generated Content
Lyu, L. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/803 · Cited by 77 · see citing works
On ChatGPT: Perspectives from Software Engineering Students
Hanifi, K., Çetin, O., & Yılmaz, C. (2023) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs60937.2023.00028 · Cited by 28 · see citing works
“The Flamingo model [3] can adeptly generate text from images, and Phenaki [4] is designed to transform texts into videos.”
MethodTell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generation
Fu, T.-J., Yu, L., Zhang, N., Fu, C.-Y., Su, J.-C., Wang, W. Y., & Bell, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01029 · Cited by 24 · see citing works
ART•V: Auto-Regressive Text-to-Video Generation with Diffusion Models
Weng, W., Feng, R., Wang, Y., Dai, Q., Wang, C., Yin, D., Zhao, Z., Qiu, K., Bao, J., Yuan, Y., et al. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00735 · Cited by 26 · see citing works
“Early T2V models demonstrated the possibility of generating videos in simple close-set domain [17, 28, 29, 31, 32, 35] and further exploited Transformer-based model [55] to achieve open-domain generation [23, 56, 64, 65].”
BackgroundInstructVideo: Instructing Video Diffusion Models with Human Feedback
Yuan, H., Zhang, S., Wang, X., Wei, Y., Feng, T., Pan, Y., Zhang, Y., Liu, Z., Albanie, S., & Ni, D. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00618 · Cited by 24 · see citing works
“Early efforts at video generation focused on GANs [24, 35, 47, 57, 67, 74, 84, 95] and VAEs [47, 53, 93].”
BackgroundComparative Analysis of Generative AI Models
Rani, G., Singh, J., & Khanna, A. (2023) · International Conference on Advances in Computation, Communication and Information Technology (ICAICCIT) · doi:10.1109/icaiccit60255.2023.10465941 · Cited by 24 · see citing works
“Yet there are certain models based upon existing architecture these are Phenaki [15], Sounify [16], VideoGPT, etc.”
BackgroundControllable Video Generation With Text-Based Instructions
Köksal, A., Ak, K. E., Sun, Y., Rajan, D., & Lim, J. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3262972 · Cited by 21 · see citing works
“Similarly, [37] employs bi-directional masked transformers-based model.”
BackgroundGINA-3D: Learning to Generate Implicit Neural Assets in the Wild
Shen, B., Yan, X., Qi, C. R., Najibi, M., Deng, B., Guibas, L. J., Zhou, Y., & Anguelov, D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00476 · Cited by 19 · see citing works
“Our framework builds upon the vector-quantized formulations [59, 60, 62, 136–140] where we focus on to-ken learning in the first stage.”
MethodVideoStudio: Generating Consistent-Content and Multi-scene Videos
Long, F., Qiu, Z., Yao, T., & Mei, T. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73027-6_27 · Cited by 30 · see citing works
“hierarchical architecture [12,25,52,53,66].”
TI2V-Zero: Zero-Shot Image Conditioning for Text-to-Video Diffusion Models
Ni, H., Egger, B., Lohit, S., Cherian, A., Wang, Y., Koike–Akino, T., Huang, X., & Marks, T. K. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00861 · Cited by 20 · see citing works
“Current TI2V generation methods [59, 63, 70] typically rely on computationally-heavy training on video-text datasets and specific architecture designs to enable text and image conditioning.”
MethodMtArtGPT: A Multi-Task Art Generation System With Pre-Trained Transformer
Jin, C., Zhu, R., Zhu, Z., Yang, L., Min, Y., & Luo, J. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3349567 · Cited by 20 · see citing works
“Phenaki [41] achieves realistic video synthesis using small discrete tokens and bidirectional masked transformers.”
BackgroundM2T: Masking Transformers Twice for Faster Decoding
Mentzer, F., Agustson, E., & Tschannen, M. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00492 · Cited by 17 · see citing works
“Recently, transformers trained for masked token prediction have successfully been applied to neural image and video generation [11, 35].”
BackgroundDriveGenVLM: Real-world Video Generation for Vision Language Model based Autonomous Driving
Fu, Y., Jain, A., Chen, X., et al. (2024) · IEEE International Automated Vehicle Validation Conference (IAVVC) · doi:10.1109/iavvc63304.2024.10786438 · Cited by 14 · see citing works
“Recently, more diffusion models have been applied in this domain and achieve better video quality and more realistic frames, such as video generation [11] and text prompt to video generation [12].”
Vision + Language Applications: A Survey
Zhou, Y., & Shimada, N. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00090 · Cited by 14 · see citing works
“With the ongoing development of transformer and diffusion models, recent methods based on transformers [68, 178] and diffusion [65, 163] improve the video quality (see Fig.”
MethodEOGT: Video Anomaly Detection with Enhanced Object Information and Global Temporal Dependency
Pi, R., Wu, P., He, X., & Peng, Y. (2024) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3662185 · Cited by 13 · see citing works
“Subsequent works such as Make-a-video [46] and Phenaki [47] applied these models to generate high-resolution and long-duration videos.”
Genie: Generative Interactive Environments
Bruce, J., Dennis, M., Edwards, A., Parker-Holder, J., Shi, Y., Hughes, E., Lai, M., Mavalankar, A., Steigerwald, R., Apps, C., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.15391 · Cited by 23 · see citing works
“By the causal nature of the ST-transformer, each discrete encoding 𝑧 𝑡 contains information from all previously seen frames of the video 𝒙 1: 𝑡 .”
BackgroundMethodInfluentialMAGVLT: Masked Generative Vision-and-Language Transformer
Kim, S., Jo, D., Lee, D., & Kim, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02235 · Cited by 12 · see citing works
“on the mask prediction has been proposed for language [22], image [10, 36], and video processing [24, 59].”
BackgroundMethodMeta-Personalizing Vision-Language Models to Find Named Instances in Video
Yeh, C.-H., Russell, B. C., Šivic, J., Heilbron, F. C., & Jenni, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01833 · Cited by 12 · see citing works
“The recent introduction of large-scale pre-trained visionlanguage models (VLMs) has enabled many new vision tasks, including zero-shot classification and retrieval [15, 20, 26], image/video generation [13, 28, 29, 31, 32, 36], or language-guided question answering [1, 21, 47].”
BackgroundGenerative AI Beyond LLMs: System Implications of Multi-Modal Generation
Golden, A., Hsia, S., Sun, F., Acun, B., Hosmer, B., Lee, Y., DeVito, Z., Johnson, J., Wei, G.-Y., Brooks, D., et al. (2024) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass61541.2024.00032 · Cited by 12 · see citing works
“We also include two TTV models: Make-a-Video [22] is built upon a diffusion model architecture, and Phenaki [23] that is derived from transformers.”
MethodGenDDS: Generating Diverse Driving Video Scenarios with Prompt-to-Video Generative Model
Fu, Y., Li, Y., & Di, X. S. (2024) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc58415.2024.10920106 · Cited by 11 · see citing works
“Applications include general video generation [8], as well as the generation of video from text prompts [9], demonstrating enhanced capabilities in creating dynamic and realistic video sequences.”
Generative Artificial Intelligence in the Metaverse Era: A Review on Models and Applications
Zhou, H., Chen, X., Li, J., Zhang, Z., Fu, Y., Pérez-Liva, M., Greenbaum, D. S., & Hui, P. (2025) · Research · doi:10.34133/research.0804 · Cited by 17 · see citing works
“Recent research has proposed neural ODE (ordinary differential equations)-based temporal modeling methods [ 132 ] and the Phenaki model [ 133 ]—the latter capable of generating variable-length, high-quality videos from a series of open-domain text prompts—offering new possibilities for open-domain…”
Scaling Instructable Agents Across Many Simulated Worlds
SIMA Team, Raad, M. A., Ahuja, A., Barros, C., Besse, F., Bolt, A., Bolton, A., Brownfield, B., Buttimore, G., Cant, M., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.10179 · Cited by 9 · see citing works
“First, our agent incorporates not only trained-from-scratch components, but also several pretrained models—including a model trained on fine-grained image-text alignment, SPARC (Bica et al., 2024), and a video prediction model, Phenaki (Villegas et al., 2022)—which we further fine-tune on our data through behavioral c…”
InfluentialStory Visualization by Online Text Augmentation with Context Memory
Ahn, D., Kim, D., Song, G., Kim, S. H., Lee, H., Kang, D., & Choi, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00290 · Cited by 8 · see citing works
“Very recently, there is a method proposed to generate a video from a long paragraph [44].”
MethodCaballero, J. (2023) · Hipertext net Revista Académica sobre Documentación Digital y Comunicación Interactiva · doi:10.31009/hipertext.net.2023.i26.08 · Cited by 8 · see citing works
Pix2Gif: Motion-Guided Diffusion for GIF Generation
Kandala, H., Gao, J., & Yang, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73013-9_3 · Cited by 6 · see citing works
Decouple Content and Motion for Conditional Image-to-Video Generation
Shen, C., Gan, Y., Chen, C., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28277 · Cited by 6 · see citing works
Creanalytics : Automating the supercut as a form of critical technical practice
Heras, D. C. (2023) · Convergence The International Journal of Research into New Media Technologies · doi:10.1177/13548565231174592 · Cited by 5 · see citing works
“Building on recent work on multi-modal systems, for example, the logic of language-guided image generation in DALL (cid:1) E 2 or Stable Diffusion are now being extended to train text-to-video models such as Make-A-Video (Singer et al., 2022), Imagen Video (Ho, 2022), and Phenaki (Villegas, 2022).”
BackgroundEhtesham, A., Kumar, S., Singh, A., et al. (2025) · IEEE Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc62904.2025.10903780 · Cited by 5 · see citing works
“Generative AI models such as DALL-E [4], Sora [5], Google Imagen [6], Phenaki [7], NUWA [8], and VideoStudio [9] are leading the way in generating consistent multi-scene videos, while models like VideoINSTA [10] excel in zero-shot long video understanding through informative spatial-temporal…”
Ouroboros-Diffusion: Exploring Consistent Content Generation in Tuning-free Long Video Diffusion
Chen, J., Long, F., An, J., Qiu, Z., Yao, T., Luo, J., & Mei, T. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i2.32205 · Cited by 5 · see citing works
Learning 3D Photography Videos via Self-supervised Diffusion on Single Images
Wang, X., Wu, C., Yin, S., Ni, M., Wang, J., Li, L., Yang, Z., Yang, F., Wang, L., Liu, Z., et al. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/167 · Cited by 4 · see citing works
Improving Subject-Driven Image Synthesis with Subject-Agnostic Guidance
Chan, K. C. K., Zhao, Y., Jia, X., Yang, M., & Wang, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00643 · Cited by 4 · see citing works
“It is widely adopted in existing works [46, 51].”
BackgroundNguyen, V. P., Ha, T. N. L., Pham, H., & Tran, Q. L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72089-5_63 · Cited by 4 · see citing works
Explaining Vision and Language through Graphs of Events in Space and Time
Masala, M., Cudlenco, N., Rebedea, T. E., et al. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00302 · Cited by 3 · see citing works
“There is a considerable amount of research at the inter-section of vision and language, such as image and video generation [22, 40, 15, 3, 33, 24, 29], captioning [11, 39, 30] or visual question answering [2, 18, 38].”
BackgroundNextGen Dynamic Video Generator using AI
Roy, A., Kishore, N., & Anand, M. V. (2023) · International Conference on Smart Structures and Systems (ICSSS) · doi:10.1109/icsss58085.2023.10407412 · Cited by 3 · see citing works
“[11] Generating high-resolution images with label editing addresses the challenge of generating high-resolution photorealistic images using generative adversarial networks (GANs).”
BackgroundImproveYourVideos: Architectural Improvements for Text-to-Video Generation Pipeline
Arkhipkin, V., Shaheen, Z., Vasilev, V., Dakhova, E., Sobolev, K., Кузнецов, А. В., & Dimitrov, D. (2024) · IEEE Access · doi:10.1109/access.2024.3522510 · Cited by 5 · see citing works
Addressing Character Consistency Challenges in AI Filmmaking
Kalinski, M., Podstawski, M., Ostrowski, K., Kudelska, M., Bartkowiak, P., Łyczko, P., Piasecki, M., Visscher, T., & Wang, H. (2025) · 2012 International Conference on Computing, Networking and Communications (ICNC) · doi:10.1109/icnc64010.2025.10993630 · Cited by 2 · see citing works
“While video models like VideoGPT [16], Imagen Video [17], and Phenaki [18] introduce mechanisms for spatio-temporal alignment, they often struggle with maintaining character consistency across longer sequences or between different videos.”
Fine-grained Controllable Video Generation via Object Appearance and Context
Huang, H., Su, Y.-C., Sun, D., Jiang, L., Jia, X., Zhu, Y., & Yan, S. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00364 · Cited by 7 · see citing works
“Recent text-to-video models [6, 18, 25, 27, 32, 49, 50, 52, 59, 60, 80] allow users to translate their creative ideas into video content easily.”
InfluentialRozo-Torres, A., Latorre-Rojas, C. J., & Sarmiento, W. J. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-91328-0_20 · Cited by 2 · see citing works
GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving
Russell, L., Hu, A., Bertoni, L., Fedoseev, G., Shotton, J., Arani, E., & Corrado, G. (2025) · arXiv · doi:10.48550/arxiv.2503.20523 · Cited by 1 · see citing works
“Autoregressive and masked generative models using quantized tokens have demonstrated strong modeling of temporal dependencies and motion dynamics [34, 35, 36, 37, 38, 39, 40, 41, 42].”
FeatsFlow: Traceable representation learning based on normalizing flows
Zhang, W., Pei, Z., & Wang, F. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107151 · Cited by 1 · see citing works
Frame by Familiar Frame: Understanding Replication in Video Diffusion Models
Rahman, A., Perera, M. V., & Patel, V. M. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00274 · Cited by 5 · see citing works
“However, video synthesis has not reached the same prominence as its image counterpart, primarily due to the immense computational demands and the scarcity of expansive, public video datasets [45].”
SpectraSpan: Zero Fine-Tuning Long Video Generation Framework and Its Frequency Domain Optimization
Zhang, W., Wang, F., & Cao, Z. (2025) · International Conference on Multimedia Retrieval · doi:10.1145/3731715.3733435 · Cited by 1 · see citing works
“Video generation models primarily fall into three categories: GAN-based models [3, 8, 22, 36, 38], Transformer-based models [17, 21, 39, 40, 44, 45], and diffusion model-based methods [5–7, 10–12, 14, 19, 26, 28, 35, 43, 48, 50].”
DynArt ChatGPT: a platform for the generation of dynamic non-traditional New Year paintings
Hong-liang, W., Manling, Z., Zhang, J., Cong, J., & Wang, J. (2023) · International Conference on Cyber-Physical Social Intelligence (ICCSI) · doi:10.1109/iccsi58851.2023.10304019 · Cited by 1 · see citing works
“Phenaki, a model capable of realistic video synthesis with a sequence of textual prompts [9].”
BackgroundTEXT TO IMAGE GENERATOR USING AI
Athawale, D. (2024) · INTERANTIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT · doi:10.55041/ijsrem31493 · Cited by 1 · see citing works
AcademicVid: Academic PDFs to video generation - Exploratory Literature Survey
Namratha, M., Kumar, M., Nisha, P., Rakshith, R., & Roshan (2024) · International Conference on Emerging Technologies in Computer Science for Interdisciplinary Applications (ICETCS) · doi:10.1109/icetcs61022.2024.10544257 · Cited by 1 · see citing works
Video modification in drone and satellite imagery
Reale, M. J., Murphy, D. P., Cornacchia, M., et al. (2024) · Disruptive Technologies in Information Sciences · doi:10.1117/12.3013881 · Cited by 1 · see citing works
“Phenaki by Villegas et al. [13] allows for long-running video generation with additional text prompts.”
BackgroundMarkovGen: Structured Prediction for Efficient Text-to-Image Generation
Jayasumana, S., Gläsner, D., Ramalingam, S., Veit, A., Chakrabarti, A., & Kumar, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00890 · Cited by 1 · see citing works
“Many of the text-to-image algorithms are also being extended to develop algorithms to handle other conditional inputs [35], and text-to-video generation [10,27,33].”
BackgroundLearning to control visual data translation
Koksal, A. (2023) · doi:10.32657/10356/165566
Data-Driven Optimization for Deposition with Degradable Tools
Zheng, T., Bujarbaruah, M., & Borrelli, F. (2023) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2023.10.1816
EditHuman: Fine-Grained Text-Driven Human Video Editing
Zhang, K., Sun, M., Hu, J., Zhang, K., & Sun, Z. (2024) · IEEE International Joint Conference on Biometrics (IJCB) · doi:10.1109/ijcb62174.2024.10744427
Generating Long-Take Videos via Effective Keyframes and Guidance
Huang, H., Su, Y.-C., & Yang, M. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00365
“…models mostly focus on generating videos with a predefined length, often shorter than real videos, or are limited to synthesizing long videos with repetitive contents [24,67], such as human actions [5, 55, 60] and sky timelapse [81] under the setting of unconditional video generation [6,57,85,91].”
Spatial Experience for the Metaverse
Paananen, V., Oppenlaender, J., van Berkel, N., & Hosio, S. J. (2026) · Springer eBooks · doi:10.1007/978-3-032-03296-6_4
Kim, S. (2026) · Journal of Broadcast Engineering · doi:10.5909/jbe.2026.31.4.700
- Authors
- Villegas, R., Babaeizadeh, M., Kindermans, P.-J., Moraldo, H., Zhang, H., Saffar, M. T., Castro, S., Kunze, J., & Erhan, D.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2210.02399
- OpenAlex
- W4303441850
- Semantic Scholar
- aa509ec67f311cd09d109356f7fa37a40072aabb
- Cited by
- 92
- References
- 62
- Citations by year
- 2026: 2 · 2025: 8 · 2024: 38 · 2023: 44
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- text-to-video generation, video generation, causal attention, spatiotemporal consistency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
