The DOI matches its DataCite record; 69 of the 73 references it cites were found in a registry or an index.
VideoGPT: Video Generation using VQ-VAE and Transformers
Open accessAbstract
We present VideoGPT: a conceptually simple architecture for scaling likelihood based generative modeling to natural videos. VideoGPT uses VQ-VAE that learns downsampled discrete latent representations of a raw video by employing 3D convolutions and axial self-attention. A simple GPT-like architecture is then used to autoregressively model the discrete latents using spatio-temporal position encodings. Despite the simplicity in formulation and ease of training, our architecture is able to generate samples competitive with state-of-the-art GAN models for video generation on the BAIR Robot dataset, and generate high fidelity natural videos from UCF-101 and Tumbler GIF Dataset (TGIF). We hope our proposed architecture serves as a reproducible reference for a minimalistic implementation of transformer based video generation models. Samples and code are available at https://wilson1yan.github.io/videogpt/index.html
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To scale to higher resolutions, several two-stage approaches [23,67,101,103] use ARMs to model a compressed latent image space instead of raw pixels.”
From High-Resolution Image Synthesis with Latent Diffusion Models (2022) · cites this paper as [101]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“While VideoGPT did not outperform the best GAN-based models at the time, namely TGAN-v2 and DVD-GAN, it achieved a respectable IS score on the UCF-101 benchmark considering its simple architecture.”
From Video Diffusion Models: A Survey (2024) · cites this paper as Yan et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“VQ-VAEs [67, 101] use autoregressive models to learn an expressive prior over a discretized latent space.”
From High-Resolution Image Synthesis with Latent Diffusion Models (2022) · cites this paper as [101]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
GAN（Generative Adversarial Nets）
淳司, 柴. (2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 20,507
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
A Style-Based Generator Architecture for Generative Adversarial Networks
Karras, T., Laine, S., & Aila, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00453 · Cited by 11,714
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593
Denoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712
Self-citation
Research works printed but not linked: 11
- Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016. · arXiv
- Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Advances in neural information processing systems , pp. 2672-2680, 2014.
- Kalchbrenner, N., Oord, A. v. d., Simonyan, K., Danihelka, I., Vinyals, O., Graves, A., and Kavukcuoglu, K. Video pixel networks. arXiv preprint arXiv:1610.00527 , 2016. · arXiv
- Kalchbrenner, N., Oord, A., Simonyan, K., Danihelka, I., Vinyals, O., Graves, A., and Kavukcuoglu, K. Video pixel networks. In International Conference on Machine Learning , pp. 1771-1779. PMLR, 2017.
- Kingma, D. P. and Dhariwal, P. Glow: Generative flow with invertible 1x1 convolutions. arXiv preprint arXiv:1807.03039 , 2018. · arXiv
- Kingma, D. P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., and Welling, M. Improving variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems , 2016.
- Le Gall, D. Mpeg: A video compression standard for multimedia applications. Communications of the ACM , 34(4): 46-58, 1991.
- Mittal, G., Marwah, T., and Balasubramanian, V. N. Syncdraw: Automatic video generation using deep recurrent attentive architectures. In Proceedings of the 25th ACM international conference on Multimedia , pp. 1096-1104, 2017.
- Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, Ł., Shazeer, N., and Ku, A. Image transformer. arXiv preprint arXiv:1802.05751 , 2018. · arXiv
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. OpenAI Blog , 1(8):9, 2019.
- Zhang, H., Goodfellow, I., Metaxas, D., and Odena, A. Selfattention generative adversarial networks. In International Conference on Machine Learning , pp. 7354-7363. PMLR, 2019.
Cited by
Sorted by citations
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models
Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S., & Kreis, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02161 · Cited by 724
VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
Tong, Z., Song, Y., Wang, J., & Wang, L. (2022) · arXiv · doi:10.48550/arxiv.2203.12602 · Cited by 909
“[84] Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas.”
BackgroundMethodStructure and Content-Guided Video Synthesis with Diffusion Models
Esser, P., Chiu, J., Atighehchian, P., Granskog, J., & Germanidis, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00675 · Cited by 526
Diffusion Autoencoders: Toward a Meaningful and Decodable Representation
Preechakul, K., Chatthee, N., Wizadwongsa, S., & Suwajanakorn, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01036 · Cited by 338
VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models
Chen, H., Zhang, Y., Cun, X., Xia, M., Wang, X., Weng, C., & Shan, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00698 · Cited by 425
Self-citationBailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory
Li, S., Yu, W., Gu, T., Lin, C., Wang, Q., Qian, C., Loy, C. C., & Liu, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01077 · Cited by 208
StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2
Skorokhodov, I., Tulyakov, S., & Elhoseiny, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00361 · Cited by 199
NaturalSpeech : End-to-End Text-to-Speech Synthesis With Human-Level Quality
Tan, X., Chen, J., Liu, H., Cong, J., Zhang, C., Liu, Y., Wang, X., Leng, Y., Yi, Y., He, L., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3356232 · Cited by 283
NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion
Wu, C., Liang, J., Ji, L., Yang, F., Fang, Y., Jiang, D., & Duan, N. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19787-1_41 · Cited by 216
Notice of Removal: VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation
Luo, Z., Chen, D., Zhang, Y., Huang, Y., Wang, L., Shen, Y., Zhao, D., Zhou, J., & Tan, T. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00984 · Cited by 144
Self-citationPreserve Your Own Correlation: A Noise Prior for Video Diffusion Models
Ge, S., Nah, S., Liu, G., Poon, T., Tao, A., Catanzaro, B., Jacobs, D. W., Huang, J., Liu, M.-Y., & Balaji, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02096 · Cited by 176
MAGVIT: Masked Generative Video Transformer
Yu, L., Cheng, Y., Sohn, K., Lezama, J., Zhang, H., Chang, H., Hauptmann, A. G., Yang, M., Yuan, H., Essa, I. A., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01008 · Cited by 190
Video Probabilistic Diffusion Models in Projected Latent Space
Yu, S., Sohn, K., Kim, S., & Shin, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01770 · Cited by 144
MARLIN: Masked Autoencoder for facial video Representation LearnINg
Cai, Z., Ghosh, S., Stefanov, K., Dhall, A., Cai, J., Rezatofighi, H., Haffari, R., & Hayat, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00150 · Cited by 122
Phenaki: Variable Length Video Generation From Open Domain Textual Description
Villegas, R., Babaeizadeh, M., Kindermans, P.-J., Moraldo, H., Zhang, H., Saffar, M. T., Castro, S., Kunze, J., & Erhan, D. (2022) · arXiv · doi:10.48550/arxiv.2210.02399 · Cited by 92
“Method FVD ↓ DVD-GAN [9] 109.8 VideoGPT [55] 103.3 TrIVD-GAN [27] 103.3 Transframer [31] 100.0 HARP [57] 99.3 CCVS [28] 99.0 Video Transformer [51] 94.0 FitVid [3] 93.6 MCVD [47] 89.5 NUWA [54] 86.9 RaMViD [19] 84.2 Phenaki (Ours) 97.0 Phenaki is not specifically designed for video prediction,…”
BackgroundInfluentialFrido: Feature Pyramid Diffusion for Complex Scene Image Synthesis
Fan, W.-C., Chen, Y., Chen, D., Yu, C., Yuan, L., & Wang, Y.-C. F. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25133 · Cited by 80
When Does Sora Show: The Beginning of TAO to Imaginative Intelligence and Scenarios Engineering
Wang, F., Miao, Q., Li, L., Ni, Q., Li, X., Li, J., Fan, L., Tian, Y., & Han, Q. (2024) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2024.124383 · Cited by 70
Understanding World or Predicting Future? A Comprehensive Survey of World Models
Ding, J., Zhang, Y., Shang, Y., Zhang, Y., Zong, Z., Feng, J., Yuan, Y., Su, H., Li, N., Sukiennik, N., et al. (2026) · ACM Computing Surveys · doi:10.1145/3746449 · Cited by 169
Self-citationVIDM: Video Implicit Diffusion Models
Mei, K., & Patel, V. M. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i8.26094 · Cited by 70
Dream Video: Composing Your Dream Videos with Customized Subject and Motion
Wei, Y., Zhang, S., Qing, Z., Yuan, H., Liu, Z., Liu, Y., Zhang, Y., Zhou, J., & Shan, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00625 · Cited by 94
Self-citationNUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation
Yin, S., Wu, C., Yang, H., Wang, J., Wang, X., Ni, M., Yang, Z., Li, L., Liu, S., Yang, F., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.73 · Cited by 166
SimDA: Simple Diffusion Adapter for Efficient Video Generation
Xing, Z., Dai, Q., Hu, H., Wu, Z., & Jiang, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00748 · Cited by 92
Modeling Atomistic Dynamic Fracture Mechanisms Using a Progressive Transformer Diffusion Model
Buehler, M. J. (2022) · Journal of Applied Mechanics · doi:10.1115/1.4055730 · Cited by 48
Make Pixels Dance: High-Dynamic Video Generation
Zeng, Y., Wei, G., Zheng, J., Zou, J., Yang, W., Zhang, Y., & Li, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00845 · Cited by 109
Self-citationDysen-VDM: Empowering Dynamics-Aware Text-to-Video Diffusion with LLMs
Fei, H., Wu, S., Ji, W., Zhang, H., & Chua, T. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00730 · Cited by 68
Zheng, C., & Vedaldi, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02084 · Cited by 61
Luo, X., Sun, J., Zhang, R., Chi, P., & Cui, R. (2024) · Computers & Geosciences · doi:10.1016/j.cageo.2024.105541 · Cited by 48
DLFormer: Discrete Latent Transformer for Video Inpainting
Ren, J., Zheng, Q., Zhao, Y., Xu, X., & Li, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00350 · Cited by 45
Video prediction by efficient transformers
Ye, X., & Bilodeau, G.-A. (2022) · Image and Vision Computing · doi:10.1016/j.imavis.2022.104612 · Cited by 51
Wang, Y., Yang, D., Brémond, F., & Dantcheva, A. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3449075 · Cited by 159
Text2Performer: Text-Driven Human Video Generation
Jiang, Y., Yang, S., Koh, T. L., Wu, W., Loy, C. C., & Liu, Z. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02079 · Cited by 54
Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis
Menapace, W., Siarohin, A., Skorokhodov, I., Deyneka, E., Chen, T.-S., Kag, A., Fang, Y., Stoliar, A., Ricci, E., Ren, J., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00672 · Cited by 68
Zheng, Q., & Zhang, D. (2022) · Computational Geosciences · doi:10.1007/s10596-022-10144-8 · Cited by 38
Beyond Deepfake Images: Detecting AI-Generated Videos
Vahdati, D. S., Nguyen, T. D., Azizpour, A., & Stamm, M. C. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00443 · Cited by 37
Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioning
Han, L., Ren, J., Lee, H.-Y., Barbieri, F., Olszewski, K., Minaee, S., Metaxas, D., & Tulyakov, S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00360 · Cited by 39
Retracted: Sora for Scenarios Engineering of Intelligent Vehicles: V&V, C&C, and Beyonds
Li, X., Miao, Q., Li, L., Ni, Q., Fan, L., Wang, Y., Tian, Y., & Wang, F. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3379989 · Cited by 39
RetractedBailando ++: 3D Dance GPT With Choreographic Memory
Li, S., Yu, W., Gu, T., Lin, C., Wang, Q., Qian, C., Loy, C. C., & Liu, Z. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3319435 · Cited by 36
VPTR: Efficient Transformers for Video Prediction
Ye, X., & Bilodeau, G.-A. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956707 · Cited by 32
Vlogger: Make Your Dream A Vlog
Zhuang, S., Li, K., Chen, X., Wang, Y., Liu, Z., Qiao, Y., & Wang, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00841 · Cited by 62
Breathing Life Into Sketches Using Text-to-Video Priors
Gal, R., Vinker, Y., Alaluf, Y., Bermano, A. H., Cohen‐Or, D., Shamir, A., & Chechik, G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00414 · Cited by 35
MoStGAN-V: Video Generation with Temporal Motion Styles
Shen, X., Li, X., & Elhoseiny, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00547 · Cited by 32
On the Content Bias in Fréchet Video Distance
Ge, S., Mahapatra, A., Parmar, G., Zhu, J.-Y., & Huang, J.-B. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00695 · Cited by 50
FitVid: Overfitting in Pixel-Level Video PredictionPreprint
Babaeizadeh, M., Saffar, M., Nair, S., Levine, S., Finn, C., & Erhan, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.13195 · Cited by 94
GAIA-1: A Generative World Model for Autonomous Driving
Hu, A., Russell, L., Yeo, H., Murez, Z., Fedoseev, G., Kendall, A., Shotton, J., & Corrado, G. (2023) · arXiv · doi:10.48550/arxiv.2309.17080 · Cited by 28
“Likewise, they can operate at the pixel level [72, 73], or in a discrete learned token space [74, 75, 76, 77].”
BackgroundRV-GAN: Recurrent GAN for Unconditional Video Generation
Gupta, S., Keshari, A., & Das, S. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00220 · Cited by 27
ART•V: Auto-Regressive Text-to-Video Generation with Diffusion Models
Weng, W., Feng, R., Wang, Y., Dai, Q., Wang, C., Yin, D., Zhao, Z., Qiu, K., Bao, J., Yuan, Y., et al. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00735 · Cited by 31
Self-citationSize-invariant 3D generation from a single 2D rock image
Phan, J., Ruspini, L. C., Kiss, G. H., & Lindseth, F. (2022) · Journal of Petroleum Science and Engineering · doi:10.1016/j.petrol.2022.110648 · Cited by 27
Tell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generation
Fu, T.-J., Yu, L., Zhang, N., Fu, C.-Y., Su, J.-C., Wang, W. Y., & Bell, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01029 · Cited by 27
InstructVideo: Instructing Video Diffusion Models with Human Feedback
Yuan, H., Zhang, S., Wang, X., Wei, Y., Feng, T., Pan, Y., Zhang, Y., Liu, Z., Albanie, S., & Ni, D. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00618 · Cited by 50
Self-citationControllable Video Generation With Text-Based Instructions
Köksal, A., Ak, K. E., Sun, Y., Rajan, D., & Lim, J. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3262972 · Cited by 22
CHeart: A Conditional Spatio-Temporal Generative Model for Cardiac Anatomy
Qiao, M., Wang, S., Qiu, H., de Marvao, A., O’Regan, D. P., Rueckert, D., & Bai, W. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3331982 · Cited by 24
Character-centric Story Visualization via Visual Planning and Token Alignment
Chen, H., Han, R., Wu, T.-L., Nakayama, H., & Peng, N. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.565 · Cited by 20
Wang, Z., Wang, J., Liu, Z., & Qiu, Q. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02162 · Cited by 22
Efficient Video Prediction via Sparsely Conditioned Flow Matching
Davtyan, A., Sameni, S., & Favaro, P. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02126 · Cited by 26
Imasato, N., Miyazawa, K., Duncan, C., & Nagai, T. (2023) · IEEE Access · doi:10.1109/access.2023.3280603 · Cited by 21
Video Diffusion Models with Local-Global Context Guidance
Yang, S., Zhang, L., Liu, Y., Jiang, Z., & He, Y. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/182 · Cited by 25
A Generative Model for Surrogates of Spatial-Temporal Wildfire Nowcasting
Cheng, S. L., Guo, Y., & Arcucci, R. (2023) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2023.3298535 · Cited by 16
MOSO: Decomposing MOtion, Scene and Object for Video Prediction
Sun, M., Wang, N., Zhu, X., & Liu, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01796 · Cited by 19
Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model
He, X., Huang, Q., Zhang, Z., Lin, Z., Wu, Z., Yang, S., Li, M., Chen, Z., Xu, S., & Wu, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00220 · Cited by 26
Mirasol3B: A Multimodal Autoregressive Model for Time-Aligned and Contextual Modalities
Piergiovanni, A., Noble, I., Kim, D., Ryoo, M. S., Gomes, V., & Angelova, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02531 · Cited by 17
MagicStick: Controllable Video Editing via Control Handle Transformations
Ma, Y., Cun, X., Liang, S., Xing, J., He, Y., Qi, C., Chen, S., & Chen, Q. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00909 · Cited by 63
A unified model for continuous conditional video prediction
Ye, X., & Bilodeau, G.-A. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00368 · Cited by 16
Skillful Precipitation Nowcasting Using Physical‐Driven Diffusion Networks
Wang, R., Fung, J. C. H., & Lau, A. K. H. (2024) · Geophysical Research Letters · doi:10.1029/2024gl110832 · Cited by 23
MS-LSTM: Exploring spatiotemporal multiscale representations in video prediction domain
Ma, Z., Zhang, H., & Liu, J. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110731 · Cited by 15
StyleInV: A Temporal Style Modulated Inversion Network for Unconditional Video Generation
Wang, Y., Jiang, L., & Loy, C. C. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02089 · Cited by 15
Continuous PDE Dynamics Forecasting with Implicit Neural RepresentationsPreprint
Yin, Y., Kirchmeyer, M., Franceschi, J.-Y., Rakotomamonjy, A., & Gallinari, P. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.14855 · Cited by 52
Co-Speech Gesture Synthesis using Discrete Gesture Token Learning
Lu, S., Yoon, Y., & Feng, A. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10342027 · Cited by 23
HARP: Autoregressive Latent Video Prediction with High-Fidelity Image Generator
Seo, Y., Lee, K., Liu, F., James, S., & Abbeel, P. (2022) · Proceedings - International Conference on Image Processing · doi:10.1109/icip46576.2022.9897982 · Cited by 20
Self-citationGenie: Generative Interactive Environments
Bruce, J., Dennis, M., Edwards, A., Parker-Holder, J., Shi, Y., Hughes, E., Lai, M., Mavalankar, A., Steigerwald, R., Apps, C., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.15391 · Cited by 23
“…video (Blattmann et al., 2023b; Brooks et al., 2024; Clark et al., 2019; Finn et al., 2016; Ho et al., 2022a,b; Höppe et al., 2022; Kalchbrenner et al., 2017; Le Moing et al., 2021; Lotter et al., 2017; Luc et al., 2020; Singer et al., 2023; Walker et al., 2021; Yan et al., 2021; Yu et al., 2023).”
Sounding Video Generator: A Unified Framework for Text-Guided Sounding Video Generation
Liu, J.-W., Wang, W., Chen, S., Zhu, X., & Liu, J. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3262180 · Cited by 12
Modeling Spatio-Temporal Dynamical Systems With Neural Discrete Learning and Levels-of-Experts
Wang, K., Wu, H., Zhang, G., Fang, J., Liang, Y., Wu, Y., Zimmermann, R., & Wang, Y. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3363711 · Cited by 12
Object Discovery from Motion-Guided Tokens
Bao, Z., Tokmakov, P., Wang, Y.-X., Gaidon, A., & Hebert, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02200 · Cited by 13
Cosmos World Foundation Model Platform for Physical AI
NVIDIA, :, Agarwal, N., Ali, A., Bala, M., Balaji, Y., Barker, E., Cai, T., Chattopadhyay, P., Chen, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.03575 · Cited by 13
Self-citationMotionVideoGAN: A Novel Video Generator Based on the Motion Space Learned From Image Pairs
Zhu, J., Ma, H., Chen, J., & Yuan, J. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3251095 · Cited by 11
Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis
Gu, Y., Wang, X., Ge, Y., Shan, Y., & Shou, M. Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00729 · Cited by 24
Time Dependent Image Generation of Plants from Incomplete Sequences with CNN-Transformer
Drees, L., Weber, I., Rußwurm, M., & Roscher, R. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16788-1_30 · Cited by 11
Transformers predicting the future. Applying attention in next-frame and time series forecastingPreprint
Cholakov, R., & Kolev, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.08224 · Cited by 22
Sign Language Production with Latent Motion Transformer
Xie, P., Peng, T., Du, Y., & Zhang, Q. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00300 · Cited by 10
MIGC++: Advanced Multi-Instance Generation Controller for Image Synthesis
Zhou, D., Li, Y., Ma, F., Yang, Z., & Yang, Y. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3510752 · Cited by 20
Neural network processing of holographic images
Schreck, J. S., Gantos, G., Hayman, M., Bansemer, A. R., & Gagne, D. J. (2022) · Atmospheric measurement techniques · doi:10.5194/amt-15-5793-2022 · Cited by 9
UniAdapter: All-in-One Control for Flexible Video Generation
Wang, C., Hu, P., Zhao, H., Guo, Y., Gu, J., Dong, X. M., Han, J., Xu, H., & Liang, X. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3532495 · Cited by 11
Interactive wearable digital devices for blind and partially sighted people
Xue, W., Ren, Y., Tang, Y. W., Gong, Z., Zhang, T., Chen, Z., Dong, X., Ma, X., Wang, Z., Xu, H., et al. (2025) · Nature Reviews Electrical Engineering · doi:10.1038/s44287-025-00170-w · Cited by 9
Ullah, A., Numan, M., Khalid, M. N. A., & Majid, A. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129767 · Cited by 10
SSGVS: Semantic Scene Graph-to-Video Synthesis
Cong, Y., Yi, J., Rosenhahn, B., & Yang, M. Y. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00254 · Cited by 10
3D PET image generation with tumour masks using TGAN
Bergen, R. V., Rajotte, J.-F., Yousefirizi, F., Klyuzhin, I. S., Rahmim, A., & Ng, R. T. (2022) · SPIE Medical Imaging: Image Processing · doi:10.1117/12.2611292 · Cited by 8
Assessing privacy leakage in synthetic 3-D PET imaging using transversal GAN
Bergen, R. V., Rajotte, J.-F., Yousefirizi, F., Rahmim, A., & Ng, R. T. (2023) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2023.107910 · Cited by 7
Post-Deployment Fine-Tunable Semantic Communication
Si, P., Liu, R., Qian, L., Zhao, J., & Lam, K. (2024) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2024.3433479 · Cited by 10
Yoo, J., Kim, S., Lee, D., Kim, C., & Hong, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02192 · Cited by 7
GSSTU: Generative Spatial Self-Attention Transformer Unit for Enhanced Video Prediction
Singh, B., Singh, D., Kaushal, R., Halder, A., & Chattopadhyay, P. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3359716 · Cited by 7
Prospects and technology of embodied intelligent humanoid robots driven by AI large models
Kai, Z. E. N. G., Wang, Y., Tan, H., Fang, Q., Wang, Y., & Yuan, L. (2025) · Scientia Sinica Informationis · doi:10.1360/ssi-2024-0350 · Cited by 6
A Two-Part Transformer Network for Controllable Motion Synthesis
Hou, S., Tao, H., Bao, H., & Xu, W. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3284402 · Cited by 9
Regeneration Learning: A Learning Paradigm for Data Generation
Tan, X., Qin, T., Bian, J., Liu, T., & Bengio, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30271 · Cited by 10
Video Prediction by Modeling Videos as Continuous Multi-Dimensional Processes
Shrivastava, G., & Shrivastava, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00691 · Cited by 9
Karaarslan, E., & Aydin, Ö. F. (2024) · doi:10.36227/techrxiv.170862194.43871446/v2 · Cited by 6
Collaboratively Self-Supervised Video Representation Learning for Action Recognition
Zhang, J., Wan, Z., Hu, L., Lin, S., Wu, S., & Shan, S. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3531772 · Cited by 7
Nowcasting of Extreme Precipitation Using Deep Generative Models
Bi, H., Kyryliuk, M., Wang, Z., Meo, C., Wang, Y., Imhoff, R. O., Uijlenhoet, R., & Dauwels, J. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094988 · Cited by 5
Mining and Applying Composition Knowledge of Dance Moves for Style-Concentrated Dance Generation
Zhang, X., Yang, S., Xu, Y., Zhang, W., & Gao, L. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i4.25673 · Cited by 5
Learning Semantic Textual Similarity via Topic-informed Discrete Latent Variables
Yu, E., Du, L., Jin, Y., Wei, Z., & Chang, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.328 · Cited by 7
Metadata
- Authors
- Yan, W., Zhang, Y., Abbeel, P., & Srinivas, A.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2104.10157
- OpenAlex
- W3152733922
- Cited by
- 156
- References
- 73
- Citations by year
- 2026: 2 · 2025: 21 · 2024: 44 · 2023: 50 · 2022: 32 · 2021: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- video generation, vector quantized variational autoencoder, discrete latent representations, 3D convolution, autoregressive models, UCF101
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
