VideoGPT: Video Generation using VQ-VAE and Transformers
Open accessAbstract
We present VideoGPT: a conceptually simple architecture for scaling likelihood based generative modeling to natural videos. VideoGPT uses VQ-VAE that learns downsampled discrete latent representations of a raw video by employing 3D convolutions and axial self-attention. A simple GPT-like architecture is then used to autoregressively model the discrete latents using spatio-temporal position encodings. Despite the simplicity in formulation and ease of training, our architecture is able to generate samples competitive with state-of-the-art GAN models for video generation on the BAIR Robot dataset, and generate high fidelity natural videos from UCF-101 and Tumbler GIF Dataset (TGIF). We hope our proposed architecture serves as a reproducible reference for a minimalistic implementation of transformer based video generation models. Samples and code are available at https://wilson1yan.github.io/videogpt/index.html
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Several two-stage approaches use autoregressive models to model a compressed latent image space instead of raw pixels to scale to higher resolutions.
“To scale to higher resolutions, several two-stage approaches [23,67,101,103] use ARMs to model a compressed latent image space instead of raw pixels.”
From High-Resolution Image Synthesis with Latent Diffusion Models (2022) · cites this paper as [101]
The Introduction (pp. 2–3) describes a two-stage approach that trains an autoregressive transformer on compressed latents rather than raw pixels, reducing computation for high-resolution inputs. The cited approach models video latents across space and time, however, not specifically a latent image space. (LLM, conf=0.85)
Experimental — accuracy not yet measuredVideoGPT did not outperform the best GAN-based models at the time, namely TGAN-v2 and DVD-GAN.
“While VideoGPT did not outperform the best GAN-based models at the time, namely TGAN-v2 and DVD-GAN, it achieved a respectable IS score on the UCF-101 benchmark considering its simple architecture.”
From Video Diffusion Models: A Survey (2024) · cites this paper as Yan et al., 2021
VQ-VAEs use autoregressive models to learn an expressive prior over a discretized latent space.
“VQ-VAEs [67, 101] use autoregressive models to learn an expressive prior over a discretized latent space.”
From High-Resolution Image Synthesis with Latent Diffusion Models (2022) · cites this paper as [101]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067 · see citing works
GAN（Generative Adversarial Nets）
淳司, 柴. (2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,268 · see citing works
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235 · see citing works
A Style-Based Generator Architecture for Generative Adversarial Networks
Karras, T., Laine, S., & Aila, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00453 · Cited by 11,714 · see citing works
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596 · see citing works
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593 · see citing works
Denoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712 · see citing works
Self-citationImproved Techniques for Training GANs
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., & Chen, X. (2016) · arXiv · doi:10.48550/arxiv.1606.03498 · Cited by 6,965 · see citing works
WaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911 · see citing works
Progressive Growing of GANs for Improved Quality, Stability, and Variation
Karras, T., Aila, T., Laine, S., & Lehtinen, J. (2017) · arXiv · doi:10.48550/arxiv.1710.10196 · Cited by 4,727 · see citing works
Ucf101: A dataset of 101 human actions classes from videos in the wild
Soomro, K., Zamir, A. R., & Shah, M. (2012) · arXiv · doi:10.48550/arxiv.1212.0402 · Cited by 4,575 · see citing works
Large Scale GAN Training for High Fidelity Natural Image Synthesis
Brock, A., Donahue, J., & Simonyan, K. (2018) · arXiv · doi:10.48550/arxiv.1809.11096 · Cited by 3,717 · see citing works
Pixel Recurrent Neural NetworksPreprint
van den Oord, A., Kalchbrenner, N., & Kavukcuoglu, K. (2016) · International Conference on Machine Learning · doi:10.48550/arxiv.1601.06759 · Cited by 3,111 · see citing works
Conditional Image Generation with PixelCNN DecodersPreprint
van den Oord, A., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., & Kavukcuoglu, K. (2016) · Neural Information Processing Systems · doi:10.48550/arxiv.1606.05328 · Cited by 3,066 · see citing works
CCNet: Criss-Cross Attention for Semantic Segmentation
Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., & Liu, W. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00069 · Cited by 3,045 · see citing works
The JPEG still picture compression standard
Wallace, G. K. (1992) · IEEE Transactions on Consumer Electronics · doi:10.1109/30.125072 · Cited by 2,917 · see citing works
Human-level concept learning through probabilistic program induction
Lake, B. M., Salakhutdinov, R., & Tenenbaum, J. B. (2015) · Science · doi:10.1126/science.aab3050 · Cited by 2,817 · see citing works
Neural Discrete Representation Learning
Oord, A. V. D., Vinyals, O., & Kavukcuoglu, K. (2017) · arXiv · doi:10.48550/arxiv.1711.00937 · Cited by 2,729 · see citing works
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., & Ganguli, S. (2015) · arXiv · doi:10.48550/arxiv.1503.03585 · Cited by 2,637 · see citing works
Zero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417 · see citing works
Density estimation using Real NVP
Dinh, L., Sohl-Dickstein, J., & Bengio, S. (2016) · arXiv · Cited by 2,067 · see citing works
Gall, D. L. (1991) · Communications of the ACM · doi:10.1145/103085.103090 · Cited by 2,027 · see citing works
Unsupervised Learning of Video Representations using LSTMs
Srivastava, N., Mansimov, E., & Salakhutdinov, R. (2015) · arXiv · doi:10.48550/arxiv.1502.04681 · Cited by 1,861 · see citing works
NICE: Non-linear Independent Components Estimation
Dinh, L., Krueger, D., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1410.8516 · Cited by 1,493 · see citing works
Generative Modeling by Estimating Gradients of the Data Distribution
Song, Y., & Ermon, S. (2019) · arXiv · doi:10.48550/arxiv.1907.05600 · Cited by 1,453 · see citing works
In-Datacenter Performance Analysis of a Tensor Processing Unit
Jouppi, N. P., Young, C., Patil, N. P., Patterson, D. A., Agrawal, G., Bajwa, R. S., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. (2017) · ACM SIGARCH Computer Architecture News · doi:10.1145/3140659.3080246 · Cited by 1,431 · see citing works
MoCoGAN: Decomposing Motion and Content for Video Generation
Tulyakov, S., Liu, M.-Y., Yang, X., & Kautz, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00165 · Cited by 1,076 · see citing works
Generating Long Sequences with Sparse Transformers
Child, R., Gray, S., Radford, A., & Sutskever, I. (2019) · arXiv · doi:10.48550/arxiv.1904.10509 · Cited by 1,063 · see citing works
(2020) · International Conference on Machine Learning · Cited by 1,012 · see citing works
“Waveglow: a flow-based generative network for speech synthesis,”
Prenger, R., Valle, R., & Catanzaro, B. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683143 · Cited by 1,004 · see citing works
Unsupervised Learning for Physical Interaction through Video Prediction
Finn, C., Goodfellow, I., & Levine, S. (2016) · arXiv · doi:10.48550/arxiv.1605.07157 · Cited by 759 · see citing works
Generating diverse high-fidelity images with vq-vae-2
Razavi, A., Oord, A. V. D., & Vinyals, O. (2019) · arXiv · doi:10.48550/arxiv.1906.00446 · Cited by 709 · see citing works
Jukebox: A Generative Model for MusicPreprint
Dhariwal, P., Jun, H., Payne, C., Kim, J. W., Radford, A., & Sutskever, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00341 · Cited by 602 · see citing works
Stochastic Video Generation with a Learned PriorPreprint
Denton, E., & Fergus, R. (2018) · International Conference on Machine Learning · doi:10.48550/arxiv.1802.07687 · Cited by 599 · see citing works
Wang, T.-C., Liu, M.-Y., Zhu, J.-Y., Liu, G., Tao, A., Kautz, J., & Catanzaro, B. (2018) · arXiv · doi:10.48550/arxiv.1808.06601 · Cited by 594 · see citing works
Salimans, T., Karpathy, A., Chen, X., & Kingma, D. P. (2017) · arXiv · doi:10.48550/arxiv.1701.05517 · Cited by 584 · see citing works
ViZDoom: A Doom-based AI research platform for visual reinforcement learning
Kempka, M., Wydmuch, M., Runc, G., Toczek, J., & Jaśkowski, W. (2016) · IEEE Conference on Computational Intelligence and Games (CIG) · doi:10.1109/cig.2016.7860433 · Cited by 550 · see citing works
NVAE: A Deep Hierarchical Variational AutoencoderPreprint
Vahdat, A., & Kautz, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.03898 · Cited by 514 · see citing works
Generating Videos with Scene Dynamics
Vondrick, C., Pirsiavash, H., & Torralba, A. (2016) · Maryland Shared Open Access Repository · doi:10.13016/m26gih-tnyz · Cited by 441 · see citing works
Temporal Generative Adversarial Nets with Singular Value Clipping
Saito, M., Matsumoto, E., & Saito, S. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.308 · Cited by 412 · see citing works
Self-Supervised Visual Planning with Temporal Skip ConnectionsPreprint
Ebert, F., Finn, C., Lee, A. X., & Levine, S. (2017) · Conference on Robot Learning · doi:10.48550/arxiv.1710.05268 · Cited by 373 · see citing works
Axial Attention in Multidimensional Transformers
Ho, J., Kalchbrenner, N., Weissenborn, D., & Salimans, T. (2019) · arXiv · doi:10.48550/arxiv.1912.12180 · Cited by 370 · see citing works
Stochastic Adversarial Video PredictionPreprint
Lee, A. X., Zhang, R., Ebert, F., Abbeel, P., Finn, C., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.01523 · Cited by 357 · see citing works
Self-citationStochastic Variational Video PredictionPreprint
Babaeizadeh, M., Finn, C., Erhan, D., Campbell, R. H., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.11252 · Cited by 351 · see citing works
Unsupervised Learning of Disentangled Representations from Video
Denton, E., & Birodkar, V. (2017) · neural information processing systems · Cited by 321 · see citing works
PixelSNAIL: An Improved Autoregressive Generative ModelPreprint
Chen, X., Mishra, N., Rohaninejad, M., & Abbeel, P. (2017) · International Conference on Machine Learning · doi:10.48550/arxiv.1712.09763 · Cited by 319 · see citing works
Self-citationMetNet: A Neural Weather Model for Precipitation ForecastingPreprint
Sønderby, C. K., Espeholt, L., Heek, J., Dehghani, M., Oliver, A., Salimans, T., Agrawal, S., Hickey, J. J., & Kalchbrenner, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.12140 · Cited by 295 · see citing works
Parallel WaveNet: Fast High-Fidelity Speech SynthesisPreprint
van den Oord, A., Li, Y., Babuschkin, I., Simonyan, K., Vinyals, O., Kavukcuoglu, K., van den Driessche, G., Lockhart, E., Cobo, L. C., Stimberg, F., et al. (2017) · arXiv (Cornell University) · Cited by 262 · see citing works
Predicting Deeper into the Future of Semantic Segmentation
Luc, P., Neverova, N., Couprie, C., Verbeek, J. J., & LeCun, Y. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.77 · Cited by 250 · see citing works
Ho, J., Chen, X., Srinivas, A., Duan, Y., & Abbeel, P. (2019) · arXiv · Cited by 245 · see citing works
Self-citationTGIF: A New Dataset and Benchmark on Animated GIF Description
Li, Y., Song, Y., Cao, L., Tetreault, J., Goldberg, L., Jaimes, A., & Luo, J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.502 · Cited by 231 · see citing works
High Fidelity Speech Synthesis with Adversarial NetworksPreprint
Bińkowski, M., Donahue, J., Dieleman, S., Clark, A., Elsen, E., Casagrande, N., Cobo, L. C., & Simonyan, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11646 · Cited by 199 · see citing works
Adversarial Video Generation on Complex Datasets
Clark, A., Donahue, J., & Simonyan, K. (2019) · arXiv · doi:10.48550/arxiv.1907.06571 · Cited by 157 · see citing works
Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images
Child, R. (2020) · arXiv · doi:10.48550/arxiv.2011.10650 · Cited by 140 · see citing works
Saito, M., Saito, S., Koyama, M., & Kobayashi, S. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-020-01333-y · Cited by 116 · see citing works
Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional UpscalingPreprint
Menick, J., & Kalchbrenner, N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.01608 · Cited by 94 · see citing works
Scaling Autoregressive Video Models
Weissenborn, D., Täckström, O., & Uszkoreit, J. (2019) · arXiv · doi:10.48550/arxiv.1906.02634 · Cited by 93 · see citing works
Rakhimov, R., Volkhonskiy, D., Artemov, A., Zorin, D., & Burnaev, E. V. (2021) · International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP) · doi:10.5220/0010241801010112 · Cited by 85 · see citing works
Attentive Semantic Video Generation Using Captions
Marwah, T., Mittal, G., & Balasubramanian, V. N. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.159 · Cited by 69 · see citing works
Mittal, G., Marwah, T., & Balasubramanian, V. N. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123309 · Cited by 58 · see citing works
Transformation-based Adversarial Video Prediction on Large-Scale DataPreprint
Luc, P., Clark, A., Dieleman, S., Casas, D. D. L., Doron, Y., Cassirer, A., & Simonyan, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04035 · Cited by 53 · see citing works
Towards High Resolution Video Generation with Progressive Growing of Sliced Wasserstein GANsPreprint
Acharya, D., Huang, Z., Paudel, D. P., & Gool, L. V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.02419 · Cited by 45 · see citing works
TGANv2: Efficient Training of Large Models for Video Generation with Multiple Subsampling LayersPreprint
Saito, M., & Saito, S. (2018) · arXiv (Cornell University) · Cited by 35 · see citing works
Lower dimensional kernels for video discriminators
Kahembwe, E., & Ramamoorthy, S. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.09.016 · Cited by 33 · see citing works
Predicting Video with VQVAEPreprint
Walker, J., Razavi, A. S., & van den Oord, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.01950 · Cited by 29 · see citing works
Markov Decision Process for Video Generation
Yushchenko, V., Araslanov, N., & Roth, S. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00190 · Cited by 10 · see citing works
Flagged first, then by citations
4 of the 73 references couldn’t be matched to a known paper.
Research works printed but not linked: 11
- Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016. · arXiv
- Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Advances in neural information processing systems , pp. 2672-2680, 2014.
- Kalchbrenner, N., Oord, A. v. d., Simonyan, K., Danihelka, I., Vinyals, O., Graves, A., and Kavukcuoglu, K. Video pixel networks. arXiv preprint arXiv:1610.00527 , 2016. · arXiv
- Kalchbrenner, N., Oord, A., Simonyan, K., Danihelka, I., Vinyals, O., Graves, A., and Kavukcuoglu, K. Video pixel networks. In International Conference on Machine Learning , pp. 1771-1779. PMLR, 2017.
- Kingma, D. P. and Dhariwal, P. Glow: Generative flow with invertible 1x1 convolutions. arXiv preprint arXiv:1807.03039 , 2018. · arXiv
- Kingma, D. P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., and Welling, M. Improving variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems , 2016.
- Le Gall, D. Mpeg: A video compression standard for multimedia applications. Communications of the ACM , 34(4): 46-58, 1991.
- Mittal, G., Marwah, T., and Balasubramanian, V. N. Syncdraw: Automatic video generation using deep recurrent attentive architectures. In Proceedings of the 25th ACM international conference on Multimedia , pp. 1096-1104, 2017.
- Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, Ł., Shazeer, N., and Ku, A. Image transformer. arXiv preprint arXiv:1802.05751 , 2018. · arXiv
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. OpenAI Blog , 1(8):9, 2019.
- Zhang, H., Goodfellow, I., Metaxas, D., and Odena, A. Selfattention generative adversarial networks. In International Conference on Machine Learning , pp. 7354-7363. PMLR, 2019.
Cited by
Sorted by citations
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 15,880 · see citing works
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737 · see citing works
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models
Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S., & Kreis, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02161 · Cited by 724 · see citing works
VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
Tong, Z., Song, Y., Wang, J., & Wang, L. (2022) · arXiv · doi:10.48550/arxiv.2203.12602 · Cited by 909 · see citing works
“[84] Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas.”
BackgroundMethodStructure and Content-Guided Video Synthesis with Diffusion Models
Esser, P., Chiu, J., Atighehchian, P., Granskog, J., & Germanidis, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00675 · Cited by 413 · see citing works
Diffusion Autoencoders: Toward a Meaningful and Decodable Representation
Preechakul, K., Chatthee, N., Wizadwongsa, S., & Suwajanakorn, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01036 · Cited by 338 · see citing works
VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models
Chen, H., Zhang, Y., Cun, X., Xia, M., Wang, X., Weng, C., & Shan, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00698 · Cited by 257 · see citing works
Self-citationBailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory
Li, S., Yu, W., Gu, T., Lin, C., Wang, Q., Qian, C., Loy, C. C., & Liu, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01077 · Cited by 208 · see citing works
StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2
Skorokhodov, I., Tulyakov, S., & Elhoseiny, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00361 · Cited by 199 · see citing works
NaturalSpeech : End-to-End Text-to-Speech Synthesis With Human-Level Quality
Tan, X., Chen, J., Liu, H., Cong, J., Zhang, C., Liu, Y., Wang, X., Leng, Y., Yi, Y., He, L., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3356232 · Cited by 221 · see citing works
NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion
Wu, C., Liang, J., Ji, L., Yang, F., Fang, Y., Jiang, D., & Duan, N. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19787-1_41 · Cited by 167 · see citing works
Notice of Removal: VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation
Luo, Z., Chen, D., Zhang, Y., Huang, Y., Wang, L., Shen, Y., Zhao, D., Zhou, J., & Tan, T. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00984 · Cited by 144 · see citing works
Self-citationPreserve Your Own Correlation: A Noise Prior for Video Diffusion Models
Ge, S., Nah, S., Liu, G., Poon, T., Tao, A., Catanzaro, B., Jacobs, D. W., Huang, J., Liu, M.-Y., & Balaji, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02096 · Cited by 138 · see citing works
MAGVIT: Masked Generative Video Transformer
Yu, L., Cheng, Y., Sohn, K., Lezama, J., Zhang, H., Chang, H., Hauptmann, A. G., Yang, M., Yuan, H., Essa, I. A., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01008 · Cited by 130 · see citing works
Video Probabilistic Diffusion Models in Projected Latent Space
Yu, S., Sohn, K., Kim, S., & Shin, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01770 · Cited by 128 · see citing works
MARLIN: Masked Autoencoder for facial video Representation LearnINg
Cai, Z., Ghosh, S., Stefanov, K., Dhall, A., Cai, J., Rezatofighi, H., Haffari, R., & Hayat, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00150 · Cited by 113 · see citing works
Phenaki: Variable Length Video Generation From Open Domain Textual Description
Villegas, R., Babaeizadeh, M., Kindermans, P.-J., Moraldo, H., Zhang, H., Saffar, M. T., Castro, S., Kunze, J., & Erhan, D. (2022) · arXiv · doi:10.48550/arxiv.2210.02399 · Cited by 92 · see citing works
“Method FVD ↓ DVD-GAN [9] 109.8 VideoGPT [55] 103.3 TrIVD-GAN [27] 103.3 Transframer [31] 100.0 HARP [57] 99.3 CCVS [28] 99.0 Video Transformer [51] 94.0 FitVid [3] 93.6 MCVD [47] 89.5 NUWA [54] 86.9 RaMViD [19] 84.2 Phenaki (Ours) 97.0 Phenaki is not specifically designed for video prediction,…”
BackgroundInfluentialFrido: Feature Pyramid Diffusion for Complex Scene Image Synthesis
Fan, W.-C., Chen, Y., Chen, D., Yu, C., Yuan, L., & Wang, Y.-C. F. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25133 · Cited by 69 · see citing works
When Does Sora Show: The Beginning of TAO to Imaginative Intelligence and Scenarios Engineering
Wang, F., Miao, Q., Li, L., Ni, Q., Li, X., Li, J., Fan, L., Tian, Y., & Han, Q. (2024) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2024.124383 · Cited by 64 · see citing works
Understanding World or Predicting Future? A Comprehensive Survey of World Models
Ding, J., Zhang, Y., Shang, Y., Zhang, Y., Zong, Z., Feng, J., Yuan, Y., Su, H., Li, N., Sukiennik, N., et al. (2026) · ACM Computing Surveys · doi:10.1145/3746449 · Cited by 58 · see citing works
Self-citationVIDM: Video Implicit Diffusion Models
Mei, K., & Patel, V. M. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i8.26094 · Cited by 60 · see citing works
Dream Video: Composing Your Dream Videos with Customized Subject and Motion
Wei, Y., Zhang, S., Qing, Z., Yuan, H., Liu, Z., Liu, Y., Zhang, Y., Zhou, J., & Shan, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00625 · Cited by 60 · see citing works
Self-citationNUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation
Yin, S., Wu, C., Yang, H., Wang, J., Wang, X., Ni, M., Yang, Z., Li, L., Liu, S., Yang, F., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.73 · Cited by 68 · see citing works
SimDA: Simple Diffusion Adapter for Efficient Video Generation
Xing, Z., Dai, Q., Hu, H., Wu, Z., & Jiang, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00748 · Cited by 59 · see citing works
Modeling Atomistic Dynamic Fracture Mechanisms Using a Progressive Transformer Diffusion Model
Buehler, M. J. (2022) · Journal of Applied Mechanics · doi:10.1115/1.4055730 · Cited by 48 · see citing works
Make Pixels Dance: High-Dynamic Video Generation
Zeng, Y., Wei, G., Zheng, J., Zou, J., Yang, W., Zhang, Y., & Li, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00845 · Cited by 59 · see citing works
Self-citationDysen-VDM: Empowering Dynamics-Aware Text-to-Video Diffusion with LLMs
Fei, H., Wu, S., Ji, W., Zhang, H., & Chua, T. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00730 · Cited by 53 · see citing works
Zheng, C., & Vedaldi, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02084 · Cited by 46 · see citing works
Luo, X., Sun, J., Zhang, R., Chi, P., & Cui, R. (2024) · Computers & Geosciences · doi:10.1016/j.cageo.2024.105541 · Cited by 46 · see citing works
DLFormer: Discrete Latent Transformer for Video Inpainting
Ren, J., Zheng, Q., Zhao, Y., Xu, X., & Li, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00350 · Cited by 45 · see citing works
Video prediction by efficient transformers
Ye, X., & Bilodeau, G.-A. (2022) · Image and Vision Computing · doi:10.1016/j.imavis.2022.104612 · Cited by 47 · see citing works
Wang, Y., Yang, D., Brémond, F., & Dantcheva, A. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3449075 · Cited by 105 · see citing works
Text2Performer: Text-Driven Human Video Generation
Jiang, Y., Yang, S., Koh, T. L., Wu, W., Loy, C. C., & Liu, Z. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02079 · Cited by 41 · see citing works
Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis
Menapace, W., Siarohin, A., Skorokhodov, I., Deyneka, E., Chen, T.-S., Kag, A., Fang, Y., Stoliar, A., Ricci, E., Ren, J., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00672 · Cited by 42 · see citing works
Zheng, Q., & Zhang, D. (2022) · Computational Geosciences · doi:10.1007/s10596-022-10144-8 · Cited by 38 · see citing works
Beyond Deepfake Images: Detecting AI-Generated Videos
Vahdati, D. S., Nguyen, T. D., Azizpour, A., & Stamm, M. C. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00443 · Cited by 37 · see citing works
Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioning
Han, L., Ren, J., Lee, H.-Y., Barbieri, F., Olszewski, K., Minaee, S., Metaxas, D., & Tulyakov, S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00360 · Cited by 38 · see citing works
Retracted: Sora for Scenarios Engineering of Intelligent Vehicles: V&V, C&C, and Beyonds
Li, X., Miao, Q., Li, L., Ni, Q., Fan, L., Wang, Y., Tian, Y., & Wang, F. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3379989 · Cited by 38 · see citing works
RetractedBailando ++: 3D Dance GPT With Choreographic Memory
Li, S., Yu, W., Gu, T., Lin, C., Wang, Q., Qian, C., Loy, C. C., & Liu, Z. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3319435 · Cited by 32 · see citing works
VPTR: Efficient Transformers for Video Prediction
Ye, X., & Bilodeau, G.-A. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956707 · Cited by 31 · see citing works
Vlogger: Make Your Dream A Vlog
Zhuang, S., Li, K., Chen, X., Wang, Y., Liu, Z., Qiao, Y., & Wang, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00841 · Cited by 32 · see citing works
Breathing Life Into Sketches Using Text-to-Video Priors
Gal, R., Vinker, Y., Alaluf, Y., Bermano, A. H., Cohen‐Or, D., Shamir, A., & Chechik, G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00414 · Cited by 35 · see citing works
MoStGAN-V: Video Generation with Temporal Motion Styles
Shen, X., Li, X., & Elhoseiny, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00547 · Cited by 30 · see citing works
On the Content Bias in Fréchet Video Distance
Ge, S., Mahapatra, A., Parmar, G., Zhu, J.-Y., & Huang, J.-B. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00695 · Cited by 31 · see citing works
FitVid: Overfitting in Pixel-Level Video PredictionPreprint
Babaeizadeh, M., Saffar, M., Nair, S., Levine, S., Finn, C., & Erhan, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.13195 · Cited by 39 · see citing works
GAIA-1: A Generative World Model for Autonomous Driving
Hu, A., Russell, L., Yeo, H., Murez, Z., Fedoseev, G., Kendall, A., Shotton, J., & Corrado, G. (2023) · arXiv · doi:10.48550/arxiv.2309.17080 · Cited by 28 · see citing works
“Likewise, they can operate at the pixel level [72, 73], or in a discrete learned token space [74, 75, 76, 77].”
BackgroundRV-GAN: Recurrent GAN for Unconditional Video Generation
Gupta, S., Keshari, A., & Das, S. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00220 · Cited by 27 · see citing works
ART•V: Auto-Regressive Text-to-Video Generation with Diffusion Models
Weng, W., Feng, R., Wang, Y., Dai, Q., Wang, C., Yin, D., Zhao, Z., Qiu, K., Bao, J., Yuan, Y., et al. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00735 · Cited by 26 · see citing works
Self-citationSize-invariant 3D generation from a single 2D rock image
Phan, J., Ruspini, L. C., Kiss, G. H., & Lindseth, F. (2022) · Journal of Petroleum Science and Engineering · doi:10.1016/j.petrol.2022.110648 · Cited by 26 · see citing works
Tell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generation
Fu, T.-J., Yu, L., Zhang, N., Fu, C.-Y., Su, J.-C., Wang, W. Y., & Bell, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01029 · Cited by 24 · see citing works
InstructVideo: Instructing Video Diffusion Models with Human Feedback
Yuan, H., Zhang, S., Wang, X., Wei, Y., Feng, T., Pan, Y., Zhang, Y., Liu, Z., Albanie, S., & Ni, D. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00618 · Cited by 24 · see citing works
Self-citationControllable Video Generation With Text-Based Instructions
Köksal, A., Ak, K. E., Sun, Y., Rajan, D., & Lim, J. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3262972 · Cited by 21 · see citing works
CHeart: A Conditional Spatio-Temporal Generative Model for Cardiac Anatomy
Qiao, M., Wang, S., Qiu, H., de Marvao, A., O’Regan, D. P., Rueckert, D., & Bai, W. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3331982 · Cited by 21 · see citing works
Character-centric Story Visualization via Visual Planning and Token Alignment
Chen, H., Han, R., Wu, T.-L., Nakayama, H., & Peng, N. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.565 · Cited by 20 · see citing works
Wang, Z., Wang, J., Liu, Z., & Qiu, Q. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02162 · Cited by 20 · see citing works
Efficient Video Prediction via Sparsely Conditioned Flow Matching
Davtyan, A., Sameni, S., & Favaro, P. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02126 · Cited by 20 · see citing works
Imasato, N., Miyazawa, K., Duncan, C., & Nagai, T. (2023) · IEEE Access · doi:10.1109/access.2023.3280603 · Cited by 18 · see citing works
Video Diffusion Models with Local-Global Context Guidance
Yang, S., Zhang, L., Liu, Y., Jiang, Z., & He, Y. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/182 · Cited by 18 · see citing works
A Generative Model for Surrogates of Spatial-Temporal Wildfire Nowcasting
Cheng, S. L., Guo, Y., & Arcucci, R. (2023) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2023.3298535 · Cited by 16 · see citing works
MOSO: Decomposing MOtion, Scene and Object for Video Prediction
Sun, M., Wang, N., Zhu, X., & Liu, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01796 · Cited by 17 · see citing works
Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model
He, X., Huang, Q., Zhang, Z., Lin, Z., Wu, Z., Yang, S., Li, M., Chen, Z., Xu, S., & Wu, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00220 · Cited by 16 · see citing works
Mirasol3B: A Multimodal Autoregressive Model for Time-Aligned and Contextual Modalities
Piergiovanni, A., Noble, I., Kim, D., Ryoo, M. S., Gomes, V., & Angelova, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02531 · Cited by 17 · see citing works
MagicStick: Controllable Video Editing via Control Handle Transformations
Ma, Y., Cun, X., Liang, S., Xing, J., He, Y., Qi, C., Chen, S., & Chen, Q. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00909 · Cited by 26 · see citing works
A unified model for continuous conditional video prediction
Ye, X., & Bilodeau, G.-A. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00368 · Cited by 16 · see citing works
Skillful Precipitation Nowcasting Using Physical‐Driven Diffusion Networks
Wang, R., Fung, J. C. H., & Lau, A. K. H. (2024) · Geophysical Research Letters · doi:10.1029/2024gl110832 · Cited by 14 · see citing works
MS-LSTM: Exploring spatiotemporal multiscale representations in video prediction domain
Ma, Z., Zhang, H., & Liu, J. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110731 · Cited by 14 · see citing works
StyleInV: A Temporal Style Modulated Inversion Network for Unconditional Video Generation
Wang, Y., Jiang, L., & Loy, C. C. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02089 · Cited by 14 · see citing works
Continuous PDE Dynamics Forecasting with Implicit Neural RepresentationsPreprint
Yin, Y., Kirchmeyer, M., Franceschi, J.-Y., Rakotomamonjy, A., & Gallinari, P. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.14855 · Cited by 16 · see citing works
Co-Speech Gesture Synthesis using Discrete Gesture Token Learning
Lu, S., Yoon, Y., & Feng, A. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10342027 · Cited by 13 · see citing works
HARP: Autoregressive Latent Video Prediction with High-Fidelity Image Generator
Seo, Y., Lee, K., Liu, F., James, S., & Abbeel, P. (2022) · Proceedings - International Conference on Image Processing · doi:10.1109/icip46576.2022.9897982 · Cited by 13 · see citing works
Self-citationGenie: Generative Interactive Environments
Bruce, J., Dennis, M., Edwards, A., Parker-Holder, J., Shi, Y., Hughes, E., Lai, M., Mavalankar, A., Steigerwald, R., Apps, C., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.15391 · Cited by 23 · see citing works
“…video (Blattmann et al., 2023b; Brooks et al., 2024; Clark et al., 2019; Finn et al., 2016; Ho et al., 2022a,b; Höppe et al., 2022; Kalchbrenner et al., 2017; Le Moing et al., 2021; Lotter et al., 2017; Luc et al., 2020; Singer et al., 2023; Walker et al., 2021; Yan et al., 2021; Yu et al., 2023).”
Sounding Video Generator: A Unified Framework for Text-Guided Sounding Video Generation
Liu, J.-W., Wang, W., Chen, S., Zhu, X., & Liu, J. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3262180 · Cited by 12 · see citing works
Modeling Spatio-Temporal Dynamical Systems With Neural Discrete Learning and Levels-of-Experts
Wang, K., Wu, H., Zhang, G., Fang, J., Liang, Y., Wu, Y., Zimmermann, R., & Wang, Y. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3363711 · Cited by 12 · see citing works
Object Discovery from Motion-Guided Tokens
Bao, Z., Tokmakov, P., Wang, Y.-X., Gaidon, A., & Hebert, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02200 · Cited by 12 · see citing works
Cosmos World Foundation Model Platform for Physical AI
NVIDIA, :, Agarwal, N., Ali, A., Bala, M., Balaji, Y., Barker, E., Cai, T., Chattopadhyay, P., Chen, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.03575 · Cited by 13 · see citing works
Self-citationMotionVideoGAN: A Novel Video Generator Based on the Motion Space Learned From Image Pairs
Zhu, J., Ma, H., Chen, J., & Yuan, J. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3251095 · Cited by 11 · see citing works
Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis
Gu, Y., Wang, X., Ge, Y., Shan, Y., & Shou, M. Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00729 · Cited by 11 · see citing works
Time Dependent Image Generation of Plants from Incomplete Sequences with CNN-Transformer
Drees, L., Weber, I., Rußwurm, M., & Roscher, R. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16788-1_30 · Cited by 11 · see citing works
Transformers predicting the future. Applying attention in next-frame and time series forecastingPreprint
Cholakov, R., & Kolev, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.08224 · Cited by 20 · see citing works
Sign Language Production with Latent Motion Transformer
Xie, P., Peng, T., Du, Y., & Zhang, Q. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00300 · Cited by 10 · see citing works
MIGC++: Advanced Multi-Instance Generation Controller for Image Synthesis
Zhou, D., Li, Y., Ma, F., Yang, Z., & Yang, Y. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3510752 · Cited by 46 · see citing works
Neural network processing of holographic images
Schreck, J. S., Gantos, G., Hayman, M., Bansemer, A. R., & Gagne, D. J. (2022) · Atmospheric measurement techniques · doi:10.5194/amt-15-5793-2022 · Cited by 10 · see citing works
UniAdapter: All-in-One Control for Flexible Video Generation
Wang, C., Hu, P., Zhao, H., Guo, Y., Gu, J., Dong, X. M., Han, J., Xu, H., & Liang, X. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3532495 · Cited by 8 · see citing works
Interactive wearable digital devices for blind and partially sighted people
Xue, W., Ren, Y., Tang, Y. W., Gong, Z., Zhang, T., Chen, Z., Dong, X., Ma, X., Wang, Z., Xu, H., et al. (2025) · Nature Reviews Electrical Engineering · doi:10.1038/s44287-025-00170-w · Cited by 8 · see citing works
Ullah, A., Numan, M., Khalid, M. N. A., & Majid, A. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129767 · Cited by 9 · see citing works
SSGVS: Semantic Scene Graph-to-Video Synthesis
Cong, Y., Yi, J., Rosenhahn, B., & Yang, M. Y. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00254 · Cited by 8 · see citing works
3D PET image generation with tumour masks using TGAN
Bergen, R. V., Rajotte, J.-F., Yousefirizi, F., Klyuzhin, I. S., Rahmim, A., & Ng, R. T. (2022) · SPIE Medical Imaging: Image Processing · doi:10.1117/12.2611292 · Cited by 8 · see citing works
Assessing privacy leakage in synthetic 3-D PET imaging using transversal GAN
Bergen, R. V., Rajotte, J.-F., Yousefirizi, F., Rahmim, A., & Ng, R. T. (2023) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2023.107910 · Cited by 7 · see citing works
Post-Deployment Fine-Tunable Semantic Communication
Si, P., Liu, R., Qian, L., Zhao, J., & Lam, K. (2024) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2024.3433479 · Cited by 7 · see citing works
Yoo, J., Kim, S., Lee, D., Kim, C., & Hong, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02192 · Cited by 7 · see citing works
GSSTU: Generative Spatial Self-Attention Transformer Unit for Enhanced Video Prediction
Singh, B., Singh, D., Kaushal, R., Halder, A., & Chattopadhyay, P. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3359716 · Cited by 7 · see citing works
Prospects and technology of embodied intelligent humanoid robots driven by AI large models
Kai, Z. E. N. G., Wang, Y., Tan, H., Fang, Q., Wang, Y., & Yuan, L. (2025) · Scientia Sinica Informationis · doi:10.1360/ssi-2024-0350 · Cited by 6 · see citing works
A Two-Part Transformer Network for Controllable Motion Synthesis
Hou, S., Tao, H., Bao, H., & Xu, W. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3284402 · Cited by 6 · see citing works
Regeneration Learning: A Learning Paradigm for Data Generation
Tan, X., Qin, T., Bian, J., Liu, T., & Bengio, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30271 · Cited by 6 · see citing works
Video Prediction by Modeling Videos as Continuous Multi-Dimensional Processes
Shrivastava, G., & Shrivastava, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00691 · Cited by 6 · see citing works
Karaarslan, E., & Aydin, Ö. F. (2024) · doi:10.36227/techrxiv.170862194.43871446/v2 · Cited by 6 · see citing works
Collaboratively Self-Supervised Video Representation Learning for Action Recognition
Zhang, J., Wan, Z., Hu, L., Lin, S., Wu, S., & Shan, S. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3531772 · Cited by 5 · see citing works
Nowcasting of Extreme Precipitation Using Deep Generative Models
Bi, H., Kyryliuk, M., Wang, Z., Meo, C., Wang, Y., Imhoff, R. O., Uijlenhoet, R., & Dauwels, J. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094988 · Cited by 5 · see citing works
Mining and Applying Composition Knowledge of Dance Moves for Style-Concentrated Dance Generation
Zhang, X., Yang, S., Xu, Y., Zhang, W., & Gao, L. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i4.25673 · Cited by 5 · see citing works
Learning Semantic Textual Similarity via Topic-informed Discrete Latent Variables
Yu, E., Du, L., Jin, Y., Wei, Z., & Chang, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.328 · Cited by 5 · see citing works
- Authors
- Yan, W., Zhang, Y., Abbeel, P., & Srinivas, A.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2104.10157
- OpenAlex
- W3152733922
- Cited by
- 156
- References
- 73
- Citations by year
- 2026: 2 · 2025: 21 · 2024: 44 · 2023: 50 · 2022: 32 · 2021: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- video generation, vector quantized variational autoencoder, discrete latent representations, 3D convolution, autoregressive models, UCF101
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
