The DOI matches its DataCite record; 69 of the 73 references it cites were found in a registry or an index.
VideoGPT: Video Generation using VQ-VAE and Transformers
Open accessAbstract
We present VideoGPT: a conceptually simple architecture for scaling likelihood based generative modeling to natural videos. VideoGPT uses VQ-VAE that learns downsampled discrete latent representations of a raw video by employing 3D convolutions and axial self-attention. A simple GPT-like architecture is then used to autoregressively model the discrete latents using spatio-temporal position encodings. Despite the simplicity in formulation and ease of training, our architecture is able to generate samples competitive with state-of-the-art GAN models for video generation on the BAIR Robot dataset, and generate high fidelity natural videos from UCF-101 and Tumbler GIF Dataset (TGIF). We hope our proposed architecture serves as a reproducible reference for a minimalistic implementation of transformer based video generation models. Samples and code are available at https://wilson1yan.github.io/videogpt/index.html
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To scale to higher resolutions, several two-stage approaches [23,67,101,103] use ARMs to model a compressed latent image space instead of raw pixels.”
From High-Resolution Image Synthesis with Latent Diffusion Models (2022) · cites this paper as [101]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“While VideoGPT did not outperform the best GAN-based models at the time, namely TGAN-v2 and DVD-GAN, it achieved a respectable IS score on the UCF-101 benchmark considering its simple architecture.”
From Video Diffusion Models: A Survey (2024) · cites this paper as Yan et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“VQ-VAEs [67, 101] use autoregressive models to learn an expressive prior over a discretized latent space.”
From High-Resolution Image Synthesis with Latent Diffusion Models (2022) · cites this paper as [101]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
GAN（Generative Adversarial Nets）
淳司, 柴. (2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 20,507
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
A Style-Based Generator Architecture for Generative Adversarial Networks
Karras, T., Laine, S., & Aila, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00453 · Cited by 11,714
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593
Denoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712
Self-citation
Research works printed but not linked: 11
- Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016. · arXiv
- Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Advances in neural information processing systems , pp. 2672-2680, 2014.
- Kalchbrenner, N., Oord, A. v. d., Simonyan, K., Danihelka, I., Vinyals, O., Graves, A., and Kavukcuoglu, K. Video pixel networks. arXiv preprint arXiv:1610.00527 , 2016. · arXiv
- Kalchbrenner, N., Oord, A., Simonyan, K., Danihelka, I., Vinyals, O., Graves, A., and Kavukcuoglu, K. Video pixel networks. In International Conference on Machine Learning , pp. 1771-1779. PMLR, 2017.
- Kingma, D. P. and Dhariwal, P. Glow: Generative flow with invertible 1x1 convolutions. arXiv preprint arXiv:1807.03039 , 2018. · arXiv
- Kingma, D. P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., and Welling, M. Improving variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems , 2016.
- Le Gall, D. Mpeg: A video compression standard for multimedia applications. Communications of the ACM , 34(4): 46-58, 1991.
- Mittal, G., Marwah, T., and Balasubramanian, V. N. Syncdraw: Automatic video generation using deep recurrent attentive architectures. In Proceedings of the 25th ACM international conference on Multimedia , pp. 1096-1104, 2017.
- Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, Ł., Shazeer, N., and Ku, A. Image transformer. arXiv preprint arXiv:1802.05751 , 2018. · arXiv
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. OpenAI Blog , 1(8):9, 2019.
- Zhang, H., Goodfellow, I., Metaxas, D., and Odena, A. Selfattention generative adversarial networks. In International Conference on Machine Learning , pp. 7354-7363. PMLR, 2019.
Cited by
Sorted by citations
Items 101–166 of 166
Learn the Force We Can: Enabling Sparse Motion Control in Multi-Object Video Generation
Davtyan, A., & Favaro, P. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i10.29056 · Cited by 6
Art creator: Steering styles in diffusion model
Tang, S., Qian, W., Liu, P., & Cao, J. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129511 · Cited by 5
Survey on Multimodal Transformers for RobotsPreprint
Miyazawa, K., & Nagai, T. (2023) · doi:10.36227/techrxiv.21993317 · Cited by 5
DiffGait: Gait Representation Augmentation of mmWave Sensing Through a Latent Diffusion Model
Li, J., Li, B., Wang, L., & Liu, W. (2024) · IEEE Sensors Journal · doi:10.1109/jsen.2024.3457946 · Cited by 6
Personalized News Recommendation Towards the Era of LLMs: Review and Prospect
Li, J., Liu, Z., Hu, L., Rao, Y., Liu, B., Fang, B., & Nie, L. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3581806 · Cited by 5
Recursive variational autoencoders for 3D blood vessel generative modeling
Feldman, P., Fainstein, M., Siless, V., Delrieux, C. A., & Iarussi, E. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103703 · Cited by 4
WALDO: Future Video Synthesis using Object Layer Decomposition and Parametric Flow Prediction
Moing, G. L., Ponce, J., & Schmid, C. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02123 · Cited by 4
Karaarslan, E., & Aydin, Ö. F. (2024) · doi:10.36227/techrxiv.170862194.43871446/v1 · Cited by 8
AI-Generated Videos and Deepfakes: A Technical PrimerPreprint
Sufian, A. (2024) · doi:10.36227/techrxiv.172348990.01007128/v1 · Cited by 5
Hou, Y., Zhang, W., Zhu, Z., & Yu, H. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113509 · Cited by 3
NextGen Dynamic Video Generator using AI
Roy, A., Kishore, N., & Anand, M. V. (2023) · International Conference on Smart Structures and Systems (ICSSS) · doi:10.1109/icsss58085.2023.10407412 · Cited by 3
Bridging Language, Vision and Action: Multimodal VAEs in Robotic Manipulation Tasks
Šejnová, G., Vavrečka, M., & Štěpánová, K. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802160 · Cited by 3
ImproveYourVideos: Architectural Improvements for Text-to-Video Generation Pipeline
Arkhipkin, V., Shaheen, Z., Vasilev, V., Dakhova, E., Sobolev, K., Кузнецов, А. В., & Dimitrov, D. (2024) · IEEE Access · doi:10.1109/access.2024.3522510 · Cited by 9
Hippocampo-neocortical interaction as compressive retrieval-augmented generation
Spens, E., & Burgess, N. (2026) · Nature Communications · doi:10.1038/s41467-026-74357-6 · Cited by 5
Liang, 林. L., & Binbin, 杨. Y. (2023) · Acta Optica Sinica · doi:10.3788/aos230758 · Cited by 4
Poredi, N., Sudarsan, M., Solomon, E., Nagothu, D., & Chen, Y. (2024) · Disruptive Technologies in Information Sciences · doi:10.1117/12.3013240 · Cited by 3
DTR: Dynamic Tree-Ring Watermarking Framework for Diffusion-Based Video Generation
Zeng, S., Yang, L., Yang, J., Wang, Y., & Gao, T. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888152 · Cited by 4
Qian, K., Suarez, G. O., Nambara, T., Kanekiyo, T., & Zhang, Y. J. (2025) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2025.118003 · Cited by 6
Self-citationPIP: Physical Interaction Prediction via Mental Imagery with Span Selection.Preprint
Duan, J., Yu, S., Poria, S., Wen, B., & Tan, C. (2021) · arXiv (Cornell University) · Cited by 5
Addressing Character Consistency Challenges in AI Filmmaking
Kalinski, M., Podstawski, M., Ostrowski, K., Kudelska, M., Bartkowiak, P., Łyczko, P., Piasecki, M., Visscher, T., & Wang, H. (2025) · 2012 International Conference on Computing, Networking and Communications (ICNC) · doi:10.1109/icnc64010.2025.10993630 · Cited by 2
HVQ-VAE: Variational auto-encoder with hyperbolic vector quantization
Chen, S., Fang, P., Harandi, M. T., Le, T., Cai, J., & Phung, D. (2025) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2025.104392 · Cited by 2
Self-Supervised Spontaneous Latent-Based Facial Expression Sequence Generation
Yap, C. H., Yap, M. H., Davison, A. K., & Cunningham, R. J. (2023) · IEEE Open Journal of Signal Processing · doi:10.1109/ojsp.2023.3275052 · Cited by 2
Modeling Wound Healing Using Vector Quantized Variational Autoencoders and Transformers
Backová, L., Bengoetxea, G., Rogalla, S., Franco-Barranco, D., Solon, J., & Arganda‐Carreras, I. (2023) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi53787.2023.10230571 · Cited by 3
TiV-ODE: A Neural ODE-based Approach for Controllable Video Generation From Text-Image Pairs
Xu, Y., Li, N., Goel, A., Yao, Z., Guo, Z., Kasaei, H., Kasaei, M., & Li, Z. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610149 · Cited by 2
STNAM: A Spatio-Temporal Non-autoregressive Model for Video Prediction
Yu, Y., & Meng, Z. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650569 · Cited by 2
Ha, S., Kim, S.-Y., & Lim, S. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3536222 · Cited by 2
Kang, J. (2025) · Journal of Digital Contents Society · doi:10.9728/dcs.2025.26.3.577 · Cited by 2
NWT: Towards natural audio-to-video generation with representation learningPreprint
Mama, R., Tyndel, M. S., Kadhim, H., Clifford, C., & Thurairatnam, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.04283 · Cited by 16
CHeart: A Conditional Spatio-Temporal Generative Model for Cardiac AnatomyPreprint
Qiao, M., Wang, S., Qiu, H., de Marvao, A., O’Regan, D. P., Rueckert, D., & Bai, W. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.13098 · Cited by 2
Shrivastava, G., & Shrivastava, A. (2024) · Qeios · doi:10.32388/dm98uz · Cited by 3
GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving
Russell, L., Hu, A., Bertoni, L., Fedoseev, G., Shotton, J., Arani, E., & Corrado, G. (2025) · arXiv · doi:10.48550/arxiv.2503.20523 · Cited by 1
“Autoregressive and masked generative models using quantized tokens have demonstrated strong modeling of temporal dependencies and motion dynamics [34, 35, 36, 37, 38, 39, 40, 41, 42].”
Video Diffusion Models: A Survey
Melnik, A., Ljubljanac, M., Lu, C., Yan, Q., Ren, W., & Ritter, H. (2024) · arXiv · doi:10.48550/arxiv.2405.03150 · Cited by 4
VILP: Imitation Learning With Latent Video Planning
Xu, Z., Qiu, Q., & She, Y. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3542317 · Cited by 6
Joint Global–Local Frames Modeling to enhance semantic alignment for zero-shot long video editing
Yu, Z., Qiao, P., Chen, J., & Zhang, X. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130836 · Cited by 1
Toward Accurate Quality Assessment of Machine-Generated Infrared Video Using Fréchet Video Distance
Lu, H., Wang, S., Zhang, D., Huang, B., Chen, E., & Sui, Y. (2024) · IEEE Access · doi:10.1109/access.2024.3453406 · Cited by 1
SNED: Superposition Network Architecture Search for Efficient Video Diffusion Model
Li, Z., Yan, K., Liu, Y., Liu, D., Hinz, T., Liu, F., & Wang, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00827 · Cited by 1
Cross Attention Transformers for Multi-modal Unsupervised Whole-Body PET Anomaly Detection
Patel, A. S., Tudosiu, P.-D., Pinaya, W. H. L., Cook, G., Goh, V., Sebastien Ourselin, A., & Cardoso, M. J. (2023) · The Journal of Machine Learning for Biomedical Imaging · doi:10.59275/j.melba.2023-18c1 · Cited by 1
Survey on Multimodal Transformers for RobotsPreprint
Miyazawa, K., & Nagai, T. (2023) · doi:10.36227/techrxiv.21993317.v1 · Cited by 2
REGIS: Refining Generated Videos via Iterative Stylistic RedesigningPreprint
Abohwo, J. (2023) · Research Square · doi:10.21203/rs.3.rs-3541408/v1 · Cited by 2
MoCha: Towards Movie-Grade Talking Character Synthesis
Wei, C., Sun, B., Ma, H., Hou, J., Juefei-Xu, F., He, Z., Dai, X., Zhang, L., Li, K., Hou, T., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.23307
“Automating film production holds immense commercial potential, promising to democratize cinematic-level story-telling by enabling content creators to effortlessly generate films through natural language [15, 36, 41, 47].”
Pathologies in priors and inference for Bayesian transformersPreprint
Cinquin, T., Immer, A., Horn, M., & Fortuin, V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04020 · Cited by 7
Layered Controllable Video GenerationPreprint
Huang, J., Jin, Y., Yi, K. M., & Sigal, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.12747
GONG, B., Langguth, M., Ji, Y., et al. (2022) · Geoscientific model development · doi:10.5194/gmd-2021-430-cec1
GONG, B. (2022) · Geoscientific model development · doi:10.5194/gmd-2021-430-ac1
GONG, B., Langguth, M., Ji, Y., et al. (2022) · Geoscientific model development · doi:10.5194/gmd-2021-430-cec2
Gong, B., Langguth, M., Ji, Y., et al. (2022) · Geoscientific model development · doi:10.5194/gmd-2021-430-rc1
Gong, B., Langguth, M., Ji, Y., et al. (2022) · Geoscientific model development · doi:10.5194/gmd-2021-430-rc2
GONG, B. (2022) · Geoscientific model development · doi:10.5194/gmd-2021-430-ac3
GONG, B. (2022) · Geoscientific model development · doi:10.5194/gmd-2021-430-ac2
Schreck, J. S., Gantos, G., Hayman, M., et al. (2022) · Atmospheric measurement techniques · doi:10.5194/amt-2022-97-rc1
Schreck, J. S., Gantos, G., Hayman, M., et al. (2022) · Atmospheric measurement techniques · doi:10.5194/amt-2022-97-rc2
Schreck, J. S. (2022) · Atmospheric measurement techniques · doi:10.5194/amt-2022-97-ac1
Schreck, J. S. (2022) · Atmospheric measurement techniques · doi:10.5194/amt-2022-97-ac2
Cascaded Video Generation for Videos In-the-Wild
Castrejón, L., Ballas, N., & Courville, A. C. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956228
Learning to control visual data translation
Koksal, A. (2023) · doi:10.32657/10356/165566
Image and video generation via deep learning
Jiang, L. (2023) · doi:10.32657/10356/172067
Deep learning application on head CT images
How, C. H. (2023) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/172627
VidStyleODE: Disentangled Video Editing via StyleGAN and NeuralODEs
Haji-Ali, M., Bond, A., Karacan, L., Birdal, T., Erdem, E., Ceylan, D., & Erdem, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00692 · Cited by 1
PredToken: Predicting Unknown Tokens and Beyond with Coarse-to-Fine Iterative Decoding
Nie, X., Jin, H., Yan, Y., Chen, X., Zhu, Z., & Qi, D. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01718
Thermal Videodiff (TVD): A Diffusion Architecture For Thermal Video Synthesis
Qazi, T., & Lall, B. (2024) · Proceedings - International Conference on Image Processing · doi:10.1109/icip51287.2024.10647272
Enhancing Image Generation by fusing Auto Encoder & Transformative Generation Approach
Sherley, S. E. F., Sanjay, T., Shri Kaanth, P., & Jeffrey Samuel, S. (2024) · International Conference on Electrical Electronics and Computing Technologies (ICEECT) · doi:10.1109/iceect61758.2024.10739002
Optimizing Semantic and Texture Consistency in Video GenerationPreprint
Yu, X., & Zhang, J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5137309
Lampl, N. M., de Freitas, J. M., Fuchs, A., Brezina, B., Klitzsch, M., & Pernkopf, F. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888765
Adaptive Large Language Models via Attention Shortcuts
Verma, P., & Pilancı, M. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889322
Gutiérrez, J. M. P., Sharma, A., Tsuda, S., & Cronin, L. (2026) · Artificial Life · doi:10.1162/artl.a.460
VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning
Tan, H., Lei, J., Wolf, T., & Bansal, M. (2021) · arXiv · doi:10.48550/arxiv.2106.11250 · Cited by 35
Metadata
- Authors
- Yan, W., Zhang, Y., Abbeel, P., & Srinivas, A.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2104.10157
- OpenAlex
- W3152733922
- Cited by
- 156
- References
- 73
- Citations by year
- 2026: 2 · 2025: 21 · 2024: 44 · 2023: 50 · 2022: 32 · 2021: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- video generation, vector quantized variational autoencoder, discrete latent representations, 3D convolution, autoregressive models, UCF101
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
