MoCoGAN: Decomposing Motion and Content for Video Generation
Open accessAbstract
Visual signals in a video can be divided into content and motion. While content specifies which objects are in the video, motion describes their dynamics. Based on this prior, we propose the Motion and Content decomposed Generative Adversarial Network (MoCoGAN) framework for video generation. The proposed framework generates a video by mapping a sequence of random vectors to a sequence of video frames. Each random vector consists of a content part and a motion part. While the content part is kept fixed, the motion part is realized as a stochastic process. To learn motion and content decomposition in an unsupervised manner, we introduce a novel adversarial learning scheme utilizing both image and video discriminators. Extensive experimental results on several challenging datasets with qualitative and quantitative comparison to the state-of-the-art approaches, verify effectiveness of the proposed framework. In addition, we show that MoCoGAN allows one to generate videos with same content but different motion as well as videos with different content and same motion. Our code is available at https://github.com/sergeytulyakov/mocogan.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
MoCoGAN achieved higher IS scores than the preceding TGAN and VGAN models on unconditional video generation on UCF-101 and Tai-Chi-HD datasets.
“MoCoGAN was evaluated on unconditional video generation on the UCF-101 and Tai-Chi-HD datasets and achieved higher IS scores than the preceding TGAN and VGAN models.”
From Video Diffusion Models: A Survey (2024) · cites this paper as Tulyakov et al., 2018
In Section 4, Table 2 reports a higher inception score (IS) for MoCoGAN than for TGAN and VGAN on UCF101. For Tai-Chi, the paper reports user-preference scores in Table 3, not IS scores, so the Tai-Chi portion of the claim is unsupported. (LLM, conf=0.90)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields
Cao, Z., Simon, T., Wei, S.-E., & Sheikh, Y. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.143 · Cited by 7,539 · see citing works
Unsupervised Learning of Video Representations using LSTMs
Srivastava, N., Mansimov, E., & Salakhutdinov, R. (2015) · arXiv · doi:10.48550/arxiv.1502.04681 · Cited by 1,861 · see citing works
Decomposing Motion and Content for Natural Video Sequence Prediction
Villegas, R., Yang, J., Hong, S., Lin, X., & Lee, H. (2017) · arXiv · doi:10.48550/arxiv.1706.08033 · Cited by 488 · see citing works
Flagged first, then by citations
Research works printed but not linked: 39
- N. Aifanti, C. Papachristou, and A. Delopoulos. The mug facial expression database. In Image Analysis for Multime- dia Interactive Services (WIAMIS), 2010 11th International Workshop on, pages 1-4, 2010.
- M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein gan. arXiv preprint arXiv:1701.07875, 2017. · arXiv
- X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS), 2016.
- J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In Advances in Neural Information Processing Systems (NIPS) Workshop, 2014.
- E. Denton and V. Birodkar. Unsupervised learning of dis- entangled representations from video. Advances in Neural Information Processing Systems (NIPS), 2017.
- E. L. Denton, S. Chintala, R. Fergus, et al. Deep genera- tive image models using a laplacian pyramid of adversarial networks. In Advances in Neural Information Processing Systems (NIPS), 2015.
- G. Doretto, A. Chiuso, Y. N. Wu, and S. Soatto. Dynamic textures. International Journal of Computer Vision (IJCV), 2003.
- C. Finn, I. Goodfellow, and S. Levine. Unsupervised learn- ing for physical interaction through video prediction. In Ad- vances in Neural Information Processing Systems (NIPS), 2016.
- I. Goodfellow. Nips 2016 tutorial: Generative adversarial networks. arXiv preprint arXiv:1701.00160, 2016. · arXiv
- I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Gen- erative adversarial nets. In Advances in Neural Information Processing Systems (NIPS), 2014.
- L. Gorelick, M. Blank, E. Shechtman, M. Irani, and R. Basri. Actions as space-time shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 29(12):2247- 2253, 2007.
- K. Gregor, I. Danihelka, A. Graves, D. J. Rezende, and D. Wierstra. Draw: A recurrent neural network for image generation. In International Conference on Machine Learn- ing (ICML), 2015.
- D. J. Im, C. D. Kim, H. Jiang, and R. Memisevic. Generating images with recurrent adversarial networks. arXiv preprint arXiv:1602.05110, 2016. · arXiv
- N. Kalchbrenner, A. v. d. Oord, K. Simonyan, I. Danihelka, O. Vinyals, A. Graves, and K. Kavukcuoglu. Video pixel networks. arXiv preprint arXiv:1610.00527, 2016. · arXiv
- T. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017. · arXiv
- D. Kingma and J. Ba. Adam: A method for stochastic op- timization. In International Conference on Learning Repre- sentations (ICLR), 2015.
- D. P. Kingma and M. Welling. Auto-encoding variational bayes. In International Conference on Learning Representa- tions (ICLR), 2014.
- M.-Y. Liu, T. Breuel, and J. Kautz. Unsupervised image-to- image translation networks. In Advances in Neural Informa- tion Processing Systems (NIPS), 2017.
- M.-Y. Liu and O. Tuzel. Coupled generative adversarial net- works. In Advances in Neural Information Processing Sys- tems (NIPS), 2016.
- X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. P. Smolley. Least squares generative adversarial networks. IEEE Inter- national Conference on Computer Vision (ICCV), 2017.
- M. Mathieu, C. Couprie, and Y. LeCun. Deep multi-scale video prediction beyond mean square error. In International Conference on Learning Representations (ICLR), 2016.
- A. Odena, C. Olah, and J. Shlens. Conditional image synthe- sis with auxiliary classifier gans. In International Conference on Machine Learning (ICML), 2017.
- J. Oh, X. Guo, H. Lee, R. L. Lewis, and S. Singh. Action- conditional video prediction using deep networks in atari games. In Advances in Neural Information Processing Sys- tems (NIPS), 2015.
- A. Radford, L. Metz, and S. Chintala. Unsupervised rep- resentation learning with deep convolutional generative ad- versarial networks. In International Conference on Learning Representations (ICLR), 2016.
- D. J. Rezende, S. Mohamed, and D. Wierstra. Stochas- tic backpropagation and variational inference in deep latent gaussian models. In International Conference on Machine Learning (ICML), 2014.
- K. Roth, A. Lucchi, S. Nowozin, and T. Hofmann. Stabiliz- ing training of generative adversarial networks through reg- ularization. In Advances in Neural Information Processing Systems (NIPS), 2017.
- M. Saito, E. Matsumoto, and S. Saito. Temporal generative adversarial nets with singular value clipping. In IEEE Inter- national Conference on Computer Vision (ICCV), 2017.
- T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Rad- ford, and X. Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems (NIPS), 2016.
- K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. · arXiv
- M. Szummer and R. W. Picard. Temporal texture modeling. In International Conference on Image Processing (ICIP), 1996.
- L. Theis, A. v. d. Oord, and M. Bethge. A note on the eval- uation of generative models. In International Conference on Learning Representations (ICLR), 2016.
- S. Tulyakov, A. Fitzgibbon, and S. Nowozin. Hybrid vae: Improving deep generative models using partial observa- tions. Advances in Neural Information Processing Systems (NIPS) Workshop, 2017.
- J. van Amersfoort, A. Kannan, M. Ranzato, A. Szlam, D. Tran, and S. Chintala. Transformation-based models of video sequences. arXiv preprint arXiv:1701.08435, 2017. · arXiv
- A. van den Oord, N. Kalchbrenner, L. Espeholt, O. Vinyals, A. Graves, et al. Conditional image generation with pixel- cnn decoders. In Advances in Neural Information Processing Systems (NIPS), 2016.
- R. Villegas, J. Yang, S. Hong, X. Lin, and H. Lee. Decom- posing motion and content for natural video sequence pre- diction. In International Conference on Learning Represen- tations (ICLR), 2017.
- C. Vondrick, H. Pirsiavash, and A. Torralba. Generating videos with scene dynamics. In Advances in Neural Infor- mation Processing Systems (NIPS), 2016.
- L.-Y. Wei and M. Levoy. Fast texture synthesis using tree- structured vector quantization. In ACM SIGGRAPH, 2000.
- T. Xue, J. Wu, K. Bouman, and B. Freeman. Probabilistic modeling of future frames from a single image. In Advances in Neural Information Processing Systems (NIPS), 2016.
- H. Zhang, T. Xu, H. Li, S. Zhang, X. Huang, X. Wang, and D. Metaxas. Stackgan: Text to photo-realistic image synthe- sis with stacked generative adversarial networks. IEEE In- ternational Conference on Computer Vision (ICCV), 2016.
Cited by
Sorted by citations
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
Gui, J., Sun, Z., Wen, Y., Tao, D., & Ye, J. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3130191 · Cited by 1,231 · see citing works
The Creation and Detection of Deepfakes: A Survey
Mirsky, Y., & Lee, W. (2022) · ACM Computing Surveys · doi:10.1145/3425780 · Cited by 863 · see citing works
The Creation and Detection of Deepfakes: A Survey
Mirsky, Y., & Lee, W. (2020) · arXiv · doi:10.48550/arxiv.2004.11138 · Cited by 863
Chan, C., Ginosar, S., Zhou, T., & Efros, A. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00603 · Cited by 737 · see citing works
Wang, T.-C., Liu, M.-Y., Zhu, J.-Y., Liu, G., Tao, A., Kautz, J., & Catanzaro, B. (2018) · arXiv · doi:10.48550/arxiv.1808.06601 · Cited by 594 · see citing works
Self-citationMotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model
Zhang, M., Cai, Z., Pan, L., Hong, F., Guo, X., Yang, L., & Liu, Z. (2022) · arXiv · doi:10.48550/arxiv.2208.15001 · Cited by 437 · see citing works
Generating Human Motion from Textual Descriptions with Discrete Representations
Zhang, J., Zhang, Y., Cun, X., Zhang, Y., Zhao, H., Lu, H., Shen, X., & Ying, S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01415 · Cited by 328 · see citing works
Video Coding for Machines: A Paradigm of Collaborative Compression and Intelligent Analytics
Duan, L., Liu, J., Yang, W., Huang, T., & Gao, W. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3016485 · Cited by 260 · see citing works
Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
Liu, M.-Y., Huang, X., Yu, J., Wang, T.-C., & Mallya, A. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3049196 · Cited by 236 · see citing works
Self-citationShape-Based Generative Modeling for de Novo Drug Design
Skalic, M., Jiménez, J., Sabbadin, D., & De Fabritiis, G. (2019) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.8b00706 · Cited by 229 · see citing works
Explainability of Deep Vision-Based Autonomous Driving Systems: Review and Challenges
Zablocki, É., Ben-Younes, H., Pérez, P., & Cord, M. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01657-x · Cited by 227 · see citing works
VideoGPT: Video Generation using VQ-VAE and Transformers
Yan, W., Zhang, Y., Abbeel, P., & Srinivas, A. (2021) · arXiv · doi:10.48550/arxiv.2104.10157 · Cited by 156 · see citing works
Self-supervised learning for videos: A survey
Schiappa, M. C., Rawat, Y. S., & Shah, M. (2023) · ACM Computing Surveys · doi:10.1145/3577925 · Cited by 155 · see citing works
Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation
Stypułkowski, M., Vougioukas, K., He, S., Zięba, M., Petridis, S., & Pantic, M. (2023) · arXiv · doi:10.48550/arxiv.2301.03396 · Cited by 140
Cai, Z., Stefanov, K., Dhall, A., & Hayat, M. (2022) · 2022 International Conference on Digital Image Computing: Techniques and Applications (DICTA) · doi:10.1109/dicta56598.2022.10034605 · Cited by 99 · see citing works
Phenaki: Variable Length Video Generation From Open Domain Textual Description
Villegas, R., Babaeizadeh, M., Kindermans, P.-J., Moraldo, H., Zhang, H., Saffar, M. T., Castro, S., Kunze, J., & Erhan, D. (2022) · arXiv · doi:10.48550/arxiv.2210.02399 · Cited by 92 · see citing works
“…42, 13, 50], variational based stochastic meth-ods [2, 10, 46, 3] and more recently by learning a discrete representation [49, 33, 31], auto-regressive models [51, 55, 28, 57], diffusion models [47, 14, 56, 19] flow based models [24], and finally adversarial based methods [48, 39, 43, 9, 40, 27].”
BackgroundDeepfakes and scientific knowledge dissemination
Doss, C., Mondschein, J., Shu, D., Wolfson, T., Kopecky, D., Fitton-Kane, V. A., Bush, L., & Tucker, C. (2023) · Scientific Reports · doi:10.1038/s41598-023-39944-3 · Cited by 90 · see citing works
GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions
Wu, C., Huang, L., Zhang, Q., Li, B., Ji, L., Yang, F., Sapiro, G., & Duan, N. (2021) · arXiv · doi:10.48550/arxiv.2104.14806 · Cited by 85 · see citing works
Seeing is Believing: Exploring Perceptual Differences in DeepFake Videos
Tahir, R., Batool, B., Jamshed, H., Jameel, M., Anwar, M., Ahmed, F., Zaffar, M. A., & Zaffar, M. F. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445699 · Cited by 79 · see citing works
Detecting Deepfake Videos with Temporal Dropout 3DCNN
Zhang, D., Li, C., Lin, F., Zeng, D., & Ge, S. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/178 · Cited by 77 · see citing works
Factorised Spatial Representation Learning: Application in Semi-supervised Myocardial Segmentation
Chartsias, A., Joyce, T., Papanastasiou, G., Semple, S., Williams, M., Newby, D., Dharmakumar, R., & Tsaftaris, S. A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-00934-2_55 · Cited by 75 · see citing works
Deep learning for visual speech analysis: A survey
Sheng, C., Kuang, G., Bai, L., Hou, C., Guo, Y., Xu, X., Pietikäinen, M., & Liu, L. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3376710 · Cited by 64 · see citing works
GAIA-1: A Generative World Model for Autonomous Driving
Hu, A., Russell, L., Yeo, H., Murez, Z., Fedoseev, G., Kendall, A., Shotton, J., & Corrado, G. (2023) · arXiv · doi:10.48550/arxiv.2309.17080 · Cited by 28 · see citing works
“GAN-based methods produce more realistic videos [59, 60, 61, 62, 63, 64] but are known to suffer from training instability and a lack of generation diversity [65].”
MethodFine-Grained Multimodal DeepFake Classification via Heterogeneous Graphs
Yin, Q., Lu, W., Cao, X., Luo, X., Zhou, Y., & Huang, J. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02128-1 · Cited by 23 · see citing works
Self-Supervised Video GANs: Learning for Appearance Consistency and Motion Coherency
Hyun, S., Kim, J., & Heo, J.-P. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01068 · Cited by 16 · see citing works
Fine-grained talking face generation with video reinterpretation
Huang, X., Wang, M., & Gong, M. (2021) · The Visual Computer · doi:10.1007/s00371-020-01982-7 · Cited by 11 · see citing works
Cosmos World Foundation Model Platform for Physical AI
NVIDIA, :, Agarwal, N., Ali, A., Bala, M., Balaji, Y., Barker, E., Cai, T., Chattopadhyay, P., Chen, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.03575 · Cited by 13 · see citing works
Self-citationVideo Diffusion Models: A Survey
Melnik, A., Ljubljanac, M., Lu, C., Yan, Q., Ren, W., & Ritter, H. (2024) · arXiv · doi:10.48550/arxiv.2405.03150 · Cited by 4 · see citing works
MoCha: Towards Movie-Grade Talking Character Synthesis
Wei, C., Sun, B., Ma, H., Hou, J., Juefei-Xu, F., He, Z., Dai, X., Zhang, L., Li, K., Hou, T., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.23307
“Automating film production holds immense commercial potential, promising to democratize cinematic-level story-telling by enabling content creators to effortlessly generate films through natural language [15, 36, 41, 47].”
- Authors
- Tulyakov, S., Liu, M.-Y., Yang, X., & Kautz, J.
- Affiliations
- Snap (United States) (US)
- Nvidia (United Kingdom) (GB)
- Venue
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2018-06
- Pages
- 1526-1535
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2737548191
- Cited by
- 1,076
- References
- 4
- Citations by year
- 2026: 57 · 2025: 105 · 2024: 123 · 2023: 134 · 2022: 139 · 2021: 172 · 2020: 156 · 2019: 128 · 2018: 53 · 2017: 8
- FWCI
- 38.35
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Keywords
- video generation, generative adversarial networks, unsupervised learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
