Train Sparsely, Generate Densely: Memory-Efficient Unsupervised Training of High-Resolution Temporal GAN
Open accessThe claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The generalized IS for video domain is highly specific to the UCF101 dataset and hardly applicable to videos in the wild due to classification difficulty.
“However, this metric is generally highly specific to the UCF101 dataset and is hardly applicable to videos in the wild due to classification difficulty.”
From Video Diffusion Models: A Survey (2024) · cites this paper as Saito et al., 2020
Section 5.7 says the paper computes video Inception Score using a classifier fine-tuned on UCF101 and an average tensor from that dataset. This supports the metric’s UCF101-specific implementation, but the paper does not say it is hardly applicable to videos in the wild or attribute such a limitation to classification difficulty. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596 · see citing works
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593 · see citing works
GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.08500 · Cited by 8,365 · see citing works
Improved training of Wasserstein GANs
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., & Courville, A. (2017) · arXiv · doi:10.48550/arxiv.1704.00028 · Cited by 6,800 · see citing works
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354 · cited 2× · see citing works
“2015),whichwas first trainedwith Sports-1M dataset (Karpathy et al. 2014) and then fine-tuned on the UCF101 dataset.”
MethodDeep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196 · see citing works
Large Scale GAN Training for High Fidelity Natural Image Synthesis
Brock, A., Donahue, J., & Simonyan, K. (2018) · arXiv · doi:10.48550/arxiv.1809.11096 · Cited by 3,717 · see citing works
Self-Attention Generative Adversarial Networks
Zhang, H., Goodfellow, I., Metaxas, D., & Odena, A. (2018) · arXiv · doi:10.48550/arxiv.1805.08318 · Cited by 3,353 · see citing works
Unsupervised Learning of Video Representations using LSTMs
Srivastava, N., Mansimov, E., & Salakhutdinov, R. (2015) · arXiv · doi:10.48550/arxiv.1502.04681 · Cited by 1,861 · see citing works
Wang, T.-C., Liu, M.-Y., Zhu, J.-Y., Liu, G., Tao, A., Kautz, J., & Catanzaro, B. (2018) · arXiv · doi:10.48550/arxiv.1808.06601 · Cited by 594 · see citing works
Flagged first, then by citations
Research works printed but not linked: 49
- Acharya, D., Huang, Z., Paudel, D.P., Gool, L.V.: To- wards High Resolution Video Generation with Progres- sive Growing of Sliced Wasserstein GANs. In: Arxiv preprint arXiv:1810.02419 (2018) · arXiv
- Akiba, T., Fukuda, K., Suzuki, S.: ChainerMN: Scalable Distributed Deep Learning Framework. In: Proceedings of Workshop on ML Systems in NIPS (2017)
- Babaeizadeh, M., Finn, C., Erhan, D., Campbell, R.H., Levine, S.: Stochastic Variational Video Prediction. In: ICLR (2018)
- Bansal, A., Ma, S., Ramanan, D., Sheikh, Y.: Recycle- GAN: Unsupervised Video Retargeting. In: ECCV (2018)
- Borji, A.: Pros and Cons of GAN Evaluation Measures. In: Arxiv preprint arXiv:1802.03446 (2018) · arXiv
- Byeon, W., Wang, Q., Srivastava, R.K., Koumoutsakos, P.: ContextVP: Fully Context-Aware Video Prediction. In: ECCV (2018)
- Cai, H., Bai, C., Tai, Y.W., Tang, C.K.: Deep Video Gen- eration, Prediction and Completion of Human Action Se- quences. In: ECCV (2018)
- Denton, E., Chintala, S., Szlam, A., Fergus, R.: Deep Generative Image Models Using a Laplacian Pyramid of Adversarial Networks. In: NIPS (2015)
- Denton, E., Fergus, R.: Stochastic Video Generation with a Learned Prior. In: Arxiv preprint arXiv:1802.07687 (2018) · arXiv
- Ebert, F., Finn, C., Lee, A.X., Levine, S.: Self-Supervised Visual Planning with Temporal Skip Connections. In: Conference on Robot Learning (CoRL) (2017)
- Finn, C., Goodfellow, I., Levine, S.: Unsupervised Learn- ing for Physical Interaction through Video Prediction. In: NIPS (2016)
- Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative Adversarial Nets. In: NIPS (2014)
- Hao, Z., Huang, X., Belongie, S.: Controllable Video Gen- eration with Sparse Trajectories. In: CVPR (2018)
- Hara, K., Kataoka, H., Satoh, Y.: Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet? In: CVPR (2018)
- Huang, X., Liu, M.Y., Belongie, S., Kautz, J.: Multi- modal Unsupervised Image-to-Image Translation. In: ECCV (2018)
- Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to- Image Translation with Conditional Adversarial Net- works. In: CVPR (2017)
- Kalchbrenner, N., van den Oord, A., Simonyan, K., Dani- helka, I., Vinyals, O., Graves, A., Kavukcuoglu, K.: Video Pixel Networks. In: arxiv preprint arXiv:1610.00527 (2016) · arXiv
- Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive Growing of GANs for Improved Quality, Stability, and Variation. In: ICLR (2018)
- Kingma, D., Ba, J.: Adam: A Method for Stochastic Op- timization. In: ICLR (2015)
- Lee, A.X., Zhang, R., Ebert, F., Abbeel, P., Finn, C., Levine, S.: Stochastic Adversarial Video Prediction. In: Arxiv preprint arXiv:1804.01523 (2018) · arXiv
- Li, Y., Fang, C., Yang, J., Wang, Z., Lu, X., Yang, M.H.: Flow-Grounded Spatial-Temporal Video Prediction from Still Images. In: ECCV (2018)
- Liang, X., Lee, L., Dai, W., Xing, E.P.: Dual Motion GAN for Future-Flow Embedded Video Prediction. In: ICCV (2017)
- Liu, M.Y., Breuel, T., Kautz, J.: Unsupervised Image-to- Image Translation Networks. In: NIPS (2017)
- Liu, Z., Yeh, R.A., Tang, X., Liu, Y., Agarwala, A.: Video Frame Synthesis using Deep Voxel Flow. In: ICCV (2017)
- Lotter, W., Kreiman, G., Cox, D.: Deep Predictive Cod- ing Networks for Video Prediction and Unsupervised Learning. In: ICLR (2017)
- Mathieu, M., Couprie, C., LeCun, Y.: Deep Multi-Scale Video Prediction beyond Mean Square Error. In: ICLR (2016)
- Mescheder, L., Nowozin, S., Geiger, A.: Which Train- ing Methods for GANs do actually Converge? In: ICML (2018)
- Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spec- tral Normalization for Generative Adversarial Networks. In: ICLR (2018)
- Miyato, T., Koyama, M.: cGANs with Projection Dis- criminator. In: ICLR (2018)
- Oh, J., Guo, X., Lee, H., Lewis, R., Singh, S.: Action- Conditional Video Prediction using Deep Networks in Atari Games. In: NIPS (2015)
- Ohnishi, K., Yamamoto, S., Ushiku, Y., Harada, T.: Hier- archical Video Generation from Orthogonal Information: Optical Flow and Texture. In: AAAI (2018)
- Radford, A., Metz, L., Chintala, S.: Unsupervised Repre- sentation Learning with Deep Convolutional Generative Adversarial Networks. In: ICLR (2016)
- Ranzato, M., Szlam, A., Bruna, J., Mathieu, M., Col- lobert, R., Chopra, S.: Video (Language) Modeling: A Baseline for Generative Models of Natural Videos. arXiv preprint arXiv:1412.6604 (2014) · arXiv
- Rössler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., Nießner, M.: FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces. In: Arxiv preprint arXiv:1803.09179 (2018) · arXiv
- Saito, M., Matsumoto, E., Saito, S.: Temporal Generative Adversarial Nets with Singular Value Clipping. In: ICCV (2017)
- Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved Techniques for Training GANs. In: NIPS (2016)
- Shi, X., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.K., Woo, W.c.: Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting. In: NIPS (2015)
- Soomro, K., Zamir, A.R., Shah, M.: UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild. arXiv preprint arXiv:1212.0402 (2012) · arXiv
- Tokui, S., Oono, K., Hido, S., Clayton, J.: Chainer: a Next-Generation Open Source Framework for Deep Learning. In: Proceedings of Workshop on Machine Learning Systems in NIPS (2015)
- Tulyakov, S., Liu, M.Y., Yang, X., Kautz, J.: MoCoGAN: Decomposing Motion and Content for Video Generation. In: CVPR (2018)
- Unterthiner, T., van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., Gelly, S.: Towards Accurate Genera- tive Models of Video: A New Metric & Challenges. In: Arxiv preprint arXiv:1812.01717 (2018) · arXiv
- Vondrick, C., Pirsiavash, H., Torralba, A.: Generating Videos with Scene Dynamics. In: NIPS (2016)
- Wang, T.C., Liu, M.Y., Zhu, J.Y., Tao, A., Kautz, J., Catanzaro, B.: High-Resolution Image Synthesis and Se- mantic Manipulation with Conditional GANs. In: CVPR (2018)
- Yang, C., Wang, Z., Zhu, X., Huang, C., Shi, J., Lin, D.: Pose Guided Human Video Generation. In: ECCV (2018)
- Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., Metaxas, D.: StackGAN++: Realistic Image Synthe- sis with Stacked Generative Adversarial Networks. In: arXiv preprint arXiv:1710.10916 (2017) · arXiv
- Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., Metaxas, D.: StackGAN: Text to Photo-realistic Im- age Synthesis with Stacked Generative Adversarial Net- works. In: ICCV (2017)
- Zhang, Z., Xie, Y., Yang, L.: Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Net- work. In: CVPR (2018)
- Zhao, L., Peng, X., Tian, Y., Kapadia, M., Metaxas, D.: Learning to Forecast and Refine Residual Motion for Image-to-Video Generation. In: ECCV (2018)
- Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired Image-to-Image Translation using Cycle-Consistent Ad- versarial Networks. In: ICCV (2017)
Cited by
Sorted by citations
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models
Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S., & Kreis, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02161 · Cited by 724 · see citing works
VideoGPT: Video Generation using VQ-VAE and Transformers
Yan, W., Zhang, Y., Abbeel, P., & Srinivas, A. (2021) · arXiv · doi:10.48550/arxiv.2104.10157 · Cited by 156 · see citing works
Phenaki: Variable Length Video Generation From Open Domain Textual Description
Villegas, R., Babaeizadeh, M., Kindermans, P.-J., Moraldo, H., Zhang, H., Saffar, M. T., Castro, S., Kunze, J., & Erhan, D. (2022) · arXiv · doi:10.48550/arxiv.2210.02399 · Cited by 92 · see citing works
“…42, 13, 50], variational based stochastic meth-ods [2, 10, 46, 3] and more recently by learning a discrete representation [49, 33, 31], auto-regressive models [51, 55, 28, 57], diffusion models [47, 14, 56, 19] flow based models [24], and finally adversarial based methods [48, 39, 43, 9, 40, 27].”
BackgroundGODIVA: Generating Open-DomaIn Videos from nAtural Descriptions
Wu, C., Huang, L., Zhang, Q., Li, B., Ji, L., Yang, F., Sapiro, G., & Duan, N. (2021) · arXiv · doi:10.48550/arxiv.2104.14806 · Cited by 85 · see citing works
Generative AI-based data augmentation for improved bioacoustic classification in noisy environments
Gibbons, A., King, E., Donohue, I., & Parnell, A. (2024) · arXiv · doi:10.48550/arxiv.2412.01530 · Cited by 6 · see citing works
Video Diffusion Models: A Survey
Melnik, A., Ljubljanac, M., Lu, C., Yan, Q., Ren, W., & Ritter, H. (2024) · arXiv · doi:10.48550/arxiv.2405.03150 · Cited by 4 · see citing works
- Authors
- Saito, M., Saito, S., Koyama, M., & Kobayashi, S.
- Affiliations
- Preferred Networks (Japan) (JP)
- Corresponding author
- Saito, Masaki
- Venue
- International Journal of Computer Vision
- Published
- 2020-05-29
- Volume
- 128
- Issue
- 10-11
- Pages
- 2586-2606
- ISSN
- 0920-5691 (ISSN-L)
- 1573-1405
- OpenAlex
- W3031246127
- Cited by
- 116
- References
- 11
- Citations by year
- 2026: 7 · 2025: 17 · 2024: 25 · 2023: 29 · 2022: 11 · 2021: 17 · 2020: 8 · 2019: 2
- FWCI
- 6.10
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- International Journal of Computer Vision (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Keywords
- high-resolution video generation, memory-efficient training, unsupervised learning, GPU memory reduction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
