CoChat
Sign Up
Cited by 162Generative Adversarial Networks and Image SynthesisAdvanced Image Processing TechniquesDigital Media Forensic Detection

Learning to Generate Time-Lapse Videos Using Multi-stage Dynamic Generative Adversarial Networks

Open access
Xiong, Wei, Luo, Wenhan, Ma, Lin, Liu, Wei, Luo, Jiebo·2018·2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
DOI 10.1109/cvpr.2018.00251
Open this paper in CoChat — free accountOpen PDF

Abstract

Taking a photo outside, can we predict the immediate future, e.g., how would the cloud move in the sky? We address this problem by presenting a generative adversarial network (GAN) based two-stage approach to generating realistic time-lapse videos of high resolution. Given the first frame, our model learns to generate long-term future frames. The first stage generates videos of realistic contents for each frame. The second stage refines the generated video from the first stage by enforcing it to be closer to real videos with regard to motion dynamics. To further encourage vivid motion in the final generated video, Gram matrix is employed to model the motion more precisely. We build a large scale time-lapse dataset, and test our approach on this new dataset. Using our model, we are able to generate realistic videos of up to 128 Ã- 128 resolution for 32 frames. Quantitative and qualitative experiment results demonstrate the superiority of our model over the state-of-the-art models.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr.2018.00251

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Sky Time-lapse is a collection of unlabeled short clips of time-lapse shots of the sky, each consisting of 32 frames at 640 × 360 pixels.

    “Sky Time-lapse (Xiong et al., 2018) is a collection of unlabeled short clips that contain time-lapse shots of the sky. The videos have been taken from YouTube and divided into smaller non-overlapping segments. Each clip consists of 32 frames of continuous video at a resolution of 640 × 360 pixels.”

    From Video Diffusion Models: A Survey (2024) · cites this paper as Xiong et al., 2018

    Section 4.1 (Dataset; page unavailable) says the authors collected YouTube time-lapse videos of sky scenes and made non-overlapping clips of 32 continuous frames; it gives the original frame size as 640 × 360 pixels. The copy calls the dataset “Sky Scene,” not “Sky Time-lapse,” and does not state that the clips are unlabeled. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 32

  1. M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein gen- erative adversarial networks. In International Conference on Machine Learning, pages 214-223, 2017.
  2. E. Denton, S. Chintala, A. Szlam, and R. Fergus. Deep gen- erative image models using a laplacian pyramid of adversar- ial networks. In Advances In Neural Information Processing Systems (NIPS), pages 1486-1494, 2015.
  3. C. M. Funke, L. A. Gatys, A. S. Ecker, and M. Bethge. Syn- thesising dynamic textures using convolutional neural net- works. arXiv preprint arXiv:1702.07006, 2017. · arXiv
  4. L. Gatys, A. S. Ecker, and M. Bethge. Texture synthesis using convolutional neural networks. In Advances In Neu- ral Information Processing Systems (NIPS), pages 262-270, 2015.
  5. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Gen- erative adversarial nets. In Advances In Neural Information Processing Systems (NIPS), pages 2672-2680, 2014.
  6. K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In European Conference on Com- puter Vision (ECCV), pages 630-645. Springer, 2016.
  7. E. Hoffer and N. Ailon. Deep metric learning using triplet network. In International Workshop on Similarity-Based Pattern Recognition, pages 84-92. Springer, 2015.
  8. S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), pages 448-456, 2015.
  9. P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image- to-image translation with conditional adversarial networks. IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2017.
  10. J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision, pages 694-711. Springer, 2016.
  11. D. P. Kingma and M. Welling. Auto-encoding variational bayes. International Conference on Learning Representa- tions (ICLR), 2014.
  12. J. Li, X. Liang, Y. Wei, T. Xu, J. Feng, and S. Yan. Percep- tual generative adversarial networks for small object detec- tion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  13. X. Liang, H. Zhang, and E. P. Xing. Generative se- mantic manipulation with contrasting gan. arXiv preprint arXiv:1708.00315, 2017. · arXiv
  14. M. Mathieu, C. Couprie, and Y. LeCun. Deep multi-scale video prediction beyond mean square error. arXiv preprint arXiv:1511.05440, 2015. · arXiv
  15. M. Mirza and S. Osindero. Conditional generative adversar- ial nets. arXiv preprint arXiv:1411.1784, 2014. · arXiv
  16. V. Nair and G. E. Hinton. Rectified linear units improve re- stricted boltzmann machines. In International Conference on Machine Learning (ICML), pages 807-814, 2010.
  17. A. Radford, L. Metz, and S. Chintala. Unsupervised rep- resentation learning with deep convolutional generative ad- versarial networks. International Conference on Learning Representations (ICLR), 2016.
  18. S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee. Generative adversarial text to image synthesis. In- ternational Conference on Learning Representations (ICLR), 2016.
  19. K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. In Advances in neural information processing systems, pages 568-576, 2014.
  20. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations (ICLR), 2015.
  21. M. Tesfaldet, M. A. Brubaker, and K. G. Derpanis. Two- stream convolutional networks for dynamic texture synthe- sis. arXiv preprint arXiv:1706.06982, 2017. · arXiv
  22. S. Tulyakov, M.-Y. Liu, X. Yang, and J. Kautz. Moco- gan: Decomposing motion and content for video generation. arXiv preprint arXiv:1707.04993, 2017. · arXiv
  23. R. Villegas, J. Yang, Y. Zou, S. Sohn, X. Lin, and H. Lee. Learning to generate long-term future via hierarchical pre- diction. International Conference on Machine Learning (ICML), 2017.
  24. C. Vondrick, H. Pirsiavash, and A. Torralba. Generating videos with scene dynamics. In Advances In Neural Infor- mation Processing Systems (NIPS), pages 613-621, 2016.
  25. C. Vondrick and A. Torralba. Generating the future with ad- versarial transformers. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  26. Z. Wu, Y. Fu, Y.-G. Jiang, and L. Sigal. Harnessing ob- ject and scene semantics for large-scale video understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3112-3121, 2016.
  27. S. Xiao, M. Farajtabar, X. Ye, J. Yan, L. Song, and H. Zha. Wasserstein learning of deep generative point process mod- els. In Advances in Neural Information Processing Systems (NIPS), pages 3247-3257, 2017.
  28. B. Xu, N. Wang, T. Chen, and M. Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853, 2015. · arXiv
  29. T. Xue, J. Wu, K. L. Bouman, and W. T. Freeman. Visual dy- namics: Probabilistic future frame synthesis via cross convo- lutional networks. In Advances In Neural Information Pro- cessing Systems (NIPS), 2016.
  30. H. Yu, J. Wang, Z. Huang, Y. Yang, and W. Xu. Video paragraph captioning using hierarchical recurrent neural net- works. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4584-4593, 2016.
  31. H. Zhang, T. Xu, H. Li, S. Zhang, X. Huang, X. Wang, and D. Metaxas. Stackgan: Text to photo-realistic image syn- thesis with stacked generative adversarial networks. Interna- tional Conference on Computer Vision (ICCV), 2017.
  32. Y. Zhou and T. L. Berg. Learning temporal transformations from time-lapse videos. In European Conference on Com- puter Vision (ECCV), pages 262-277. Springer, 2016.

Cited by

Sorted by citations

Authors
Xiong, W., Luo, W., Ma, L., Liu, W., & Luo, J.
Affiliations
  • University of Rochester (US)
  • Tencent (China) (CN)
ORCID
Venue
2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
Publisher
IEEE
Published
2018-06
Pages
2364-2373
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr.2018.00251
OpenAlex
W2964318715
Cited by
162
References
6
Citations by year
2026: 8 · 2025: 17 · 2024: 12 · 2023: 24 · 2022: 15 · 2021: 25 · 2020: 28 · 2019: 23 · 2018: 10
FWCI
6.48
Type
conference paper
Language
en
Open access
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
Keywords
generative adversarial networks, motion dynamics, Gram matrix

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review