CoChat
Sign Up
PreprintCited by 488Generative Adversarial Networks and Image SynthesisHuman Pose and Action RecognitionVideo Coding and Compression Technologies

Decomposing Motion and Content for Natural Video Sequence Prediction

Open access
Villegas, Ruben, Yang, Jimei, Hong, Seunghoon, Lin, Xunyu, Lee, Honglak·2017·arXiv
DOI 10.48550/arxiv.1706.08033arXiv 1706.08033
Open this paper in CoChat — free accountOpen PDF

Abstract

We propose a deep neural network for the prediction of future frames in natural video sequences. To effectively handle complex evolution of pixels in videos, we propose to decompose the motion and content, two key components generating dynamics in videos. Our model is built upon the Encoder-Decoder Convolutional Neural Network and Convolutional LSTM for pixel-level prediction, which independently capture the spatial layout of an image and the corresponding temporal dynamics. By independently modeling motion and content, predicting the next frame reduces to converting the extracted content features into the next frame content by the identified motion features, which simplifies the task of prediction. Our model is end-to-end trainable over multiple time steps, and naturally learns to decompose motion and content without separate training. We evaluate the proposed network architecture on human activity videos using KTH, Weizmann action, and UCF-101 datasets. We show state-of-the-art performance in comparison to recent approaches. To the best of our knowledge, this is the first end-to-end trainable network architecture with motion and content separation to model the spatiotemporal dynamics for pixel-level future prediction in natural videos.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1706.08033

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Prior methods for affordance learning are tested primarily on human video datasets with no physical robot or in-the-wild experiments.

    “prior methods for affordance learning are tested primarily on human video datasets with no physical robot or in-the-wild experiments.”

    From Affordances from Human Videos as a Versatile Representation for Robotics (2023) · cites this paper as [107]

    The Experiments section (p. 6) evaluates video prediction on human-action datasets, supporting the human-video portion of the claim. But the paper does not characterize these as prior affordance-learning methods, and §6.2 (p. 7) explicitly reports testing on UCF-101 videos of human actions taken in the wild. It therefore does not support the blanket assertion of no in-the-wild experiments. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

1 of the 10 references couldn’t be matched to a known paper.

Research works printed but not linked: 22

  1. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In NIPS . 2014.
  2. R. Goroshin, M. Mathieu, and Y. LeCun. Learning to linearize under uncertainty. In NIPS . 2015.
  3. W. Lotter, G. Kreiman, and D. Cox. Unsupervised learning of visual structure using predictive generative networks. arXiv preprint arXiv:1504.08023 , 2015. · arXiv
  4. M. Mathieu, C. Couprie, and Y. LeCun. Deep multi-scale video prediction beyond mean square error. arXiv preprint arXiv:1511.05440 , 2015. · arXiv
  5. J. Oh, X. Guo, H. Lee, R. L. Lewis, and S. Singh. Action-conditional video prediction using deep networks in atari games. In NIPS . 2015.
  6. V. Patraucean, A. Handa, and R. Cipolla. Spatio-temporal video autoencoder with differentiable memory. CoRR , abs/1511.06309, 2015. · arXiv
  7. L. C. Pickup, Z. Pan, D. Wei, Y. Shih, C. Zhang, A. Zisserman, B. Scholkopf, and W. T. Freeman. Seeing the arrow of time. In CVPR , 2014.
  8. S. L. Pintea, J. C. van Gemert, and A. W. M. Smeulders. Dejavu: Motion prediction in static images. In European Conference on Computer Vision , 2014.
  9. M. Ranzato, A. Szlam, J. Bruna, M. Mathieu, R. Collobert, and S. Chopra. Video (language) modeling: a baseline for generative models of natural videos. arXiv preprint arXiv:1412.6604 , 2014. · arXiv
  10. M. S. Ryoo. Human activity prediction: Early recognition of ongoing activities from streaming videos. In ICCV , 2011.
  11. X. Shi, Z. Chen, H. Wang, D.-Y. Yeung, W.-k. Wong, and W.-c. WOO. Convolutional lstm network: A machine learning approach for precipitation nowcasting. In Advances in Neural Information Processing Systems 28 . 2015.
  12. K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. In NIPS . 2014.
  13. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR , 2015.
  14. K. Soomro, A. R. Zamir, and M. Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 , 2012. · arXiv
  15. N. Srivastava, E. Mansimov, and R. Salakhudinov. Unsupervised learning of video representations using lstms. In ICML , 2015.
  16. C. Vondrick, H. Pirsiavash, and A. Torralba. Generating videos with scene dynamics. In NIPS . 2016.
  17. J. Walker, A. Gupta , and M. Hebert . Patch to the future: Unsupervised visual prediction. In CVPR , 2014.
  18. J. Walker, C. Doersch, A. Gupta, and M. Hebert. An uncertain future: Forecasting from static images using variational autoencoders. CoRR , abs/1606.07873, 2016. · arXiv
  19. P. Weinzaepfel, J. Revaud, Z. Harchaoui, and C. Schmid. DeepFlow: Large displacement optical flow with deep matching. In ICCV , 2013.
  20. T. Xue, J. Wu, K. L. Bouman, and W. T. Freeman. Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks. NIPS , 2016.
  21. J. Yuen and A. Torralba. A data-driven approach for event prediction. In ECCV , 2010.
  22. M. D. Zeiler, G. W. Taylor, and R. Fergus. Adaptive deconvolutional networks for mid and high level feature learning. In ICCV , 2011.

Cited by

Sorted by citations

Related

Authors
Villegas, R., Yang, J., Hong, S., Lin, X., & Lee, H.
ORCID
Lee, Honglak
Venue
arXiv
Publisher
arXiv
Published
2017
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1706.08033
arXiv
1706.08033
OpenAlex
W2615413256
Cited by
488
References
10
Citations by year
2026: 1 · 2025: 13 · 2024: 32 · 2023: 50 · 2022: 58 · 2021: 78 · 2020: 100 · 2019: 84 · 2018: 64 · 2017: 8
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Computer Science › Signal Processing › Video Coding and Compression Technologies
Keywords
video prediction, spatiotemporal dynamics, convolutional encoder-decoder, convolutional LSTM, pixel-level prediction, end-to-end learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review