Visual Forecasting by Imitating Dynamics in Natural Sequences
Open accessThe DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
We introduce a general framework for visual forecasting, which directly imitates visual sequences without additional supervision. As a result, our model can be applied at several semantic levels and does not require any domain knowledge or handcrafted features. We achieve this by formulating visual forecasting as an inverse reinforcement learning (IRL) problem, and directly imitate the dynamics in natural sequences from their raw pixel values. The key challenge is the high-dimensional and continuous state-action space that prohibits the application of previous IRL algorithms. We address this computational bottleneck by extending recent progress in model-free imitation with trainable deep feature representations, which (1) bypasses the exhaustive state-action pair visits in dynamic programming by using a dual formulation and (2) avoids explicit state sampling at gradient computation using a deep feature reparametrization. This allows us to apply IRL at scale and directly imitate the dynamics in high-dimensional continuous visual sequences from the raw pixel values. We evaluate our approach at three different level-of-abstraction, from low level pixels to higher level semantics: future frame generation, action anticipation, visual story forecasting. At all levels, our approach outperforms existing methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Markov decision processes: Discrete stochastic dynamic programming
Hazeghi, K., & Puterman, M. L. (1995) · Journal of the American Statistical Association · doi:10.2307/2291177 · Cited by 8,037
Generative adversarial imitation learning
Ho, J., & Ermon, S. (2016) · arXiv · doi:10.48550/arxiv.1606.03476 · Cited by 1,942
Unsupervised Learning of Video Representations using LSTMs
Srivastava, N., Mansimov, E., & Salakhutdinov, R. (2015) · arXiv · doi:10.48550/arxiv.1502.04681 · Cited by 1,861
(2000) · doi:10.2460/ajvr.67.2.323 · Cited by 1,621
Unsupervised Learning of Visual Representations Using Videos
Wang, X., & Gupta, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.320 · Cited by 1,005
Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification
Misra, I., Zitnick, C. L., & Hebert, M. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_32 · Cited by 779
Research works printed but not linked: 39
- soomro, 2012, Ucf101 A Dataset of 101 Human Actions Classes from Videos in the Wild
- sigurdsson, 2016, Learning visual storylines with skipping recurrent neural networks, ECCV
- 10.1109/CVPR.2016.508 · DOI · Paper page
- 10.1145/1102351.1102433 · DOI
- mnih, 2014, Recurrent models of visual attention, NIPS
- radford, 2016, Unsupervised representation learning with deep convolutional generative adversarial networks, ICLRE
- patron-perez, 2010, High five: Recognising human interactions in tv shows, BMVC
- pathak, 2017, Learning features by watching objects move, CVPR
- mathieu, 2016, Deep multiscale video prediction beyond mean square error, ICLRE
- mathe, 2013, Action from still image dataset and inverse optimal control to learn task specific visual scanpaths, NIPS
- 10.1109/CDC.2014.7040156 · DOI
- 10.1109/CVPR.2016.110 · DOI · Paper page
- 10.1177/0278364915619772 · DOI · Paper page
- lan, 2014, A hierarchical representation for future action prediction, ECCV
- ma, 2017, Forecasting interactive dynamics of pedestrians with fictitious play, CVPR
- li, 2017, Adversarial learning for neural dialogue generation, EMNLP
- xue, 2016, Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks, NIPS
- yan, 2016, Attribute2image: Conditional image generation from visual attributes, ECCV
- ziebart, 2008, Maximum entropy inverse reinforcement learning, AAAI
- zeng, 2017, Agent-centric risk assessment: Accident anticipation and risky region localization, CVPR
- yu, 2017, Seqgan: sequence generative adversarial nets with policy gradient, AAAI, 10.1609/aaai.v31i1.10804 · DOI · Paper page
- 10.1109/ICCV.2015.530 · DOI
- ho, 2016, Model-free imitation learning with policy optimization, ICML
- huang, 2014, Action-reaction: Forecasting the dynamics of human interaction, ECCV
- 10.18653/v1/N16-1147 · DOI · Paper page
- isola, 2017, Image-to-image translation with conditional adversarial networks, CVPR
- kalchbrenner, 2016, Video pixel networks
- chan, 2016, Anticipating accidents in dashcam videos, ACCV
- 10.1109/CVPR.2009.5206540 · DOI
- finn, 2016, Guided cost learning: Deep inverse optimal control via policy optimization, ICML
- gorban, 2015, THUMOS Challenge Action Recognition with A Large Number of Classes
- goodfellow, 2014, Generative adversarial nets, NIPS
- xie, 2013, Inferring “dark matter” and “dark energy” from videos, ICCV
- 10.1109/ICRA.2017.7989385 · DOI · Paper page
- 10.1109/CVPR.2014.416 · DOI · Paper page
- walker, 2016, An uncertain future: Forecasting from variational autoencoders, ECCV
- 10.1109/ICCV.2015.281 · DOI · Paper page
- 10.1109/CVPR.2016.18 · DOI · Paper page
- vondrick, 2017, Generating the future with adversarial transformers, CVPR
Cited by
A Review on Deep Learning Techniques for Video Prediction
Oprea, S., Martinez-Gonzalez, P., Garcia-Garcia, A., Castro-Vargas, J. A., Orts-Escolano, S., Garcia-Rodriguez, J., & Argyros, A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3045007 · Cited by 301
Furnari, A., & Farinella, G. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00635 · Cited by 182
“Previous works investigated different forms of action and activity anticipation from third person video [1, 15, 18, 22, 24, 25, 28, 29, 30, 37, 54, 59].”
BackgroundCrowd Density Forecasting by Modeling Patch-Based Dynamics
Minoura, H., Yonetani, R., Nishimura, M., & Ushiku, Y. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.3043169 · Cited by 16
Metadata
- Authors
- Zeng, K.-H., Shen, W. B., Huang, D.-A., Sun, M., & Niebles, J. C.
- Venue
- 2017 IEEE International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2017-10
- Pages
- 3018-3027
- OpenAlex
- W2963302737
- Cited by
- 53
- References
- 15
- FWCI
- 2.10
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.