CoChat
Sign Up
Cited by 301Generative Adversarial Networks and Image SynthesisHuman Pose and Action RecognitionMultimodal Machine Learning Applications

A Review on Deep Learning Techniques for Video Prediction

Open access
Oprea, Sergiu, Martinez-Gonzalez, Pablo, Garcia-Garcia, Alberto, Castro-Vargas, John Alejandro, Orts-Escolano, Sergio, Garcia-Rodriguez, Jose, Argyros, Antonis·2022·IEEE Transactions on Pattern Analysis and Machine Intelligence
DOI 10.1109/tpami.2020.3045007
Open this paper in CoChat — free accountOpen PDF

Abstract

The ability to predict, anticipate and reason about future outcomes is a key component of intelligent decision-making systems. In light of the success of deep learning in computer vision, deep-learning-based video prediction emerged as a promising research direction. Defined as a self-supervised learning task, video prediction represents a suitable framework for representation learning, as it demonstrated potential capabilities for extracting meaningful representations of the underlying patterns in natural videos. Motivated by the increasing interest in this task, we provide a review on the deep learning methods for prediction in video sequences. We first define the video prediction fundamentals, as well as mandatory background concepts and the most used datasets. Next, we carefully analyze existing video prediction models organized according to a proposed taxonomy, highlighting their contributions and their significance in the field. The summary of the datasets and methods is accompanied with experimental results that facilitate the assessment of the state of the art on a quantitative basis. The paper is summarized by drawing some general conclusions, identifying open research challenges and by pointing out future research directions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tpami.2020.3045007

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Video prediction is a much-studied task, including predicting how visual input will develop conditional on the system's outputs, but this is not sufficient for the kind of use in perception that includes making sense of exogenous changes not predictable from outputs.

    “For example, video prediction is a much-studied task (Oprea et al. 2020), and this includes predicting how visual input will develop conditional on the system's outputs (Finn et al. 2016).But this is not sufficient for the kind of use in perception that we have described, which also includes making sense of exogenous changes which are not predictable from outputs.”

    From Consciousness in Artificial Intelligence: Insights from the Science of Consciousness (2023) · cites this paper as Oprea et al. 2020

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 310 · Cited by 301

Research works printed but not linked: 205

  1. [1][1] M. H. Nguyen and F. D. la Torre, 'Max-margin early event detectors,' in CVPR , 2012. -
  2. [2][2] K. M. Kitani, B. D. Ziebart, J. A. Bagnell, and M. Hebert, 'Activity Forecasting,' in ECCV , 2012. -
  3. [3][3] C. Vondrick, H. Pirsiavash, and A. Torralba, 'Anticipating Visual Representations from Unlabeled Video,' in CVPR , 2016. -
  4. [4][4] K. Zeng, W. B. Shen, D. Huang, M. Sun, and J. C. Niebles, 'Visual Forecasting by Imitating Dynamics in Natural Sequences,' in ICCV , 2017. -
  5. [5][5] S. Shalev-Shwartz, N. Ben-Zrihem, A. Cohen, and A. Shashua, 'Long-term planning by short-term prediction,' arXiv:1602.01580 , 2016. - · arXiv
  6. [6][6] O. Makansi, E. Ilg, O. Cicek, and T. Brox, 'Overcoming limitations of mixture density networks: A sampling and fitting framework for multimodal future prediction,' in CVPR , 2019. -
  7. [7][7] Z. Liu, R. A. Yeh, X. Tang, Y. Liu, and A. Agarwala, 'Video frame synthesis using deep voxel flow,' in ICCV , 2017. -
  8. [8][8] P. Luc, C. Couprie, Y. LeCun, and J. Verbeek, 'Predicting Future Instance Segmentation by Forecasting Convolutional Features,' in ECCV , 2018, pp. 593-608. -
  9. [9][9] A. Bhattacharyya, M. Fritz, and B. Schiele, 'Bayesian prediction of future street scenes using synthetic likelihoods,' in ICLR , 2019. -
  10. [10][10] A. Terwilliger, G. Brazil, and X. Liu, 'Recurrent flow-guided semantic forecasting,' in WACV , 2019. -
  11. [11][11] A. Bhattacharyya, M. Fritz, and B. Schiele, 'Long-Term On-Board Prediction of People in Traffic Scenes Under Uncertainty,' in CVPR , 2018. -
  12. [12][12] W. Liu, W. Luo, D. Lian, and S. Gao, 'Future frame prediction for anomaly detection - A new baseline,' in CVPR . IEEE, 2018. -
  13. [13][13] X. Shi, Z. Chen, H. Wang, D. Yeung, W. Wong, and W. Woo, 'Convolutional LSTM network: A machine learning approach for precipitation nowcasting,' in NeurIPS , 2015. -
  14. [14][14] X. Shi, Z. Gao, L. Lausen, H. Wang, D.-Y. Yeung, W.-k. Wong, and W.-c. WOO, 'Deep learning for precipitation nowcasting: A benchmark and a new model,' in NeurIPS , 2017. -
  15. [15][15] A. Hu, F. Cotter, N. Mohan, C. Gurau, and A. Kendall, 'Probabilistic future prediction for video scene understanding,' arXiv:2003.06409 , 2020. - · arXiv
  16. [16][16] A. Garcia-Garcia, P. Martinez-Gonzalez, S. Oprea, J. A. CastroVargas, S. Orts-Escolano, J. Garcia-Rodriguez, and A. JoverAlvarez, 'The robotrix: An extremely photorealistic and verylarge-scale indoor dataset of sequences with robot trajectories and interactions,' in IROS , 2018, pp. 6790-6797. -
  17. [17][17] Y. Kong and Y. Fu, 'Human action recognition and prediction: A survey,' arXiv:1806.11230 , 2018. - · arXiv
  18. [18][18] C. Sahin, G. Garcia-Hernando, J. Sock, and T. Kim, 'A review on object pose recovery: from 3d bounding box detectors to full 6d pose estimators,' arXiv:2001.10609 , 2020. - · arXiv
  19. [19][19] V. Villena-Martinez, S. Oprea, M. Saval-Calvo, J. A. L´ opez, A. F. Guill´ o, and R. B. Fisher, 'When deep learning meets data alignment: A review on deep registration networks (DRNs),' arXiv:2003.03167 , 2020. - · arXiv
  20. [20][20] Y. LeCun, Y. Bengio, and G. E. Hinton, 'Deep learning,' Nature , vol. 521, no. 7553, 2015. -
  21. [23][23] D. Mumford, 'On the computational architecture of the neocortex,' Biological Cybernetics , vol. 66, no. 3, 1992. -
  22. [24][24] A. Cleeremans and J. L. McClelland, 'Learning the structure of event sequences.' Journal of Experimental Psychology: General , vol. 120, no. 3, 1991. -
  23. [25][25] A. Cleeremans and J. Elman, Mechanisms of implicit learning: Connectionist models of sequence processing . MIT press, 1993. -
  24. [26][26] R. Baker, M. Dexter, T. E. Hardwicke, A. Goldstone, and Z. Kourtzi, 'Learning to predict: Exposure to temporal sequences facilitates prediction of future events,' Vision Research , vol. 99, 2014. -
  25. [28][28] W. R. Softky, 'Unsupervised pixel-prediction,' in NeurIPS , 1995. -
  26. [29][29] G. Deco and B. Sch¨ urmann, 'Predictive coding in the visual cortex by a recurrent network with gabor receptive fields,' Neural Processing Letters , vol. 14, no. 2, 2001. -
  27. [30][30] A. Hollingworth, 'Constructing visual representations of natural scenes: the roles of short- and long-term visual memory.' Journal of experimental psychology. Human perception and performance , vol. 30 3, 2004. -
  28. [31][31] Y. Bengio, A. C. Courville, and P. Vincent, 'Representation learning: A review and new perspectives,' Trans. on PAMI , vol. 35, no. 8, 2013. -
  29. [33][33] P. Agrawal, J. Carreira, and J. Malik, 'Learning to see by moving,' in ICCV , 2015. -
  30. [34][34] D.-A. Huang, V. Ramanathan, D. Mahajan, L. Torresani, M. Paluri, L. Fei-Fei, and J. Carlos Niebles, 'What makes a video a video: Analyzing temporal information in video understanding models and datasets,' in CVPR , June 2018. -
  31. [35][35] L. C. Pickup, Z. Pan, D. Wei, Y. Shih, C. Zhang, A. Zisserman, B. Sch¨ olkopf, and W. T. Freeman, 'Seeing the arrow of time,' in CVPR , 2014. -
  32. [36][36] D. Wei, J. J. Lim, A. Zisserman, and W. T. Freeman, 'Learning and using the arrow of time,' in CVPR , 2018. -
  33. [38][38] M. Babaeizadeh, C. Finn, D. Erhan, R. H. Campbell, and S. Levine, 'Stochastic variational video prediction,' in ICLR , 2018. -
  34. [39][39] H. Zhao, O. Gallo, I. Frosio, and J. Kautz, 'Loss functions for image restoration with neural networks,' IEEE Trans. Computational Imaging , vol. 3, no. 1, 2017. -
  35. [40][40] K. Janocha and W. M. Czarnecki, 'On loss functions for deep neural networks in classification,' arXiv:1702.05659 , 2017. - · arXiv
  36. [42][42] J.-J. Hwang, T.-W. Ke, J. Shi, and S. X. Yu, 'Adversarial structure matching for structured prediction tasks,' in CVPR , 2019. -
  37. [43][43] M. Mathieu, C. Couprie, and Y. LeCun, 'Deep multi-scale video prediction beyond mean square error,' in ICLR (Poster) , 2016. -
  38. [44][44] A. Dosovitskiy and T. Brox, 'Generating images with perceptual similarity metrics based on deep networks,' in NIPS , 2016. -
  39. [45][45] J. Johnson, A. Alahi, and L. Fei-Fei, 'Perceptual losses for realtime style transfer and super-resolution,' in ECCV , vol. 9906, 2016. -
  40. [47][47] M. S. M. Sajjadi, B. Sch¨ olkopf, and M. Hirsch, 'Enhancenet: Single image super-resolution through automated texture synthesis,' in ICCV , 2017. -
  41. [48][48] J. Zhu, P. Kr¨ ahenb¨ uhl, E. Shechtman, and A. A. Efros, 'Generative visual manipulation on the natural image manifold,' in ECCV , ser. Lecture Notes in Computer Science, vol. 9909, 2016. -
  42. [49][49] W. Lotter, G. Kreiman, and D. D. Cox, 'Unsupervised learning of visual structure using predictive generative networks,' arXiv:1511.06380 , 2015. - · arXiv
  43. [50][50] X. Chen, W. Wang, J. Wang, and W. Li, 'Learning object-centric transformation for video prediction,' in ACM-MM , ser. MM '17. New York, NY, USA: ACM, 2017. -
  44. [51][51] X. Jin, X. Li, H. Xiao, X. Shen, Z. Lin, J. Yang, Y. Chen, J. Dong, L. Liu, Z. Jie, J. Feng, and S. Yan, 'Video Scene Parsing with Predictive Feature Learning,' in ICCV , 2017. -
  45. [52][52] N. Wichers, R. Villegas, D. Erhan, and H. Lee, 'Hierarchical long-term video prediction without supervision,' in ICML , ser. Proceedings of Machine Learning Research, vol. 80, 2018. -
  46. [53][53] R. Villegas, J. Yang, Y. Zou, S. Sohn, X. Lin, and H. Lee, 'Learning to generate long-term future via hierarchical prediction,' in ICML , 2017. -
  47. [54][54] J. Walker, K. Marino, A. Gupta, and M. Hebert, 'The pose knows: Video forecasting by generating pose futures,' in ICCV , 2017. -
  48. [55][55] X. Liang, L. Lee, W. Dai, and E. P. Xing, 'Dual motion GAN for future-flow embedded video prediction,' in ICCV , 2017. -
  49. [56][56] P. Luc, N. Neverova, C. Couprie, J. Verbeek, and Y. LeCun, 'Predicting Deeper into the Future of Semantic Segmentation,' in ICCV , 2017. -
  50. [57][57] Z. Hu and J. Wang, 'A novel adversarial inference framework for video prediction with action control,' in ICCV Workshops , Oct 2019. -
  51. [58][58] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, 'Gradient-based learning applied to document recognition,' Proceedings of the IEEE , vol. 86, no. 11, Nov 1998. -
  52. [59][59] V. Jain, J. F. Murray, F. Roth, S. C. Turaga, V. P. Zhigulin, K. L. Briggman, M. Helmstaedter, W. Denk, and H. S. Seung, 'Supervised learning of image restoration with convolutional networks,' in ICCV , 2007. -
  53. [60][60] E. L. Denton, S. Chintala, A. Szlam, and R. Fergus, 'Deep generative image models using a laplacian pyramid of adversarial networks,' in NeurIPS , 2015. -
  54. [61][61] F. Yu, V. Koltun, and T. A. Funkhouser, 'Dilated residual networks,' in CVPR . IEEE, 2017. -
  55. [62][62] L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, 'Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,' TPAMI , vol. 40, no. 4, 2018. -
  56. [66][66] Y. Wang, L. Jiang, M.-H. Yang, L.-J. Li, M. Long, and L. Fei-Fei, 'Eidetic 3d LSTM: A model for video prediction and beyond,' in ICLR , 2019. -
  57. [67][67] C. Vondrick, H. Pirsiavash, and A. Torralba, 'Generating Videos with Scene Dynamics,' in NeurIPS , 2016. -
  58. [68][68] S. Tulyakov, M.-Y. Liu, X. Yang, and J. Kautz, 'MoCoGAN: Decomposing motion and content for video generation,' in CVPR , June 2018. -
  59. [69][69] S. Aigner and M. K¨ orner, 'Futuregan: Anticipating the future frames of video sequences using spatio-temporal 3d convolutions in progressively growing autoencoder gans,' arXiv:1810.01325 , 2018. - · arXiv
  60. [71][71] J. R. van Amersfoort, A. Kannan, M. Ranzato, A. Szlam, D. Tran, and S. Chintala, 'Transformation-based models of video sequences,' arXiv:1701.08435 , 2017. - · arXiv
  61. [72][72] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, 'Learning representations by back-propagating errors,' Nature , vol. 323, no. 6088, 1986. -
  62. [73][73] M. Ranzato, A. Szlam, J. Bruna, M. Mathieu, R. Collobert, and S. Chopra, 'Video (language) modeling: a baseline for generative models of natural videos,' arXiv:1412.6604 , 2014. - · arXiv
  63. [75][75] W. Lotter, G. Kreiman, and D. Cox, 'Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning,' in ICLR (Poster) , 2017. -
  64. [76][76] W. Byeon, Q. Wang, R. K. Srivastava, and P. Koumoutsakos, 'Contextvp: Fully context-aware video prediction,' in CVPR (Workshops) , 2018. -
  65. [77][77] V. Patraucean, A. Handa, and R. Cipolla, 'Spatio-temporal video autoencoder with differentiable memory,' (ICLR) Workshop , 2015. -
  66. [78][78] C. Lu, M. Hirsch, and B. Sch¨ olkopf, 'Flexible Spatio-Temporal Networks for Video Prediction,' in CVPR , 2017. -
  67. [79][79] E. L. Denton and V. Birodkar, 'Unsupervised learning of disentangled representations from video,' in NeurIPS , 2017. -
  68. [80][80] J. Oh, X. Guo, H. Lee, R. L. Lewis, and S. P. Singh, 'ActionConditional Video Prediction using Deep Networks in Atari Games,' in NeurIPS , 2015. -
  69. [81][81] E. Denton and R. Fergus, 'Stochastic video generation with a learned prior,' in ICML , ser. Proceedings of Machine Learning Research, J. G. Dy and A. Krause, Eds., vol. 80, 2018. -
  70. [82][82] S. shahabeddin Nabavi, M. Rochan, and Y. Wang, 'Future Semantic Segmentation with Convolutional LSTM,' in BMVC , 2018. -
  71. [83][83] S. Vora, R. Mahjourian, S. Pirk, and A. Angelova, 'Future segmentation using 3d structure,' arXiv:1811.11358 , 2018. - · arXiv
  72. [84][84] J. Sun, J. Xie, J. Hu, Z. Lin, J. Lai, W. Zeng, and W. Zheng, 'Predicting future instance segmentation with contextual pyramid convLSTMs,' in ACM Multimedia . ACM, 2019. -
  73. [85][85] M. Minderer, C. Sun, R. Villegas, F. Cole, K. P. Murphy, and H. Lee, 'Unsupervised learning of object structure and dynamics from videos,' in NeurIPS , 2019. -
  74. [86][86] S. Hochreiter and J. Schmidhuber, 'Long short-term memory,' Neural Computation , vol. 9, no. 8, 1997. -
  75. [87][87] K. Cho, B. van Merrienboer, C ¸ . G¨ ulc ¸ehre, F. Bougares, H. Schwenk, and Y. Bengio, 'Learning phrase representations using RNN encoder-decoder for statistical machine translation,' EMNLP , pp. 1724-1734, 2014. -
  76. [88][88] A. Graves, S. Fern´ andez, and J. Schmidhuber, 'Multidimensional recurrent neural networks,' in ICANN , vol. 4668, 2007. -
  77. [89][89] C. Finn, I. J. Goodfellow, and S. Levine, 'Unsupervised Learning for Physical Interaction through Video Prediction,' in NeurIPS , 2016. -
  78. [90][90] E. Zhan, S. Zheng, Y. Yue, L. Sha, and P. Lucey, 'Generating multi-agent trajectories using programmatic weak supervision,' in ICLR , 2019. -
  79. [91][91] A. van den Oord, N. Kalchbrenner, and K. Kavukcuoglu, 'Pixel Recurrent Neural Networks,' in ICML , 2016. -
  80. [92][92] R. M. Neal, 'Connectionist learning of belief networks,' Artif. Intell. , vol. 56, no. 1, 1992. -
  81. [94][94] A. van den Oord, N. Kalchbrenner, L. Espeholt, K. Kavukcuoglu, O. Vinyals, and A. Graves, 'Conditional image generation with pixelcnn decoders,' in NIPS , 2016. -
  82. [95][95] N. Kalchbrenner, A. van den Oord, K. Simonyan, I. Danihelka, O. Vinyals, A. Graves, and K. Kavukcuoglu, 'Video pixel networks,' in ICML , 2017, pp. 1771-1779. -
  83. [96][96] K. Fragkiadaki, J. Huang, A. Alemi, S. Vijayanarasimhan, S. Ricco, and R. Sukthankar, 'Motion prediction under multimodality with conditional stochastic networks,' arXiv:1705.02082 , 2017. - · arXiv
  84. [97][97] L. Castrejon, N. Ballas, and A. Courville, 'Improved conditional vrnns for video prediction,' in ICCV , 2019. -
  85. [98][98] J. Chung, K. Kastner, L. Dinh, K. Goel, A. C. Courville, and Y. Bengio, 'A recurrent latent variable model for sequential data,' in NIPS , 2015. -
  86. [99][99] M. Henaff, J. J. Zhao, and Y. LeCun, 'Prediction under uncertainty with error-encoding networks,' arXiv:1711.04994 , 2017. - · arXiv
  87. [100][100] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. WardeFarley, S. Ozair, A. Courville, and Y. Bengio, 'Generative adversarial nets,' in NIPS , 2014, pp. 2672-2680. -
  88. [101][101] Y.-H. Kwon and M.-G. Park, 'Predicting future frames using retrospective cycle gan,' in CVPR , 2019. -
  89. [102][102] C. Vondrick and A. Torralba, 'Generating the Future with Adversarial Transformers,' in CVPR , 2017. -
  90. [103][103] Y. Zhou and T. L. Berg, 'Learning Temporal Transformations from Time-Lapse Videos,' in ECCV , 2016. -
  91. [104][104] P. Bhattacharjee and S. Das, 'Temporal coherency based criteria for predicting video frames using deep multi-stage generative adversarial networks,' in NIPS , 2017. -
  92. [105][105] M. Saito, E. Matsumoto, and S. Saito, 'Temporal generative adversarial nets with singular value clipping,' in ICCV , 2017. -
  93. [106][106] B. Chen, W. Wang, and J. Wang, 'Video imagination from a single image with transformation generation,' in ACM Multimedia , 2017. -
  94. [107][107] M. Mirza and S. Osindero, 'Conditional generative adversarial nets,' arXiv:1411.1784 , 2014. - · arXiv
  95. [108][108] A. X. Lee, R. Zhang, F. Ebert, P. Abbeel, C. Finn, and S. Levine, 'Stochastic adversarial video prediction,' arXiv:1804.01523 , 2018. - · arXiv
  96. [109][109] A. Radford, L. Metz, and S. Chintala, 'Unsupervised representation learning with deep convolutional generative adversarial networks,' in ICLR , 2016. -
  97. [110][110] M. Arjovsky and L. Bottou, 'Towards principled methods for training generative adversarial networks,' in ICLR , 2017. -
  98. [111][111] C. Sch¨ uldt, I. Laptev, and B. Caputo, 'Recognizing human actions: A local SVM approach,' in ICPR . IEEE, 2004. -
  99. [112][112] L. Gorelick, M. Blank, E. Shechtman, M. Irani, and R. Basri, 'Actions as space-time shapes,' Trans. on PAMI , vol. 29, no. 12, 2007. -
  100. [114][114] H. Jhuang, J. Gall, S. Zuffi, C. Schmid, and M. J. Black, 'Towards understanding action recognition,' in ICCV , 2013. -
  101. [115][115] K. Soomro, A. R. Zamir, and M. Shah, 'UCF101: A dataset of 101 human actions classes from videos in the wild,' arXiv:1212.0402 , 2012. - · arXiv
  102. [116][116] W. Zhang, M. Zhu, and K. G. Derpanis, 'From actemes to action: A strongly-supervised representation for detailed action understanding,' in ICCV , 2013. -
  103. [118][118] H. Idrees, A. R. Zamir, Y. Jiang, A. Gorban, I. Laptev, R. Sukthankar, and M. Shah, 'The THUMOS challenge on action recognition for videos 'in the wild',' CVIU , vol. 155, 2017. -
  104. [119][119] P. Doll´ ar, C. Wojek, B. Schiele, and P. Perona, 'Pedestrian detection: A benchmark,' in CVPR , 2009. -
  105. [121][121] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, 'The cityscapes dataset for semantic urban scene understanding,' in CVPR , 2016. -
  106. [124][124] S. Abu-El-Haija, N. Kothari, J. Lee, P. Natsev, G. Toderici, B. Varadarajan, and S. Vijayanarasimhan, 'Youtube-8m: A largescale video classification benchmark,' arXiv:1609.08675 , 2016. - · arXiv
  107. [125][125] B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L. Li, 'YFCC100M: the new data in multimedia research,' Commun. ACM , vol. 59, no. 2, 2016. -
  108. [126][126] I. Sutskever, G. E. Hinton, and G. W. Taylor, 'The recurrent temporal restricted boltzmann machine,' in NIPS , 2008. -
  109. [127][127] C. F. Cadieu and B. A. Olshausen, 'Learning intermediate-level representations of form and motion from natural movies,' Neural Computation , vol. 24, no. 4, 2012. -
  110. [128][128] R. Memisevic and G. Exarchakis, 'Learning invariant features by harnessing the aperture problem,' in ICML , vol. 28, 2013. -
  111. [129][129] F. Ebert, C. Finn, A. X. Lee, and S. Levine, 'Self-supervised visual planning with temporal skip connections,' in CoRL , ser. Proceedings of Machine Learning Research, vol. 78, 2017. -
  112. [130][130] S. Dasari, F. Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, S. Singh, S. Levine, and C. Finn, 'Robonet: Large-scale multirobot learning,' arXiv:1910.11215 , 2019. - · arXiv
  113. [131][131] R. Vezzani and R. Cucchiara, 'Video surveillance online repository (visor): an integrated framework,' Multimedia Tools Appl. , vol. 50, no. 2, 2010. -
  114. [132][132] J. Santner, C. Leistner, A. Saffari, T. Pock, and H. Bischof, 'PROST: parallel robust online simple tracking,' in CVPR , 2010. -
  115. [133][133] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, 'The arcade learning environment: An evaluation platform for general agents,' J. Artif. Intell. Res. , vol. 47, 2013. -
  116. [134][134] G. Seguin, P. Bojanowski, R. Lajugie, and I. Laptev, 'Instancelevel video segmentation from object tracks,' in CVPR , 2016. -
  117. [135][135] Z. Bauer, F. Gomez-Donoso, E. Cruz, S. Orts-Escolano, and M. Cazorla, 'UASOL, a large-scale high-resolution outdoor stereo dataset,' Scientific Data , vol. 6, no. 1, 2019. -
  118. [137][137] E. Santana and G. Hotz, 'Learning a driving simulator,' arXiv:1608.01230 , 2016. - · arXiv
  119. [141][141] J. Walker, C. Doersch, A. Gupta, and M. Hebert, 'An Uncertain Future: Forecasting from Static Images Using Variational Autoencoders,' in ECCV , 2016. -
  120. [142][142] Z. Hao, X. Huang, and S. J. Belongie, 'Controllable video generation with sparse trajectories,' in CVPR , 2018. -
  121. [143][143] Y. Ye, M. Singh, A. Gupta, and S. Tulsiani, 'Compositional video prediction,' in ICCV , October 2019. -
  122. [144][144] S. Mozaffari, O. Y. Al-Jarrah, M. Dianati, P. A. Jennings, and A. Mouzakitis, 'Deep learning-based vehicle behaviour prediction for autonomous driving applications: A review,' arXiv:1912.11676 , 2019. - · arXiv
  123. [146][146] K. Simonyan and A. Zisserman, 'Very deep convolutional networks for large-scale image recognition,' in ICLR , 2015. -
  124. [147][147] T. Brox, A. Bruhn, N. Papenberg, and J. Weickert, 'High accuracy optical flow estimation based on a theory for warping,' in ECCV , T. Pajdla and J. Matas, Eds., vol. 3024, 2004. -
  125. [148][148] T. Karras, T. Aila, S. Laine, and J. Lehtinen, 'Progressive growing of gans for improved quality, stability, and variation,' in ICLR , 2018. -
  126. [150][150] B. Jin, Y. Hu, Q. Tang, J. Niu, Z. Shi, Y. Han, and X. Li, 'Exploring spatial-temporal multi-frequency analysis for high-fidelity and temporal-consistency video prediction,' arXiv:2002.09905 , 2020. - · arXiv
  127. [151][151] O. Shouno, 'Photo-realistic video prediction on natural videos of largely changing frames,' arXiv:2003.08635 , 2020. - · arXiv
  128. [153][153] M. Oliu, J. Selva, and S. Escalera, 'Folded recurrent neural networks for future video prediction,' in ECCV , 2018. -
  129. [154][154] W. Yu, Y. Lu, S. Easterbrook, and S. Fidler, 'Efficient and information-preserving future frame prediction and beyond,' in ICLR , 2020. -
  130. [155][155] F. A. Reda, G. Liu, K. J. Shih, R. Kirby, J. Barker, D. Tarjan, A. Tao, and B. Catanzaro, 'SDC-Net: Video prediction using spatiallydisplaced convolution,' in ECCV , 2018. -
  131. [156][156] R. Memisevic and G. E. Hinton, 'Learning to represent spatial transformations with factored higher-order boltzmann machines,' Neural Computation , vol. 22, no. 6, 2010. -
  132. [157][157] R. Memisevic, 'Gradient-based learning of higher-order image features,' in ICCV , 2011. -
  133. [158][158] --, 'Learning to relate images,' Trans. on PAMI , vol. 35, no. 8, 2013. -
  134. [159][159] V. Michalski, R. Memisevic, and K. Konda, 'Modeling deep temporal dependencies with recurrent grammar cells,' in NeurIPS , 2014. -
  135. [160][160] M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu, 'Spatial Transformer Networks,' in NeurIPS , 2015. -
  136. [161][161] B. Klein, L. Wolf, and Y. Afek, 'A dynamic convolutional layer for short rangeweather prediction,' in CVPR , 2015. -
  137. [162][162] B. D. Brabandere, X. Jia, T. Tuytelaars, and L. V. Gool, 'Dynamic filter networks,' in NeurIPS , 2016. -
  138. [164][164] P. Luc, A. Clark, S. Dieleman, D. de Las Casas, Y. Doron, A. Cassirer, and K. Simonyan, 'Transformation-based adversarial video prediction on large-scale data,' arXiv:2003.04035 , 2020. - · arXiv
  139. [165][165] K. He, X. Zhang, S. Ren, and J. Sun, 'Spatial pyramid pooling in deep convolutional networks for visual recognition,' Trans. on PAMI , vol. 37, no. 9, 2015. -
  140. [166][166] K. Simonyan and A. Zisserman, 'Two-stream convolutional networks for action recognition in videos,' in NeurIPS , Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, Eds., 2014. -
  141. [167][167] H. Gao, H. Xu, Q. Cai, R. Wang, F. Yu, and T. Darrell, 'Disentangling propagation and generation for video prediction,' in ICCV , 2019. -
  142. [169][169] J. Hsieh, B. Liu, D. Huang, F. Li, and J. C. Niebles, 'Learning to decompose and disentangle representations for video prediction,' in NeurIPS , 2018. -
  143. [170][170] S. Chiappa, S. Racani` ere, D. Wierstra, and S. Mohamed, 'Recurrent environment simulators,' in ICLR , 2017. -
  144. [171][171] K. Fragkiadaki, P. Agrawal, S. Levine, and J. Malik, 'Learning visual predictive models of physics for playing billiards,' in ICLR (Poster) , 2016. -
  145. [172][172] A. Dosovitskiy and V. Koltun, 'Learning to Act by Predicting the Future,' in ICLR , 2017. -
  146. [174][174] H.-k. Chiu, E. Adeli, and J. C. Niebles, 'Segmenting the future,' arXiv:1904.10666 , 2019. - · arXiv
  147. [175][175] X. Jin, H. Xiao, X. Shen, J. Yang, Z. Lin, Y. Chen, Z. Jie, J. Feng, and S. Yan, 'Predicting Scene Parsing and Motion Dynamics in the Future,' in NeurIPS , 2017. -
  148. [176][176] J. Ba and R. Caruana, 'Do deep nets really need to be deep?' in NIPS , 2014. -
  149. [177][177] G. E. Hinton, O. Vinyals, and J. Dean, 'Distilling the knowledge in a neural network,' arXiv:1503.02531 , 2015. - · arXiv
  150. [178][178] J. Revaud, P. Weinzaepfel, Z. Harchaoui, and C. Schmid, 'EpicFlow: Edge-preserving interpolation of correspondences for optical flow,' in CVPR , 2015. -
  151. [179][179] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, 'Pyramid scene parsing network,' in CVPR , 2017. -
  152. [180][180] M. Rosca, B. Lakshminarayanan, D. Warde-Farley, and S. Mohamed, 'Variational approaches for auto-encoding generative adversarial networks,' arXiv:1706.04987 , 2017. - · arXiv
  153. [181][181] K. He, G. Gkioxari, P. Doll´ ar, and R. B. Girshick, 'Mask R-CNN,' in ICCV , 2017. -
  154. [182][182] S. E. Reed, Y. Zhang, Y. Zhang, and H. Lee, 'Deep visual analogymaking,' in NIPS , 2015. -
  155. [183][183] N. Fushishita, A. Tejero-de-Pablos, Y. Mukuta, and T. Harada, 'Long-term video generation of multiple futures using human poses,' arXiv:1904.07538 , 2019. - · arXiv
  156. [185][185] Y. Bengio, R. Ducharme, P. Vincent, and C. Janvin, 'A neural probabilistic language model,' J. Mach. Learn. Res. , vol. 3, no. null, p. 11371155, Mar. 2003. -
  157. [186][186] I. Sutskever, O. Vinyals, and Q. V. Le, 'Sequence to sequence learning with neural networks,' in NeurIPS , 2014. -
  158. [187][187] A. Mahendran and A. Vedaldi, 'Understanding deep image representations by inverting them,' in CVPR , 2015. -
  159. [188][188] R. Chalasani and J. C. Pr´ ıncipe, 'Deep predictive coding networks,' in ICLR (Workshop Poster) , 2013. -
  160. [189][189] M. F. Stollenga, W. Byeon, M. Liwicki, and J. Schmidhuber, 'Parallel multi-dimensional lstm, with application to fast biomedical volumetric image segmentation,' in NeurIPS , 2015. -
  161. [190][190] J. Zhang, Y. Zheng, and D. Qi, 'Deep spatio-temporal residual networks for citywide crowd flows prediction,' in AAAI , 2017.
  162. [192][192] Z. Yi, H. R. Zhang, P. Tan, and M. Gong, 'Dualgan: Unsupervised dual learning for image-to-image translation,' in ICCV , 2017.
  163. [194][194] W. Luo, W. Liu, and S. Gao, 'A revisit of sparse coding based anomaly detection in stacked RNN framework,' in ICCV . IEEE, 2017.
  164. [196][196] L. Dinh, D. Krueger, and Y. Bengio, 'NICE: non-linear independent components estimation,' in ICLR (Workshop) , 2015.
  165. [197][197] Y. Wang, M. Long, J. Wang, Z. Gao, and P. S. Yu, 'Predrnn: Recurrent neural networks for predictive learning using spatiotemporal lstms,' in NeurIPS , 2017.
  166. [199][199] S. Niklaus, L. Mai, and F. Liu, 'Video frame interpolation via adaptive separable convolution,' in ICCV . IEEE, 2017.
  167. [200][200] --, 'Video frame interpolation via adaptive convolution,' in CVPR . IEEE, 2017.
  168. [201][201] J. Carreira, E. Noland, A. Banki-Horvath, C. Hillier, and A. Zisserman, 'A short note about kinetics-600,' arXiv:1808.01340 , 2018. · arXiv
  169. [202][202] T. Xue, J. Wu, K. L. Bouman, and B. Freeman, 'Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks,' in NeurIPS , 2016.
  170. [203][203] S. Song, F. Yu, A. Zeng, A. X. Chang, M. Savva, and T. A. Funkhouser, 'Semantic scene completion from a single depth image,' in CVPR . IEEE, 2017.
  171. [204][204] M. Menze and A. Geiger, 'Object scene flow for autonomous vehicles,' in CVPR , 2015.
  172. [205][205] J. Janai, F. G¨ uney, A. Ranjan, M. J. Black, and A. Geiger, 'Unsupervised learning of multi-frame optical flow with occlusions,' in ECCV , vol. 11220, 2018.
  173. [206][206] S. E. Reed, Z. Akata, S. Mohan, S. Tenka, B. Schiele, and H. Lee, 'Learning what and where to draw,' in NIPS , 2016.
  174. [208][208] K. Fragkiadaki, S. Levine, P. Felsen, and J. Malik, 'Recurrent Network Models for Human Dynamics,' in ICCV , 2015.
  175. [209][209] T. Jakab, A. Gupta, H. Bilen, and A. Vedaldi, 'Conditional image generation for learning the structure of visual objects,' arXiv:1806.07823 , 2018. · arXiv
  176. [210][210] Y. Zhang, Y. Guo, Y. Jin, Y. Luo, Z. He, and H. Lee, 'Unsupervised discovery of object landmarks as structural representations,' in CVPR , 2018.
  177. [211][211] R. Goroshin, M. Mathieu, and Y. LeCun, 'Learning to linearize under uncertainty,' in NeurIPS , 2015.
  178. [212][212] G. E. Hinton, A. Krizhevsky, and S. D. Wang, 'Transforming autoencoders,' in ICANN , vol. 6791. Springer, 2011.
  179. [213][213] R. Goroshin, J. Bruna, J. Tompson, D. Eigen, and Y. LeCun, 'Unsupervised learning of spatiotemporally coherent metrics,' in ICCV , 2015.
  180. [214][214] T. Brox and J. Malik, 'Object segmentation by long term analysis of point trajectories,' in ECCV , vol. 6315, 2010.
  181. [215][215] J. Schmidhuber, 'Learning complex, extended sequences using the principle of history compression,' Neural Computation , vol. 4, no. 2, 1992.
  182. [216][216] P. Agrawal, A. Nair, P. Abbeel, J. Malik, and S. Levine, 'Learning to poke by poking: Experiential learning of intuitive physics,' in NeurIPS , 2016, p. 50925100.
  183. [217][217] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. P. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, 'Asynchronous methods for deep reinforcement learning,' in ICML , vol. 48, 2016.
  184. [218][218] J. Zhang and K. Cho, 'Query-efficient imitation learning for endto-end simulated driving,' in AAAI , 2017.
  185. [219][219] S. Kohl, B. Romera-Paredes, C. Meyer, J. De Fauw, J. R. Ledsam, K. Maier-Hein, S. A. Eslami, D. J. Rezende, and O. Ronneberger, 'A probabilistic u-net for segmentation of ambiguous images,' in NeurIPS , 2018.
  186. [220][220] F. Yu, W. Xian, Y. Chen, F. Liu, M. Liao, V. Madhavan, and T. Darrell, 'BDD100K: A diverse driving video database with scalable annotation tooling,' arXiv:1805.04687 , 2018. · arXiv
  187. [222][222] D. P. Kingma and M. Welling, 'Auto-encoding variational bayes,' in ICLR , 2014.
  188. [223][223] X. Yan, J. Yang, K. Sohn, and H. Lee, 'Attribute2image: Conditional image generation from visual attributes,' in ECCV , vol. 9908, 2016.
  189. [224][224] H. Wu, M. Rubinstein, E. Shih, J. V. Guttag, F. Durand, and W. T. Freeman, 'Eulerian video magnification for revealing subtle changes in the world,' ToG , vol. 31, no. 4, 2012.
  190. [225][225] R. Villegas, A. Pathak, H. Kannan, D. Erhan, Q. V. Le, and H. Lee, 'High fidelity video prediction with large stochastic recurrent neural networks,' in NeurIPS , 2019, pp. 81-91.
  191. [226][226] C. K. Sønderby, T. Raiko, L. Maaløe, S. K. Sønderby, and O. Winther, 'Ladder variational autoencoders,' in NIPS , 2016.
  192. [227][227] R. Pottorff, J. Nielsen, and D. Wingate, 'Video extrapolation with an invertible linear embedding,' arXiv:1903.00133 , 2019. · arXiv
  193. [228][228] D. P. Kingma and P. Dhariwal, 'Glow: Generative flow with invertible 1x1 convolutions,' in NeurIPS , 2018.
  194. [230][230] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, 'The unreasonable effectiveness of deep features as a perceptual metric,' in CVPR , 2018.
  195. [231][231] T. Unterthiner, S. van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly, 'Towards accurate generative models of video: A new metric & challenges,' arXiv:1812.01717 , 2018. · arXiv
  196. [232][232] T. Salimans, I. J. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, 'Improved techniques for training gans,' in NIPS , 2016.
  197. [233][233] O. Breuleux, Y. Bengio, and P. Vincent, 'Quickly generating representative samples from an rbm-derived process,' Neural Computation , vol. 23, no. 8, 2011.
  198. [234][234] E. Hildreth, 'Theory of edge detection,' Proc. of Royal Society of London , vol. 207, no. 187-217, 1980.
  199. [235][235] Y. Wang, Z. Gao, M. Long, J. Wang, and P. S. Yu, 'Predrnn++: Towards A resolution of the deep-in-time dilemma in spatiotemporal predictive learning,' in ICML , ser. Proceedings of Machine Learning Research, vol. 80, 2018.
  200. [236][236] F. Cricri, X. Ni, M. Honkala, E. Aksu, and M. Gabbouj, 'Video ladder networks,' arXiv:1612.01756 , 2016. · arXiv

200 of 205 shown

Authors
Oprea, S., Martinez-Gonzalez, P., Garcia-Garcia, A., Castro-Vargas, J. A., Orts-Escolano, S., Garcia-Rodriguez, J., & Argyros, A.
Affiliations
  • University of Alicante (ES)
  • Department of Computer Technology, University of Alicante, Alicante, Spain
  • Campus UAB, Institute of Space Sciences (ICE-CSIC), Barcelona, Spain
  • Department of Computer Science and Artificial Intelligence, University of Alicante, Alicante, Spain
  • University of Crete (GR)
ORCID
Venue
IEEE Transactions on Pattern Analysis and Machine Intelligence
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2022-06-01
Volume
44
Issue
6
Pages
2806-2826
ISSN
  • 0162-8828 (ISSN-L)
  • 2160-9292
  • 1939-3539
DOI
10.1109/tpami.2020.3045007
OpenAlex
W3015879326
Cited by
301
References
310
Citations by year
2026: 26 · 2025: 63 · 2024: 63 · 2023: 52 · 2022: 64 · 2021: 27 · 2020: 6
FWCI
3.17
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DIGITAL.CSIC (Spanish National Research Council (CSIC)) (submittedVersion)
  • RUA, Repositorio Institucional de la Universidad de Alicante (Universidad de Alicante) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • MoDeaAS (PID2019-104818RB-I00)
  • Spanish national grants for PhD studies (FPU17/00166, ACIF/2018/197)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
video prediction, self-supervised learning, representation learning, open problems
MeSH terms
Deep Learning, Algorithms

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review