CoChat
Sign Up
Cited by 129Generative Adversarial Networks and Image SynthesisHuman Motion and AnimationHuman Pose and Action Recognition

Video Prediction Recalling Long-term Motion Context via Memory Alignment Learning

Open access
Lee, Sangmin, Kim, Hak Gu, Hwi Choi, Dae, Kim, Hyung-Il, Ro, Yong Man·2021·2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr46437.2021.00307
Open this paper in CoChat — free accountOpen PDF

Abstract

Our work addresses long-term motion context issues for predicting future frames. To predict the future precisely, it is required to capture which long-term motion context (e.g., walking or running) the input motion (e.g., leg movement) belongs to. The bottlenecks arising when dealing with the long-term motion context are: (i) how to predict the long-term motion context naturally matching input sequences with limited dynamics, (ii) how to predict the long-term motion context with high-dimensionality (e.g., complex motion). To address the issues, we propose novel motion context-aware video prediction. To solve the bottle-neck (i), we introduce a long-term motion context memory (LMC-Memory) with memory alignment learning. The pro-posed memory alignment learning enables to store long-term motion contexts into the memory and to match them with sequences including limited dynamics. As a result, the long-term context can be recalled from the limited in-put sequence. In addition, to resolve the bottleneck (ii), we propose memory query decomposition to store local motion context (i.e., low-dimensional dynamics) and recall the suitable local context for each local part of the input individually. It enables to boost the alignment effects of the memory. Experimental results show that the proposed method outperforms other sophisticated RNN-based methods, especially in long-term condition. Further, we validate the effectiveness of the proposed network designs by conducting ablation studies and memory feature analysis. The source code of this work is available†.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr46437.2021.00307

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LMC-Memory improves video prediction by matching long-term motion contexts with sequences that exhibit limited dynamics.

    “The LMC-Memory model (Lee et al., 2021) utilizes memory alignment learning to store long-term motion contexts, improving video prediction by matching these contexts with sequences that exhibit limited dynamics.”

    From The AI Hippocampus: How Far are We From Human Memory? (2026) · cites this paper as Lee et al., 2021

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 42

  1. [1][1] M. Babaeizadeh, C. Finn, D. Erhan, R. H. Campbell, and S. Levine. Stochastic variational video prediction. In International Conference on Learning Representations (ICLR) , 2018. 2 -
  2. [2][2] W. Byeon, Q. Wang, R. Kumar Srivastava, and P. Koumoutsakos. Contextvp: Fully context-aware video prediction. In European Conference on Computer Vision (ECCV) , pages 753-769, 2018. 2 -
  3. [3][3] Q. Cai, Y. Pan, T. Yao, C. Yan, and T. Mei. Memory matching networks for one-shot image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4080-4088, 2018. 3 -
  4. [4][4] L. Castrejon, N. Ballas, and A. Courville. Improved conditional vrnns for video prediction. In International Conference on Computer Vision (ICCV) , pages 7608-7617, 2019. 1 -
  5. [5][5] R. Chandra, U. Bhattacharya, A. Bera, and D. Manocha. Traphic: Trajectory prediction in dense and heterogeneous traffic using weighted interactions. In Conference on Computer Vision and Pattern Recognition (CVPR) , pages 84838492, 2019. 1 -
  6. [6][6] E. Denton and R. Fergus. Stochastic video generation with a learned prior. In International Conference on Machine Learning (ICML) , 2018. 2 -
  7. [7][7] C. Finn, I. Goodfellow, and S. Levine. Unsupervised learning for physical interaction through video prediction. In Neural Information Processing Systems (NeurIPS) , pages 64-72, 2016. 1, 2, 6 -
  8. [8][8] H. Gao, H. Xu, Q. Cai, R. Wang, F. Yu, and T. Darrell. Disentangling propagation and generation for video prediction. In International Conference on Computer Vision (ICCV) , pages 9006-9015, 2019. 2 -
  9. [9][9] D. Gong, L. Liu, V. Le, B. Saha, M.R. Mansour, S. Venkatesh, and A. Hengel. Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In International Conference on Computer Vision (ICCV) , pages 1705-1714, 2019. 3, 4 -
  10. [10][10] T. Han, W. Xie, and A. Zisserman. Memory-augmented dense predictive coding for video representation learning. In European Conference on Computer Vision (ECCV) , 2020. 3 -
  11. [11][11] J. Hsieh, B. Liu, D. Huang, L.F. Fei-Fei, and J.C. Niebles. Learning to decompose and disentangle representations for video prediction. In Neural Information Processing Systems (NeurIPS) , pages 517-526, 2018. 1 -
  12. [13][13] B. Jin, Y. Hu, Q. Tang, J. Niu, Z. Shi, Y. Han, and X. Li. Exploring spatial-temporal multi-frequency analysis for high-fidelity and temporal-consistency video prediction. In Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4554-4563, 2020. 2 -
  13. [14][14] Ł. Kaiser, O. Nachum, A. Roy, and S. Bengio. Learning to remember rare events. In International Conference on Learning Representations (ICLR) , 2017. 3 -
  14. [15][15] N. Kalchbrenner, A¨ a. van den Oord, K. Simonyan, I. Danihelka, O. Vinyals, A. Graves, and K. Kavukcuoglu. Video pixel networks. In International Conference on Machine Learning (ICML) , pages 1771-1779. JMLR. org, 2017. 2, 6 -
  15. [16][16] Y. Kim, S. Nam, I. Cho, and S.J. Kim. Unsupervised keypoint learning for guiding class-conditional video prediction. In Neural Information Processing Systems (NeurIPS) , pages 3809-3819, 2019. 2 -
  16. [17][17] D.P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR) , 2015. 5 -
  17. [18][18] Z. Lai, E. Lu, and W. Xie. Mast: A memory-augmented selfsupervised tracker. In Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6479-6488, 2020. 3 -
  18. [19][19] A. X Lee, R. Zhang, F. Ebert, P. Abbeel, C. Finn, and S. Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:1804.01523 , 2018. 2 - · arXiv
  19. [20][20] S. Lee, J. Sung, Y. Yu, and Gu. Kim. A memory network approach for story-based temporal summarization of 360 videos. In Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1410-1419, 2018. 3 -
  20. [21][21] M. Minderer, C. Sun, R. Villegas, F. Cole, K.P. Murphy, and H. Lee. Unsupervised learning of object structure and dynamics from videos. In Neural Information Processing Systems (NeurIPS) , pages 92-102, 2019. 2 -
  21. [22][22] M. Oliu, J. Selva, and S. Escalera. Folded recurrent neural networks for future video prediction. In European Conference on Computer Vision (ECCV) , pages 716-731, 2018. 7 -
  22. [23][23] H. Park, J. Noh, and B. Ham. Learning memory-guided normality for anomaly detection. In Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1437214381, 2020. 3 -
  23. [24][24] V. Patraucean, A. Handa, and R. Cipolla. Spatio-temporal video autoencoder with differentiable memory. In International Conference on Learning Representations Workshop (ICLRW) , 2016. 2 -
  24. [25][25] W. Pei, J. Zhang, X. Wang, L. Ke, X. Shen, and Y. Tai. Memory-attended recurrent network for video captioning. In Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8347-8356, 2019. 3 -
  25. [26][26] C. Schuldt, I. Laptev, and B. Caputo. Recognizing human actions: a local svm approach. In International Conference on Pattern Recognition (ICPR) , volume 3, pages 32-36. IEEE, 2004. 5 -
  26. [27][27] X. Shi, Z. Gao, L. Lausen, H. Wang, D. Yeung, W. Wong, and W. Woo. Deep learning for precipitation nowcasting: A benchmark and a new model. In Neural Information Processing Systems (NeurIPS) , pages 5617-5627, 2017. 6 -
  27. [29][29] J. Su, W. Byeon, F. Huang, J. Kautz, and A. Anandkumar. Convolutional tensor-train lstm for spatio-temporal learning. In Neural Information Processing Systems (NeurIPS) , 2020. 1, 2, 5, 6, 7 -
  28. [31][31] R. Villegas, D. Erhan, and H. Lee. Hierarchical long-term video prediction without supervision. In International Conference on Machine Learning (ICML) , pages 6038-6046, 2018. 2 -
  29. [32][32] R. Villegas, A. Pathak, H. Kannan, D. Erhan, Quoc V. Le, and H. Lee. High fidelity video prediction with large stochastic recurrent neural networks. In Neural Information Processing Systems (NeurIPS) , pages 81-91, 2019. 2 -
  30. [34][34] R. Villegas, J. Yang, Y. Zou, S. Sohn, X. Lin, and H. Lee. Learning to generate long-term future via hierarchical prediction. In International Conference on Machine Learning (ICML) , pages 3560-3569. JMLR. org, 2017. 1, 2, 5 -
  31. [35][35] Y. Wang, Z. Gao, M. Long, J. Wang, and S.Y. Philip. Predrnn++: Towards a resolution of the deep-in-time dilemma in spatiotemporal predictive learning. In International Conference on Machine Learning (ICML) , pages 5123-5132, 2018. 1, 2, 5, 6, 7 -
  32. [36][36] Y. Wang, L. Jiang, M.H. Yang, L.J. Li, M. Long, and L. FeiFei. Eidetic 3d lstm: A model for video prediction and beyond. In International Conference on Learning Representations (ICLR) , 2019. 1, 2, 5, 6, 7 -
  33. [37][37] Y. Wang, M. Long, J. Wang, Z. Gao, and S.Y. Philip. Predrnn: Recurrent neural networks for predictive learning using spatiotemporal lstms. In Neural Information Processing Systems (NeurIPS) , pages 879-888, 2017. 2, 5, 6, 7 -
  34. [39][39] Y. Wu, R. Gao, J. Park, and Q. Chen. Future video synthesis with object motion prediction. In Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5539-5548, 2020. 2 -
  35. [40][40] S. Xingjian, Z. Chen, H. Wang, D. Yeung, W. Wong, and W. Woo. Convolutional lstm network: A machine learning approach for precipitation nowcasting. In Neural Information Processing Systems (NeurIPS) , pages 802-810, 2015. 1, 3 -
  36. [41][41] J. Xu, B. Ni, and X. Yang. Video prediction via selective sampling. In Neural Information Processing Systems (NeurIPS) , pages 1705-1715, 2018. 2 -
  37. [42][42] Q. Xu, H. Zhang, W. Wang, P. Belhumeur, and U. Neumann. Stochastic dynamics for video infilling. In Winter Conference on Applications of Computer Vision (WACV) , pages 2714-2723, 2020. 2 -
  38. [43][43] Y. Ye, M. Singh, A. Gupta, and S. Tulsiani. Compositional video prediction. In International Conference on Computer Vision (ICCV) , pages 10353-10362, 2019. 2 -
  39. [44][44] W. Yu, Y. Lu, S. Easterbrook, and S. Fidler. Efficient and information-preserving future frame prediction and beyond. In International Conference on Learning Representations (ICLR) , volume 2020, 2020. 2 -
  40. [45][45] R. Zhang, P. Isola, A.A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Conference on Computer Vision and Pattern Recognition (CVPR) , pages 586-595, 2018. 5, 6 -
  41. [46][46] L. Zhu and Y. Yang. Inflated episodic memory with region self-attention for long-tailed visual recognition. In Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4344-4353, 2020. 3 -
  42. [47][47] M. Zhu, P. Pan, W. Chen, and Y. Yang. Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis. In Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5802-5810, 2019. 3

Cited by

Sorted by citations

Authors
Lee, S., Kim, H. G., Hwi Choi, D., Kim, H.-I., & Ro, Y. M.
Affiliations
  • Korea Advanced Institute of Science and Technology (KR)
  • Electronics and Telecommunications Research Institute (KR)
ORCID
Venue
2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2021-06
Pages
3053-3062
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr46437.2021.00307
OpenAlex
W3142930505
Cited by
129
References
5
Citations by year
2026: 5 · 2025: 32 · 2024: 31 · 2023: 31 · 2022: 26 · 2021: 4
FWCI
7.38
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Funders
Institute for Information and Communications Technology Promotion (2020-0-00004)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Engineering › Control and Systems Engineering › Human Motion and Animation
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
Keywords
video prediction, future frame prediction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review