Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learning
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
Spatiotemporal predictive learning aims to generate future frames by learning from historical frames. In this paper, we investigate existing methods and present a general framework of spatiotemporal predictive learning, in which the spatial encoder and decoder capture intra-frame features and the middle temporal module catches inter-frame correlations. While the mainstream methods employ recurrent units to capture long-term temporal dependencies, they suffer from low computational efficiency due to their unparallelizable architectures. To parallelize the temporal module, we propose the Temporal Attention Unit (TAU), which decomposes temporal attention into intra-frame statical attention and inter-frame dynamical attention. Moreover, while the mean squared error loss focuses on intra-frame errors, we introduce a novel differential divergence regularization to take inter-frame variations into account. Extensive experiments demonstrate that the proposed method enables the derived model to achieve competitive performance on various spatiotemporal prediction benchmarks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072 · cited 1×
“The last decade has witnessed revolutionary advances in deep learning across various supervised learning tasks such as image classification [30, 74], object detection [64, 66], and etc.”
BackgroundMomentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913 · cited 1×
“We investigate the such ability of our method by training the model on the KITTI [25] dataset and evaluating it on the Caltech Pedestrian [16] dataset.”
MethodLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816
Masked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489
Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-K., & Woo, W.-C. (2015) · arXiv · doi:10.48550/arxiv.1506.04214 · Cited by 7,709 · cited 5×
“We evaluate our proposed method against strong recent baselines, including competitive recurrent architectures: ConvLSTM [78], PredRNN [95], PredRNN++ [93], MIM [98], LMC [50], E3D-LSTM [94], Conv-TT-LSTM [82], and CrevNet [103].”
BackgroundMethodInfluentialXLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
Context Encoders: Feature Learning by Inpainting
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.278 · Cited by 5,691
Deep learning and process understanding for data-driven Earth system science
Reichstein, M., Camps-Valls, G., Stevens, B., Jung, M., Denzler, J., Carvalhais, N., & Prabhat (2019) · Nature · doi:10.1038/s41586-019-0912-1 · Cited by 5,673 · cited 1×
“Accurate spatiotemporal predictive learning can benefit broad practical applications in climate change [74, 77], human motion forecasting [91, 107], traffic flow prediction [18, 97], and representation learning [39, 71].”
BackgroundBootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.07733 · Cited by 3,846 · cited 2×
“Contrastive self-supervised learning [7, 8, 28, 33, 92, 104] is a dominant manner in visual self-supervised learning that aims at a pretext task of grouping similar samples closer and diverse samples away from each other.”
BackgroundConvolutional Sequence to Sequence Learning
Gehring, J., Auli, M., Grangier, D., Yarats, D., & Dauphin, Y. N. (2017) · arXiv · doi:10.48550/arxiv.1705.03122 · Cited by 2,850 · cited 1×
“Most of the existing methods [2,3,9,11,12,19,24,29,41, 44,65,78,79,82,88,89,93–96, 98, 100 , 103] in spatiotemporal spatiotemporal predictive learning employ hybrid Based on the general framework, we argue that the temporal module plays an essential role in spatiotemporal predictive learning.”
MethodUnsupervised Learning of Visual Representations by Solving Jigsaw Puzzles
Noroozi, M., & Favaro, P. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46466-4_5 · Cited by 2,775 · cited 1×
“Early works on visual self-supervised learning design pretext tasks such as colorization [108], inpainting [69], rotation [26], jigsaw [66].”
BackgroundUnsupervised Representation Learning by Predicting Image Rotations
Gidaris, S., Singh, P., & Komodakis, N. (2018) · arXiv · doi:10.48550/arxiv.1803.07728 · Cited by 2,477
Unsupervised Learning of Video Representations using LSTMs
Srivastava, N., Mansimov, E., & Salakhutdinov, R. (2015) · arXiv · doi:10.48550/arxiv.1502.04681 · Cited by 1,861
NICE: Non-linear Independent Components Estimation
Dinh, L., Krueger, D., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1410.8516 · Cited by 1,493
Wang, T., & Isola, P. (2020) · arXiv · doi:10.48550/arxiv.2005.10242 · Cited by 921
Unsupervised Learning for Physical Interaction through Video Prediction
Finn, C., Goodfellow, I., & Levine, S. (2016) · arXiv · doi:10.48550/arxiv.1605.07157 · Cited by 759
Rethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · cited 1×
“Through the competition mechanism of the softmax function [10,70,99], the high difference frames are penalized.”
MethodPredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning
Wang, Y., Wu, H., Zhang, J., Gao, Z., Wang, J., Yu, P. S., & Long, M. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3165153 · Cited by 677
Self-citationDecomposing Motion and Content for Natural Video Sequence Prediction
Villegas, R., Yang, J., Hong, S., Lin, X., & Lee, H. (2017) · arXiv · doi:10.48550/arxiv.1706.08033 · Cited by 488 · cited 2×
“Caltech Pedestrian Method SSIM ↑ PSNR ↑ BeyondMSE [64] 0.847 - MCnet [90] 0.879 - DVF [59] 0.897 26.2 Dual-GAN [53] 0.899 - CtrlGen [31] 0.900 26.5 PredNet [61] 0.905 27.6 ContextVP [4] 0.921 28.7 GAN-VGG [80] 0.916 - G-VGG [80] 0.917 - SDC-Net [72] 0.918 - rCycleGan [47] 0.919 29.2 DPG [20] 0.923…”
MethodInfluentialA Review on Deep Learning Techniques for Video Prediction
Oprea, S., Martinez-Gonzalez, P., Garcia-Garcia, A., Castro-Vargas, J. A., Orts-Escolano, S., Garcia-Rodriguez, J., & Argyros, A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3045007 · Cited by 301
Video Prediction Recalling Long-term Motion Context via Memory Alignment Learning
Lee, S., Kim, H. G., Hwi Choi, D., Kim, H.-I., & Ro, Y. M. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00307 · Cited by 129 · cited 2×
“We evaluate our proposed method against strong recent baselines, including competitive recurrent architectures: ConvLSTM [78], PredRNN [95], Pre-dRNN++ [93], MIM [98], LMC [50], E3D-LSTM [94], Conv-TT-LSTM [82], and CrevNet [103].”
BackgroundMethod
Research works printed but not linked: 81
- liu, 2022, Automix: Unveiling the power of mixup for stronger classifiers, Computer Vision- ECCV 2022 17th European Conference Tel Aviv Israel October 23–27 2022 Proceedings Part XXIV, 441
- 10.1109/ICCV.2017.478 · DOI · Paper page
- 10.1109/CVPR52688.2022.01167 · DOI · Paper page
- 10.1109/ICCV.2017.194 · DOI · Paper page
- li, 2022, Efficient multi -order gated aggregation network, ar Xiv preprint
- liu, 2022, Decoupled mixup for data-efficient learning, ArXiv Preprint
- 10.1109/TCSVT.2020.3021409 · DOI
- khemakhem, 2020, Variational autoencoders and nonlinear ica: A unifying framework, International Conference on Artificial Intelligence and Statistics, 2207
- lee, 2018, Stochastic adversarial video prediction, ArXiv Preprint
- 10.1109/CVPR.2019.00191 · DOI · Paper page
- 10.1109/IROS.2018.8594264 · DOI
- 10.1109/CVPR42600.2020.00461 · DOI · Paper page
- kalchbrenner, 0, Video pixel networks, International Conference on Machine Learning, 1771
- lee, 2018, Mutual suppression network for video prediction using disentangled features, ArXiv Preprint
- chen, 2020, Big self-supervised mod-els are strong semi-supervised learners, Advances in neural information processing systems, 33, 22243
- chen, 2020, A simple framework for contrastive learning of visual representations, International Conference on Machine Learning, 1597
- 10.1609/aaai.v35i2.16202 · DOI
- byeon, 2018, Contextvp: Fully context-aware video prediction, Proceedings of the European Conference on Computer Vision (ECCV), 753
- 10.1007/978-3-030-58592-1_7 · DOI
- castrejon, 2019, Im-proved conditional vrnns for video prediction, Proceedings of the IEEEICVF International Conference on Computer Vision, 7608
- cao, 2022, A survey on generative diffusion model, ArXiv Preprint
- 10.1109/CVPR46437.2021.01518 · DOI · Paper page
- 10.24963/ijcai.2018/408 · DOI
- jia, 2016, Dynamic filter networks, Advances in neural information processing systems, 29
- hinton, 2021, How to represent part-whole hierarchies in a neural network, ArXiv Preprint
- hsieh, 2018, Learning to decompose and dis-entangle representations for video prediction, Advances in neural information processing systems, 31
- 10.1109/CVPR.2018.00819 · DOI
- guo, 2022, Visual attention network, ArXiv Preprint
- jenni, 2020, Video representation learning by recognizing temporal transformations, European Conference on Computer Vision, 425
- 10.1109/CVPR52688.2022.00317 · DOI · Paper page
- 10.1109/ICCV.2019.00910 · DOI · Paper page
- gao, 2022, Pifold: To-ward effective and efficient protein inverse folding, ar Xiv preprint
- gao, 2022, Alphadesign: A graph protein design method and benchmark on alphafolddb, ArXiv Preprint
- greff, 2019, Multi-object representation learning with iterative variational inference, International Conference on Machine Learning, 2424
- 10.1109/CVPR42600.2020.01149 · DOI · Paper page
- ding, 2022, Scaling up your kernels to 31x31: Revisiting large kernel design in cnns, ArXiv Preprint
- denton, 2017, Unsupervised learning of disentangled representations from video, Advances in neural information processing systems, 30
- dinh, 0, Density estimation using real nvp, International Conference on Learning Representations
- dinh, 2015, Nice: Nonlinear independent components estimation, In International Conference on Learning Representations Workshop
- 10.1109/CVPR.2019.00937 · DOI · Paper page
- denton, 2018, Stochastic video generation with a learned prior, International Conference on Machine Learning, 1174
- wu, 2022, Flowformer: Linearizing transformers with conservation flows, Proceedings of the 39th International Conference on Machine Learning volume 162 of Proceedings of Machine Learning Research, 24226
- 10.1109/CVPR.2019.00937 · DOI · Paper page
- 10.1109/CVPR.2009.5206631 · DOI · Paper page
- 10.24963/ijcai.2019/317 · DOI · Paper page
- wang, 2018, Predrnn++: Towards a resolution of the deep-in-time dilemma in spatiotemporal predictive learning, International Conference on Machine Learning, 5123
- wang, 2017, Predrnn: Recurrent neural networks for predictive learning using spatiotemporal lstms, Advances in Neural IInformation Processing Systems
- wang, 2018, Eidetic 3d lstm: A model for video prediction and beyond, International Conference on Learning Representations
- 10.1016/j.cviu.2018.04.007 · DOI · Paper page
- villegas, 2019, High fidelity video prediction with large stochastic recurrent neural networks, Advances in Neural IInformation Processing Systems
- 10.1145/3474085.3475622 · DOI
- 10.1109/CVPR52688.2022.00710 · DOI
- villegas, 2018, Hi-erarchicallong-term video prediction without supervision, International Conference on Machine Learning, 6038
- su, 2020, Convolutional tensor-train lstm for spatiotemporallearning, Advances in neural information processing systems, 33, 13714
- tan, 2022, Target-aware molecular graph generation, ar Xiv preprint
- tan, 2022, Rfold: Towards simple yet effective rna secondary structure prediction, ArXiv Preprint
- shouno, 2020, Photo-realistic video prediction on natural videos of largely changing frames, ArXiv Preprint
- shi, 2017, Deep learning for precipitation nowcasting: A benchmark and a new model, Advances in neural information processing systems
- zhang, 2016, Colorful image colorization, European Conference on Computer Vision, 649
- 10.1016/j.neucom.2021.08.079 · DOI
- zhang, 2017, Deep spatiotemporal residual networks for citywide crowd flows prediction, Thirty-First AAAI Conference on Artificial Intelligence, 10.1609/aaai.v31i1.10735 · DOI · Paper page
- zhang, 2017, Learning spatiotemporal features using 3dcnn and convolutionallstm for gesture recognition, Proceedings of the IEEE International Conference on Computer Vision Workshops, 3120
- ren, 2015, Faster r-cnn: Towards real-time object detection with re-gion proposal networks, Advances in neural information processing systems, 28, 91
- zbontar, 2021, Barlow twins: Self-supervised learning via redundancy reduction, ArXiv Preprint
- 10.1109/ICME.2019.00048 · DOI
- shi, 2015, Convolutionallstm network: A machine learning approach for precipitation nowcasting, Advances in neural information processing systems
- 10.1109/ICPR.2004.1334462 · DOI · Paper page
- yu, 2019, Efficient and information-preserving future frame prediction and beyond, International Conference on Learning Representations
- babaeizadeh, 2021, Fitvid: Overfitting in pixel-level video prediction, ArXiv Preprint
- babaeizadeh, 2018, Stochastic variational video prediction, International Conference on Learning Representations
- 10.1109/CVPR46437.2021.00689 · DOI · Paper page
- peng, 2020, Random feature attention, International Conference on Learning Representations
- 10.1007/978-3-030-01234-2_44 · DOI · Paper page
- 10.18653/v1/2022.acl-long.561 · DOI
- 10.1007/978-3-030-01264-9_44 · DOI
- mathieu, 2015, Deep multi-scale video prediction beyond mean square error, ArXiv Preprint
- mathieu, 2019, Disentangling disentanglement in variational autoencoders, International Conference on Machine Learning, 4402
- minderer, 2019, Unsupervised learning of object structure and dynamics from videos, Advances in Neural IInformation Processing Systems
- locatello, 2019, Challenging common assumptions in the unsupervised learning of disentangled representations, International Conference on Machine Learning, 4114
- szlam marc’ aurelio ranzato, 2014, Video (language) modeling: a baseline for generative models of natural videos, CoRR abs/1412 6604
- lotter, 2017, Deep predictive coding networks for video prediction and unsupervised learning, International Conference on Learning Representations
Cited by
VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecasting
Tang, Y., Dong, P., Tang, Z., Chu, X., & Liang, J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00575 · Cited by 70
Metadata
- Authors
- Tan, C., Gao, Z., Wu, L., Xu, Y., Xia, J., Li, S., & Li, S. Z.
- Venue
- 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2023-06
- Pages
- 18770-18782
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4386076121
- Cited by
- 219
- References
- 32
- FWCI
- 24.90
- Type
- conference paper
- Language
- en
- Open access
- green
- Funders
- National Key R&D Program of (2022ZDOl15100)
- National Natural Science Foundation of China (U21A20427)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.