Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learning
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
Spatiotemporal predictive learning aims to generate future frames by learning from historical frames. In this paper, we investigate existing methods and present a general framework of spatiotemporal predictive learning, in which the spatial encoder and decoder capture intra-frame features and the middle temporal module catches inter-frame correlations. While the mainstream methods employ recurrent units to capture long-term temporal dependencies, they suffer from low computational efficiency due to their unparallelizable architectures. To parallelize the temporal module, we propose the Temporal Attention Unit (TAU), which decomposes temporal attention into intra-frame statical attention and inter-frame dynamical attention. Moreover, while the mean squared error loss focuses on intra-frame errors, we introduce a novel differential divergence regularization to take inter-frame variations into account. Extensive experiments demonstrate that the proposed method enables the derived model to achieve competitive performance on various spatiotemporal prediction benchmarks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072 · cited 1×
“The last decade has witnessed revolutionary advances in deep learning across various supervised learning tasks such as image classification [30, 74], object detection [64, 66], and etc.”
Background
Research works printed but not linked: 81
- liu, 2022, Automix: Unveiling the power of mixup for stronger classifiers, Computer Vision- ECCV 2022 17th European Conference Tel Aviv Israel October 23–27 2022 Proceedings Part XXIV, 441
- 10.1109/ICCV.2017.478 · DOI · Paper page
- 10.1109/CVPR52688.2022.01167 · DOI · Paper page
- 10.1109/ICCV.2017.194 · DOI · Paper page
- li, 2022, Efficient multi -order gated aggregation network, ar Xiv preprint
- liu, 2022, Decoupled mixup for data-efficient learning, ArXiv Preprint
- 10.1109/TCSVT.2020.3021409 · DOI
- khemakhem, 2020, Variational autoencoders and nonlinear ica: A unifying framework, International Conference on Artificial Intelligence and Statistics, 2207
- lee, 2018, Stochastic adversarial video prediction, ArXiv Preprint
- 10.1109/CVPR.2019.00191 · DOI · Paper page
- 10.1109/IROS.2018.8594264 · DOI
- 10.1109/CVPR42600.2020.00461 · DOI · Paper page
- kalchbrenner, 0, Video pixel networks, International Conference on Machine Learning, 1771
- lee, 2018, Mutual suppression network for video prediction using disentangled features, ArXiv Preprint
- chen, 2020, Big self-supervised mod-els are strong semi-supervised learners, Advances in neural information processing systems, 33, 22243
- chen, 2020, A simple framework for contrastive learning of visual representations, International Conference on Machine Learning, 1597
- 10.1609/aaai.v35i2.16202 · DOI
- byeon, 2018, Contextvp: Fully context-aware video prediction, Proceedings of the European Conference on Computer Vision (ECCV), 753
- 10.1007/978-3-030-58592-1_7 · DOI
- castrejon, 2019, Im-proved conditional vrnns for video prediction, Proceedings of the IEEEICVF International Conference on Computer Vision, 7608
- cao, 2022, A survey on generative diffusion model, ArXiv Preprint
- 10.1109/CVPR46437.2021.01518 · DOI · Paper page
- 10.24963/ijcai.2018/408 · DOI
- jia, 2016, Dynamic filter networks, Advances in neural information processing systems, 29
- hinton, 2021, How to represent part-whole hierarchies in a neural network, ArXiv Preprint
- hsieh, 2018, Learning to decompose and dis-entangle representations for video prediction, Advances in neural information processing systems, 31
- 10.1109/CVPR.2018.00819 · DOI
- guo, 2022, Visual attention network, ArXiv Preprint
- jenni, 2020, Video representation learning by recognizing temporal transformations, European Conference on Computer Vision, 425
- 10.1109/CVPR52688.2022.00317 · DOI · Paper page
- 10.1109/ICCV.2019.00910 · DOI · Paper page
- gao, 2022, Pifold: To-ward effective and efficient protein inverse folding, ar Xiv preprint
- gao, 2022, Alphadesign: A graph protein design method and benchmark on alphafolddb, ArXiv Preprint
- greff, 2019, Multi-object representation learning with iterative variational inference, International Conference on Machine Learning, 2424
- 10.1109/CVPR42600.2020.01149 · DOI · Paper page
- ding, 2022, Scaling up your kernels to 31x31: Revisiting large kernel design in cnns, ArXiv Preprint
- denton, 2017, Unsupervised learning of disentangled representations from video, Advances in neural information processing systems, 30
- dinh, 0, Density estimation using real nvp, International Conference on Learning Representations
- dinh, 2015, Nice: Nonlinear independent components estimation, In International Conference on Learning Representations Workshop
- 10.1109/CVPR.2019.00937 · DOI · Paper page
- denton, 2018, Stochastic video generation with a learned prior, International Conference on Machine Learning, 1174
- wu, 2022, Flowformer: Linearizing transformers with conservation flows, Proceedings of the 39th International Conference on Machine Learning volume 162 of Proceedings of Machine Learning Research, 24226
- 10.1109/CVPR.2019.00937 · DOI · Paper page
- 10.1109/CVPR.2009.5206631 · DOI · Paper page
- 10.24963/ijcai.2019/317 · DOI · Paper page
- wang, 2018, Predrnn++: Towards a resolution of the deep-in-time dilemma in spatiotemporal predictive learning, International Conference on Machine Learning, 5123
- wang, 2017, Predrnn: Recurrent neural networks for predictive learning using spatiotemporal lstms, Advances in Neural IInformation Processing Systems
- wang, 2018, Eidetic 3d lstm: A model for video prediction and beyond, International Conference on Learning Representations
- 10.1016/j.cviu.2018.04.007 · DOI · Paper page
- villegas, 2019, High fidelity video prediction with large stochastic recurrent neural networks, Advances in Neural IInformation Processing Systems
- 10.1145/3474085.3475622 · DOI
- 10.1109/CVPR52688.2022.00710 · DOI
- villegas, 2018, Hi-erarchicallong-term video prediction without supervision, International Conference on Machine Learning, 6038
- su, 2020, Convolutional tensor-train lstm for spatiotemporallearning, Advances in neural information processing systems, 33, 13714
- tan, 2022, Target-aware molecular graph generation, ar Xiv preprint
- tan, 2022, Rfold: Towards simple yet effective rna secondary structure prediction, ArXiv Preprint
- shouno, 2020, Photo-realistic video prediction on natural videos of largely changing frames, ArXiv Preprint
- shi, 2017, Deep learning for precipitation nowcasting: A benchmark and a new model, Advances in neural information processing systems
- zhang, 2016, Colorful image colorization, European Conference on Computer Vision, 649
- 10.1016/j.neucom.2021.08.079 · DOI
- zhang, 2017, Deep spatiotemporal residual networks for citywide crowd flows prediction, Thirty-First AAAI Conference on Artificial Intelligence, 10.1609/aaai.v31i1.10735 · DOI · Paper page
- zhang, 2017, Learning spatiotemporal features using 3dcnn and convolutionallstm for gesture recognition, Proceedings of the IEEE International Conference on Computer Vision Workshops, 3120
- ren, 2015, Faster r-cnn: Towards real-time object detection with re-gion proposal networks, Advances in neural information processing systems, 28, 91
- zbontar, 2021, Barlow twins: Self-supervised learning via redundancy reduction, ArXiv Preprint
- 10.1109/ICME.2019.00048 · DOI
- shi, 2015, Convolutionallstm network: A machine learning approach for precipitation nowcasting, Advances in neural information processing systems
- 10.1109/ICPR.2004.1334462 · DOI · Paper page
- yu, 2019, Efficient and information-preserving future frame prediction and beyond, International Conference on Learning Representations
- babaeizadeh, 2021, Fitvid: Overfitting in pixel-level video prediction, ArXiv Preprint
- babaeizadeh, 2018, Stochastic variational video prediction, International Conference on Learning Representations
- 10.1109/CVPR46437.2021.00689 · DOI · Paper page
- peng, 2020, Random feature attention, International Conference on Learning Representations
- 10.1007/978-3-030-01234-2_44 · DOI · Paper page
- 10.18653/v1/2022.acl-long.561 · DOI
- 10.1007/978-3-030-01264-9_44 · DOI
- mathieu, 2015, Deep multi-scale video prediction beyond mean square error, ArXiv Preprint
- mathieu, 2019, Disentangling disentanglement in variational autoencoders, International Conference on Machine Learning, 4402
- minderer, 2019, Unsupervised learning of object structure and dynamics from videos, Advances in Neural IInformation Processing Systems
- locatello, 2019, Challenging common assumptions in the unsupervised learning of disentangled representations, International Conference on Machine Learning, 4114
- szlam marc’ aurelio ranzato, 2014, Video (language) modeling: a baseline for generative models of natural videos, CoRR abs/1412 6604
- lotter, 2017, Deep predictive coding networks for video prediction and unsupervised learning, International Conference on Learning Representations
Cited by
VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecasting
Tang, Y., Dong, P., Tang, Z., Chu, X., & Liang, J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00575 · Cited by 70
Metadata
- Authors
- Tan, C., Gao, Z., Wu, L., Xu, Y., Xia, J., Li, S., & Li, S. Z.
- Venue
- 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2023-06
- Pages
- 18770-18782
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4386076121
- Cited by
- 219
- References
- 32
- FWCI
- 24.90
- Type
- conference paper
- Language
- en
- Open access
- green
- Funders
- National Key R&D Program of (2022ZDOl15100)
- National Natural Science Foundation of China (U21A20427)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.