CoChat

Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learning

Open access
Tan, Cheng, Gao, Zhangyang, Wu, Lirong, Xu, Yongjie, Xia, Jun, Li, Siyuan, Li, Stan Z. · 2023 · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Cited by 219

The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.

Abstract

Spatiotemporal predictive learning aims to generate future frames by learning from historical frames. In this paper, we investigate existing methods and present a general framework of spatiotemporal predictive learning, in which the spatial encoder and decoder capture intra-frame features and the middle temporal module catches inter-frame correlations. While the mainstream methods employ recurrent units to capture long-term temporal dependencies, they suffer from low computational efficiency due to their unparallelizable architectures. To parallelize the temporal module, we propose the Temporal Attention Unit (TAU), which decomposes temporal attention into intra-frame statical attention and inter-frame dynamical attention. Moreover, while the mean squared error loss focuses on intra-frame errors, we introduce a novel differential divergence regularization to take inter-frame variations into account. Extensive experiments demonstrate that the proposed method enables the derived model to achieve competitive performance on various spatiotemporal prediction benchmarks.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr52729.2023.01800

Cites

32 of 32 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 32 references

Research works printed but not linked: 81

  1. liu, 2022, Automix: Unveiling the power of mixup for stronger classifiers, Computer Vision- ECCV 2022 17th European Conference Tel Aviv Israel October 23–27 2022 Proceedings Part XXIV, 441
  2. 10.1109/ICCV.2017.478 · DOI · Paper page
  3. 10.1109/CVPR52688.2022.01167 · DOI · Paper page
  4. 10.1109/ICCV.2017.194 · DOI · Paper page
  5. li, 2022, Efficient multi -order gated aggregation network, ar Xiv preprint
  6. liu, 2022, Decoupled mixup for data-efficient learning, ArXiv Preprint
  7. 10.1109/TCSVT.2020.3021409 · DOI
  8. khemakhem, 2020, Variational autoencoders and nonlinear ica: A unifying framework, International Conference on Artificial Intelligence and Statistics, 2207
  9. lee, 2018, Stochastic adversarial video prediction, ArXiv Preprint
  10. 10.1109/CVPR.2019.00191 · DOI · Paper page
  11. 10.1109/IROS.2018.8594264 · DOI
  12. 10.1109/CVPR42600.2020.00461 · DOI · Paper page
  13. kalchbrenner, 0, Video pixel networks, International Conference on Machine Learning, 1771
  14. lee, 2018, Mutual suppression network for video prediction using disentangled features, ArXiv Preprint
  15. chen, 2020, Big self-supervised mod-els are strong semi-supervised learners, Advances in neural information processing systems, 33, 22243
  16. chen, 2020, A simple framework for contrastive learning of visual representations, International Conference on Machine Learning, 1597
  17. 10.1609/aaai.v35i2.16202 · DOI
  18. byeon, 2018, Contextvp: Fully context-aware video prediction, Proceedings of the European Conference on Computer Vision (ECCV), 753
  19. 10.1007/978-3-030-58592-1_7 · DOI
  20. castrejon, 2019, Im-proved conditional vrnns for video prediction, Proceedings of the IEEEICVF International Conference on Computer Vision, 7608
  21. cao, 2022, A survey on generative diffusion model, ArXiv Preprint
  22. 10.1109/CVPR46437.2021.01518 · DOI · Paper page
  23. 10.24963/ijcai.2018/408 · DOI
  24. jia, 2016, Dynamic filter networks, Advances in neural information processing systems, 29
  25. hinton, 2021, How to represent part-whole hierarchies in a neural network, ArXiv Preprint
  26. hsieh, 2018, Learning to decompose and dis-entangle representations for video prediction, Advances in neural information processing systems, 31
  27. 10.1109/CVPR.2018.00819 · DOI
  28. guo, 2022, Visual attention network, ArXiv Preprint
  29. jenni, 2020, Video representation learning by recognizing temporal transformations, European Conference on Computer Vision, 425
  30. 10.1109/CVPR52688.2022.00317 · DOI · Paper page
  31. 10.1109/ICCV.2019.00910 · DOI · Paper page
  32. gao, 2022, Pifold: To-ward effective and efficient protein inverse folding, ar Xiv preprint
  33. gao, 2022, Alphadesign: A graph protein design method and benchmark on alphafolddb, ArXiv Preprint
  34. greff, 2019, Multi-object representation learning with iterative variational inference, International Conference on Machine Learning, 2424
  35. 10.1109/CVPR42600.2020.01149 · DOI · Paper page
  36. ding, 2022, Scaling up your kernels to 31x31: Revisiting large kernel design in cnns, ArXiv Preprint
  37. denton, 2017, Unsupervised learning of disentangled representations from video, Advances in neural information processing systems, 30
  38. dinh, 0, Density estimation using real nvp, International Conference on Learning Representations
  39. dinh, 2015, Nice: Nonlinear independent components estimation, In International Conference on Learning Representations Workshop
  40. 10.1109/CVPR.2019.00937 · DOI · Paper page
  41. denton, 2018, Stochastic video generation with a learned prior, International Conference on Machine Learning, 1174
  42. wu, 2022, Flowformer: Linearizing transformers with conservation flows, Proceedings of the 39th International Conference on Machine Learning volume 162 of Proceedings of Machine Learning Research, 24226
  43. 10.1109/CVPR.2019.00937 · DOI · Paper page
  44. 10.1109/CVPR.2009.5206631 · DOI · Paper page
  45. 10.24963/ijcai.2019/317 · DOI · Paper page
  46. wang, 2018, Predrnn++: Towards a resolution of the deep-in-time dilemma in spatiotemporal predictive learning, International Conference on Machine Learning, 5123
  47. wang, 2017, Predrnn: Recurrent neural networks for predictive learning using spatiotemporal lstms, Advances in Neural IInformation Processing Systems
  48. wang, 2018, Eidetic 3d lstm: A model for video prediction and beyond, International Conference on Learning Representations
  49. 10.1016/j.cviu.2018.04.007 · DOI · Paper page
  50. villegas, 2019, High fidelity video prediction with large stochastic recurrent neural networks, Advances in Neural IInformation Processing Systems
  51. 10.1145/3474085.3475622 · DOI
  52. 10.1109/CVPR52688.2022.00710 · DOI
  53. villegas, 2018, Hi-erarchicallong-term video prediction without supervision, International Conference on Machine Learning, 6038
  54. su, 2020, Convolutional tensor-train lstm for spatiotemporallearning, Advances in neural information processing systems, 33, 13714
  55. tan, 2022, Target-aware molecular graph generation, ar Xiv preprint
  56. tan, 2022, Rfold: Towards simple yet effective rna secondary structure prediction, ArXiv Preprint
  57. shouno, 2020, Photo-realistic video prediction on natural videos of largely changing frames, ArXiv Preprint
  58. shi, 2017, Deep learning for precipitation nowcasting: A benchmark and a new model, Advances in neural information processing systems
  59. zhang, 2016, Colorful image colorization, European Conference on Computer Vision, 649
  60. 10.1016/j.neucom.2021.08.079 · DOI
  61. zhang, 2017, Deep spatiotemporal residual networks for citywide crowd flows prediction, Thirty-First AAAI Conference on Artificial Intelligence, 10.1609/aaai.v31i1.10735 · DOI · Paper page
  62. zhang, 2017, Learning spatiotemporal features using 3dcnn and convolutionallstm for gesture recognition, Proceedings of the IEEE International Conference on Computer Vision Workshops, 3120
  63. ren, 2015, Faster r-cnn: Towards real-time object detection with re-gion proposal networks, Advances in neural information processing systems, 28, 91
  64. zbontar, 2021, Barlow twins: Self-supervised learning via redundancy reduction, ArXiv Preprint
  65. 10.1109/ICME.2019.00048 · DOI
  66. shi, 2015, Convolutionallstm network: A machine learning approach for precipitation nowcasting, Advances in neural information processing systems
  67. 10.1109/ICPR.2004.1334462 · DOI · Paper page
  68. yu, 2019, Efficient and information-preserving future frame prediction and beyond, International Conference on Learning Representations
  69. babaeizadeh, 2021, Fitvid: Overfitting in pixel-level video prediction, ArXiv Preprint
  70. babaeizadeh, 2018, Stochastic variational video prediction, International Conference on Learning Representations
  71. 10.1109/CVPR46437.2021.00689 · DOI · Paper page
  72. peng, 2020, Random feature attention, International Conference on Learning Representations
  73. 10.1007/978-3-030-01234-2_44 · DOI · Paper page
  74. 10.18653/v1/2022.acl-long.561 · DOI
  75. 10.1007/978-3-030-01264-9_44 · DOI
  76. mathieu, 2015, Deep multi-scale video prediction beyond mean square error, ArXiv Preprint
  77. mathieu, 2019, Disentangling disentanglement in variational autoencoders, International Conference on Machine Learning, 4402
  78. minderer, 2019, Unsupervised learning of object structure and dynamics from videos, Advances in Neural IInformation Processing Systems
  79. locatello, 2019, Challenging common assumptions in the unsupervised learning of disentangled representations, International Conference on Machine Learning, 4114
  80. szlam marc’ aurelio ranzato, 2014, Video (language) modeling: a baseline for generative models of natural videos, CoRR abs/1412 6604
  81. lotter, 2017, Deep predictive coding networks for video prediction and unsupervised learning, International Conference on Learning Representations

Cited by

See all 219

Metadata

Authors
Tan, C., Gao, Z., Wu, L., Xu, Y., Xia, J., Li, S., & Li, S. Z.
Venue
2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2023-06
Pages
18770-18782
ISSN
1063-6919 (ISSN-L)
OpenAlex
W4386076121
Cited by
219
References
32
FWCI
24.90
Type
conference paper
Language
en
Open access
green
Funders
  • National Key R&D Program of (2022ZDOl15100)
  • National Natural Science Foundation of China (U21A20427)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learning. https://app.cochat.ai/papers/10.1109/cvpr52729.2023.01800
The paper
Tan, C., Gao, Z., Wu, L., Xu, Y., Xia, J., Li, S., & Li, S. Z. (2023). Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learning. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/cvpr52729.2023.01800

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.