Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Cited by 169

The DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index.

Video Transformers: A Survey

Open access
Selva, Javier, Johansen, Anders S., Escalera, Sergio, Nasrollahi, Kamal, Moeslund, Thomas B., Clapés, Albert · 2023 · IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract

Transformer models have shown great success handling long-range interactions, making them a promising tool for modeling video. However, they lack inductive biases and scale quadratically with input length. These limitations are further exacerbated when dealing with the high dimensionality introduced by the temporal dimension. While there are surveys analyzing the advances of Transformers for vision, none focus on an in-depth analysis of video-specific designs. In this survey, we analyze the main contributions and trends of works leveraging Transformers to model video. Specifically, we delve into how videos are handled at the input level first. Then, we study the architectural changes made to deal with video more efficiently, reduce redundancy, re-introduce useful inductive biases, and capture long-term temporal dynamics. In addition, we provide an overview of different training regimes and explore effective self-supervised learning strategies for video. Finally, we conduct a performance comparison on the most common benchmark for Video Transformers (i.e., action classification), finding them to outperform 3D ConvNets even with less computational complexity.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tpami.2023.3243465

Cites

74 of 74 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 74 references

Research works printed but not linked: 129

  1. 10.1016/j.imavis.2021.104216 · DOI
  2. 10.1109/CVPR46437.2021.00193 · DOI · Paper page
  3. 10.1109/ICCV48922.2021.00808 · DOI
  4. zbontar, 2021, Barlow twins: Self-supervised learning via redundancy reduction, Proc 38th Int Conf Mach Learn, 12310
  5. 10.1109/ICCV48922.2021.00256 · DOI
  6. bardes, 2022, VICReg: Variance-invariance-covariance regularization for self-supervised learning, Proc Int Conf Learn Representations, 5228
  7. 10.1109/CVPR52688.2022.00517 · DOI · Paper page
  8. xie, 2017, Rethinking spatiotemporal feature learning for video understanding, Proc Eur Conf Comput Vis, 318
  9. 10.1609/aaai.v33i01.33018545 · DOI · Paper page
  10. 10.1109/ICIP.2019.8803639 · doi:10.1109/icip.2019.8803639
  11. 10.1007/978-3-030-58598-3_27 · DOI
  12. 10.1080/21681163.2020.1835550 · DOI
  13. 10.1609/aaai.v34i07.6718 · DOI · Paper page
  14. 10.1109/TASLP.2021.3065823 · DOI
  15. 10.1109/CVPR42600.2020.00481 · DOI · Paper page
  16. ridnik, 2021, ImageNet-21k pretraining for the masses, Proc Int Conf Neural Inf Process
  17. 10.1109/ICCVW.2019.00125 · DOI
  18. 10.1109/ICCV48922.2021.01378 · DOI · Paper page
  19. 10.1109/CVPR.2019.00033 · DOI
  20. plizzari, 2021, Skeleton-based action recognition via spatial and temporal transformer networks, Comput Vis Image Understanding, 208
  21. 10.1109/CVPR52688.2022.01940 · DOI
  22. yun, 2022, Time is MattEr: Temporal self-supervision for video transformers, Proc 39th Int Conf Mach Learn, 25804
  23. ging, 2020, COOT: Cooperative hierarchical transformer for video-text representation learning, Proc 34th Int Conf Neural Inf Process Syst
  24. 10.1109/CVPR46437.2021.01248 · DOI
  25. 10.1007/978-3-030-68238-5_48 · DOI · Paper page
  26. 10.1109/ICCV48922.2021.00747 · DOI
  27. radford, 2018, Improving language understanding by generative pre-training, OpenAI Preprint
  28. 10.1109/ICCV48922.2021.01327 · DOI · Paper page
  29. 10.1109/CVPR.2017.634 · DOI · Paper page
  30. 10.1016/j.aiopen.2022.01.001 · DOI
  31. 10.1007/s11263-021-01547-8 · DOI
  32. yang, 2022, Transformers meet visual learning understanding: A comprehensive review
  33. jain, 2019, Attention is not explanation, Proc Conf North Amer Chapter Assoc Comput Linguistics Hum Lang Technol, 3543
  34. bulat, 2021, Space-time mixing attention for video transformer, Proc Int Conf Neural Inf Process, 19594
  35. 10.1109/CVPR46437.2021.00725 · DOI · Paper page
  36. 10.1109/CVPR52688.2022.00290 · DOI
  37. 10.1109/ICCV.2019.00015 · DOI
  38. 10.1109/CVPR46437.2021.00585 · DOI · Paper page
  39. qi, 2017, PointNet: Deep hierarchical feature learning on point sets in a metric space, Proc Int Conf Neural Inf Process, 5099
  40. yu, 2021, Learning from inside: Self-driven siamese sampling and reasoning for video question answering, Proc Int Conf Neural Inf Process, 26462
  41. 10.1109/ICCV48922.2021.01606 · DOI · Paper page
  42. 10.1109/CVPR46437.2021.00610 · DOI
  43. 10.1109/CVPR42600.2020.00028 · DOI · Paper page
  44. zha, 2021, Shifted chunk transformer for spatio-temporal representational learning, Proc Int Conf Neural Inf Process, 11384
  45. yang, 2021, Associating objects with transformers for video object segmentation, Proc Int Conf Neural Inf Process, 2491
  46. 10.1109/ICCV.2019.00718 · DOI · Paper page
  47. 10.1109/WACV48630.2021.00058 · DOI
  48. 10.18653/v1/D16-1244 · DOI · Paper page
  49. 10.1109/CVPR52688.2022.01170 · DOI · Paper page
  50. 10.1109/ICCV48922.2021.00988 · DOI · Paper page
  51. 10.1109/CVPR52688.2022.01179 · DOI · Paper page
  52. ye, 2022, Referring segmentation in images and videos with cross-modal self-attention network, IEEE Trans Pattern Anal Mach Intell, 44, 3719
  53. 10.1109/TCSVT.2021.3082763 · DOI · Paper page
  54. 10.1109/ICCV48922.2021.01325 · DOI · Paper page
  55. 10.1016/j.neucom.2018.03.080 · DOI · Paper page
  56. 10.1109/ICCV48922.2021.00871 · DOI
  57. 10.1109/ICCV.2019.00756 · DOI · Paper page
  58. iashin, 2020, Multi-modal dense video captioning, Proc IEEE Conf Comput Vis and Pattern Recog, 4117
  59. 10.1109/ICTAI50040.2020.00176 · DOI
  60. luo, 2020, Synchronous bidirectional learning for multilingual lip reading, Proc Brit Mach Vis Conf, 8122
  61. 10.1109/ICCV48922.2021.01039 · DOI
  62. fajtl, 2018, Summarizing videos with attention, Proc Asian Conf Comput Vis, 39
  63. 10.1109/CVPR52688.2022.00476 · DOI · Paper page
  64. 10.1109/CVPR42600.2020.01151 · DOI
  65. hwang, 2021, Video instance segmentation using inter-frame communication transformers, Proc Int Conf Neural Inf Process, 13352
  66. 10.1109/CVPR.2018.00911 · DOI · Paper page
  67. patrick, 2021, Keeping your eye on the ball: Trajectory attention in video transformers, Proc Int Conf Neural Inf Process, 12493
  68. patrick, 2021, Support-set bottlenecks for video-text representation learning, Proc Int Conf Learn Representations, 8116
  69. ryoo, 2021, TokenLearner: Adaptive space-time tokenization for videos, Proc Int Conf Neural Inf Process, 12786
  70. 10.1109/CVPR46437.2021.00054 · DOI
  71. 10.1109/CVPR52688.2022.00289 · DOI
  72. 10.1016/j.neucom.2021.03.026 · DOI
  73. 10.1109/TPAMI.2011.157 · DOI · Paper page
  74. 10.1109/ICCV48922.2021.01341 · DOI
  75. 10.1109/CVPR52688.2022.01362 · DOI
  76. 10.1109/TIP.2021.3113114 · DOI
  77. 10.1109/CVPR52688.2022.00315 · DOI
  78. 10.1109/CVPR52688.2022.01322 · DOI · Paper page
  79. 10.1109/CVPR46437.2021.00192 · DOI
  80. iashin, 2020, A better use of audio-visual cues: Dense video captioning with bi-modal transformer, Proc Brit Mach Vis Conf, 698
  81. 10.1109/CVPR42600.2020.00092 · DOI
  82. 10.1109/CVPR52688.2022.01827 · DOI
  83. 10.1007/978-3-030-01267-0_41 · DOI
  84. 10.1109/ICCV48922.2021.00156 · DOI
  85. zhou, 2021, Hopper: Multi-hop transformer for spatiotemporal reasoning, Proc Int Conf Learn Representations, 1553
  86. akbari, 2021, Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text, Proc Int Conf Neural Inf Process, 24206
  87. 10.1109/CVPR46437.2021.00863 · DOI · Paper page
  88. li, 2022, UniFormer: Unified transformer for efficient spatial-temporal representation learning, Proc Int Conf Learn Representations, 14733
  89. 10.1109/CVPR46437.2021.00162 · DOI · Paper page
  90. 10.1109/ICCVW.2019.00194 · DOI
  91. bozic, 2021, TransformerFusion: Monocular RGB scene reconstruction using transformers, Proc Int Conf Neural Inf Process, 1403
  92. 10.1007/978-3-030-58517-4_31 · DOI · Paper page
  93. 10.1109/CVPR52688.2022.01367 · DOI · Paper page
  94. 10.1109/ICCV48922.2021.01332 · DOI · Paper page
  95. xu, 2021, Long short-term transformer for online action detection, Proc Int Conf Neural Inf Process, 1086
  96. 10.1109/ICCV48922.2021.01170 · DOI · Paper page
  97. 10.1109/CVPR46437.2021.01334 · DOI
  98. 10.1145/3394171.3416291 · DOI
  99. 10.1109/CVPR52688.2022.01563 · DOI · Paper page
  100. 10.1109/ICCVW.2019.00186 · DOI · Paper page
  101. 10.1109/CVPR52688.2022.00950 · DOI
  102. 10.1109/CVPR.2019.01232 · DOI · Paper page
  103. 10.1109/ICCV.2019.00565 · DOI · Paper page
  104. 10.1109/CVPR52688.2022.00293 · DOI
  105. 10.1109/CVPR52688.2022.01359 · DOI
  106. 10.1109/CVPR.2018.00769 · DOI
  107. 10.1109/LSP.2019.2923918 · DOI
  108. 10.1007/978-3-030-66823-5_18 · DOI
  109. 10.1109/CVPR42600.2020.01004 · DOI
  110. 10.1145/3343031.3356051 · DOI
  111. 10.1109/ICCV48922.2021.00948 · DOI · Paper page
  112. 10.1109/ICCV48922.2021.01471 · DOI
  113. hendrycks, 2019, Using self-supervised learning can improve model robustness and uncertainty, Proc 33rd Int Conf Neural Inf Process Syst
  114. 10.1609/aaai.v32i1.11506 · DOI · Paper page
  115. 10.1109/CVPR52688.2022.01867 · DOI
  116. chen, 2020, A simple framework for contrastive learning of visual representations, Proc 37th Int Conf Mach Learn, 1597
  117. 10.1109/WACV48630.2021.00331 · DOI
  118. tian, 2021, Understanding self-supervised learning dynamics without contrastive pairs, Proc 38th Int Conf Mach Learn, 10268
  119. purushwalkam, 2020, Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases, Proc 34th Int Conf Neural Inf Process Syst
  120. 10.1109/CVPR46437.2021.00689 · DOI · Paper page
  121. 10.1007/s41095-021-0247-3 · DOI
  122. 10.1109/CVPR42600.2020.00877 · DOI · Paper page
  123. tay, 2020, Efficient transformers: A survey, ACM Comput Surv, 55
  124. vaswani, 2017, Attention is all you need, Proc 31st Int Conf Neural Inf Process Syst, 6000
  125. 10.1109/CVPR52688.2022.01171 · DOI
  126. dong, 2021, Attention is not all you need: Pure attention loses rank doubly exponentially with depth, Proc 38th Int Conf Mach Learn, 2793
  127. 10.18653/v1/2020.acl-main.244 · DOI · Paper page
  128. 10.1109/ICCV48922.2021.00774 · DOI · Paper page
  129. 10.1109/CVPR52688.2022.00713 · DOI · Paper page

Cited by

Metadata

Authors
Selva, J., Johansen, A. S., Escalera, S., Nasrollahi, K., Moeslund, T. B., & Clapés, A.
Venue
IEEE Transactions on Pattern Analysis and Machine Intelligence
Published
2023-11-01
Volume
45
Issue
11
Pages
12922-12943
ISSN
  • 0162-8828 (ISSN-L)
  • 2160-9292
  • 1939-3539
OpenAlex
W4319777935
Cited by
169
References
74
FWCI
15.09
Type
journal article
Language
en
Open access
green
Funders
  • Pioneer Centre for AI
  • Danmarks Grundforskningsfond
  • Spanish project (PID2019-105093GB-I00)
  • Institució Catalana de Recerca i Estudis Avançats
  • ICREA Academia program
  • Aalborg Universitet

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.