The DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index.
Video Transformers: A Survey
Open accessAbstract
Transformer models have shown great success handling long-range interactions, making them a promising tool for modeling video. However, they lack inductive biases and scale quadratically with input length. These limitations are further exacerbated when dealing with the high dimensionality introduced by the temporal dimension. While there are surveys analyzing the advances of Transformers for vision, none focus on an in-depth analysis of video-specific designs. In this survey, we analyze the main contributions and trends of works leveraging Transformers to model video. Specifically, we delve into how videos are handled at the input level first. Then, we study the architectural changes made to deal with video more efficiently, reduce redundancy, re-introduce useful inductive biases, and capture long-term temporal dynamics. In addition, we provide an overview of different training regimes and explore effective self-supervised learning strategies for video. Finally, we conduct a performance comparison on the most common benchmark for Video Transformers (i.e., action classification), finding them to outperform 3D ConvNets even with less computational complexity.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623
Research works printed but not linked: 129
- 10.1016/j.imavis.2021.104216 · DOI
- 10.1109/CVPR46437.2021.00193 · DOI · Paper page
- 10.1109/ICCV48922.2021.00808 · DOI
- zbontar, 2021, Barlow twins: Self-supervised learning via redundancy reduction, Proc 38th Int Conf Mach Learn, 12310
- 10.1109/ICCV48922.2021.00256 · DOI
- bardes, 2022, VICReg: Variance-invariance-covariance regularization for self-supervised learning, Proc Int Conf Learn Representations, 5228
- 10.1109/CVPR52688.2022.00517 · DOI · Paper page
- xie, 2017, Rethinking spatiotemporal feature learning for video understanding, Proc Eur Conf Comput Vis, 318
- 10.1609/aaai.v33i01.33018545 · DOI · Paper page
- 10.1109/ICIP.2019.8803639 · doi:10.1109/icip.2019.8803639
- 10.1007/978-3-030-58598-3_27 · DOI
- 10.1080/21681163.2020.1835550 · DOI
- 10.1609/aaai.v34i07.6718 · DOI · Paper page
- 10.1109/TASLP.2021.3065823 · DOI
- 10.1109/CVPR42600.2020.00481 · DOI · Paper page
- ridnik, 2021, ImageNet-21k pretraining for the masses, Proc Int Conf Neural Inf Process
- 10.1109/ICCVW.2019.00125 · DOI
- 10.1109/ICCV48922.2021.01378 · DOI · Paper page
- 10.1109/CVPR.2019.00033 · DOI
- plizzari, 2021, Skeleton-based action recognition via spatial and temporal transformer networks, Comput Vis Image Understanding, 208
- 10.1109/CVPR52688.2022.01940 · DOI
- yun, 2022, Time is MattEr: Temporal self-supervision for video transformers, Proc 39th Int Conf Mach Learn, 25804
- ging, 2020, COOT: Cooperative hierarchical transformer for video-text representation learning, Proc 34th Int Conf Neural Inf Process Syst
- 10.1109/CVPR46437.2021.01248 · DOI
- 10.1007/978-3-030-68238-5_48 · DOI · Paper page
- 10.1109/ICCV48922.2021.00747 · DOI
- radford, 2018, Improving language understanding by generative pre-training, OpenAI Preprint
- 10.1109/ICCV48922.2021.01327 · DOI · Paper page
- 10.1109/CVPR.2017.634 · DOI · Paper page
- 10.1016/j.aiopen.2022.01.001 · DOI
- 10.1007/s11263-021-01547-8 · DOI
- yang, 2022, Transformers meet visual learning understanding: A comprehensive review
- jain, 2019, Attention is not explanation, Proc Conf North Amer Chapter Assoc Comput Linguistics Hum Lang Technol, 3543
- bulat, 2021, Space-time mixing attention for video transformer, Proc Int Conf Neural Inf Process, 19594
- 10.1109/CVPR46437.2021.00725 · DOI · Paper page
- 10.1109/CVPR52688.2022.00290 · DOI
- 10.1109/ICCV.2019.00015 · DOI
- 10.1109/CVPR46437.2021.00585 · DOI · Paper page
- qi, 2017, PointNet: Deep hierarchical feature learning on point sets in a metric space, Proc Int Conf Neural Inf Process, 5099
- yu, 2021, Learning from inside: Self-driven siamese sampling and reasoning for video question answering, Proc Int Conf Neural Inf Process, 26462
- 10.1109/ICCV48922.2021.01606 · DOI · Paper page
- 10.1109/CVPR46437.2021.00610 · DOI
- 10.1109/CVPR42600.2020.00028 · DOI · Paper page
- zha, 2021, Shifted chunk transformer for spatio-temporal representational learning, Proc Int Conf Neural Inf Process, 11384
- yang, 2021, Associating objects with transformers for video object segmentation, Proc Int Conf Neural Inf Process, 2491
- 10.1109/ICCV.2019.00718 · DOI · Paper page
- 10.1109/WACV48630.2021.00058 · DOI
- 10.18653/v1/D16-1244 · DOI · Paper page
- 10.1109/CVPR52688.2022.01170 · DOI · Paper page
- 10.1109/ICCV48922.2021.00988 · DOI · Paper page
- 10.1109/CVPR52688.2022.01179 · DOI · Paper page
- ye, 2022, Referring segmentation in images and videos with cross-modal self-attention network, IEEE Trans Pattern Anal Mach Intell, 44, 3719
- 10.1109/TCSVT.2021.3082763 · DOI · Paper page
- 10.1109/ICCV48922.2021.01325 · DOI · Paper page
- 10.1016/j.neucom.2018.03.080 · DOI · Paper page
- 10.1109/ICCV48922.2021.00871 · DOI
- 10.1109/ICCV.2019.00756 · DOI · Paper page
- iashin, 2020, Multi-modal dense video captioning, Proc IEEE Conf Comput Vis and Pattern Recog, 4117
- 10.1109/ICTAI50040.2020.00176 · DOI
- luo, 2020, Synchronous bidirectional learning for multilingual lip reading, Proc Brit Mach Vis Conf, 8122
- 10.1109/ICCV48922.2021.01039 · DOI
- fajtl, 2018, Summarizing videos with attention, Proc Asian Conf Comput Vis, 39
- 10.1109/CVPR52688.2022.00476 · DOI · Paper page
- 10.1109/CVPR42600.2020.01151 · DOI
- hwang, 2021, Video instance segmentation using inter-frame communication transformers, Proc Int Conf Neural Inf Process, 13352
- 10.1109/CVPR.2018.00911 · DOI · Paper page
- patrick, 2021, Keeping your eye on the ball: Trajectory attention in video transformers, Proc Int Conf Neural Inf Process, 12493
- patrick, 2021, Support-set bottlenecks for video-text representation learning, Proc Int Conf Learn Representations, 8116
- ryoo, 2021, TokenLearner: Adaptive space-time tokenization for videos, Proc Int Conf Neural Inf Process, 12786
- 10.1109/CVPR46437.2021.00054 · DOI
- 10.1109/CVPR52688.2022.00289 · DOI
- 10.1016/j.neucom.2021.03.026 · DOI
- 10.1109/TPAMI.2011.157 · DOI · Paper page
- 10.1109/ICCV48922.2021.01341 · DOI
- 10.1109/CVPR52688.2022.01362 · DOI
- 10.1109/TIP.2021.3113114 · DOI
- 10.1109/CVPR52688.2022.00315 · DOI
- 10.1109/CVPR52688.2022.01322 · DOI · Paper page
- 10.1109/CVPR46437.2021.00192 · DOI
- iashin, 2020, A better use of audio-visual cues: Dense video captioning with bi-modal transformer, Proc Brit Mach Vis Conf, 698
- 10.1109/CVPR42600.2020.00092 · DOI
- 10.1109/CVPR52688.2022.01827 · DOI
- 10.1007/978-3-030-01267-0_41 · DOI
- 10.1109/ICCV48922.2021.00156 · DOI
- zhou, 2021, Hopper: Multi-hop transformer for spatiotemporal reasoning, Proc Int Conf Learn Representations, 1553
- akbari, 2021, Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text, Proc Int Conf Neural Inf Process, 24206
- 10.1109/CVPR46437.2021.00863 · DOI · Paper page
- li, 2022, UniFormer: Unified transformer for efficient spatial-temporal representation learning, Proc Int Conf Learn Representations, 14733
- 10.1109/CVPR46437.2021.00162 · DOI · Paper page
- 10.1109/ICCVW.2019.00194 · DOI
- bozic, 2021, TransformerFusion: Monocular RGB scene reconstruction using transformers, Proc Int Conf Neural Inf Process, 1403
- 10.1007/978-3-030-58517-4_31 · DOI · Paper page
- 10.1109/CVPR52688.2022.01367 · DOI · Paper page
- 10.1109/ICCV48922.2021.01332 · DOI · Paper page
- xu, 2021, Long short-term transformer for online action detection, Proc Int Conf Neural Inf Process, 1086
- 10.1109/ICCV48922.2021.01170 · DOI · Paper page
- 10.1109/CVPR46437.2021.01334 · DOI
- 10.1145/3394171.3416291 · DOI
- 10.1109/CVPR52688.2022.01563 · DOI · Paper page
- 10.1109/ICCVW.2019.00186 · DOI · Paper page
- 10.1109/CVPR52688.2022.00950 · DOI
- 10.1109/CVPR.2019.01232 · DOI · Paper page
- 10.1109/ICCV.2019.00565 · DOI · Paper page
- 10.1109/CVPR52688.2022.00293 · DOI
- 10.1109/CVPR52688.2022.01359 · DOI
- 10.1109/CVPR.2018.00769 · DOI
- 10.1109/LSP.2019.2923918 · DOI
- 10.1007/978-3-030-66823-5_18 · DOI
- 10.1109/CVPR42600.2020.01004 · DOI
- 10.1145/3343031.3356051 · DOI
- 10.1109/ICCV48922.2021.00948 · DOI · Paper page
- 10.1109/ICCV48922.2021.01471 · DOI
- hendrycks, 2019, Using self-supervised learning can improve model robustness and uncertainty, Proc 33rd Int Conf Neural Inf Process Syst
- 10.1609/aaai.v32i1.11506 · DOI · Paper page
- 10.1109/CVPR52688.2022.01867 · DOI
- chen, 2020, A simple framework for contrastive learning of visual representations, Proc 37th Int Conf Mach Learn, 1597
- 10.1109/WACV48630.2021.00331 · DOI
- tian, 2021, Understanding self-supervised learning dynamics without contrastive pairs, Proc 38th Int Conf Mach Learn, 10268
- purushwalkam, 2020, Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases, Proc 34th Int Conf Neural Inf Process Syst
- 10.1109/CVPR46437.2021.00689 · DOI · Paper page
- 10.1007/s41095-021-0247-3 · DOI
- 10.1109/CVPR42600.2020.00877 · DOI · Paper page
- tay, 2020, Efficient transformers: A survey, ACM Comput Surv, 55
- vaswani, 2017, Attention is all you need, Proc 31st Int Conf Neural Inf Process Syst, 6000
- 10.1109/CVPR52688.2022.01171 · DOI
- dong, 2021, Attention is not all you need: Pure attention loses rank doubly exponentially with depth, Proc 38th Int Conf Mach Learn, 2793
- 10.18653/v1/2020.acl-main.244 · DOI · Paper page
- 10.1109/ICCV48922.2021.00774 · DOI · Paper page
- 10.1109/CVPR52688.2022.00713 · DOI · Paper page
Cited by
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072
A comprehensive survey on applications of transformers for deep learning tasks
Islam, S., Elmekki, H., Elsebai, A., Bentahar, J., Drawel, N., Rjoub, G., & Pedrycz, W. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122666 · Cited by 487
Transformers in Remote Sensing: A Survey
Aleissaee, A. A., Kumar, A., Anwer, R. M., Khan, S., Cholakkal, H., Xia, G.-S., & Khan, F. S. (2023) · Remote Sensing · doi:10.3390/rs15071860 · Cited by 417
ChatGPT in healthcare: A taxonomy and systematic review
Li, J., Dada, A., Puladi, B., Kleesiek, J., & Egger, J. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108013 · Cited by 452
Expanding Language-Image Pretrained Models for General Video Recognition
Ni, B., Peng, H., Chen, M., Zhang, S., Meng, G., Fu, J., Xiang, S., & Ling, H. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19772-7_1 · Cited by 326
Neural Architecture Search for Transformers: A Survey
Chitty-Venkata, K. T., Emani, M., Vishwanath, V., & Somani, A. K. (2022) · IEEE Access · doi:10.1109/access.2022.3212767 · Cited by 118
Lai, D. K.-H., Cheng, E. S.-W., So, B. P.-H., Mao, Y.-J., Cheung, S. M.-Y., Cheung, D. S. K., Wong, D. W.-C., & Cheung, J. C.-W. (2023) · Mathematics · doi:10.3390/math11143081 · Cited by 18
Sparks of Large Audio Models: A Survey and Outlook
Latif, S., Shoukat, M., Shamshad, F., Usama, M., Ren, Y., Cuayáhuitl, H., Wang, W., Zhang, X., Togneri, R., Cambria, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12792 · Cited by 17
Chen, N., Xu, T., Sun, M., Yao, C., & Yang, D. (2025) · Intelligent Computing · doi:10.34133/icomputing.0143 · Cited by 5
Metadata
- Authors
- Selva, J., Johansen, A. S., Escalera, S., Nasrollahi, K., Moeslund, T. B., & Clapés, A.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2023-11-01
- Volume
- 45
- Issue
- 11
- Pages
- 12922-12943
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W4319777935
- Cited by
- 169
- References
- 74
- FWCI
- 15.09
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- Funders
- Pioneer Centre for AI
- Danmarks Grundforskningsfond
- Spanish project (PID2019-105093GB-I00)
- Institució Catalana de Recerca i Estudis Avançats
- ICREA Academia program
- Aalborg Universitet
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.