The DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index.
Video Transformers: A Survey
Open accessAbstract
Transformer models have shown great success handling long-range interactions, making them a promising tool for modeling video. However, they lack inductive biases and scale quadratically with input length. These limitations are further exacerbated when dealing with the high dimensionality introduced by the temporal dimension. While there are surveys analyzing the advances of Transformers for vision, none focus on an in-depth analysis of video-specific designs. In this survey, we analyze the main contributions and trends of works leveraging Transformers to model video. Specifically, we delve into how videos are handled at the input level first. Then, we study the architectural changes made to deal with video more efficiently, reduce redundancy, re-introduce useful inductive biases, and capture long-term temporal dynamics. In addition, we provide an overview of different training regimes and explore effective self-supervised learning strategies for video. Finally, we conduct a performance comparison on the most common benchmark for Video Transformers (i.e., action classification), finding them to outperform 3D ConvNets even with less computational complexity.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Masked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489
Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-K., & Woo, W.-C. (2015) · arXiv · doi:10.48550/arxiv.1506.04214 · Cited by 7,709
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242
Context Encoders: Feature Learning by Inpainting
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.278 · Cited by 5,691
Emerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638 · cited 1×
“Another example is seen in VT works introducing asymmetries in the networks computing the different views’ representations: using additional predictors [112], momentum encoders [103], [112] (originally proposed in [165], are believed to behave as network ensembles [166]) and even CNNs [160] (probably”
BackgroundRao, R. P. N., & Ballard, D. H. (1999) · Nature Neuroscience · doi:10.1038/4580 · Cited by 5,483 · cited 1×
“In this sense, all tokens are tasked with solving feature-based MTM by trying to predict the next token representation (in a predictive coding setting [159], [172]).”
BackgroundHan (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682
SlowFast Networks for Video Recognition
Feichtenhofer, C., Fan, H., Malik, J., & He, K. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00630 · Cited by 4,007
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.07733 · Cited by 3,846 · cited 1×
“Self-supervised learning (SSL) has been recently shown to alleviate data needs for an equivalent supervision-based pre-training (e.g., [28], [29]), while providing more robust [152] and general features [153], [154], [155].”
BackgroundCatastrophic Interference in Connectionist Networks: The Sequential Learning Problem
McCloskey, M., & Cohen, N. J. (1989) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60536-8 · Cited by 3,770
Exploring Simple Siamese Representation Learning
Chen, X., & He, K. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01549 · Cited by 3,725
A Closer Look at Spatiotemporal Convolutions for Action Recognition
Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., & Paluri, M. (2017) · arXiv · doi:10.48550/arxiv.1711.11248 · Cited by 3,664 · cited 4×
“'21 Post-LN ReLU 12 2 – – – – – Faster R-CNN [89], Mask R-CNN [133] FA F X – X – [134] SMT '19 Post-LN ReLU 8 2 8 1 – X – RN-18 [84] FA F X – – – [78] SLT '20 Post-LN ReLU 8 3 8 3 – – – InceptionV4 [135] FA F X – – – [136] JSLT '20 Pre-LN ReLU 1 2 1 2 – – – InceptionV4 [135] FA F X – – – [137] SBL '20 Post-LN NA 8 6 N…”
InfluentialTransformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2021) · arXiv · doi:10.48550/arxiv.2101.01169 · Cited by 3,594
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · cited 1×
“Relative PEs are generally added as an additional bias term (as in [42], [88], [89], [90]) in the dot-product betweenQ andK (modifying (1)).”
BackgroundViViT: A Video Vision Transformer
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lucic, M., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00676 · Cited by 2,767
Swin Transformer V2: Scaling Up Capacity and Resolution
Liu, Z., Hu, H., Lin, Y., Yao, Z., Xie, Z., Wei, Y., Ning, J., Cao, Y., Zhang, Z., Dong, L., et al. (2021) · arXiv · doi:10.48550/arxiv.2111.09883 · Cited by 2,679
Self-Attention with Relative Position Representations
Shaw, P., Uszkoreit, J., & Vaswani, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2074 · Cited by 2,505
Zero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417
Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., & Hu, H. (2021) · arXiv · doi:10.48550/arxiv.2106.13230 · Cited by 2,294
Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., & Hu, H. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00320 · Cited by 2,294
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281
Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey
Jing, L., & Tian, Y. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2992393 · Cited by 2,036
Is Space-Time Attention All You Need for Video Understanding?
Bertasius, G., Wang, H., & Torresani, L. (2021) · arXiv · doi:10.48550/arxiv.2102.05095 · Cited by 1,663
Chen, X., Yan, B., Zhu, J., Wang, D., Yang, X., & Lu, H. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00803 · Cited by 1,588 · cited 1×
“Other works train Transformer and deep CNN embedding layers end-to-end either with a pre-trained embedding network [64], [118], [139], fine-tuning just the later layers [51], [114], or training end-to-end from scratch [57], [65], [140], [141].”
BackgroundThe “Something Something” Video Database for Learning and Evaluating Visual Common Sense
Goyal, R., Kahou, S. E., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., et al. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.622 · Cited by 1,471
Lin, T., Wang, Y., Liu, X., & Qiu, X. (2022) · AI Open · doi:10.1016/j.aiopen.2022.10.001 · Cited by 1,422
Multiscale vision transformers
Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., & Feichtenhofer, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00675 · Cited by 1,407 · cited 10×
“For instance, [7], [9], [30] adopted an encoder-only architecture (along with the inclusion of [CLS] ) for video classification following [5].”
BackgroundMethodInfluentialAST: Audio Spectrogram Transformer
Gong, Y., Chung, Y.-A., & Glass, J. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-698 · Cited by 1,159
TrackFormer: Multi-Object Tracking with Transformers
Meinhardt, T., Kirillov, A., Leal-Taixe, L., & Feichtenhofer, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00864 · Cited by 1,081 · cited 6×
“However, these approaches may excel on applications that only require low-level reasoning of the current observation, enhanced with the forwarded high-level past context (such as for tracking [116], segmentation [94] or dense video captioning [42]).”
BackgroundMethodInfluentialUnsupervised Learning of Visual Representations Using Videos
Wang, X., & Gupta, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.320 · Cited by 1,005
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
Miech, A., Zhukov, D., Alayrac, J.-B., Tapaswi, M., Laptev, I., & Sivic, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00272 · Cited by 986 · cited 1×
“These are generally instantiated as 3D CNNs (such as I3D [25] and S3D [52]), commonly pre-trained on large video datasets such as Kinetics [25], [53] or HowTo100M [54] to produce features involving temporal relationships.”
BackgroundStand-Alone Self-Attention in Vision Models
Ramachandran, P., Parmar, N., Vaswani, A., Bello, I., Levskaya, A., & Shlens, J. (2019) · arXiv · doi:10.48550/arxiv.1906.05909 · Cited by 977 · cited 1×
“Empirical studies on both image [142], [143] and video [113] Transformers have consistently found improvementswhentrainingTransformersandCNNembeddinglayersend-to-end.”
BackgroundAttention is not not Explanation
Wiegreffe, S., & Pinter, Y. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1002 · Cited by 966 · cited 1×
“Restrictions apply. explanations on what the model deems relevant [199], [200], [201].”
BackgroundCoAtNet: Marrying Convolution and Attention for All Data Sizes
Dai, Z., Liu, H., Le, Q. V., & Tan, M. (2021) · arXiv · doi:10.48550/arxiv.2106.04803 · Cited by 925 · cited 1×
“Empirical studies on both image [142], [143] and video [113] Transformers have consistently found improvementswhentrainingTransformersandCNNembeddinglayersend-to-end.”
BackgroundVideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
Tong, Z., Song, Y., Wang, J., & Wang, L. (2022) · arXiv · doi:10.48550/arxiv.2203.12602 · Cited by 909 · cited 10×
“For instance, [7], [9], [30] adopted an encoder-only architecture (along with the inclusion of [CLS] ) for video classification following [5].”
BackgroundMethodInfluentialSerrano, S., & Smith, N. A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1282 · Cited by 690
Multi-modal Transformer for Video Retrieval
Gabeur, V., Sun, C., Alahari, K., & Schmid, C. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58548-8_13 · Cited by 615
Masked Feature Prediction for Self-Supervised Visual Pre-Training
Wei, C., Fan, H., Xie, S., Wu, C.-Y., Yuille, A., & Feichtenhofer, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01426 · Cited by 587
Neimark, D., Bar, O., Zohar, M., & Asselmann, D. (2021) · 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw54120.2021.00355 · Cited by 551
Understanding Robustness of Transformers for Image Classification
Bhojanapalli, S., Chakrabarti, A., Glasner, D., Li, D., Unterthiner, T., & Veit, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01007 · Cited by 440 · cited 1×
“It has been shown that vision Transformers are robust to various perturbations [192], [193], suggesting they may be better able to form abstract semantic representations [194], probably due to their ability to leverage non-local contexts [195].”
BackgroundIntriguing Properties of Vision Transformers
Naseer, M., Ranasinghe, K., Khan, S., Hayat, M., Khan, F. S., & Yang, M.-H. (2021) · arXiv · doi:10.48550/arxiv.2105.10497 · Cited by 431 · cited 1×
“Possible future venues could analyze the learned patterns of attention preferred by different heads (as in [202]), which may clue on relevant design choices that favor such patterns.”
BackgroundPerceiver: General Perception with Iterative Attention
Jaegle, A., Gimeno, F., Brock, A., Zisserman, A., Vinyals, O., & Carreira, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03206 · Cited by 414
HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
Li, L., Chen, Y.-C., Cheng, Y., Gan, Z., Yu, L., & Liu, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.161 · Cited by 408
A Short Note on the Kinetics-700-2020 Human Action Dataset
Smaira, L., Carreira, J., Noland, E., Clancy, E., Wu, A., & Zisserman, A. (2020) · arXiv · doi:10.48550/arxiv.2010.10864 · Cited by 334
VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks
Sung, Y.-L., Cho, J., & Bansal, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00516 · Cited by 329
Multiview Transformers for Video Recognition
Yan, S., Xiong, X., Arnab, A., Lu, Z., Zhang, M., Sun, C., & Schmid, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00333 · Cited by 312 · cited 6×
“This can be addressed by allowing to form spatiotemporal interactions before aggregation, either locally (by explicitly sharing information between neighborhoods [95], [96], [97], [107] as well as by using 3D patches [9], [128]) or globally [105].”
BackgroundMethodInfluentialA Review on Deep Learning Techniques for Video Prediction
Oprea, S., Martinez-Gonzalez, P., Garcia-Garcia, A., Castro-Vargas, J. A., Orts-Escolano, S., Garcia-Rodriguez, J., & Argyros, A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3045007 · Cited by 301
Vision Transformers Are Robust Learners
Paul, S., & Chen, P.-Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i2.20103 · Cited by 244 · cited 1×
“better able to form abstract semantic representations [197], probably due to their ability to leverage non-local contexts [198].”
BackgroundBEVT: BERT Pretraining of Video Transformers
Wang, R., Chen, D., Wu, Z., Chen, Y., Dai, X., Liu, M., Jiang, Y.-G., Zhou, L., & Yuan, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01432 · Cited by 234 · cited 7×
“To solve this issue in the context of VTs, we roughly find three families of approaches, categorized by the type of target: (1) working at feature level either through regression [12], [28], [169], [170] or contrasting [12], [51], [80], [147], as well as (2) quantization of visual tokens [81], [112].”
BackgroundMethodInfluentialA large-scale study on unsupervised spatiotemporal representation learning
Feichtenhofer, C., Fan, H., Xiong, B., Girshick, R., & He, K. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00331 · Cited by 222
MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning
Lei, J., Wang, L., Shen, Y., Yu, D., Berg, T., & Bansal, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.233 · Cited by 179
Scene Memory Transformer for Embodied Agents in Long-Horizon Tasks
Fang, K., Toshev, A., Fei-Fei, L., & Savarese, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00063 · Cited by 178 · cited 6×
“Memories are mostly accessed via either cross-attention [94], [121], [122] or self-attention [110], [123].”
BackgroundMethodInfluentialThinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers
Miech, A., Alayrac, J.-B., Laptev, I., Sivic, J., & Zisserman, A. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00970 · Cited by 149
A Practical Survey on Faster and Lighter Transformers
Fournier, Q., Caron, G. M., & Aloise, D. (2023) · ACM Computing Surveys · doi:10.1145/3586074 · Cited by 145 · cited 1×
“Existing surveys focus on design choices for Transformers in general [13], NLP [14], images [15], [16], or efficient designs [17], [18].”
BackgroundEpisodic Transformer for Vision-and-Language Navigation
Pashevich, A., Schmid, C., & Sun, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01564 · Cited by 135
When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations
Chen, X., Hsieh, C.-J., & Gong, B. (2021) · arXiv · doi:10.48550/arxiv.2106.01548 · Cited by 128 · cited 1×
“On the one hand, large-scalepre-trainingaidsTransformerstoovercometheirlack ofinductivebiases[2],[5],[135],butrecentstudiessuggestthat self-supervisedpre-training(seeSectionV-B)alleviatestheneed forlargesuperviseddatasets[28],[29].”
Frozen Pretrained Transformers as Universal Computation Engines
Lu, K., Grover, A., Abbeel, P., & Mordatch, I. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i7.20729 · Cited by 113 · cited 1×
“It has been found that high-level semantic features learned by language-based Transformers generalize to other modalities [197], [198].”
BackgroundScaling Autoregressive Video Models
Weissenborn, D., Täckström, O., & Uszkoreit, J. (2019) · arXiv · doi:10.48550/arxiv.1906.02634 · Cited by 93
Parameter Efficient Multimodal Transformers for Video Representation Learning
Lee, S., Yu, Y., Kim, G., Breuel, T., Kautz, J., & Song, Y. (2020) · arXiv · doi:10.48550/arxiv.2012.04124 · Cited by 49 · cited 8×
“Modeling temporal interactions requires special considerations not present when only modeling appearance (i.e., with image Transformers).”
BackgroundMethodInfluential
Research works printed but not linked: 129
- 10.1016/j.imavis.2021.104216 · DOI
- 10.1109/CVPR46437.2021.00193 · DOI · Paper page
- 10.1109/ICCV48922.2021.00808 · DOI
- zbontar, 2021, Barlow twins: Self-supervised learning via redundancy reduction, Proc 38th Int Conf Mach Learn, 12310
- 10.1109/ICCV48922.2021.00256 · DOI
- bardes, 2022, VICReg: Variance-invariance-covariance regularization for self-supervised learning, Proc Int Conf Learn Representations, 5228
- 10.1109/CVPR52688.2022.00517 · DOI · Paper page
- xie, 2017, Rethinking spatiotemporal feature learning for video understanding, Proc Eur Conf Comput Vis, 318
- 10.1609/aaai.v33i01.33018545 · DOI · Paper page
- 10.1109/ICIP.2019.8803639 · doi:10.1109/icip.2019.8803639
- 10.1007/978-3-030-58598-3_27 · DOI
- 10.1080/21681163.2020.1835550 · DOI
- 10.1609/aaai.v34i07.6718 · DOI · Paper page
- 10.1109/TASLP.2021.3065823 · DOI
- 10.1109/CVPR42600.2020.00481 · DOI · Paper page
- ridnik, 2021, ImageNet-21k pretraining for the masses, Proc Int Conf Neural Inf Process
- 10.1109/ICCVW.2019.00125 · DOI
- 10.1109/ICCV48922.2021.01378 · DOI · Paper page
- 10.1109/CVPR.2019.00033 · DOI
- plizzari, 2021, Skeleton-based action recognition via spatial and temporal transformer networks, Comput Vis Image Understanding, 208
- 10.1109/CVPR52688.2022.01940 · DOI
- yun, 2022, Time is MattEr: Temporal self-supervision for video transformers, Proc 39th Int Conf Mach Learn, 25804
- ging, 2020, COOT: Cooperative hierarchical transformer for video-text representation learning, Proc 34th Int Conf Neural Inf Process Syst
- 10.1109/CVPR46437.2021.01248 · DOI
- 10.1007/978-3-030-68238-5_48 · DOI · Paper page
- 10.1109/ICCV48922.2021.00747 · DOI
- radford, 2018, Improving language understanding by generative pre-training, OpenAI Preprint
- 10.1109/ICCV48922.2021.01327 · DOI · Paper page
- 10.1109/CVPR.2017.634 · DOI · Paper page
- 10.1016/j.aiopen.2022.01.001 · DOI
- 10.1007/s11263-021-01547-8 · DOI
- yang, 2022, Transformers meet visual learning understanding: A comprehensive review
- jain, 2019, Attention is not explanation, Proc Conf North Amer Chapter Assoc Comput Linguistics Hum Lang Technol, 3543
- bulat, 2021, Space-time mixing attention for video transformer, Proc Int Conf Neural Inf Process, 19594
- 10.1109/CVPR46437.2021.00725 · DOI · Paper page
- 10.1109/CVPR52688.2022.00290 · DOI
- 10.1109/ICCV.2019.00015 · DOI
- 10.1109/CVPR46437.2021.00585 · DOI · Paper page
- qi, 2017, PointNet: Deep hierarchical feature learning on point sets in a metric space, Proc Int Conf Neural Inf Process, 5099
- yu, 2021, Learning from inside: Self-driven siamese sampling and reasoning for video question answering, Proc Int Conf Neural Inf Process, 26462
- 10.1109/ICCV48922.2021.01606 · DOI · Paper page
- 10.1109/CVPR46437.2021.00610 · DOI
- 10.1109/CVPR42600.2020.00028 · DOI · Paper page
- zha, 2021, Shifted chunk transformer for spatio-temporal representational learning, Proc Int Conf Neural Inf Process, 11384
- yang, 2021, Associating objects with transformers for video object segmentation, Proc Int Conf Neural Inf Process, 2491
- 10.1109/ICCV.2019.00718 · DOI · Paper page
- 10.1109/WACV48630.2021.00058 · DOI
- 10.18653/v1/D16-1244 · DOI · Paper page
- 10.1109/CVPR52688.2022.01170 · DOI · Paper page
- 10.1109/ICCV48922.2021.00988 · DOI · Paper page
- 10.1109/CVPR52688.2022.01179 · DOI · Paper page
- ye, 2022, Referring segmentation in images and videos with cross-modal self-attention network, IEEE Trans Pattern Anal Mach Intell, 44, 3719
- 10.1109/TCSVT.2021.3082763 · DOI · Paper page
- 10.1109/ICCV48922.2021.01325 · DOI · Paper page
- 10.1016/j.neucom.2018.03.080 · DOI · Paper page
- 10.1109/ICCV48922.2021.00871 · DOI
- 10.1109/ICCV.2019.00756 · DOI · Paper page
- iashin, 2020, Multi-modal dense video captioning, Proc IEEE Conf Comput Vis and Pattern Recog, 4117
- 10.1109/ICTAI50040.2020.00176 · DOI
- luo, 2020, Synchronous bidirectional learning for multilingual lip reading, Proc Brit Mach Vis Conf, 8122
- 10.1109/ICCV48922.2021.01039 · DOI
- fajtl, 2018, Summarizing videos with attention, Proc Asian Conf Comput Vis, 39
- 10.1109/CVPR52688.2022.00476 · DOI · Paper page
- 10.1109/CVPR42600.2020.01151 · DOI
- hwang, 2021, Video instance segmentation using inter-frame communication transformers, Proc Int Conf Neural Inf Process, 13352
- 10.1109/CVPR.2018.00911 · DOI · Paper page
- patrick, 2021, Keeping your eye on the ball: Trajectory attention in video transformers, Proc Int Conf Neural Inf Process, 12493
- patrick, 2021, Support-set bottlenecks for video-text representation learning, Proc Int Conf Learn Representations, 8116
- ryoo, 2021, TokenLearner: Adaptive space-time tokenization for videos, Proc Int Conf Neural Inf Process, 12786
- 10.1109/CVPR46437.2021.00054 · DOI
- 10.1109/CVPR52688.2022.00289 · DOI
- 10.1016/j.neucom.2021.03.026 · DOI
- 10.1109/TPAMI.2011.157 · DOI · Paper page
- 10.1109/ICCV48922.2021.01341 · DOI
- 10.1109/CVPR52688.2022.01362 · DOI
- 10.1109/TIP.2021.3113114 · DOI
- 10.1109/CVPR52688.2022.00315 · DOI
- 10.1109/CVPR52688.2022.01322 · DOI · Paper page
- 10.1109/CVPR46437.2021.00192 · DOI
- iashin, 2020, A better use of audio-visual cues: Dense video captioning with bi-modal transformer, Proc Brit Mach Vis Conf, 698
- 10.1109/CVPR42600.2020.00092 · DOI
- 10.1109/CVPR52688.2022.01827 · DOI
- 10.1007/978-3-030-01267-0_41 · DOI
- 10.1109/ICCV48922.2021.00156 · DOI
- zhou, 2021, Hopper: Multi-hop transformer for spatiotemporal reasoning, Proc Int Conf Learn Representations, 1553
- akbari, 2021, Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text, Proc Int Conf Neural Inf Process, 24206
- 10.1109/CVPR46437.2021.00863 · DOI · Paper page
- li, 2022, UniFormer: Unified transformer for efficient spatial-temporal representation learning, Proc Int Conf Learn Representations, 14733
- 10.1109/CVPR46437.2021.00162 · DOI · Paper page
- 10.1109/ICCVW.2019.00194 · DOI
- bozic, 2021, TransformerFusion: Monocular RGB scene reconstruction using transformers, Proc Int Conf Neural Inf Process, 1403
- 10.1007/978-3-030-58517-4_31 · DOI · Paper page
- 10.1109/CVPR52688.2022.01367 · DOI · Paper page
- 10.1109/ICCV48922.2021.01332 · DOI · Paper page
- xu, 2021, Long short-term transformer for online action detection, Proc Int Conf Neural Inf Process, 1086
- 10.1109/ICCV48922.2021.01170 · DOI · Paper page
- 10.1109/CVPR46437.2021.01334 · DOI
- 10.1145/3394171.3416291 · DOI
- 10.1109/CVPR52688.2022.01563 · DOI · Paper page
- 10.1109/ICCVW.2019.00186 · DOI · Paper page
- 10.1109/CVPR52688.2022.00950 · DOI
- 10.1109/CVPR.2019.01232 · DOI · Paper page
- 10.1109/ICCV.2019.00565 · DOI · Paper page
- 10.1109/CVPR52688.2022.00293 · DOI
- 10.1109/CVPR52688.2022.01359 · DOI
- 10.1109/CVPR.2018.00769 · DOI
- 10.1109/LSP.2019.2923918 · DOI
- 10.1007/978-3-030-66823-5_18 · DOI
- 10.1109/CVPR42600.2020.01004 · DOI
- 10.1145/3343031.3356051 · DOI
- 10.1109/ICCV48922.2021.00948 · DOI · Paper page
- 10.1109/ICCV48922.2021.01471 · DOI
- hendrycks, 2019, Using self-supervised learning can improve model robustness and uncertainty, Proc 33rd Int Conf Neural Inf Process Syst
- 10.1609/aaai.v32i1.11506 · DOI · Paper page
- 10.1109/CVPR52688.2022.01867 · DOI
- chen, 2020, A simple framework for contrastive learning of visual representations, Proc 37th Int Conf Mach Learn, 1597
- 10.1109/WACV48630.2021.00331 · DOI
- tian, 2021, Understanding self-supervised learning dynamics without contrastive pairs, Proc 38th Int Conf Mach Learn, 10268
- purushwalkam, 2020, Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases, Proc 34th Int Conf Neural Inf Process Syst
- 10.1109/CVPR46437.2021.00689 · DOI · Paper page
- 10.1007/s41095-021-0247-3 · DOI
- 10.1109/CVPR42600.2020.00877 · DOI · Paper page
- tay, 2020, Efficient transformers: A survey, ACM Comput Surv, 55
- vaswani, 2017, Attention is all you need, Proc 31st Int Conf Neural Inf Process Syst, 6000
- 10.1109/CVPR52688.2022.01171 · DOI
- dong, 2021, Attention is not all you need: Pure attention loses rank doubly exponentially with depth, Proc 38th Int Conf Mach Learn, 2793
- 10.18653/v1/2020.acl-main.244 · DOI · Paper page
- 10.1109/ICCV48922.2021.00774 · DOI · Paper page
- 10.1109/CVPR52688.2022.00713 · DOI · Paper page
Cited by
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072
A comprehensive survey on applications of transformers for deep learning tasks
Islam, S., Elmekki, H., Elsebai, A., Bentahar, J., Drawel, N., Rjoub, G., & Pedrycz, W. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122666 · Cited by 487
Transformers in Remote Sensing: A Survey
Aleissaee, A. A., Kumar, A., Anwer, R. M., Khan, S., Cholakkal, H., Xia, G.-S., & Khan, F. S. (2023) · Remote Sensing · doi:10.3390/rs15071860 · Cited by 417
ChatGPT in healthcare: A taxonomy and systematic review
Li, J., Dada, A., Puladi, B., Kleesiek, J., & Egger, J. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108013 · Cited by 452
Expanding Language-Image Pretrained Models for General Video Recognition
Ni, B., Peng, H., Chen, M., Zhang, S., Meng, G., Fu, J., Xiang, S., & Ling, H. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19772-7_1 · Cited by 326
Metadata
- Authors
- Selva, J., Johansen, A. S., Escalera, S., Nasrollahi, K., Moeslund, T. B., & Clapés, A.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2023-11-01
- Volume
- 45
- Issue
- 11
- Pages
- 12922-12943
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W4319777935
- Cited by
- 169
- References
- 74
- FWCI
- 15.09
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- Funders
- Pioneer Centre for AI
- Danmarks Grundforskningsfond
- Spanish project (PID2019-105093GB-I00)
- Institució Catalana de Recerca i Estudis Avançats
- ICREA Academia program
- Aalborg Universitet
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.