The DOI matches its Crossref record; 106 of the 111 references it cites were found in a registry or an index.
Multiscale Vision Transformers
Open accessAbstract
We present Multiscale Vision Transformers (MViT) for video and image recognition, by connecting the seminal idea of multiscale feature hierarchies with transformer models. Multiscale Transformers have several channel-resolution scale stages. Starting from the input resolution and a small channel dimension, the stages hierarchically expand the channel capacity while reducing the spatial resolution. This creates a multiscale pyramid of features with early layers operating at high spatial resolution to model simple low-level visual information, and deeper layers at spatially coarse, but complex, high-dimensional features. We evaluate this fundamental architectural prior for modeling the dense nature of visual signals for a variety of video recognition tasks where it outperforms concurrent vision transformers that rely on large scale external pre-training and are 5-10× more costly in computation and parameters. We further remove the temporal dimension and apply our model for image classification where it outperforms prior work on vision transformers. Code is available at: https://github.com/facebookresearch/SlowFast.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“As in original ViT [25], we use residual connections [45] and Layer Normalization (LN) [2] in the pre-normalization configuration that applies LN at the beginning of the residual function, and our MLPs consist of two linear layers with GELU activation [48], where the first layer expands the dimension from D to 4D, and…”
MethodResultVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 6×
“In contrast to original Multi Head Attention (MHA) operators [98], where the channel dimension and the spatiotemporal resolution remains fixed, MHPA pools the sequence of latent tensors to reduce the sequence length (resolution) of the attended input.”
BackgroundMethodInfluentialVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“Incorporating downsampling, shift invariance, and shared weights, ConvNets are de-facto standard backbones for computer vision tasks for image [65, 62, 88, 90, 46, 12, 15, 34, 93, 81, 41] and video [87, 31, 11, 79, 69, 106, 96, 30, 105, 35, 29, 117, 57].”
BackgroundImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“Incorporating downsampling, shift invariance, and shared weights, ConvNets are de-facto standard backbones for computer vision tasks for image [70, 67, 94, 96, 51, 15, 18, 39, 99, 87, 46] and video [93, 36, 14, 85, 74, 112, 102, 34, 111, 40, 33, 123, 62].”
BackgroundFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · cited 1×
“Finally, compared to the best model of concurrent Swin [79] (which was designed for image recognition), MViT has +0.”
ResultRethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Research works printed but not linked: 11
- gao, 2019, Res2net: A new multi-scale backbone architecture, IEEE PAMI
- fan, 2020, Pyslowfast
- feichtenhofer, 2019, SlowFast networks for video recognition in ActivityNet challenge 2019
- 10.1109/CVPR.2009.5206848 · DOI · Paper page
- 10.1109/CVPR.2019.00033 · DOI
- hang, 2020, ResNeSt Split-Attention Networks
- 10.1109/ICCV.2019.00718 · DOI
- 10.1109/CVPR.2015.7298594 · DOI · Paper page
- carion, 2020, Endto-end object detection with transformers, Proc ECCV, 213
- loshchilov, 2018, Fixing weight decay regularization in adam
- qiu, 2017, Learning spatiotemporal representation with pseudo-3d residual networks, Proc ICCV
Cited by
Sorted by citations
Items 401–500 of 1,003
E2E-LOAD: End-to-End Long-form Online Action Detection
Cao, S., Luo, W., Wang, B., Zhang, W., & Ma, L. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00956 · Cited by 19
“We explore various efficient attention mechanisms proposed by the previous video models, such as the Video Swin Transformer [17], MeMViT [27], and MViT [6].”
MethodInfluentialVideo Action Recognition with Attentive Semantic Units
Chen, Y., Chen, D., Liu, R., Li, H., & Peng, W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00933 · Cited by 20
“Methods with ImageNet or web-scale image pretraining MViTv1-B, 64×3 [11] 64 81.”
MethodVray, G., Tomar, D., Bozorgtabar, B., & Thiran, J. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3355645 · Cited by 19
“Table II compares Distill-SODA against SOTA methods for closed-set domain adaptation scenario via ACC metric and open-set recognition capability through AUC metric on three splits for adapting source model initially trained through CE + , MC-Dropout and T3PO across various target domains.”
MethodGrim, L. F. L., & Gradvohl, A. L. S. (2024) · Solar Physics · doi:10.1007/s11207-024-02276-0 · Cited by 20
Deep Unfolding Transformers for Sparse Recovery of Video
Weerdt, B. D., Eldar, Y. C., & Deligiannis, N. (2024) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2024.3381749 · Cited by 20
“In recent years, Transformer models have achieved state-of-the-art results in language modeling [16], [34], [35], computer vision [36], [37], [38], and image processing tasks [40], [41] (see [39] for an overview in vision).”
BackgroundEgocentric Action Recognition by Capturing Hand-Object Contact and Object State
Shiota, T., Takagi, M., Kumagai, K., Seshimo, H., & Aono, Y. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00641 · Cited by 23
“In the following sections, we describe each loss function and the learning/inference procedure in detail.”
Triplet Attention Transformer for Spatiotemporal Predictive Learning
Nie, X., Chen, X., Jin, H., Zhu, Z., Yan, Y., & Qi, D. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00688 · Cited by 20
“MViT [11] introduces multiscale features for video sequences, and Video Swin Transformer [28] adapts the model to 3D settings.”
BackgroundM2VAD : Multiview multimodality transformer-based weakly supervised video anomaly detection
Paulraj, S., & Subramaniyaswamy, V. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.105139 · Cited by 17
Dual-Scale Transformer for Large-Scale Single-Pixel Imaging
Qu, G., Wang, P., & Yuan, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02393 · Cited by 17
“Due to the remarkable performance of SA, researchers are extending Transformer into low-level vision tasks [7, 12, 35, 55, 60, 66].”
BackgroundPALM: Predicting Actions through Language Models
Kim, S., Huang, D., Xian, Y., Hilliges, O., Gool, L. V., & Wang, X. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73007-8_9 · Cited by 32
“We evaluate various video encoders trained on egocentric videos, including MViT [18], SlowFast [19], StillFast [52], EgoVLP [37], and EgoVLPv2 [49].”
BackgroundMethodLearning Motion and Temporal Cues for Unsupervised Video Object Segmentation
Zhuge, Y., Gu, H., Zhang, L., Qi, J., & Lu, H. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3418980 · Cited by 16
“In MViT [14], a hierarchical architecture is presented, which efficiently captures information across multiple scales, resulting in enhanced performance for a range of vision tasks such as image classification and video understanding.”
Human-Centric Transformer for Domain Adaptive Action Recognition
Lin, K.-Y., Zhou, J., & Zheng, W. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3429387 · Cited by 19
“Also, we conduct experiments using advanced Vision Transformers as backbones, i.e., DeiT [116], Swin [46] and MViTv2 [117] pretrained on Ima-geNet,aswellasMViTv1[118]andMViTv2[117]pretrainedon Kinetics-400.”
Ego4D: Around the World in 3,600 Hours of Egocentric Video
Grauman, K. L., Westbury, A., Byrne, E., Cartillier, V., Chavis, Z. Q., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Kukreja, D., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3381075 · Cited by 20
Self-citationWhat Makes for Hierarchical Vision Transformer?
Fang, Y. K., Wang, X., Wu, R., & Liu, W. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3282019 · Cited by 20
“Recently, the impregnable position of convolutional neural networks (CNNs) in computer vision seems to be weakened by the emerging hierarchical Vision Transformer families [1], [2], [3], [4].”
BackgroundDynamic Graph Message Passing Networks for Visual Recognition
Zhang, L., Chen, M., Arnab, A., Xue, X., & Torr, P. H. S. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3207500 · Cited by 18
“Similarly, MViT [14] also performs pooling to reduce the number of tokens on which self-attention is applied.”
BackgroundA Novel Action Transformer Network for Hybrid Multimodal Sign Language Recognition
Javaid, S., & Rizvi, S. (2022) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2023.031924 · Cited by 16
K-centered Patch Sampling for Efficient Video Recognition
Park, S. H., Tack, J., Heo, B., Ha, J.-W., & Shin, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19833-5_10 · Cited by 17
“Following this line of research, recent works considered the temporal extension of vision transformer (ViT) [11] for an effective spatiotemporal understanding of videos [12,23,26,34,35].”
BackgroundA Simple and Efficient Pipeline to Build an End-to-End Spatial-Temporal Action Detector
Sui, L., Zhang, C.-L., Gu, L., & Han, F. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00594 · Cited by 16
“Most existing works often follow a traditional FastRCNN [10] pipeline with pre-extracted proposals to perform STAD [13, 20, 42, 29, 6].”
MethodTAMformer: Multi-Modal Transformer with Learned Attention Mask for Early Intent Prediction
Osman, N., Camporese, G., & Ballan, L. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095781 · Cited by 17
“Transformers [22] have been also investigated for spatio-temporal modeling [23, 24, 20] improving the state-of-the-art performances on video related problems including video action recognition.”
BackgroundAutoFocusFormer: Image Segmentation off the Grid
Chen, Z., Patnaik, K., Zhai, S., Wan, A., Ren, Z., Schwing, A. G., Colburn, A., & Li, F. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01748 · Cited by 18
“Various improvements have since been proposed, e.g., MViT and PVT [8, 43] propose using feature hierarchies similar to those of standard convolutional networks.”
The Wisdom of Crowds: Temporal Progressive Attention for Early Action Prediction
Stergiou, A., & Damen, D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01413 · Cited by 17
“The recent introduction of Vision Transformer [10] has inspired subsequent works on action recognition by either focusing on how spatio-temporal information can be processed [1,2] or architectural optimizations for spatio-temporal data [12,40,48,68,70].”
BackgroundYuan, K., Kong, Z., Zheng, C., Sun, M.-T., & Wen, X. (2023) · ACM Multimedia · doi:10.1145/3581783.3612023 · Cited by 25
“Following ViT, many Transformer-based models [2, 3, 13, 35, 65] were developed for video classification tasks.”
MethodEventful Transformers: Leveraging Temporal Redundancy in Vision Transformers
Dutson, M., Li, Y., & Gupta, M. (2023) · IEEE International Conference on Computer Vision · doi:10.1109/iccv51070.2023.01551 · Cited by 28
“Vision Transformers [16] have achieved impressive accuracy across a range of visual recognition problems, attaining state-of-the-art performance in tasks including image classification [16], video classification [1, 2, 18], and object detection [8, 37, 40, 61].”
BackgroundSelf-citationExploiting Optical Flow Guidance for Transformer-Based Video Inpainting
Zhang, K., Peng, J., Fu, J., & Liu, D. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3361010 · Cited by 16
“Transformer has been integrated to numerous fields and achieved promising performance, such as basic architecture design [29], [30], [31], image classification [50], [51], [52], [53], object detection [54], [55], action detection [56], segmentation [57], etc.”
BackgroundIntxausti, E., Skočaj, D., Cernuda, C., & Zugasti, E. (2024) · Applied Sciences · doi:10.3390/app14072785 · Cited by 17
Yang, Z., An, G., Zheng, Z., Cao, S., & Ruan, Q. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3390133 · Cited by 16
BiF-DETR:Remote sensing object detection based on Bidirectional information fusion
Xu, Z., Wang, C., & Huang, K. (2024) · Displays · doi:10.1016/j.displa.2024.102802 · Cited by 19
Chen, Y., Chen, D., Liu, R., Zhou, S., Xue, W., & Peng, W. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01768 · Cited by 16
“By lightly fine-tuning the model, VLP-based methods outperform the previous end-to-end network architectures, including two-stream networks [55, 64, 80], 3D convolutional neural networks [6, 14, 15, 22, 48, 59, 60, 68], and vision-transformer-based (ViT) networks [3, 12, 38, 46, 71].”
Beyond the Premier: Assessing Action Spotting Transfer Capability Across Diverse Domains
Cabado, B., Cioppa, A., Giancola, S., Villa, A., Guijarro‐Berdiñas, B., Padrón, E. J., Ghanem, B., & Droogenbroeck, M. V. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00343 · Cited by 17
“A more recent innovation, the Multiscale Vision Trans-former (MViT) [21, 42], synergizes convolutional neural networks (CNNs) and transformers in video classification to capture spatial and temporal nuances.”
MethodTransformer-Based Explainable Model for Breast Cancer Lesion Segmentation
Wang, H., Wei, L., Liu, B., Li, J., Li, J., Fang, J., & Mooney, C. (2025) · Applied Sciences · doi:10.3390/app15031295 · Cited by 19
Lee, Y., Kim, H.-I., Yun, K., & Moon, J. (2021) · IEEE Access · doi:10.1109/access.2021.3132916 · Cited by 16
“5 MViT-B, 32× 3 [6] Kinetics-400 32 36.”
MethodAkdag, E., Zhu, Z., Bondarev, E., & de With, P. H. N. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00576 · Cited by 16
“In recent years, ViT and its variants, including MViT [9] and Swin Transformer [24], have achieved outstanding performance in action recognition.”
BackgroundMulti-Scale Adaptive Skeleton Transformer for action recognition
Wang, X., Chen, K., Zhao, Z., Shi, G., Xie, X., Jiang, X. W., & Yang, Y. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104229 · Cited by 15
Liu, Y., Zhao, J., Luo, Q., Shen, C., Wang, R., & Ding, X. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105821 · Cited by 16
Cross-domain few-shot action recognition with unlabeled videos
Wang, X., Zhang, S., Qing, Z., Lv, Y., Gao, C., & Sang, N. (2023) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2023.103737 · Cited by 17
Werewolf Among Us: Multimodal Resources for Modeling Persuasion Behaviors in Social Deduction Games
Lai, B., Zhang, H., Liu, M., Pariani, A., Ryan, F., Jia, W., Hayati, S. A., Rehg, J. M., & Yang, D. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.411 · Cited by 21
“We first leverage a pre-trained language model (Devlin et al., 2019; Liu et al., 2019) as the text encoder to obtain the ut-terance embedding, and a vision transformer (Fan et al., 2021) to obtain the visual embedding.”
MethodRyoo, M. S., Gopalakrishnan, K., Kahatapitiya, K., Xiao, T. G., Rao, K., Stone, A., Lu, Y., Ibarz, J., & Arnab, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01828 · Cited by 21
“Transformers have since become state-of-the-art in video modeling, being able to handle multiple modalities like audio/text [2,49] and scaling efficiently [24,47].”
BackgroundEfficient Multi-Scale Cosine Attention Transformer for Image Super-Resolution
Chen, Y., Wang, G., & Chen, R. (2023) · IEEE Signal Processing Letters · doi:10.1109/lsp.2023.3321558 · Cited by 15
“MViT [4] improves the classification performance by using a multiscale strategy to model input at different scales.”
BackgroundMethodGhostViT: Expediting Vision Transformers via Cheap Operations
Cao, H., Qu, Z., Chen, G., Li, X., Thiele, L., & Knoll, A. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3326795 · Cited by 15
“…1) using a hierarchical pyramid structure to reduce the spatial resolution of the input (the token sequence length) across the stages [7], [13]; 2) restricting the attention computation in a small local window [9], [14]; 3) reducing the spatial structure of key and value via down-sampling…”
MethodSemantic-Disentangled Transformer With Noun-Verb Embedding for Compositional Action Recognition
Huang, P. F., Yan, R., Shu, X., Tu, Z., Dai, G., & Tang, J. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3341297 · Cited by 15
“Recently, some ViT-based approaches achieve superior performance compared to CNN-based methods on the large image recognition task ( e.g. ImageNet [55]), such as Swin Transformer [45], MViT [16], Improved MViT [37], etc.”
MethodKnowledge-guided pre-training and fine-tuning: Video representation learning for action recognition
Wang, G., Zhou, Y., He, Z., Lu, K., Feng, Y., Liu, Z., & Wang, G. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.127136 · Cited by 15
Temporal context video compression with flow-guided feature prediction
Wang, Y., Huang, Q., Tang, B., Sun, H., Guo, X., & Miao, Z. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123322 · Cited by 15
PPTtrack: Pyramid pooling based Transformer backbone for visual tracking
Wang, J., Yang, S., Wang, Y., & Yang, G. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123716 · Cited by 15
Gibril, M. B. A., Shafri, H. Z. M., Shanableh, A., Al-Ruzouq, R. I., Hashim, S. J., Wayayok, A., & Sachit, M. S. (2024) · Remote Sensing Applications Society and Environment · doi:10.1016/j.rsase.2024.101195 · Cited by 15
Rousta, F., Esteki, A., Shalbaf, A., Sadeghi, A., Moghadam, P. K., & Voshagh, A. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108205 · Cited by 16
Li, K., Wang, D., Liu, G., Zhu, W., Zhong, H., & Wang, Q. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106653 · Cited by 19
Asymmetric Masked Distillation for Pre-Training Small Foundation Models
Zhao, Z., Huang, B., Xing, S., Wu, G., Qiao, Y., & Wang, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01752 · Cited by 17
Beyond Prompt Learning: Continual Adapter for Efficient Rehearsal-Free Continual Learning
Gao, X., Dong, S., He, Y., Wang, Q., & Gong, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73013-9_6 · Cited by 15
Mao, J., Shen, Y., Guo, J., Yao, Y., Hua, X., & Shen, H. T. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3535405 · Cited by 24
Vu, D.-Q., Phung, T. T. T., & Wang, J. (2021) · IEEE International Conference on Visual Communications and Image Processing (VCIP) · doi:10.1109/vcip53242.2021.9675335 · Cited by 16
“Besides, several approaches based on transformer have been proposed and achieved state-of-the-art performance, such as MViT [28], VTN [29], etc.”
BackgroundRobotic Grasp Detection Based on Transformer
Dong, M., Bai, Y., Wei, S., & Yu, X. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-13841-6_40 · Cited by 15
ViGAT: Bottom-Up Event Recognition and Explanation in Video Using Factorized Graph Attention Network
Gkalelis, N., Daskalakis, D., & Mezaris, V. (2022) · IEEE Access · doi:10.1109/access.2022.3213652 · Cited by 15
“In [13], similarly to SlowFast [65], andX3D [75] in the 3D convolutional paradigm, multiscale ViT (MViT) introduces several channel resolution scale stages into transformer models.”
BackgroundMethodInfluentialLearning visual language models for video understanding
Yang, A. (2023) · theses.fr (ABES) · doi:10.70675/802872a7z8092z4556za1fdz2c7af45f0e72 · Cited by 26
Yang, Z., Zhao, X., Yao, C., Zhang, Q., & Xiao, J. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111348 · Cited by 15
About Time: Advances, Challenges, and Outlooks of Action Understanding
Stergiou, A., & Poppe, R. W. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02478-4 · Cited by 17
GPPT: Graph pyramid pooling transformer for visual scene
Li, Z., Liu, W., Sun, X., Pan, Y., Gribova, V. V., Filaretov, V. F., Cohn, A. G., & Huang, D.-S. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130729 · Cited by 14
ROI-aware multiscale cross-attention vision transformer for pest image identification
Kim, G.-E., Son, C., & Lee, S. (2025) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2025.110732 · Cited by 15
What Makes for Good Tokenizers in Vision Transformer?
Qian, S., Zhu, Y., Li, W., Li, M., & Jia, J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3231442 · Cited by 15
“For intra-token refinement, we adopt the multi-scale feature extraction as in [46], [47].”
BackgroundMethodSpatio-Temporal Action Detection Under Large Motion
Singh, G., Choutas, V., Saha, S., Yu, F., & Gool, L. V. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00595 · Cited by 14
Extreme Low-Resolution Action Recognition with Confident Spatial-Temporal Attention Transfer
Bai, Y., Zou, Q., Chen, X., Li, L., Ding, Z., & Chen, L. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01771-4 · Cited by 18
“MViT (Fan et al., 2021) proposed a multi-scale strategy for ViTbased video recognition, and reduced the computation cost by pooling spatial-temporal attention, which leads to a stateof-the-art performance.”
BackgroundCampus Abnormal Behavior Recognition With Temporal Segment Transformers
Liu, H. L., Chuah, J. H., Khairuddin, A. S. M., Zhao, X. M., & Wang, X. D. (2023) · IEEE Access · doi:10.1109/access.2023.3266440 · Cited by 14
“Therefore, researchers have attempted to solve the inductive bias problem of pure transformers [28], [31], [32].”
BackgroundMethodInfluentialHA-Transformer: Harmonious aggregation from local to global for object detection
Chen, Y., Chen, S., Deng, Y., & Wang, K. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120539 · Cited by 14
An Optimal Transport View of Class-Imbalanced Visual Recognition
Jin, L., Lang, D., & Lei, N. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01831-9 · Cited by 14
CA-CMT: Coordinate Attention for Optimizing CMT Networks
Bang, J. H., Park, S.-W., Kim, J.-Y., Park, J., Huh, J., Jung, S.-H., & Sim, C.-B. (2023) · IEEE Access · doi:10.1109/access.2023.3297206 · Cited by 14
“In particular, Vanilla ViT models that do not use convolutional layers [13], [14], [15], [16], [17] have shown strength in long range dependency, although they still require a lot of computational resources compared to CNN.”
MethodResearch on Students’ Action Behavior Recognition Method Based on Classroom Time-Series Images
Shou, Z., Yan, M., Wen, H., Liu, J., Mo, J., & Zhang, H. (2023) · Applied Sciences · doi:10.3390/app131810426 · Cited by 14
“[16] proposed a Multi-scale Transformer, which combines the multiscale feature hierarchies with the transformer to extract the video features at different levels so that the model can better understand the video content.”
BackgroundMethodWang, C., Zhang, X., Yang, W., Wang, G., Zhao, Z., Liu, X., & Lu, B. (2023) · Remote Sensing · doi:10.3390/rs15225272 · Cited by 14
Towards Transferable Adversarial Attacks on Image and Video Transformers
Wei, Z., Chen, J., Goldblum, M., Wu, Z., Goldstein, T., Jiang, Y., & Davis, L. S. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3331582 · Cited by 15
“MViT [41] turns to directly shortening the length of tokens via pooling “query”, “key”, and “ value” before inputting them to MSHA.”
BackgroundGated Multi-Scale Transformer for Temporal Action Localization
Yang, J., Wei, P., Ren, Z., & Zheng, N. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3338082 · Cited by 14
“These have been proven to be effective in previous studies [52], [57], [58], [59].”
ResultYang, M., Chengke, W., Guo, Y., He, Y. Z., Jiang, R., Jiang, J., & Yang, Z. (2023) · Advanced Engineering Informatics · doi:10.1016/j.aei.2023.102294 · Cited by 14
Appearance-Agnostic Representation Learning for Compositional Action Recognition
Huang, P. F., Shu, X., Yan, R., Tu, Z., & Tang, J. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3384392 · Cited by 14
“By designing for task-specific attention, generic ViT-based video understanding approaches [2], [10], [30], [36], [37], [44] exhibits compelling spatio-temporal modeling competencies.”
Chen, Z., Liu, S., Liu, S., Chang, Y., Chen, J., Feng, G., & He, S. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111922 · Cited by 14
Compare and Focus: Multi-Scale View Aggregation for Crowd Counting
Jiang, S., Cai, J., Zhang, H., Liu, Y., & Liu, Q. (2024) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2024.3432789 · Cited by 14
“MViT [30] combined multi-scale feature hierarchies with transformer.”
BackgroundWu, J., Ma, J., Xi, H., Li, J., & Zhu, J. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106914 · Cited by 15
Hu, W., Yang, X., Ma, P., Fu, T., & Zhou, C. (2024) · Aquaculture · doi:10.1016/j.aquaculture.2024.742008 · Cited by 14
Cao, M., Zheng, J., & Zhang, C. (2025) · EURASIP Journal on Audio Speech and Music Processing · doi:10.1186/s13636-025-00397-3 · Cited by 17
Zhang, Z., Wu, Q., Ding, S., Wang, X., & Ye, J. (2025) · Nature Communications · doi:10.1038/s41467-025-66340-4 · Cited by 19
“To address this, we followed a standard cube embedding procedure 62,63 for transforming raw video data into input embeddings, but we reduced the temporal grid stride to be smaller than the 2D spatial grid stride.”
EQUI-VOCAL: Synthesizing Queries for Compositional Video Events from Limited User Interactions
Zhang, E., Daum, M., He, D., Haynes, B. M., Krishna, R., & Bałazińska, M. (2023) · Proceedings of the VLDB Endowment · doi:10.14778/3611479.3611482 · Cited by 14
“We use bothMViT [21] and CLIP [60] as the feature extractors.”
MethodMalagoli, E., & Persio, L. D. (2025) · Mathematics · doi:10.3390/math13060893 · Cited by 19
Gong, C., Qin, H., & El‐Yacoubi, M. A. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3585379 · Cited by 15
“Transformer-based models, such as HVT [57], PVT [58], and MViT [59], have incorporated pyramid structures into ViT to improve performance.”
Classification of Left and Right Coronary Arteries in Coronary Angiographies Using Deep Learning
Eschen, C. K., Banasik, K., Christensen, A. H., Chmura, P. J., Pedersen, F., Køber, L., Engstrøm, T., Dahl, A. B., Brunak, S., & Bundgaard, H. (2022) · Electronics · doi:10.3390/electronics11132087 · Cited by 13
Full Contextual Attention for Multi-resolution Transformers in Semantic Segmentation
Themyr, L., Rambour, C., Thome, N., Collins, T., & Hostettler, A. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00324 · Cited by 13
“Among these approaches, window-based patch extraction vision transformers recently provided a simple yet efficient approach to compute attention [24, 37, 13].”
MethodMultimodal Vision Transformers with Forced Attention for Behavior Analysis
Agrawal, T., Balazia, M., Müller, P., & Brémond, F. (2022) · IEEE Workshop/Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00339 · Cited by 15
“MViT [21] is a multi-scale vision transformer for video recognition trained from scratch that reduces computation by pooling attention for spatiotemporal modeling.”
BackgroundTwinFormer: Fine-to-Coarse Temporal Modeling for Long-Term Action Recognition
Zhou, J., Lin, K.-Y., Qiu, Y.-K., & Zheng, W. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3302471 · Cited by 13
“Recently, Transformer [50] and its visual variants [8], [9], [51], [52], [53], [54], [55], [56], [57] have attracted the attention of many researchers in the video action understanding community.”
BackgroundHow You Feelin’? Learning Emotions and Mental States in Movie Scenes
Srivastava, D., Singh, A. K., & Tapaswi, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00248 · Cited by 12
Hierarchical Feature Pooling Transformer for Efficient UAV Object Tracking
Wang, H., Ma, W., Zhang, S., & Hao, W. (2023) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2023.3314435 · Cited by 13
“pooling transformer [7], [8], [9] in reducing the computational cost of MHSA, we attempt to draw upon its efficient encoder–decoder architecture in UAV object tracking to effectively fuse spatial and semantics features to obtain promising tracking performance.”
MethodZhang, W., Wang, J., Tang, M., Ma, H., Wang, L., Zhang, Q., & Fan, S. (2023) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2023.3314071 · Cited by 14
MUP: Multi-granularity Unified Perception for Panoramic Activity Recognition
Cao, M., Yan, R., Shu, X., Zhang, J., Wang, J., & Xie, G.-S. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612435 · Cited by 13
Li, Z., Ouyang, B., Qiu, S., Xu, X., Cui, X., & Hua, X. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3374050 · Cited by 13
“PVT [47] and MViT [48] used a single pooling operation in the MHSA module to downsample the feature maps, using the pooled feature to simulate the token-to-region relationship.”
MethodA Hybrid CNN-Transformer for Focal Liver Lesion Classification
Zhao, L., Liu, S., Li, B., Cai, W., Liang, P., Yu, J., & Zhao, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447765 · Cited by 13
CCMR: High Resolution Optical Flow Estimation via Coarse-to-Fine Context-Guided Motion Reasoning
Jahedi, A., Luz, M., Rivinius, M., & Bruhn, A. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00675 · Cited by 13
“Methods attempting to retain global reasoning via hierarchical structures (e.g., [8,21,24,43]; generally local at a scale) do inherently not retain their input resolution.”
BackgroundA Cross-Attention-Based Multi-Information Fusion Transformer for Hyperspectral Image Classification
Yang, J., Li, A., Qian, J., Jia, Q., & Wang, L. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3429492 · Cited by 15
“[34]proposedamultiscalevisualtransformerforimage and video recognition by integrating the hierarchical structure of multiscale features with the transformer model.”
BackgroundOST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition
Chen, T., Yu, H., Yang, Z., Li, Z., Sun, W., & Chen, C. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01787 · Cited by 15
“…recognition pipelines obtain video-level representation via pooling over image-level [CLS] tokens and then classify the video into a category by calculating the matching score using co-sine similarity with category embeddings [47, 53, 56, 60], this pipeline can be formulated as: where cos ( · , ·…”
MethodSegformer++: Efficient Token-Merging Strategies for High-Resolution Semantic Segmentation
Kienzle, D., Kantonis, M., Schön, R., & Lienhart, R. W. (2024) · IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr62202.2024.00019 · Cited by 13
Hierarchical Context Transformer for Multi-Level Semantic Scene Understanding
Hao, L., Hu, Y., Yue, Y., Wu, L., Fu, H., Duan, J., & Liu, J. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3502755 · Cited by 13
“For the self-attention part on primary task i , multi-head pooling attention (MHPA) [40] is employed to enable progressive change at the spatial-temporal resolution in the transformer backbone.”
STCA: an action recognition network with spatio-temporal convolution and attention
Tian, Q., Miao, W., Zhang, L., Yang, Z., Yu, Y., Zhao, Y., & Yao, L. (2024) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-024-00350-8 · Cited by 13
Liu, Z., Gao, A., Sheng, H., & Wang, X. (2025) · PLoS ONE · doi:10.1371/journal.pone.0318264 · Cited by 18
Multitask learning in minimally invasive surgical vision: A review
Alabi, O., Vercauteren, T. K., & Shi, M. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103480 · Cited by 14
“The first, the Multiscale Vision Transformer (MViT) (Fan et al., 2021), for extracting global and temporal features from sequential video frames.”
SwinNowcast: A Swin Transformer-Based Model for Radar-Based Precipitation Nowcasting
Li, Z., Lu, Z., Li, Y., & Liu, X. (2025) · Remote Sensing · doi:10.3390/rs17091550 · Cited by 13
“This approach not only aggregates relevant information across multiple scales [36,37] but also enhances prediction accuracy while maintaining computational efficiency.”
Self-citationAurora Classification in All-Sky Images via CNN–Transformer
Lian, J., Liu, T., & Zhou, Y. (2023) · Universe · doi:10.3390/universe9050230 · Cited by 14
Egocentric Video Task Translation
Xue, Z., Song, Y., Grauman, K. L., & Torresani, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00229 · Cited by 15
“, Kinetics [6, 33]) to benefit action recognition in smaller-scale datasets [54] such as UCF-101 [52] and HMDB-51 [37] or to improve other video tasks, such as spatiotemporal action localization [2,17,19,27,49] and video anomaly detection [25,41].”
Background
Metadata
- Authors
- Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., & Feichtenhofer, C.
- Affiliations
- Meta (Israel) (IL)
- Meta (United States) (US)
- Venue
- 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2021-10
- Pages
- 6804-6815
- OpenAlex
- W3202698164
- Cited by
- 1,407
- References
- 111
- Citations by year
- 2026: 147 · 2025: 345 · 2024: 380 · 2023: 349 · 2022: 176 · 2021: 10
- FWCI
- 74.81
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University)
- TIB Data Manager
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- video recognition, image classification, feature pyramid, spatial resolution reduction, computational efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
