The DOI matches its Crossref record; 106 of the 111 references it cites were found in a registry or an index.
Multiscale Vision Transformers
Open accessAbstract
We present Multiscale Vision Transformers (MViT) for video and image recognition, by connecting the seminal idea of multiscale feature hierarchies with transformer models. Multiscale Transformers have several channel-resolution scale stages. Starting from the input resolution and a small channel dimension, the stages hierarchically expand the channel capacity while reducing the spatial resolution. This creates a multiscale pyramid of features with early layers operating at high spatial resolution to model simple low-level visual information, and deeper layers at spatially coarse, but complex, high-dimensional features. We evaluate this fundamental architectural prior for modeling the dense nature of visual signals for a variety of video recognition tasks where it outperforms concurrent vision transformers that rely on large scale external pre-training and are 5-10× more costly in computation and parameters. We further remove the temporal dimension and apply our model for image classification where it outperforms prior work on vision transformers. Code is available at: https://github.com/facebookresearch/SlowFast.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“As in original ViT [25], we use residual connections [45] and Layer Normalization (LN) [2] in the pre-normalization configuration that applies LN at the beginning of the residual function, and our MLPs consist of two linear layers with GELU activation [48], where the first layer expands the dimension from D to 4D, and…”
MethodResultVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 6×
“In contrast to original Multi Head Attention (MHA) operators [98], where the channel dimension and the spatiotemporal resolution remains fixed, MHPA pools the sequence of latent tensors to reduce the sequence length (resolution) of the attended input.”
BackgroundMethodInfluentialVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“Incorporating downsampling, shift invariance, and shared weights, ConvNets are de-facto standard backbones for computer vision tasks for image [65, 62, 88, 90, 46, 12, 15, 34, 93, 81, 41] and video [87, 31, 11, 79, 69, 106, 96, 30, 105, 35, 29, 117, 57].”
BackgroundImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“Incorporating downsampling, shift invariance, and shared weights, ConvNets are de-facto standard backbones for computer vision tasks for image [70, 67, 94, 96, 51, 15, 18, 39, 99, 87, 46] and video [93, 36, 14, 85, 74, 112, 102, 34, 111, 40, 33, 123, 62].”
BackgroundFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · cited 1×
“Finally, compared to the best model of concurrent Swin [79] (which was designed for image recognition), MViT has +0.”
ResultRethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Research works printed but not linked: 11
- gao, 2019, Res2net: A new multi-scale backbone architecture, IEEE PAMI
- fan, 2020, Pyslowfast
- feichtenhofer, 2019, SlowFast networks for video recognition in ActivityNet challenge 2019
- 10.1109/CVPR.2009.5206848 · DOI · Paper page
- 10.1109/CVPR.2019.00033 · DOI
- hang, 2020, ResNeSt Split-Attention Networks
- 10.1109/ICCV.2019.00718 · DOI
- 10.1109/CVPR.2015.7298594 · DOI · Paper page
- carion, 2020, Endto-end object detection with transformers, Proc ECCV, 213
- loshchilov, 2018, Fixing weight decay regularization in adam
- qiu, 2017, Learning spatiotemporal representation with pseudo-3d residual networks, Proc ICCV
Cited by
Sorted by citations
Items 701–800 of 1,003
Emotion recognition in user‐generated videos with long‐range correlation‐aware network
Yi, Y., Zhou, J., Wang, H., Tang, P., & Wang, M. (2024) · IET Image Processing · doi:10.1049/ipr2.13174 · Cited by 9
Automated Hand Prehension Assessment From Egocentric Video After Spinal Cord Injury
Zhao, N., & Zariffa, J. (2024) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2024.3438436 · Cited by 6
“The MViT architecture, proposed by Fan et al. in 2021 is a transformer-based architecture that hierarchically expands the channel capacity while reducing spatial resolution [28].”
MethodUnsupervised Video Domain Adaptation with Masked Pre-Training and Collaborative Self-Training
Reddy, A., Paul, W., Rivera, C. G., Shah, K., de Melo, C. M., & Chellappa, R. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01790 · Cited by 9
“These include approaches based on convolutional neural networks (CNNs) [6, 14, 15, 47, 51, 53] and vision transformers (ViTs) [1, 5, 13, 33, 39, 42].”
MethodMixed-scale cross-modal fusion network for referring image segmentation
Pan, X., Xie, X., & Yang, J. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128793 · Cited by 7
Yang, X., Wu, J., Wang, X., Yuan, Y., Li, J., Chen, G., Wang, N. L., & Heng, P. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3523298 · Cited by 6
“In addition, a masked self-attention module is introduced to maintain causality of the sequence, ensuring that the prediction at the current position depends only on the known outputs at positions before.”
Edge Guided Network With Motion Enhancement for Few-Shot Action Recognition
Du, K., Ye, W., Guo, H., Yan, Y., & Wang, H. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3533573 · Cited by 7
“With the advancements in Transformer architectures, there have been a number of meth-ods using the Transformer architectures [28], [29].”
Toward Unifying Saliency Transformer for Video Saliency Prediction and Detection
Xiong, J., Li, C., Liu, T., Zhang, P., Huo, Y., Huang, W., & Zha, Y. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3544031 · Cited by 8
“For the visual feature encoder, the pre-trained MViT-small model [27] on ImageNet [45] is employed.”
Improving esports viewing experience through hierarchical scene detection and tracking
Joo, H.-T., Lee, S.-H., Chung, I., & Kim, K.-J. (2025) · Scientific Reports · doi:10.1038/s41598-025-93692-0 · Cited by 6
Transformer Compressed Sensing Via Global Image Tokens
Lorenzana, M. B., ENGSTROM, C. M., & Chandra, S. S. (2022) · Proceedings - International Conference on Image Processing · doi:10.1109/icip46576.2022.9897630 · Cited by 6
“More specifically, image patches alone present a high level of difficulty for capturing global structures such as edges and lines [3], and restrict the ViT to a single scale of vision [3,4].”
BackgroundMulti-view 3D Reconstruction from Video with Transformer
Zhong, Y., Sun, Z., Sun, Y., Luo, S., Wang, Y., & Zhang, W. (2022) · Proceedings - International Conference on Image Processing · doi:10.1109/icip46576.2022.9897753 · Cited by 6
“Some works [20, 21] have also used the Transformer architecture for multi-image or video encoding, which can be seen in Fig.”
MethodPerson Re-identification with a Cloth-Changing Aware Transformer
Ren, X., Zhang, D., & Bao, X. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892916 · Cited by 6
“A host of works have been proposed to design the transformer componets to reduce quadratic computation complexity [26], or produce multiscale features [1], [7], or learn attentions among local tokens [2], [8].”
BackgroundPooling Transformer for Detection of Risk Events in In-The-Wild Video Ego Data
Mallick, R., Benois‐Pineau, J., Zemmari, A., Yebda, T., Pech, M., Amieva, H., & Middleton, L. E. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956675 · Cited by 6
“In [25], the pooling operation is used to hierarchically expand the channel capacity and pool spatial resolution, similar to CNN.”
BackgroundMethodShah, S. M. A. H., Khan, M. Q., Ghadi, Y. Y., Jan, S. U., Mzoughi, O., & Hamdi, M. (2023) · IEEE Access · doi:10.1109/access.2023.3302253 · Cited by 7
“Before, that, in 2010, Zhai et al. introduced Multiscale Vision Transformers (MViT) for video and image recognition, combining the concept of multiscale feature hierarchies with transformer models [4], [13].”
BackgroundEmre, T., Oghbaie, M., Chakravarty, A., Rivail, A., Riedl, S., Mai, J., Scholl, H. P. N., Sivaprasad, S., Rueckert, D., Lotery, A. J., et al. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44013-7_14 · Cited by 6
“Similar to the LSTM based approaches, hybrid ViTs have been successfully applied to video recognition tasks for memory efficiency and speed.”
InfluentialZhou, X., Huang, Y., Xue, W., Yang, X., Zou, Y., Ying, Q., Zhang, Y., Liu, J., Ren, J., & Ni, D. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43895-0_48 · Cited by 6
Multimodal Design for Interactive Collaborative Problem-Solving Support
VanderHoeven, H., Bradford, M., Jung, C., Khebour, I., Lai, K., Pustejovsky, J. D., Krishnaswamy, N., & Blanchard, N. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-60107-1_6 · Cited by 6
“Gesture Recognition There are many possible solutions to gesture recognition [14, 17, 36, 53], but nearly all suffer from difficulties in recognizing gestures at unusual angles or that may be far from camera, and deep learning approaches come with a high training and data overhead, making them…”
BackgroundSWAT: Spatial Structure Within and Among Tokens
Kahatapitiya, K., & Ryoo, M. S. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/106 · Cited by 10
Multi-scale space-time transformer for driving behavior detection
Jun, G., Yi, J., & Murphey, Y. L. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14499-7 · Cited by 6
“Transformer-based Models MViT-B [13] 37.”
How to Improve Video Analytics with Action Recognition: A Survey
Gayathri, T., & Mamatha, H. R. (2024) · ACM Computing Surveys · doi:10.1145/3679011 · Cited by 8
Geo‐spatial traffic behaviour analysis and anomaly detection for ITS applications
Akdag, E., D’Amicantonio, G., Vijverberg, J. A., Stajan, D., Beers, B. J., de With, P. H. N., & Bondarev, E. (2024) · IET Intelligent Transport Systems · doi:10.1049/itr2.12591 · Cited by 5
Empowering Efficient Spatio-Temporal Learning with a 3D CNN for Pose-Based Action Recognition
Ren, Z., Xiao, X., & Nie, H. (2024) · Sensors · doi:10.3390/s24237682 · Cited by 5
Barbosa, R. Z., & Oliveira, H. S. (2025) · IEEE Access · doi:10.1109/access.2025.3557948 · Cited by 5
“Transformer Models have evolved from basic attention mechanisms [44] to sophisticated architectures combining spatial-temporal local self-attention [45], [46], [47], [48], while other combine self-attention and convolution [49], [50], [51], [52], [53], or 1D temporal attention [54], [55], [56].”
AAT-DA: Accident Anticipation Transformer with Driver Attention
Kumamoto, Y., Ohtani, K., Suzuki, D., Yamataka, M., & Takeda, K. (2025) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw65960.2025.00127 · Cited by 5
“In recent years, following the success of transformers in various fields, numerous models have been proposed that apply transformers to video processing [21–23], demonstrating high performance across various tasks.”
InfluentialEAVFormer: an end-to-end audio and visual emotion recognition network based on transformers
Liu, H., Sun, Z., Li, H., Li, Y., Zhang, W., & Song, T. (2025) · Multimedia Systems · doi:10.1007/s00530-025-01845-y · Cited by 5
“With the increasing adoption of self-attention in the image domain, Vision Transformer [61] and its variants (e.g., MViT [15], Video Swin [44]) have achieved promising results in video analysis by capturing long-range spatiotemporal dependencies [42, 62].”
Self-citationCRxK dataset: a multi-view surveillance video dataset for re-enacted crimes in Korea
An, C., Lee, M., & Park, E. (2025) · Scientific Reports · doi:10.1038/s41598-025-15058-w · Cited by 5
LoKi: Low-dimensional KAN for Efficient Fine-tuning Image Models
Cai, X., Pan, R., & Yang, H. S. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01385 · Cited by 6
“Since KAN includes at least 2 layers of B-Spline functions, which corresponds to 3 layers of neurons, [16 , 33 , 16] corresponds to the minimum number of layers in LoKi’s activation layer.”
Alshalawi, A., Abdul, W., & Muhammad, G. (2025) · Journal of Engineering Research · doi:10.1016/j.jer.2025.08.016 · Cited by 5
Alshaabi, T., Milkie, D. E., Liu, G., Shirazinejad, C. R., Hong, J. L., Achour, K., Görlitz, F., Milunovic-Jevtic, A., Simmons, C., Abuzahriyeh, I. S., et al. (2025) · Nature Methods · doi:10.1038/s41592-025-02844-7 · Cited by 6
“Multiscale (or hierarchical) vision transformers, such as Swin 25 and MViT 26 , are designed with specialized modules (for example, shifted-window partitioning 25 and hybrid window attention 27 ) to excel at a variety of detection tasks for 2D natural images using supervised training on ImageNet 28 .”
Enhancing action recognition by leveraging the hierarchical structure of actions and textual context
Benavent-Lledó, M., Mulero-Pérez, D., Ortiz-Pérez, D., Garcia-Rodriguez, J., & Argyros, A. (2025) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2025.104560 · Cited by 5
“Specifically, using only contextual information for coarse-grained action recognition (1) achieves the best performance, with a top-1 accuracy of 54.95% for fine-grained actions and competitive results for coarse-grained actions.”
InfluentialOuYang, C., Zhong, C., & Li, H. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3628315 · Cited by 5
Lasri, K., Fazazy, K. E., Mahraz, M. A., Tairi, H., & Riffi, J. (2025) · Computation · doi:10.3390/computation13120293 · Cited by 5
“The influence of video-centric transformers has been significant, with Bertasius et al. [33] proposing TimeSformer, Arnab et al. [34] developing ViViT, and Fan et al. [35] introducing MViT.”
Hori, N., & Yamamoto, M. (2023) · Sensors · doi:10.3390/s23094321 · Cited by 5
“proposed a real-time recognition system using the Video Swin transformer [25] and Multiscale Vision Transformers (MViT) [26] models.”
MethodShifted GCN-GAT and Cumulative-Transformer based Social Relation Recognition for Long Videos
Wang, H., Hu, Y., Zhu, Y., Qi, J., & Wu, B. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612175 · Cited by 7
“Motivated by the pyramid-like design of [8], hierarchical architectures empowered the memory mechanism to largely reduce computation costs and provide multi-scale receptive fields.”
BackgroundSTMixer: A One-Stage Sparse Action Detector
Tao, W., Cao, M., Gao, Z., Wu, G., & Wang, L. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3387127 · Cited by 5
“SlowFast [11] and MViT [9] directly use the RoI features for action classification.”
BackgroundMethodLearning Temporal Dynamics in Videos With Image Transformer
Shu, Y., Qiu, Z., Long, F., Yao, T., Ngo, C., & Mei, T. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3383662 · Cited by 7
FETR: Feature Transformer for vehicle-infrastructure cooperative 3D object detection
Yan, W., Cao, H., Chen, J., & Wu, T. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128147 · Cited by 5
Recognizing Video Activities in the Wild via View-to-Scene Joint Learning
Yu, J., Chen, Y., Wang, X., Cheng, X., Ju, Z., & Xu, Y. (2024) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2024.3431128 · Cited by 5
“For scene understanding, most methods learn comprehensive features with fixed-length input while ignoring the correlation between bodies and scenes of interest [5], [7], [22], [23].”
LKSMN: Large Kernel Spatial Modulation Network for Lightweight Image Super-Resolution
Zhang, Y., Xu, L., Xiang, H., Kong, H., Bi, J., & Han, C. (2024) · The Visual Computer · doi:10.1007/s00371-024-03562-5 · Cited by 6
“In addition, we provide a comparison of the reconstruction performance of LKSMN with other networks based on the transformer architecture, such as SwinIR [28] and MViT [11].”
M2Former: Multiscale Patch Selection for Fine-Grained Visual Recognition
Moon, J., & Park, S. (2024) · Applied Sciences · doi:10.3390/app14198710 · Cited by 8
“ViT uses fixed-size image patches throughout the entire network, which ensures that the receptive field remains the same across all layers, and it prevents the ViT from obtaining multi-scale feature representations [14, 38, 63].”
BackgroundMethodInfluentialSIGMA: Sinkhorn-Guided Masked Video Modeling
Salehi, M., Dorkenwald, M., Thoker, F. M., Gavves, E., Snoek, C. G. M., & Asano, Y. M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72691-0_17 · Cited by 7
MCANet: a lightweight action recognition network with multidimensional convolution and attention
Tian, Q., Miao, W., Zhang, L., Yang, Z., Yu, Y., Zhao, Y., & Yao, L. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02454-3 · Cited by 5
“The MVit, proposed by Fan et al. [14], divides data into different scales according to channel size and resolution.”
Scene-Adaptive SVAD Based On Multi-modal Action-Based Feature Extraction
Gao, S., Yang, P., & Huang, L. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0908-6_19 · Cited by 6
“3D CNNs [5] directly learn spatio-temporal features from RGB frames, adding an additional temporal dimension to the conventional 2D CNNs. Transformer-based networks [8, 36] adopt and modify the latest transformer architecture to jointly encode spatial and temporal features.”
Song, X., Tian, Y., Liu, H., Wang, L., & Niu, J. (2025) · Sensors · doi:10.3390/s25030828 · Cited by 6
“The Pyramid Vision Transformer (PVT) [42] and the Multi-Scale Vision Transformer (MViT) [43] implement single-layer pooling to reduce the number of tokens required for the multi-head self-attention mechanism in Transformers.”
MFF‐YOLOv8: Small Object Detection Based on Multi‐Scale Feature Fusion for UAV Remote Sensing Images
Hu, K., Lu, J., Zheng, C., Xiang, Q., & Miao, L. (2025) · IET Image Processing · doi:10.1049/ipr2.70066 · Cited by 5
Anomaly detection in cropland monitoring using multiple view vision transformer
Liu, X., Liu, Y., Sui, H., Qin, C., Che, Y., & Guo, Z. (2025) · Scientific Reports · doi:10.1038/s41598-025-98405-1 · Cited by 5
“Traditionally, transformer models like Swin 18 , MViT 19 , PVT 20 , and PiT 21 did not use ConvNet concepts like convolution and pooling.”
Jiang, H., Hao, X., Huang, Y., Ma, C., Zhang, J., Pan, Y. F., & Zhang, R. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-91721-9_2 · Cited by 6
CycleACR: Cycle Modeling of Actor-Context Relations for Video Action Detection
Chen, L., Zhan, T., Song, Y., Wu, G., & Wang, L. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3595393 · Cited by 7
Research on single-character image classification of Tibetan ancient books based on deep learning
Zhao, Z. (2022) · 2022 3rd International Conference on Computer Vision, Image and Deep Learning & International Conference on Computer Engineering and Applications (CVIDL & ICCEA) · doi:10.1109/cvidliccea56201.2022.9824018 · Cited by 5
I Know How You Move: Explicit Motion Estimation for Human Action Recognition
Shen, Z., Wu, X., Li, H., Xu, T., & Wu, C. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3211423 · Cited by 5
“In particular, Swin Transformer [37], [38] uses various forms of sliding windows to enhance the locality of attention operations, and MViT [39], [40] utilizes spatiotemporal pooling operation to implement multi-scale hierarchical modelling.”
Combining frequency transformer and CNNs for medical image segmentation
Labbihi, I., Meslouhi, O. E., Benaddy, M., Kardouchi, M., & Akhloufi, M. A. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16279-9 · Cited by 5
“Theyusetheattentionoperation tomodelinter-patchesrelationshipsinanimage[4]andthemulti-layerperceptron(MLP) layers[54]tomodelrelationshipsinsideapatch[16].”
COPILOT: Human-Environment Collision Prediction and Localization from Egocentric Videos
Pan, B., Shen, B., Rempe, D., Paschalidou, D., Mo, K., Yang, Y., & Guibas, L. J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00485 · Cited by 5
“Recently, several works [2, 19, 35, 6, 39] have adapted powerful transformer architectures [54] to the video domain.”
BackgroundAutomated Emotional Valence Estimation in Infants with Stochastic and Strided Temporal Sampling
Ning, M., Ertuğrul, I. Ö., Messinger, D. S., Cohn, J. F., & Salah, A. A. (2023) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii59096.2023.10388096 · Cited by 5
MgMViT: Multi-Granularity and Multi-Scale Vision Transformer for Efficient Action Recognition
Huo, H., & Li, B. (2024) · Electronics · doi:10.3390/electronics13050948 · Cited by 5
DTF-AT: Decoupled Time-Frequency Audio Transformer for Event Classification
Alex, T., Ahmed, S. A. A., Mustafa, A., Awais, M., & Jackson, P. J. B. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29716 · Cited by 6
“Additionally, the success of various hybrid vision architectures (Dai et al. 2021; Tu et al. 2022; Fan et al. 2021) serves as a testament to synergistic learning capabilities brought by convolutions and self-attention.”
BackgroundESC-ZSAR: Expanded Semantics from Categories with Cross-Attention for Zero-Shot Action Recognition
Lee, J.-C., & Lee, D.-G. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124786 · Cited by 6
A Survey on Backbones for Deep Video Action Recognition
Tang, Z., Zhao, Y., Wen, Y., & Liu, M. (2024) · IEEE International Conference on Multimedia and Expo workshops · doi:10.1109/icmew63481.2024.10645423 · Cited by 8
“2(d) highlights Fan et al. ’s [13] blend of transformers with CNN’s multi-scale features, leading to MViT.”
BackgroundMethodInfluentialZhang, Z., Yale, Y., Huifang, H., Fanman, M., Fan, Z., Kangzhan, X., & Chunsheng, Z. (2024) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2024.3455356 · Cited by 5
“Recognizing the limitations of CNNs in certain aspects, such as the inability to naturally integrate multimodal features, we have turned to transformer-based models for segmentation tasks [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37].”
Ünal, P., Hatipoglu, O. I., Türker, A. K., Unal, A. F., Deveci, B. U., Kirci, P., & Özbayoğlu, A. M. (2024) · International Transactions on Electrical Energy Systems · doi:10.1155/2024/5548146 · Cited by 5
Kowal, M. D., Wildes, R. P., & Derpanis, K. G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01036 · Cited by 12
“MViT [18] and ViT-b [51].”
MethodVideo Recognition in Portrait Mode
Han, M., Yang, L., Jin, X., Feng, J., Chang, X., & Wang, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02062 · Cited by 5
“MViTv2-S [12] is trained for 200 epochs with a learning rate of 0.0001 and a batch size of 32 × 4.”
BackgroundMethodInfluentialZhu, G., Qin, Z., Xiong, H., Kumari, S., Alenazi, M. J. F., Guo, Y., & Chen, C. (2025) · Information Sciences · doi:10.1016/j.ins.2024.121868 · Cited by 6
A Multiscale Vision Transformer-Based Multimodal GeoAI Model for Mapping Arctic Permafrost Thaw
Li, W., Hsu, C.-Y., Wang, S., Gu, Z., Yang, Y., Rogers, B. M., & Liljedahl, A. (2025) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2025.3564310 · Cited by 6
“This is achieved by increasing the number of channels and decreasing the sequence length (or resolution) at various scaled stages [40].”
The Journey of Action Recognition
Xi, D., & Wang, L. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3717746 · Cited by 6
“More recently, transformers [7, 54, 170, 177] have emerged as a powerful alternative.”
Multi-Scale Memetic Image Registration
COCIANU, C. L., & Uscatu, C. R. (2022) · Electronics · doi:10.3390/electronics11020278 · Cited by 5
“The identification of objects in static or motion images is also accomplished using multi-scale image processing in [13].”
MethodFractional Variation Network for THz Spectrum Denoising without Clean Data
Jiao, Q., Xu, J., Liu, M., Zhao, F., Dong, L., Hui, M., Kong, L., & Zhao, Y. (2022) · Fractal and Fractional · doi:10.3390/fractalfract6050246 · Cited by 5
“Recently, the Vision Transformer [39] has also proved that a pure Transformer architecture can achieve state-ofthe-art performance in the computer vision field when the training data are large enough.”
BackgroundInfluentialMemory Transformers for Full Context and High-Resolution 3D Medical Segmentation
Themyr, L., Rambour, C., Thome, N., Collins, T., & Hostettler, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-21014-3_13 · Cited by 5
SUMLP: A Siamese U-shaped MLP-based network for change detection
Zhang, C., Wang, L., Cheng, S., & Li, Y. (2022) · Applied Soft Computing · doi:10.1016/j.asoc.2022.109766 · Cited by 5
Self-citationWang, H., Li, H., Liu, W., & Gu, X. D. (2023) · Image and Vision Computing · doi:10.1016/j.imavis.2023.104785 · Cited by 5
Vasilescu, M. A. O. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78189-6_27 · Cited by 6
“3There have been a number of related transformer architectures engineered and empirically tested with success [29], [113], [66].”
Transtl: Spatial-Temporal Localization Transformer for Multi-Label Video Classification
Wu, H., Li, M., Liu, Y., Liu, H., Xu, C., & Li, X. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747849 · Cited by 5
Action Detail Matters: Refining Video Recognition with Local Action Queries
Wang, M., Huang, Z., Kong, X., Shen, G., Dai, G., Wang, J., & Liu, Y. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01782 · Cited by 4
“Early Video Trans-formers focused on modifying Transformer architectures for better temporal modeling [2, 7, 10, 19], such as designing temporal input blocks or temporal Attention mechanisms.”
Instance-Adaptive Spatial-Temporal Enhancement for Efficient Video Compression
Zhao, Y. J., Cheng, Z., Li, J., Feng, D., Gu, Q., (22418), Q. W., Lu, G. S., & Song, L. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3602648 · Cited by 4
“Some 3D transformer structures are proposed for video processing [43], [52], [53], [54], [55], [56], [57].”
AgriSwin: A Swin Transformer-Based Deep Learning Approach for Crop Disease Classification
Sunil, C., Gokul, P. G., Sujith, G., Paul, H., & Anjali, T. (2025) · International Conference for Emerging Technology (INCET) · doi:10.1109/incet64471.2025.11139968 · Cited by 4
“All images in the dataset were resized to a uniform dimension of 224 × 224 pixels, which ensures the compatibility with the model’s input requirements [9].”
Behind the Magic, MERLIM: Multi-Modal Evaluation Benchmark for Large Image-Language Models
Villa, A., Alcázar, J. L., Soto, Á., & Ghanem, B. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00054 · Cited by 5
“…advances in computer vision and NLP fields have converged in the development of Large Vision and Language models (LVLMs) [2, 27, 29, 40, 45, 53], which have achieved state-of-the-art performance on several downstream visual tasks while enabling significant improvements in the zero-shot setups.”
Zhang, T., Yuan, X., & Xu, H. (2025) · Journal of Imaging · doi:10.3390/jimaging11100364 · Cited by 4
“Recent transformer-based methods, such as MATIS [16], leverage Multiscale Vision Transformers [17] to extract global temporal consistency information from video sequences for mask classification.”
Bai, H., Tong, W., Duan, C., Zhou, Y., & Dong, S. (2026) · Scientific Reports · doi:10.1038/s41598-026-37914-z · Cited by 4
“Despite its advantages, deep learning is limited by its high cost and low interpretability, restricting its application in industrial settings 33 .”
Revisiting Kernel Temporal Segmentation as an Adaptive Tokenizer for Long-form Video Understanding
Afham, M., Shukla, S. N., Poursaeed, O., Zhang, P., Shah, A., & Lim, S.-N. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00128 · Cited by 4
“Most of the video understanding models are devised to learn the representations of short-form videos ranging from 5 to 10 seconds [10, 37, 19, 43, 9, 3, 22, 26].”
BackgroundMethodFang, S., Lu, X., Huang, Y., Sun, G., & Liu, X. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-18234-8 · Cited by 4
Markov Progressive Framework, a Universal Paradigm for Modeling Long Videos
Pang, B., Gao, P., Li, Y., & Lu, C. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3426998 · Cited by 4
Self-citationBayesian Evidential Deep Learning for Online Action Detection
Guo, H., Wang, H., & Ji, Q. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72640-8_16 · Cited by 4
“To better capture the spatial-temporal dependencies, we adopt the multi-scale vision Transformer [17] to extract RGB features.”
PaPr: Training-Free One-Step Patch Pruning with Lightweight ConvNets for Faster Inference
Mahmud, T., Yaman, B., Liu, C.-H., & Marculescu, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73337-6_7 · Cited by 4
Sparse Refinement for Efficient High-Resolution Semantic Segmentation
Liu, Z., Zhang, Z., Khaki, S., Yang, S. W., Tang, H., Xu, C., Keutzer, K., & Han, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72855-6_7 · Cited by 4
Style Transfer of Chinese Wuhu Iron Paintings Using Hierarchical Visual Transformer
Zhou, Y., Ren, Y., Wu, C., & Xue, M. (2024) · Sensors · doi:10.3390/s24248103 · Cited by 4
MFFOD: multidomain feature fusion object detector for infrared images
Xu, W., Yin, Q., Xu, C., Zhao, Z., Li, Y., & Huang, D. (2025) · Measurement Science and Technology · doi:10.1088/1361-6501/adafd3 · Cited by 4
Self-citationPatera, P., Chen, Y., & Fang, W. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3552895 · Cited by 4
“I, we also include several Transformer-based approaches [54], [55], [56], commonly used for video accident prediction and general video analysis tasks to highlight the advantages of our multi-modal architecture.”
VideoMamba++: Integrating state space model with dual attention for enhanced video understanding
Song, X., Tian, W., Zhu, Q., & Zhang, X. (2025) · Image and Vision Computing · doi:10.1016/j.imavis.2025.105609 · Cited by 4
Combining spatio-temporal attention and multi-level feature fusion for video saliency prediction
Luo, H. (2025) · Image and Vision Computing · doi:10.1016/j.imavis.2025.105678 · Cited by 4
Huang, S., Yang, B., Huang, X., Zhang, H., Luo, D., Tong, G., Wang, Y., Shao, Y., Chen, M., Gao, Q., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01904-8 · Cited by 5
Not all patches are crucial to image recognition: Window patch clustering attention for transformers
Wu, R., Wang, Y., Li, D., & Liu, J. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114647 · Cited by 4
Sequential modeling for cystoscopic image classification
Laurie, M. A., Eminağa, O., Shkolyar, E., Jia, X., Lee, T. K., Long, J., Islam, M. T., Lau, H. D., Xing, L., & Liao, J. C. (2023) · Advanced Photonics in Urology · doi:10.1117/12.2649334 · Cited by 4
Survey on Transformer for image classification
Shi, Z., Li, C., Zhou, L., Zhang, Z., Wu, C., You, Z., & Ren, W. (2023) · Journal of Image and Graphics · doi:10.11834/jig.220799 · Cited by 4
Automated anomalous child repetitive head movement identification through transformer networks
Wedasingha, N., Samarasinghe, P. D., Seneviratne, L., Papandrea, M., Puiatti, A., & Rankin, D. (2023) · Physical and Engineering Sciences in Medicine · doi:10.1007/s13246-023-01309-5 · Cited by 5
“…model that monitors the spatial and temporal features of [ l 𝛽 M ] S i when classifying head movements; Multi-scale Vision Transformer (MiViT) [22], which analyzes spatial resolutions by scaling the video for multiple resolution channels and classifying the head movement; and Video Vision…”
BackgroundMethodInfluentialLocal Compressed Video Stream Learning for Generic Event Boundary Detection
Zhang, L., Gu, X., Li, C., Luo, T., & Fan, H. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01921-8 · Cited by 6
“Transformers were successfully applied in computer vision [9], the new trend in video understanding is using the Transformers, including [2, 10, 11, 33, 67], which achieve competitive results.”
Self-citationDeep Learning-Based Knee MRI Classification for Common Peroneal Nerve Palsy with Foot Drop
Chung, K. M., Yu, H., Kim, J. H., Lee, J. J., Sohn, J., Lee, S., Sung, J. H., Han, S., Yang, J. S., & Kim, C. (2023) · Biomedicines · doi:10.3390/biomedicines11123171 · Cited by 4
“One of the reasons for this is that EfficientNet is a model considering multi-scale parameters [35].”
BackgroundVOCALExplore: Pay-as-You-Go Video Data Exploration and Model Building
Daum, M., Zhang, E., He, D. J., Mussmann, S., Haynes, B. M., Krishna, R., & Bałazińska, M. (2023) · Proceedings of the VLDB Endowment · doi:10.14778/3625054.3625057 · Cited by 4
Jointly Modeling Spatio-Temporal Features of Tactile Signals for Action Classification
Lin, J., Li, J., Gao, J., Ma, W., & Liu, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i12.29288 · Cited by 4
VLG: General Video Recognition with Web Textual Knowledge
Lin, J., Liu, Z., Wang, W., Wu, W., & Wang, L. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02081-z · Cited by 5
“Recently, several works (Arnab et al., 2021; Bertasius et al., 2021; Neimark et al., 2021; Fan et al., 2021) try to employ and adapt strong vision transformers to encode the spatial and temporal features jointly.”
Liu, S., Wang, F., You, H., Jiao, N., Zhou, G., & Zhang, T. (2024) · Remote Sensing · doi:10.3390/rs16132472 · Cited by 4
Compositional Video Understanding with Spatiotemporal Structure-based Transformers
Yun, H., Ahn, J., Kim, M., & Kim, E. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01774 · Cited by 4
“Recently, transformers [33], which are powerful in learning long-range temporal dependencies, have been applied because the ConvNet-based models do not have good temporal dependencies on long-term videos [1, 6].”
BackgroundAbnormal Activity Detection Using Video Action Recognition: A Review
Ojha, A., Liu, Y., & Hao, Y. (2024) · doi:10.1109/icnlp60986.2024.10692366 · Cited by 5
“When applied to videos [56][57], they meticulously analyze each frame's content, making them quite thorough in examining spatial details.”
Method
Metadata
- Authors
- Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., & Feichtenhofer, C.
- Affiliations
- Meta (Israel) (IL)
- Meta (United States) (US)
- Venue
- 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2021-10
- Pages
- 6804-6815
- OpenAlex
- W3202698164
- Cited by
- 1,407
- References
- 111
- Citations by year
- 2026: 147 · 2025: 345 · 2024: 380 · 2023: 349 · 2022: 176 · 2021: 10
- FWCI
- 74.81
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University)
- TIB Data Manager
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- video recognition, image classification, feature pyramid, spatial resolution reduction, computational efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
