The DOI matches its Crossref record; 106 of the 111 references it cites were found in a registry or an index.
Multiscale Vision Transformers
Open accessAbstract
We present Multiscale Vision Transformers (MViT) for video and image recognition, by connecting the seminal idea of multiscale feature hierarchies with transformer models. Multiscale Transformers have several channel-resolution scale stages. Starting from the input resolution and a small channel dimension, the stages hierarchically expand the channel capacity while reducing the spatial resolution. This creates a multiscale pyramid of features with early layers operating at high spatial resolution to model simple low-level visual information, and deeper layers at spatially coarse, but complex, high-dimensional features. We evaluate this fundamental architectural prior for modeling the dense nature of visual signals for a variety of video recognition tasks where it outperforms concurrent vision transformers that rely on large scale external pre-training and are 5-10× more costly in computation and parameters. We further remove the temporal dimension and apply our model for image classification where it outperforms prior work on vision transformers. Code is available at: https://github.com/facebookresearch/SlowFast.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“As in original ViT [25], we use residual connections [45] and Layer Normalization (LN) [2] in the pre-normalization configuration that applies LN at the beginning of the residual function, and our MLPs consist of two linear layers with GELU activation [48], where the first layer expands the dimension from D to 4D, and…”
MethodResultVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 6×
“In contrast to original Multi Head Attention (MHA) operators [98], where the channel dimension and the spatiotemporal resolution remains fixed, MHPA pools the sequence of latent tensors to reduce the sequence length (resolution) of the attended input.”
BackgroundMethodInfluentialVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“Incorporating downsampling, shift invariance, and shared weights, ConvNets are de-facto standard backbones for computer vision tasks for image [65, 62, 88, 90, 46, 12, 15, 34, 93, 81, 41] and video [87, 31, 11, 79, 69, 106, 96, 30, 105, 35, 29, 117, 57].”
BackgroundImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“Incorporating downsampling, shift invariance, and shared weights, ConvNets are de-facto standard backbones for computer vision tasks for image [70, 67, 94, 96, 51, 15, 18, 39, 99, 87, 46] and video [93, 36, 14, 85, 74, 112, 102, 34, 111, 40, 33, 123, 62].”
BackgroundFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · cited 1×
“Finally, compared to the best model of concurrent Swin [79] (which was designed for image recognition), MViT has +0.”
ResultRethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Research works printed but not linked: 11
- gao, 2019, Res2net: A new multi-scale backbone architecture, IEEE PAMI
- fan, 2020, Pyslowfast
- feichtenhofer, 2019, SlowFast networks for video recognition in ActivityNet challenge 2019
- 10.1109/CVPR.2009.5206848 · DOI · Paper page
- 10.1109/CVPR.2019.00033 · DOI
- hang, 2020, ResNeSt Split-Attention Networks
- 10.1109/ICCV.2019.00718 · DOI
- 10.1109/CVPR.2015.7298594 · DOI · Paper page
- carion, 2020, Endto-end object detection with transformers, Proc ECCV, 213
- loshchilov, 2018, Fixing weight decay regularization in adam
- qiu, 2017, Learning spatiotemporal representation with pseudo-3d residual networks, Proc ICCV
Cited by
Sorted by citations
Items 501–600 of 1,003
Ai, S., Song, J., & Cai, G. (2022) · Mathematics · doi:10.3390/math10101733 · Cited by 11
A generalizable deep learning system for cardiac MRI
Shad, R., Zakka, C., Kaur, D., Mathur, M., Fong, R., Cho, J., Filice, R. W., Mongan, J., Kallianos, K., Khandwala, N., et al. (2026) · Nature Biomedical Engineering · doi:10.1038/s41551-026-01637-3 · Cited by 16
“While pre-training itself is computationally expensive, fine tuning on each task can be achieved within a few hours on consumer grade GPUs.”
Movies2Scenes: Using Movie Metadata to Learn Scene Representation
Chen, S., Liu, C.-H., Hao, X., Nie, X., Arap, M., & Hamid, R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00632 · Cited by 16
Comprehensive Transformer-Based Model Architecture for Real-World Storm Prediction
Lin, F., Yuan, X., Zhang, Y., Sigdel, P., Chen, L., Peng, L., & Tzeng, N.-F. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43430-3_4 · Cited by 12
“Some studies, including MViT [13, 30], PiT [50], PVT [19], among many others [5, 11, 29, 53, 28, 57, 56, 54], also address the limitations of original ViT for better performance on vision tasks.”
BackgroundMulti-Task Learning of Object States and State-Modifying Actions From Web Videos
Souček, T., Alayrac, J.-B., Miech, A., Laptev, I., & Šivic, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3362288 · Cited by 17
“Suchmodelsemployamixof2Dand3D convolutions[12],[28],[29],[82]ortransformersandtempo-ralattention[7],[26],[91].”
Guo, W., Li, S., Chen, F., Sun, Y., & Gu, Y. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3374100 · Cited by 12
“Multiscale Vision Transformer (MViT) [40], is a transformer architecture for video recognition.”
BackgroundPGVT: Pose-Guided Video Transformer for Fine-Grained Action Recognition
Zhang, H., Leong, M. C., Li, L., & Lin, W. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00651 · Cited by 13
CMC: Video Transformer Acceleration via CODEC Assisted Matrix Condensing
Song, Z., Qi, C., Liu, F., Jing, N., & Liang, X. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620665.3640393 · Cited by 15
A Video Mosaicing-Based Sensing Method for Chicken Behavior Recognition on Edge Computing Devices
Teterja, D., García‐Rodríguez, J., Azorín-López, J., Sebastián‐González, E., Nedić, D., Leković, D., Knežević, P., Drajic, D. D., & Vukobratović, D. (2024) · Sensors · doi:10.3390/s24113409 · Cited by 12
Toward Long Form Audio-Visual Video Understanding
Hou, W., Li, G., Tian, Y., & Hu, D. (2024) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3672079 · Cited by 14
“To validate the superiority of our proposed framework, we choose 13 related methods for comparison, including weakly supervised temporal action localization methods: STPN [35], RSKP [23]; long sequence modeling methods: Longformer [3], Transformer-LS [71], ActionFormer [66]; audio-visual learning methods: AVE [49], AV…”
A Fusion Deep Learning Model of ResNet and Vision Transformer for 3D CT Images
Liu, C., & Sun, C. (2024) · IEEE Access · doi:10.1109/access.2024.3423689 · Cited by 12
“To evaluate the classification performance of the ‘‘Reswin’’ model, we compared it with the current main-stream 3D data processing models, including MViT [29], R2 + 1D-50 [30], SlowFast-50 [31], X3D [32], and CSN101 [33], and unfused individual models.”
MethodHe, K., Li, L., Zhou, J. Y., Gou, F., & Wu, J. (2024) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2024.109493 · Cited by 12
Selective, Interpretable and Motion Consistent Privacy Attribute Obfuscation for Action Recognition
Ilic, F., Zhao, H., Pock, T., & Wildes, R. P. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01772 · Cited by 12
Ntinoutl, I., Sanchez, E., & Tzimiropoulos, G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01781 · Cited by 12
“Two-stage spatio-temporal action localization: Most existing works on action detection [6–8,25,36,39,41–43,46] depend on a supplementary person detector for actor localization.”
BackgroundMethodSportsHHI: A Dataset for Human-Human Interaction Detection in Sports Videos
Wu, T., He, R., Wu, G., & Wang, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01754 · Cited by 14
“Remarkable progress has been made in the action recognition task [3, 4, 13, 22, 23, 39, 40] with the advances of video convolution neural networks [3, 11, 45, 46, 49, 50, 52, 55] and video transformers [1, 2, 10, 29, 44, 51, 58].”
MuST: Multi-scale Transformers for Surgical Phase Recognition
Pérez, A., Rodríguez, S., Ayobi, N., Aparicio, N., Dessevres, E., & Arbeláez, P. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72089-5_40 · Cited by 12
ZeroI2V: Zero-Cost Adaptation of Pre-trained Transformers from Image to Video
Li, X., Zhu, Y., & Wang, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73010-8_25 · Cited by 20
Shen, G., Ouyang, Y., Lu, J., Yang, Y., & Sánchez, V. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3512369 · Cited by 15
GCN-Based Multi-Modality Fusion Network for Action Recognition
Liu, S., Wang, X., Xiong, R., & Xiaopeng Fan, B. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3521749 · Cited by 12
“Action recognition on RGB videos has been explored extensively in the past decades, from hand-crafted models [16], [17], to more recent deep learning based approaches [18], [19].”
Transformer-Based Multi-Player Tracking and Skill Recognition Framework for Volleyball Analytics
Jiang, L., Yang, Z., & Gang, L. (2025) · IEEE Access · doi:10.1109/access.2025.3526775 · Cited by 12
PitVis-2023 challenge: Workflow recognition in videos of endoscopic pituitary surgery
Das, A., Khan, D. Z., Psychogyios, D., Zhang, Y., HANRAHAN, J. G., Vasconcelos, F., Pang, Y., Chen, Z., Wu, J., Zou, X., et al. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103716 · Cited by 12
Reduction of Class Activation Uncertainty With Background Information
Kabir, H. M. D. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3570282 · Cited by 27
“SOTA performances with significant improvement over traditional CNNs [59], [60].”
InfluentialAction Probability Calibration for Efficient Naturalistic Driving Action Localization
Li, R., Wu, C., Li, L., Shen, Z., Xu, T., Wu, X., Li, X., Lu, J., & Kittler, J. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00555 · Cited by 12
“Recently proposed transformer-based methods split videos into 3D tokens and utilize transformer layer to extract the spatial-temporal features [1, 2, 8, 21].”
MethodScene adaptive mechanism for action recognition
Wu, C., Wu, X., Wu, X., & Kittler, J. (2023) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2023.103854 · Cited by 13
LIP: Local Importance-Based Pooling
Gao, Z., Wang, L., & Wu, G. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01707-4 · Cited by 15
“2019) and vision transformers (Liu et al. 2021; Wang et al. 2021; Fan et al. 2021) show that the query-key attentional operators popular in the NLP area (Vaswani et al.”
BackgroundHAR-time: human action recognition with time factor analysis on worker operating time
Yang, C.-L., Hsu, S.-C., Hsu, Y.-W., & Kang, Y.-C. (2023) · International Journal of Computer Integrated Manufacturing · doi:10.1080/0951192x.2023.2177736 · Cited by 11
“Multiscale Vision Transformers (Fan et al. 2021) Multiscale Vision Transformers Kinetics 400 Kinetics 600 2020 M.”
BackgroundImproved deep learning image classification algorithm based on Swin Transformer V2
Wei, J., Chen, J.-R., Wang, Y., Luo, H., & Li, W. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1665 · Cited by 13
“MViTs ( Fan et al., 2021 ; Li et al., 2022a ) performed pooling operations on the Q, K, and V components of the attention mechanism in Transformers to reduce the number of tokens, signi fi cantly reducing the computational overhead of the multi-scale Transformer architecture.”
MethodMultimodal action recognition: a comprehensive survey on temporal modeling
Shabaninia, E., Nezamabadi–pour, H., & Shafizadegan, F. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17345-y · Cited by 31
Music-evoked emotions classification using vision transformer in EEG signals
Wang, D., Lian, J., Cheng, H., & Zhou, Y. (2024) · Frontiers in Psychology · doi:10.3389/fpsyg.2024.1275142 · Cited by 11
Sun, W., Zhang, J., Lei, Y., & Hong, D. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3404922 · Cited by 11
“Current state-of-the-art general semantic segmentation methods mainly address multiscale variations in target objects by constructing multiscale feature representations [10], [11].”
BackgroundKim, H., Jeon, H., Kim, D., & Kim, J. (2024) · Sustainable Cities and Society · doi:10.1016/j.scs.2024.105793 · Cited by 11
Enhanced Motion-Text Alignment for Image-to-Video Transfer Learning
Zhang, W., Wan, C., Liu, T., Tian, X., Shen, X., & Ye, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01751 · Cited by 13
“‘VL’ is the abbreviation of ‘vision-language’. networks [7, 17, 21, 59, 64, 67–69, 71, 73], and vision trans-formers [1, 2, 6, 16, 32, 33, 38, 42, 48, 54, 55, 62].”
Hernandez, J., Villegas, R., & Ordóñez, V. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73235-5_25 · Cited by 15
“It also exceeds MViTv1 [23], TimeSformer [7], and ViViT [2] by margins up to 7 .”
BackgroundLiao, H., Li, Y., Wang, C., Guan, Y., Tam, K., Tian, C., Li, L., Xu, C., & Li, Z. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681326 · Cited by 14
“Other enhancements include the adoption of attention mechanisms [2, 22–24, However,existing frameworks for accident anticipation and object detection often operate independently.”
Self-citationReSpike: residual frames-based hybrid spiking neural networks for efficient action recognition
Xiao, S., Li, Y., Kim, Y., Lee, D., & Panda, P. (2025) · Neuromorphic Computing and Engineering · doi:10.1088/2634-4386/adb070 · Cited by 16
Self-citationContinuous sign language recognition using intra-inter gloss attention
Ranjbar, H., & Taheri, A. (2025) · Multimedia Tools and Applications · doi:10.1007/s11042-025-20721-5 · Cited by 18
“In the vanilla MSA, relations between query-key pairs are calculated among all embedded vectors, incurring a high computational cost.”
M2DETR: A Multi-band Multi-scale Detection Transformer for Pest Detection
Tan, W., Zhang, L., Huang, Y., Peng, K., & Qu, Y. (2025) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2025.110325 · Cited by 14
Long-term Leap Attention, Short-term Periodic Shift for Video Classification
Zhang, H., Cheng, L., Hao, Y., & Ngo, C. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3547908 · Cited by 13
“%) and MViT (81.2%).”
BackgroundInfluentialResidual spatiotemporal convolutional networks for face anti-spoofing
da Silva, V. L., Lérida, J. L., Sarret, M., Valls, M., & Giné, F. (2023) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2022.103744 · Cited by 11
3D Deformable Convolution Temporal Reasoning network for action recognition
Ou, Y., & Chen, Z. (2023) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2023.103804 · Cited by 11
DifFAR: Differentiable Frequency-based Disentanglement for Aerial Video Action Recognition
Kothandaraman, D., Lin, M. C., & Manocha, D. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10160271 · Cited by 11
“We compare against 3D CNNs (I3D [4], X3D [3]), transformer models (MVIT [51]) and frequency-based methods (FNet [52], FAR [15]).”
MethodLi, R., Chen, T., Liu, Y., & Jiang, H. (2023) · International Journal of Remote Sensing · doi:10.1080/01431161.2023.2255353 · Cited by 13
“Compared to models such as TransUNet, Deeplabv3+, and PVT, CoupleUNet has a lower parameter scale, but higher than UNet, Swin Transformer, and RoadFormer.”
MethodInfluentialYang, F., Xue, Q., Cao, Y., Li, X., Zhang, W., & Li, G. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.123081 · Cited by 11
Learnable Cube-based Video Encryption for Privacy-Preserving Action Recognition
Ishikawa, Y., Kondo, M., & Kataoka, H. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00685 · Cited by 12
“ViT uses joint space-time attention [3, 35] alongside joint space-time cube embedding [3,12,35], which treats a cube of size 2 × 16 × 16 as a single token embedding.”
MethodAutomated Grooming Detection of Mouse by Three-Dimensional Convolutional Neural Network
Sakamoto, N., Kobayashi, K., Yamamoto, T., Masuko, S., Yamamoto, M., & Murata, T. (2022) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2022.797860 · Cited by 11
“As deep learning methods have been rapidly evolved, many architectures have been continuously proposed (Qiu et al., 2017; Feichtenhofer et al., 2019; Fan et al., 2021).”
BackgroundZhou, Y., & Lian, J. (2023) · Frontiers in Neuroscience · doi:10.3389/fnins.2023.1188696 · Cited by 10
“, 2017) used for natural language processing and the vision transformers (He et al., 2017; Chen et al., 2019; Zhou et al., 2019; Dosovitskiy et al., 2020; Wu et al., 2020; Fan et al., 2021; Heo et al., 2021; Liu et al., 2021; Wang et al., 2021) designed for image analysis, our method is primarily exploited to classify…”
MethodBoosting Breast Ultrasound Video Classification by the Guidance of Keyframe Feature Centers
Sun, A., Zhang, Z., Lei, M., Dai, Y., Wang, D., & Wang, L. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43904-9_43 · Cited by 11
A lightweight vision transformer with symmetric modules for vision tasks
Liang, S., Yu, M., Lu, W., Ji, X., Tang, X., Liu, X., & You, R. (2023) · Intelligent Data Analysis · doi:10.3233/ida-227205 · Cited by 10
“Then, many scholars designed enhanced ViT models [17,50] and achieved comparable performance to convolutional neural networks.”
A Multi-Scale Video Longformer Network for Action Recognition
Chen, C., Zhang, C., & Dong, X. (2024) · Applied Sciences · doi:10.3390/app14031061 · Cited by 10
Wang, Y., Ye, Z., Wen, M., Liang, H., & Zhang, X. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103130 · Cited by 10
Pyramid transformer-based triplet hashing for robust visual place recognition
Li, Z., & Xu, P. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104167 · Cited by 10
Han, J., Zhao, J., Yue, Y., & Che, X. (2024) · IEEE Access · doi:10.1109/access.2024.3475372 · Cited by 10
“And for video recognition task, some state-of-the-art methods are also compared, such as multiscale vision transformers (MVT) [32], ViT-Shift [33], TRX-IDP [34], X3D-XXL [35], R(2+1)D [36], CorrNet-101 [37], K-MCN [38].”
MethodLiao, H., Sun, H., Shen, H., Wang, C., Tian, C., Tam, K., Li, L., Xu, C., & Li, Z. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680672 · Cited by 10
Zhang, K., Wang, T., Xu, L., Yang, L., Thé, J. V. G., & Yu, H. (2024) · Powder Technology · doi:10.1016/j.powtec.2024.120558 · Cited by 10
STAN: Spatio-Temporal Analysis Network for efficient video action recognition
Chen, S., Wang, X., Sun, Y., & Yang, K. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126255 · Cited by 10
Personalizing Federated Instrument Segmentation With Visual Trait Priors in Robotic Surgery
Xu, J., Wang, J., Yu, L., Stoyanov, D. V., Jin, Y., & Mazomenos, E. B. (2025) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2025.3526667 · Cited by 10
“MATIS [27] incorporates long-term temporal information using MViT [28] with Mask2Former [29].”
Gao, J. G., Lin, S., Wang, S., Kou, Y., Li, Z., Li, L., Zhang, C., Zhang, X., Wang, Y., & Hu, W. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-024-02327-w · Cited by 12
“The average attention entropy for one head reveals the concentration of the attention distribution, and a lower entropy indicates that on average each token attends to fewer tokens.”
MVTN: A Multiscale Video Transformer Network for Hand Gesture Recognition
Garg, M., Ghosh, D., & Pradhan, P. M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-92089-9_2 · Cited by 13
Multi-Scale Transformers with dual attention and adaptive masking for sequential recommendation
Li, H., Yang, Y., Zeng, J., Gao, M., & Wen, J. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104318 · Cited by 10
PromptonomyViT: Multi-Task Prompt Learning Improves Video Transformers using Synthetic Scene Data
Herzig, R., Abramovich, O., Avraham, E. B., Arbelle, A., Karlinsky, L., Shamir, A., Darrell, T., & Globerson, A. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00666 · Cited by 10
“Our approach employs Vision Transformers (ViT) [25], which have recently emerged as the leading model for many vision applications [2, 13, 27], including for video understanding [5,40,66,112].”
BackgroundMethodInfluentialMulti-Granular Transformer for Motion Prediction with LiDAR
Gan, Y., Xiao, H., Zhao, Y., Zhang, E., Huang, Z., Ye, X., & Ge, L. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610704 · Cited by 10
“InceptionNet [26] and FPN [7] have shown great success in image classification tasks by incorporating multi-granularity techniques into CNN models, while MViT [29] proves the effectiveness of multi-granularity representation in Transformer-based models.”
BackgroundLook, Listen, and Attack: Backdoor Attacks Against Video Action Recognition
Hammoud, H. A. A. K., Liu, S., Alkhrashi, M., AlBalawi, F., & Ghanem, B. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00348 · Cited by 15
Phan, L., Nguyen, H. T. H., Warrier, H., & Gupta, Y. K. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-26284-5_13 · Cited by 10
Bi-calibration Networks for Weakly-Supervised Video Representation Learning
Long, F., Yao, T., Qiu, Z., Tian, X., Luo, J., & Mei, T. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01779-w · Cited by 10
“, 2021) explores five different variants of space-time attention and MViT (Fan et al., 2021) provides an alternative to formulate video Transformer in a multi-scale manner.”
BackgroundExpediting Large-Scale Vision Transformer for Dense Prediction Without Fine-Tuning
Yuan, Y., Liang, W., Ding, H., Liang, Z., Zhang, C., & Hu, H. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3327511 · Cited by 14
“…object detection and semantic segmentation, more and more efforts have explored different advanced vision trans-former architecture designs [69], [79], [80], [81], [82], [83], [84], [85], [86], [87], [88], [89], [90], [91], [92], [93], [94], [95], [96] suitable for various dense prediction tasks.”
Blanc, C., Ahar, A., & Grave, K. D. (2023) · Additive Manufacturing Letters · doi:10.1016/j.addlet.2023.100161 · Cited by 9
“Despite the recent breakthroughs of Transformers in computer vision applications, the evaluated MViT [41] and Swin3D [42] performed worst on the benchmark.”
Sarraf, S., & Kabia, M. (2023) · Machine Learning and Knowledge Extraction · doi:10.3390/make5040067 · Cited by 9
Data-Efficient Masked Video Modeling for Self-supervised Action Recognition
Li, Q., Huang, X., Wan, Z., Hu, L., Wu, S., Zhang, J., Shan, S., & Wang, Z. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612496 · Cited by 13
Spatio-Temporal Convolution-Attention Video Network
Diba, A., Sharma, V., Arzani, M. M., & Gool, L. V. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00093 · Cited by 9
“…[15] 80.4 94.6 5.82 SlowFast R101-NL [17] 79.8 93.9 7.02 MFormer-HR [47] 81 X3D-XL [15] 81.9 95.5 TimeSformer-L [3] 82.2 95.6 MFormer-HR [47] 82.7 96.1 SlowFast R101-NL [17] 81.8 95.1 ViViT-L FE [2] 82.9 94.6 MViT-B [14] 83.8 96.3 MoViNet-A6 [36] 83.5 - MTV-B [70] 83.6 96.1 Uniformer-B [39] 84.”
InfluentialPosMLP-Video: Spatial and Temporal Relative Position Encoding for Efficient Video Recognition
Hao, Y., Zhou, D., Wang, Z., Ngo, C., He, X., & Wang, M. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02154-z · Cited by 9
SICFormer: A 3D-Swin Transformer for Sea Ice Concentration Prediction
Jiang, Z., Guo, B., Zhao, H., Jiang, Y., & Sun, Y. H. (2024) · Journal of Marine Science and Engineering · doi:10.3390/jmse12081424 · Cited by 9
“The third kind is CNN-ViTs-CNN ViTs, referring to various types of attention mechanisms that have evolved on the basis of Vision Transformers [33], including the work of ViViT, TimesFormer, and MViT [34–36].”
TrVLR: A Transformer-Based Vehicle Light Recognition Method in Vehicle Inspection
Zhou, J., Yang, J., Wu, X., Zhou, W., & Wang, Y. (2024) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2024.3447586 · Cited by 9
“Similarly, the MViT [48] introduces multi-head pooling attention with specific spatial-temporal resolution and achieves encouraging performance.”
Xu, G., Yin, J., Zhang, S., & Gong, M. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112453 · Cited by 9
Acoustic source localization by deep-learning attention-based modulation of microphone array data
Kocur, G. K., Thaler, D., & Markert, B. A. (2024) · NDT & E International · doi:10.1016/j.ndteint.2024.103233 · Cited by 9
Lee, S., Lai, B., Ryan, F., Boote, B., & Rehg, J. M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01382 · Cited by 11
“We compare our proposed base-lines with the recent multimodal model [28] ( i.e., Language Model + MViT [15]) for social deduction games.”
Making Vision Transformers Truly Shift-Equivariant
Rojas-Gómez, R. A., Lim, T.-Y., Do, M. N., & Yeh, R. A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00532 · Cited by 17
“Seminal works include: ViT [13], which treats an image as 16 × 16 tokens; Swin [28, 29], which introduces locally shifted windows to the attention mechanism; CvT [49], which introduces convolutional layers to ViTs; and MViT [14, 24], which introduces a multi-scale pyramid Figure 1.”
BackgroundMethodAdaptive Multiscale Slimming Network Learning for Remote Sensing Image Feature Extraction
Ye, D., Peng, J., Guo, W., & Li, H. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3490666 · Cited by 9
“They can capture rich local feature scale information due to the direct expansion of the CNN perceptual field [8], [12], [13], [29], [30], [31], [32].”
P2TC: A Lightweight Pyramid Pooling Transformer-CNN Network for Accurate 3D Whole Heart Segmentation
Cui, H., Wang, Y., Zheng, F., Li, Y., Zhang, Y., & Xia, Y. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3526727 · Cited by 9
“Other researchworkshaveproposedincorporatingpyramidpooling intothevisualTransformerstructure[28],[29],[30].”
Self-citationHSPFormer: Hierarchical Spatial Perception Transformer for Semantic Segmentation
Chen, S., Han, T., Zhang, C., Su, J., Wang, R., Chen, Y., Wang, Z., & Cai, G. (2025) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2025.3525542 · Cited by 11
“Sub-sequently, there are many methods to introduce the pyramid structure in CNNs to the design of Transformer backbones [18], [31], [32], [33], [34], [35].”
Qin, J., Chen, S., Ye, Z., Liu, J., & Liu, Z. (2025) · PLoS ONE · doi:10.1371/journal.pone.0327717 · Cited by 9
“MViT [28] further balanced local-global information via hierarchical multiscale tokens, attaining 81.2% Kinetics-400 accuracy, though its fixed-scale tokenization struggled with abrupt actions like sneezing (73.1%).”
InfluentialLi, L., Wu, Z., Liu, J., Wang, L., Jin, Y., Jiang, P., Feng, J. Q., & Wu, M. (2022) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm55620.2022.9994966 · Cited by 9
“Recently, in order to model the long-range dependencies of image data, a novel ViTs architecture [9] has been proposed, and subsequently various variants [10]–[15] of ViT have been introduced to computer vision tasks.”
BackgroundDual Attention Poser: Dual Path Body Tracking Based on Attention
Di, X., Dai, X., Zhang, X., & Chen, X. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00280 · Cited by 9
“Due to the success of transformer in the vision tasks, attention modules are widely used in the vision tasks and the performance is improved significantly in the following tasks particularly, including image classification [7, 8, 19], detection [4, 31, 45], image restoration [16, 37, 42].”
BackgroundZhang, Y., Bai, Y., Liu, C., Wang, H., Li, S., & Fu, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01012 · Cited by 9
“3) Transformer networks: based on Vision Transformers [4, 19], many approaches [5,16,20] have been proposed recently for spatial-temporal learning and shown powerful performance.”
BackgroundMei, Z., Ye, P., Li, B., Chen, T., Fan, J., & Ouyang, W. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3315872 · Cited by 9
“Extensive research efforts have led to the development of versatile vision Transformer backbones [43], self-attention variants [20], multiscale feature hierarchies [8], and positional encodings [4], specifically tailored to address the unique challenges in computer vision.”
BackgroundXu, B., Zheng, S., & Jin, Q. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612484 · Cited by 13
“We initialize our POV using an MViT-S model [10] that has been trained on Kinetics [2].”
MethodEgoPCA: A New Framework for Egocentric Hand-Object Interaction Understanding
Xu, Y., Li, Y., Huang, Z., Liu, M. X., Lu, C., Tai, Y., & Tang, C. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00486 · Cited by 9
“Recently, deep learning based methods [50, 12, 4], especially Transformers and visual-language models [11, 1, 2, 51] have achieved high performances on these benchmarks.”
BackgroundMethodInfluentialSelf-citationViLP: Knowledge Exploration using Vision, Language, and Pose Embeddings for Video Action Recognition
Chaudhuri, S., & Bhattacharya, S. (2023) · Indian Conference on Computer Vision, Graphics and Image Processing · doi:10.1145/3627631.3627637 · Cited by 9
“Recently, with the introduction of Vision Trans-former [11, 17, 24] for image recognition, a plethora of transformers are emerging in the video domain as well, such as VideoSwin [26], TimeSFormer [3], ViViT [1] and MViT [13].”
BackgroundSparsifying Transformer Models with Trainable Representation Pooling
Pietruszka, M., Borchmann, Ł., & Garncarek, Ł. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.590 · Cited by 10
“These, similarly to our Pyramidion model, introduce the bottleneck gradually along the encoding process of videos and images, leading to better results, and complexity (Fan et al., 2021).”
MethodLin, C.-W., Chen, Z., & Lin, M. (2022) · Ecological Indicators · doi:10.1016/j.ecolind.2022.109141 · Cited by 9
Automatic pancreas anatomical part detection in endoscopic ultrasound videos
Fleurentin, A., Mazellier, J.-P., Meyer, A., Montanelli, J., Swanström, L. L., Gallix, B., Valencia, L. S., & Padoy, N. (2022) · Computer Methods in Biomechanics and Biomedical Engineering Imaging & Visualization · doi:10.1080/21681163.2022.2154274 · Cited by 9
“…(Touvron et al. 2021), ConvVit (S, T) (D’Ascoli et al. 2021), Twins (S) (Chu et al. 2021), XCiT (S) (Ali et al. 2021), BEiT (Bao et al. 2021), ResNet 3D (Karpathy et al. 2014), ResNet2d+1 (Tran et al. 2018), SlowFast (Feichtenhofer et al. 2019), X3D (Feichtenhofer 2020) and MViT (Fan et al. 2021).”
BackgroundMethodInfluentialSpace or time for video classification transformers
Wu, X., Tao, C., Zhang, J., Sun, Q., Wang, J., Li, W., Liu, Y., & Guo, Y. (2023) · Applied Intelligence · doi:10.1007/s10489-023-04756-5 · Cited by 8
“Zhang et al. [30] [31, 32], or the way self-attention mechanisms are computed [33, 34], but few explore the importance of temporal and spatial attention.”
BackgroundSun, B., Ye, X., Wang, Z., Li, H., & Wang, Z. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612206 · Cited by 8
“Our proposed C2F-ALIN shows a strong recognition performance: its 90.6% top-1 accuracy on FineGym99 and 84.5% top-1 accuracy on FineGym288 clearly surpasses the state-of-the-art results (MDCN [32]) by +0.7%/+1.4%, with less computational cost than full transformer-based methods (ViViT [2], MViT [8] and UniformerV2 [21…”
BackgroundMethodInfluentialVideo-based face outline recognition
Dong, X., Yang, J., Teoh, A. B. J., Yu, D., Li, X., & Jin, Z. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110482 · Cited by 8
A*: Atrous Spatial Temporal Action Recognition for Real Time Applications
Kim, M., Spinola, F., Benz, J. P., & Kim, T. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00686 · Cited by 9
“Given its success in image understanding tasks, several approaches using the Transformer architecture for video understanding [1, 10, 33] and video action classification [27,59] For the first set of networks (detector-based networks) [10,11,13,48], a Faster-RCNN [39] network trained on the COCO [30]…”
MethodScalableTrack: Scalable One-Stream Tracking via Alternating Learning
Liu, H., Cai, Y., Fan, B., & Xu, J. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3402994 · Cited by 8
“However, Transformer-based backbone networks [5], [6], [7] have demonstrated that dimensionally scalable features bring more expressive features, leading to improved performance in various tasks.”
Tu, P., Ye, H., Shi, H., Young, J., Xie, M., Zhao, P., Zheng, C., Jiang, X., & Chen, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102604 · Cited by 12
Zhao, R., Heng, Y., Wang, H., Gao, Y., Liu, S., Yao, C., Chen, J., & Cai, W. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610087 · Cited by 8
“Analogous to 2D image processing, we employ a Vision Transformer (ViT) [24], [25] with a multi-scale receptive field to learn discriminative BEV feature maps F bev ∈ R L bev × W bev × C bev where L bev and W bev denote the size of the BEV feature map and C bev denotes the feature channel dimension.”
MethodCap4Video++: Enhancing Video Understanding With Auxiliary Captions
Wu, W., Wang, X., Luo, H., Wang, J., Yang, Y., & Ouyang, W. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3410329 · Cited by 9
From Global to Hybrid: A Review of Supervised Deep Learning for 2-D Image Feature Representation
Dong, X., Wang, Q., Deng, H., Yang, Z., Ruan, W., Liu, W., Liang, L., Wu, X. M., & Tian, Y. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3526138 · Cited by 10
“For example, MViT [55] generates a multiscale feature pyramid that captures both high-resolution spatial information and complex high-dimensional features through a tiered approach to channel and spatial resolution.”
A4-Unet: Deformable Multi-Scale Attention Network for Brain Tumor Segmentation
Wang, R., Tang, T., Du, H., Cheng, Y., Wang, Y., Yang, L., Duan, X., Yu, Y., Zhou, Y., & Chen, D. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10821912 · Cited by 8
“CrossViT [20] introduces a dual-branch transformer with cross-attention, and MViT [21] embeds a multi-scale feature pyramid into the transformer.”
Metadata
- Authors
- Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., & Feichtenhofer, C.
- Affiliations
- Meta (Israel) (IL)
- Meta (United States) (US)
- Venue
- 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2021-10
- Pages
- 6804-6815
- OpenAlex
- W3202698164
- Cited by
- 1,407
- References
- 111
- Citations by year
- 2026: 147 · 2025: 345 · 2024: 380 · 2023: 349 · 2022: 176 · 2021: 10
- FWCI
- 74.81
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University)
- TIB Data Manager
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- video recognition, image classification, feature pyramid, spatial resolution reduction, computational efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
