The DOI matches its Crossref record; 106 of the 111 references it cites were found in a registry or an index.
Multiscale Vision Transformers
Open accessAbstract
We present Multiscale Vision Transformers (MViT) for video and image recognition, by connecting the seminal idea of multiscale feature hierarchies with transformer models. Multiscale Transformers have several channel-resolution scale stages. Starting from the input resolution and a small channel dimension, the stages hierarchically expand the channel capacity while reducing the spatial resolution. This creates a multiscale pyramid of features with early layers operating at high spatial resolution to model simple low-level visual information, and deeper layers at spatially coarse, but complex, high-dimensional features. We evaluate this fundamental architectural prior for modeling the dense nature of visual signals for a variety of video recognition tasks where it outperforms concurrent vision transformers that rely on large scale external pre-training and are 5-10× more costly in computation and parameters. We further remove the temporal dimension and apply our model for image classification where it outperforms prior work on vision transformers. Code is available at: https://github.com/facebookresearch/SlowFast.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“As in original ViT [25], we use residual connections [45] and Layer Normalization (LN) [2] in the pre-normalization configuration that applies LN at the beginning of the residual function, and our MLPs consist of two linear layers with GELU activation [48], where the first layer expands the dimension from D to 4D, and…”
MethodResultVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 6×
“In contrast to original Multi Head Attention (MHA) operators [98], where the channel dimension and the spatiotemporal resolution remains fixed, MHPA pools the sequence of latent tensors to reduce the sequence length (resolution) of the attended input.”
BackgroundMethodInfluentialVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“Incorporating downsampling, shift invariance, and shared weights, ConvNets are de-facto standard backbones for computer vision tasks for image [65, 62, 88, 90, 46, 12, 15, 34, 93, 81, 41] and video [87, 31, 11, 79, 69, 106, 96, 30, 105, 35, 29, 117, 57].”
BackgroundImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“Incorporating downsampling, shift invariance, and shared weights, ConvNets are de-facto standard backbones for computer vision tasks for image [70, 67, 94, 96, 51, 15, 18, 39, 99, 87, 46] and video [93, 36, 14, 85, 74, 112, 102, 34, 111, 40, 33, 123, 62].”
BackgroundFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · cited 1×
“Finally, compared to the best model of concurrent Swin [79] (which was designed for image recognition), MViT has +0.”
ResultRethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Research works printed but not linked: 11
- gao, 2019, Res2net: A new multi-scale backbone architecture, IEEE PAMI
- fan, 2020, Pyslowfast
- feichtenhofer, 2019, SlowFast networks for video recognition in ActivityNet challenge 2019
- 10.1109/CVPR.2009.5206848 · DOI · Paper page
- 10.1109/CVPR.2019.00033 · DOI
- hang, 2020, ResNeSt Split-Attention Networks
- 10.1109/ICCV.2019.00718 · DOI
- 10.1109/CVPR.2015.7298594 · DOI · Paper page
- carion, 2020, Endto-end object detection with transformers, Proc ECCV, 213
- loshchilov, 2018, Fixing weight decay regularization in adam
- qiu, 2017, Learning spatiotemporal representation with pseudo-3d residual networks, Proc ICCV
Cited by
Sorted by citations
Items 301–400 of 1,003
Low-Resolution Self-Attention for Semantic Segmentation
Wu, Y.-H., Zhang, S., Liu, Y., Zhang, L., Zhan, X., Zhou, D., Feng, J., Cheng, M., & Zhen, L. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3577035 · Cited by 36
“Taking a natural image as input, the encoder first downsamples it by a factor of 1/4, following prevailing literature in this field [17], [19], [20], [23], [26].”
InfluentialUniRGB-IR: A Unified Framework for Visible-Infrared Semantic Tasks via Adapter Tuning
Yuan, M., Cui, B., Zhao, T. C., Wang, J., Fu, S., Yang, X., & Wei, X. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3754806 · Cited by 35
“Subsequently, various hierarchical transformers [11, 28, 39, 65] have been proposed for different down-stream vision tasks.”
Video Capsule Endoscopy Classification using Focal Modulation Guided Convolutional Neural Network
Srivastava, A., Tomar, N. K., Bağcı, U., & Jha, D. (2022) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms55023.2022.00064 · Cited by 29
“Additional work has been done on further reducing architecture complexity and increasing efficiency by; employing sparse attention [26]–[28], Pyramidal designs [29], [30], integration with CNNs [25], [31].”
BackgroundKowal, M. D., Siam, M., Islam, M. A., Bruce, N. D. B., Wildes, R. P., & Derpanis, K. G. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01361 · Cited by 24
“Recently, attention based approaches have proven to be suited to both static and time-series visual data, including action recognition, with variants of the transformer architecture [2, 12, 36, 45].”
BackgroundMethodWang, Q., Chen, K., Dou, W., & Ma, Y. (2023) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2023.3246979 · Cited by 29
“We employed the standard CNNs, including VGG19 [38], ResNet50 [39], and ResNet101 [39], and the Transformer-based models, including ViT [22], Swin Transformer [24], and MViT [40], [41], as the backbones for image classification.”
MethodTong, G., Ge, Z., & Peng, D. (2024) · Applied Intelligence · doi:10.1007/s10489-023-05250-8 · Cited by 27
Ong, K. L., Lee, C. P., Lim, H. S., Lim, K. M., & Alqahtani, A. (2024) · IEEE Access · doi:10.1109/access.2024.3360483 · Cited by 28
“In contrast to Vision Transformers [29] characterized by a fixed channel capacity and spatial resolution across the network, Multiscale Vision Transformers (MViT) [30] introduce multiple stages with different channel-resolution scales.”
BackgroundFast-iTPN: Integrally Pre-Trained Transformer Pyramid Network With Token Migration
Tian, Y., Xie, L., Qiu, J., Jiao, J., Wang, Y., Tian, Q. C., & Ye, Q. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3429508 · Cited by 35
“…were also inherited, such as introducing convolution to ViTs so that the relationship between neighborhood tokens is better formulated [15], [21], [23], [24], [25], [26], interacting between hybrid information [21], using window [7], [14], [27] or local [28] self-attentions to replace global…”
BackgroundExploiting Attention-Consistency Loss For Spatial-Temporal Stream Action Recognition
Xu, H., Jin, X., Wang, Q., Hussain, A., & Huang, K. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3538749 · Cited by 25
“Transformer-based networks [1, 3, 9, 14, 29] in computer vision tasks have aroused much interest among researchers.”
BackgroundWang, Y., Pan, Z., Li, X., Cao, Z., Xian, K., & Zhang, J. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3547978 · Cited by 33
“Some works [7, 11, 13, 33, 62] have shown the superiority of transformer in image-based tasks.”
BackgroundStarvqa: Space-Time Attention for Video Quality Assessment
Xing, F., Wang, Y., Wang, H., Li, L., & Zhu, G. (2022) · Proceedings - International Conference on Image Processing · doi:10.1109/icip46576.2022.9897881 · Cited by 27
“Inspired by ViT, several Transformer-based models [13, 20, 21] were developed for video classification tasks.”
MethodLook for the Change: Learning Object States and State-Modifying Actions from Untrimmed Web Videos
Souček, T., Alayrac, J.-B., Miech, A., Laptev, I., & Šivic, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01357 · Cited by 25
“Such models employ a mix of 2D and 3D convolutions [14, 25, 26, 61] or transformers and temporal attention [9, 23, 68].”
MethodUltrasound Localization Microscopy Using Deep Neural Network
Liu, X., & Almekkawy, M. K. (2023) · IEEE Transactions on Ultrasonics Ferroelectrics and Frequency Control · doi:10.1109/tuffc.2023.3276634 · Cited by 25
“detection [29], [30], [31], [32], [33], [34].”
MED-VT: Multiscale Encoder-Decoder Video Transformer with Application to Object Segmentation
Karim, R., Zhao, H., Wildes, R. P., & Siam, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00612 · Cited by 25
Early identification of stroke through deep learning with multi-modal human speech and movement data
Ou, Z., Wang, H., Zhang, B., Liang, H., Hu, B., Ren, L., Liu, Y., Zhang, Y., Dai, C., Wu, H., et al. (2024) · Neural Regeneration Research · doi:10.4103/1673-5374.393103 · Cited by 27
“As baseline measurements, we selected six prior models that achieved good performance in action classification methods, including the I3D (Carreira and Zisserman, 2017), SlowFast (Feichtenhofer et al., 2018), X3D (Feichtenhofer, 2020), TPN (Yang et al., 2020), TimeSformer (Bertasius et al., 2021), and MViT (Fan et al.…”
InfluentialA Semantic and Motion-Aware Spatiotemporal Transformer Network for Action Detection
Korban, M., Youngs, P. A., & Acton, S. T. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3377192 · Cited by 25
When Do We Not Need Larger Vision Models?
Shi, B., Wu, Z., Mao, M., Wang, X., & Darrell, T. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73242-3_25 · Cited by 54
Singh, M., Sharma, S. K., Sharma, S. K., & Singh, J. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126863 · Cited by 24
SAPFormer: Shape-aware propagation Transformer for point clouds
Xiao, G., Ge, S. S., Zhong, Y., Xiao, Z., Song, J., & Lu, J. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111578 · Cited by 24
Computer vision for primate behavior analysis in the wild
Vogg, R., Lüddecke, T., Henrich, J., Dey, S., Nuske, M., Hassler, V., Murphy, D., Fischer, J., Ostner, J., Schülke, O., et al. (2024) · Nature Methods · doi:10.1038/s41592-025-02653-y · Cited by 30
“The transformed vector can then be used as input to another fully-connected layer.”
MiM: Mask in Mask Self-Supervised Pre-Training for 3D Medical Image Analysis
Zhuang, J., Wu, L., Wang, Q., Peng, F., Vardhanabhuti, V., Luo, L., & Chen, H. (2025) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3564382 · Cited by 24
“While incorporating multiscale features could enhance classification performance [45], [67], [68], we opted not to follow this approach to ensure a fair comparison with other methods.”
A Content-Driven Micro-Video Recommendation Dataset at Scale
Ni, Y., Cheng, Y., Liu, X., Fu, J., Li, Y., He, X., Zhang, Y., & Yuan, F. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761655 · Cited by 24
“…I3D-r50 [4], X3D-s [10], Slow-r50 [8], X3D-m [10], R3D-r50 [47], SlowFast-r50 [11], CSN-r101 [46], X3D-l [10], SlowFast-r101 [11], MViT-B-16x4 [9], MViT-B-32x3 [9], and VideoMAE [45] with details in Appendix Section D. Q1: Can the knowledge learned from video understanding be beneficial for…”
Self-citationM2DAR: Multi-View Multi-Scale Driver Action Recognition with Vision Transformer
Ma, Y., Yuan, L., Abdelraouf, A., Han, K., Gupta, R., Li, Z., & Wang, Z.-R. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00557 · Cited by 20
“MViT has achieved state-of-the-art performance in video tasks [6].”
BackgroundFAFuse: A Four-Axis Fusion framework of CNN and Transformer for medical image segmentation
Xu, S.-K., Xiao, D., Yuan, B., Liu, Y., Wang, X., Li, N., Shi, L., Chen, J., Zhang, J.-X., Wang, Y., et al. (2023) · Comput. Biol. Medicine · doi:10.1016/j.compbiomed.2023.107567 · Cited by 20
Torres, L. H. M., Ribeiro, B., & Arrais, J. P. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.111268 · Cited by 26
Xian, R., Wang, X., & Manocha, D. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00649 · Cited by 24
“(b) Because of the UAV’s motion, the position of the human actor in F t +1 appears to be relatively behind compared to F t .”
BackgroundMethodGlobal semantic-guided network for saliency prediction
Xie, J., Liu, Z., Li, G., Lu, X., & Chen, T. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111279 · Cited by 24
A novel YOLO LSTM approach for enhanced human action recognition in video sequences
Elnady, M., & Munim, H. E. A. E. (2025) · Scientific Reports · doi:10.1038/s41598-025-01898-z · Cited by 30
A Transformer-based System for Action Spotting in Soccer Videos
Zhu, H., Liang, J., Lin, C.-H., Zhang, J., & Hu, J.-M. (2022) · MMSports@MM · doi:10.1145/3552437.3555693 · Cited by 30
“The model is based on the improved multi-scale vision transformers(MViT v2) [12, 25], and it can learn a hierarchy of dense (in space) and simple (in channels) to coarse and complex features.”
BackgroundMethodInfluentialFAR: Fourier Aerial Video Recognition
Kothandaraman, D., Guan, T., Wang, X., Hu, S., Lin, M. C., & Manocha, D. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_37 · Cited by 27
Object-centric Video Representation for Long-term Action Anticipation
Zhang, C., Fu, C., Wang, S., Agarwal, N., Lee, K., Choi, C., & Sun, C. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00661 · Cited by 25
“MViT [14, 34] and VideoSwin Transformer [38] re-introduce resolution-pooling as in convolution nets in video transformers for efficiency.”
MethodTowards Universal Soccer Video Understanding
Rao, J., Wu, H., Jiang, H., Zhang, Y., Wang, Y., & Xie, W. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00785 · Cited by 22
“For foul recognition, we follow previous work [22, 23] and adopt ResNet [21], R(2+1)D [46], and MViT [13] jointly finetuned with classifiers, as baselines.”
Zhou, Z., Ding, C., Li, J., Mohammadi, E., Liu, G., Yang, Y., & Wu, Q. M. J. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3224877 · Cited by 22
Quantformer: Learning Extremely Low-Precision Vision Transformers
Wang, Z., Wang, C., Xu, X., Zhou, J., & Lu, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3229313 · Cited by 22
“To address this, efficient architecture design [5], [17], [58], architecture search [6], [8], sparse attention [7], [11], [41], [50], [57] and network quantization are presented to reduce the storage and computational cost.”
MethodDual encoder network with transformer-CNN for multi-organ segmentation
Hong, Z., Chen, M., Hu, W., Yan, S., Qu, A., Chen, L., & Chen, J. (2022) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-022-02723-9 · Cited by 22
“Very recently, VIT has been gaining extensive research on computer vision (CV) tasks [15] and semantic segmentation [16, 17].”
BackgroundSketch-Segformer: Transformer-Based Segmentation for Figurative and Creative Sketches
Zheng, Y., Xie, J., Sain, A., Song, Y.-Z., & Ma, Z. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3302521 · Cited by 22
“Besides its significant success in natural language processing, Transformer [37] has also been applied to many computer vision tasks, such as image classification [42], [43], object detection [44], [45], semantic segmentation [34], [46], and point cloud analysis [35], [47].”
MethodDave, I. R., Rizve, M. N., Chen, C., & Shah, M. A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00232 · Cited by 22
“Semi-supervised learning in videos Although there is a tremendous recent development in action recognition [2, 6, 9, 13, 20, 22, 32, 48, 52, 61, 73, 75, 87], semi-supervised learning in videos is not explored as in the image domain.”
Jiang, S., Shuai, X., & Xing, G. (2024) · ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN) · doi:10.1109/ipsn61024.2024.00007 · Cited by 22
“Moreover, considering the training and inference as two closely coupled stage, multi-scale training [11, 24] is shown to be effective in centralized training setting.”
Adapting Short-Term Transformers for Action Detection in Untrimmed Videos
Yang, M., Gao, H., Guo, P., & Wang, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01757 · Cited by 28
“Unlike CNN, these transformer backbones [11, 12, 25] present challenges when applied directly to untrimmed video modeling due to the This CVPR paper is the Open Access version, provided by the Computer Vision Foundation.”
InfluentialMa, X., Zhang, S., Wang, Y., Li, R., Chen, X.-D., & Yu, D. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.119268 · Cited by 21
Learning to Anticipate Future with Dynamic Context Removal
Xu, X., Li, Y., & Lu, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01240 · Cited by 22
Self-citationZhang, J., Wang, X., Wan, Y., Wang, L., Wang, J., & Yu, P. S. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.02.045 · Cited by 21
MSTA-SlowFast: A Student Behavior Detector for Classroom Environments
Zhang, S., Liu, H., Sun, C., Wu, X., Wen, P., Yu, F., & Zhang, J. (2023) · Sensors · doi:10.3390/s23115205 · Cited by 21
“[22] proposed an MViT (Multiscale Vision Transformers) model for video and image recognition by combining multi-scale feature pyramid structures to achieve the extraction of video features at different levels, and encoding the features using Transformer to enable the model to better understand the visual content.”
MethodEdge Enhanced Image Style Transfer via Transformers
Zhang, C., Dai, Z., Cao, P., & Yang, J. (2023) · International Conference on Multimedia Retrieval · doi:10.1145/3591106.3592257 · Cited by 22
“Inherited the ability to capture the longrange dependencies from transformers in natural language processing (NLP), vision transformers have been developed in a wide variety of vision tasks, including image classification [12, 18, 21, 26, 28, 41, 48, 49, 59, 60, 66, 72, 74], object detection [5, 7, 13, 42, 79], semant…”
BackgroundZhou, W., Wen, L., Zhan, Y., & Wang, C. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3297589 · Cited by 22
“Although DeiT has exhibited a higher accuracy on the ImageNet dataset, it is proved to require a large-scale dataset for model convergence [42].”
BackgroundPanoSwin: a Pano-style Swin Transformer for Panorama Understanding
Ling, Z., Xing, Z., Zhou, X., Cao, M., & Zhou, G. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01703 · Cited by 22
“Vision Transformers.”
MethodUnlocking Fine-Grained Details with Wavelet-Based High-Frequency Enhancement in Transformers
Azad, R., Kazerouni, A., Sulaiman, A., Bozorgpour, A., Aghdam, E. K., Jose, A., & Merhof, D. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-45673-2_21 · Cited by 21
Limited Data, Unlimited Potential: A Study on ViTs Augmented by Masked Autoencoders
Das, S., Jain, T., Reilly, D., Balaji, P., Karmakar, S., Marjit, S., Li, X., Das, A., & Ryoo, M. S. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00673 · Cited by 23
“Hierarchical transformers [14, 32, 34, 55] have introduced inductive bias by reducing the number of tokens through patch merging and thus operating at different scales.”
BackgroundFishTrack23: An Ensemble Underwater Dataset for Multi-Object Tracking
Dawkins, M., Prior, J. H., Lewis, B., Faillettaz, R., Banez, T., Salvi, M., Rollo, A., Simon, J., Campbell, M. D., Lucero, M., et al. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00701 · Cited by 22
“Some examples include enlarging the number of tracking algorithms we are comparing [16,19,21,28,37], improved temporal handling in classifiers besides naive averaging along a track, and refining the kinematic components of the tracking algorithms.”
MethodBonyani, M., Soleymani, M., & Wang, C. (2024) · Automation in Construction · doi:10.1016/j.autcon.2024.105508 · Cited by 21
HRPVT: High-Resolution Pyramid Vision Transformer for medium and small-scale human pose estimation
Xu, Z., Dai, M., Zhang, Q., & Jiang, X. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129154 · Cited by 22
“The simultaneous developments of MViT [27], PVT [11], and Swin [28] incorporate multi-scale feature hierarchies into the transformer design, mirroring the spatial arrangement found in conventional convolutional architectures like ResNet-50.”
BackgroundImage Dehazing Based on Local and Non-Local Features
Jiao, Q., Liu, M., Ning, B., Zhao, F., Dong, L., Kong, L., Hui, M., & Zhao, Y. (2022) · Fractal and Fractional · doi:10.3390/fractalfract6050262 · Cited by 21
“Mathematically, this process can be described as follows: T Q K Unlike Multiscale Vision Transformers (MViT) [46], the fusion of different scale tokens is more complex than the shifted windows of the Swin-Transformer. soft max outputs V = (14 Hence, the Swin Transformer is a…”
BackgroundSwin Transformer-Based Object Detection Model Using Explainable Meta-Learning Mining
Baek, J.-W., & Chung, K. (2023) · Applied Sciences · doi:10.3390/app13053213 · Cited by 20
Local Reversible Transformer for semantic segmentation of grape leaf diseases
Zhang, X., Li, F., Jin, H., & Mu, W. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110392 · Cited by 20
2D Semantic Segmentation: Recent Developments and Future Directions
Guo, Y., Nie, G., Gao, W., & Liao, M. (2023) · Future Internet · doi:10.3390/fi15060205 · Cited by 24
“…in various vision tasks, their appearance provides a promising research direction for semantic segmentation [53]. direction for encoder–decoder architectures is the emergence of network architectures based on transformers, such as the Visual Transformer (Vit) [48] and its variants [49–52].”
BackgroundEGA-Depth: Efficient Guided Attention for Self-Supervised Multi-Camera Depth Estimation
Shi, Y., Cai, H., Ansari, A., & Porikli, F. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00017 · Cited by 20
“, improving the photometric matching [25, 40, 41], handling dynamic objects [6,8,18,30], utilizing semantic segmentation [4, 21, 43], supplying temporal information to the network [13, 36, 48], etc.”
BackgroundZhao, Y., Luo, C., Tang, C., Chen, D., Codella, N., & Zha, Z.-J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01403 · Cited by 20
“The augmentation and regularization in SSv2 include random augmentation [13], repeated augmentation [22], random erasing [75], Mixup [71], and CutMix [69], which follow the practice in MViT [16].”
BackgroundTemporal segment dropout for human action video recognition
Zhang, Y., Chen, Z., Xu, T., Zhao, J., Mi, S., Geng, X., & Zhang, M.-L. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109985 · Cited by 20
Uncertainty-Aware Denoising Network for Artifact Removal in EEG Signals
Jin, X., Wang, J., Liu, L., & Lin, Y. (2023) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2023.3330963 · Cited by 23
“Correspondingly, the attention mechanism is famous for its high flexibility in modeling global dependencies, which has been widely applied to various domains [27], [30].”
BackgroundPassable area segmentation for open-pit mine road from vehicle perspective
Zheng, C., Liu, L., Meng, Y., Wang, M., & Jiang, X. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107610 · Cited by 20
Actor-agnostic Multi-label Action Recognition with Multi-modal Query
Mondal, A., Nag, S., Prada, J. M., Zhu, X., & Dutta, A. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00086 · Cited by 22
“For the Charades dataset, we compare our model with AFAC [69], MViT [15], and ActionCLIP [57].”
MethodAdaptive Cross-Modal Transferable Adversarial Attacks From Images to Videos
Wei, Z., Chen, J., Wu, Z., & Jiang, Y. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3347835 · Cited by 21
“MViT[31]fuses multiscalefeaturesbytheproposedpoolingattentionoperation.”
BackgroundChen, J., Xia, Y., Yao, J., Yan, K., Zhang, J., Lü, L., Wang, F., Zhou, B., Qiu, M., Yu, Q., et al. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01950 · Cited by 26
“Transformers [50] have advanced the state-of-the-art performance in various computer vision tasks [18, 6, 32, 8, 19, 79, 48], by capturing global interactions between image patches and having no built-in inductive prior.”
BackgroundComplex-Valued Multiscale Vision Transformer on Space Target Recognition by ISAR Image Sequence
Yuan, H., Li, H., Zhang, Y., Wei, C., & Gao, R. (2024) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2024.3388427 · Cited by 20
“Fan et al. [16] proposed multiscale vision transformers (MViTs) that aim to connect the multiscale feature hierarchies with transformer, which shows an advantage on the recognition tasks.”
BackgroundChen, S., Feng, Y., Mao, Q., Li, H., Zhao, Y., Liu, J., Wang, H., & Ma, D. (2024) · Journal of Hydrology · doi:10.1016/j.jhydrol.2024.131733 · Cited by 20
RoboLLM: Robotic Vision Tasks Grounded on Multimodal Large Language Models
Long, Z., Killick, G., McCreadie, R., & Aragón-Camarasa, G. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610797 · Cited by 24
“Therefore, some models aim to mitigate these challenges by incorporating ConvNet designs into ViT, integrating hierarchical structures and translation-equivariant priors such as convolutions, pooling, and sliding windows (e.g., Swin [13], MViT [14], PVT [15]).”
BackgroundDEMF-Net: A dual encoder multi-scale feature fusion network for polyp segmentation
Cao, X., Yu, H., Kang, Y., Cui, R., Guo, J., Li, X., Xing, X., & Huang, T. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106487 · Cited by 19
SMART-vision: survey of modern action recognition techniques in vision
AlShami, A. K., Rabinowitz, R., Lâm, K. N., Shleibik, Y., Mersha, M. A., Boult, T. E., & Kalita, J. K. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-20484-5 · Cited by 23
End-to-End Compressed Video Representation Learning for Generic Event Boundary Detection
Li, C., Wang, X., Wen, L., Hong, D., Luo, T., & Zhang, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01358 · Cited by 22
“The new trend in recent years is the introduction of transformers [1, 5, 6, 24, 51], which achieving promising results on various datasets in video understanding.”
BackgroundMethodPyramid-based self-supervised learning for histopathological image classification
Wang, J., Quan, H., Wang, C., & Yang, G. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107336 · Cited by 20
What Can Simple Arithmetic Operations Do for Temporal Modeling?
Wu, W., Song, Y., Sun, Z., Wang, J., Xu, C., & Ouyang, W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01261 · Cited by 19
“𝑋 ∈ [1, 2, 10, 33, 37] propose different spatial and temporal attention designs using the large-scale ImageNet-21K [42] pre-trained model or designing new models from scratch, demonstrating improved performance for spatiotemporal modeling for video recognition.”
MethodEgo-Only: Egocentric Action Detection without Exocentric Transferring
Wang, H., Singh, M. K., & Torresani, L. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00484 · Cited by 27
“On top of the long sequence of frozen features, a detector is then trained to temporally localize the actions. tures [28] or hierarchical transformers [51, 52, 25, 44] that require adaptation of the MAE algorithm.”
BackgroundMotion-Guided Masking for Spatiotemporal Representation Learning
Fan, D., Wang, J., Liao, S., Zhu, Y., Bhat, V., Santos-Villalobos, H., Rohith, M., & Li, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00517 · Cited by 26
“models trained in a supervised manner: Our MGM using vanilla ViT-B pretrained on only SSv2 outperforms previous SOTA such as MViTv1 [10] (+4.”
MethodInfluentialAn Investigation Into the Advancements of Edge-AI Capabilities for Structural Health Monitoring
Mishra, A., Gangisetti, G., & Khazanchi, D. (2024) · IEEE Access · doi:10.1109/access.2024.3365550 · Cited by 21
“Vision transformers (ViTs) are a neural network type utilized for self-supervised learning that achieves state-of-the-art results on various computer vision tasks, including image classification, object detection, and semantic segmentation [49].”
BackgroundMDFL: Multi-Domain Diffusion-Driven Feature Learning
Li, D., Xie, W., Zhang, J., & Li, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i8.28710 · Cited by 20
Self-citationKoala: Key Frame-Conditioned Long Video-LLM
Tan, R., Sun, X., Hu, P., Wang, J.-H., Deilamsalehy, H., Plummer, B. A., Russell, B. C., & Saenko, K. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01289 · Cited by 25
“Moreover, understanding action sequences has often been constrained to short video clips. vLLMs are also similar to more recent fully attentional video encoders [5, 7, 15, 47] that leverage self-attention between spatiotemporal regions to compute more effective video representations.”
BackgroundVLPA-CLIP: Video Language Prompting and Adapting CLIP for efficient video action recognition
Wang, H., Liu, F., Jiao, L., Wang, J., Li, S., Li, L., Chen, P., Liu, X., & Ma, W. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111770 · Cited by 18
EchoFM: Foundation Model for Generalizable Echocardiogram Analysis
Kim, S., Jin, P., Song, S., Chen, C., Li, Y., Ren, H., Li, X., Liu, T., & Li, Q. (2025) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3580713 · Cited by 35
“Based on this, EchoPrime adopts the CLIP framework and incorporates a modified encoder using mVIT [33] to process video embeddings.”
InfluentialSelf-citationLaVin-DiT: Large Vision Diffusion Transformer
Wang, Z., Xia, X., Chen, R., Yu, D., Wang, C., Gong, M., & Liu, T. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01868 · Cited by 35
“By resorting to vision trans-former (ViT) [20, 31, 36], recent advancements [5, 6, 12, 16, 22, 40, 70] in generative modeling achieves significant improvements in scalability and performance for both image [11, 19, 44, 54, 72] and video generation [25, 39, 41, 63].”
Vu, D.-Q., Le, N., & Wang, J. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956634 · Cited by 19
“Besides, several approaches based on transformer have been proposed and achieved SOTA performance, such as MViT [25], VTN [26], etc.”
MethodMLP-3D: A MLP-like 3D Architecture with Grouped Time Mixing
Qiu, Z., Yao, T., Ngo, C., & Mei, T. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00307 · Cited by 19
“In the training stage, following [10,39], we exploit label smoothing, random augment [8], random erasing [74] and drop path [20] to reduce the over-fitting effect.”
BackgroundMethodInfluentialWeakly Supervised Video Representation Learning with Unaligned Text for Sequential Videos
Dong, S., Hu, H., Lian, D., Luo, W., Qian, Y., & Gao, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00241 · Cited by 18
“Due to the ability to capture long-term dependency of the self-attention mechanism [52], vision transformer models [3, 7, 11, 13, 22, 32, 33] show competitive performances against 3D-CNNs in video representation learning.”
BackgroundXu, P., Tian, L., Liu, J., Luo, D., & Jahanshahi, H. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107647 · Cited by 18
Prune Spatio-temporal Tokens by Semantic-aware Temporal Accumulation
Ding, S., Zhao, P., Zhang, X., Qian, R., Xiong, H., & Tian, Q. C. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01554 · Cited by 23
“Various variants of self-attention have been introduced in prior works [2, 29, 54, 1, 4, 31, 10, 26, 52, 22] to capture the spatiotemporal relationship.”
BackgroundNguyen, S., Le, D. V., & Havinga, P. (2024) · Pervasive and Mobile Computing · doi:10.1016/j.pmcj.2024.101912 · Cited by 21
EPK-CLIP: External and Priori Knowledge CLIP for action recognition
Yang, Z., An, G., Zheng, Z., Cao, S., & Wang, F. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124183 · Cited by 18
AI-based framework for early detection and segmentation of green citrus fruits in orchards
Akrouchi, M. E., Mhada, M., Bayad, M., Hawkesford, M. J., & Gerard, B. G. (2025) · Smart Agricultural Technology · doi:10.1016/j.atech.2025.100834 · Cited by 18
A Survey on Spatio-Temporal Prediction: From Transformers to Foundation Models
Mao, Y., Zhou, H., Chen, L., Qi, R., Sun, Z., Rong, Y., He, X., Chen, M., Mumtaz, S., Frascolla, V., et al. (2025) · ACM Computing Surveys · doi:10.1145/3766546 · Cited by 20
“In the second stage, a Vision Transformer-based encoder [116] processes input sequences packaged by the input representation learning stage for storm prediction.”
ScaleFusionNet: transformer-guided multi-scale feature fusion for skin lesion segmentation
Qamar, S., Qadri, S. F., Alroobaea, R., Alshmrani, G. M., Fazil, M., & Jiang, R. (2025) · Scientific Reports · doi:10.1038/s41598-025-17300-x · Cited by 18
“The Deformable Convolutional Network (DCN) [29], proposed by Dai et al. in 2017, is an extension of traditional convolutional networks designed to improve the adapt-ability of convolutional kernels to object deformations.”
A Spatio-Temporal Motion Network for Action Recognition Based on Spatial Attention
Yang, Q. Z., Lu, T., & Zhou, H. (2022) · Entropy · doi:10.3390/e24030368 · Cited by 17
Is Appearance Free Action Recognition Possible?
Ilic, F., Pock, T., & Wildes, R. P. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19772-7_10 · Cited by 17
Wang, J., Zhao, H., Liang, W., Wang, S., & Zhang, Y. (2023) · Physics in Medicine and Biology · doi:10.1088/1361-6560/acb19a · Cited by 17
Ding, C., Tang, D., Zheng, X., Wang, Q., He, Y., & Zhang, L. (2023) · IEEE Instrumentation & Measurement Magazine · doi:10.1109/mim.2023.10083000 · Cited by 17
“It calculates all the local attention within a window, and by this way, the input features contain global information [25].”
BackgroundHybridBranchNet: A novel structure for branch hybrid convolutional neural networks architecture
Parcham, E., & Fateh, M. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.05.025 · Cited by 17
Hammou, B. A., Antaki, F., Boucher, M. C., & Duval, R. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105178 · Cited by 17
Olabanjo, O. A., Ashiribo, W. U. S. U., Asokere, M., Afisi, O. T., Okugbesan, B., Olabanjo, O., Folorunso, O. O., & Mazzara, M. (2023) · Analytics · doi:10.3390/analytics2030039 · Cited by 21
SCT: A Simple Baseline for Parameter-Efficient Fine-Tuning via Salient Channels
Zhao, H., Wang, P., Zhao, Y., Luo, H., Wang, F., & Shou, M. Z. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01918-3 · Cited by 21
“…remarkable success in computer vision tasks (Dosovitskiy et al., 2020; Liu et al., 2021; Yuan et al., 2022; Zhou et al., 2021a; Carion et al., 2020; Li et al., 2021; Strudel et al., 2021) by using large-scale training data, such as ImageNet21K and JFT-300M, to create strong representations.”
SGTR+: End-to-End Scene Graph Generation With Transformer
Li, R., Zhang, S., & He, X. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3332246 · Cited by 17
MSViT: Dynamic Mixed-scale Tokenization for Vision Transformers
Havtorn, J. D., Royer, A., Blankevoort, T., & Bejnordi, B. E. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00091 · Cited by 17
“Hierarchical (or Multi-scale) Vision Transformers [27, 26, 15, 10, 58] is a popular family of models that draw inspiration from the inductive bias of CNNs to build efficient ViT architectures: For instance in Swin, the image is initially split in very small tokens (4x4) which interact through lo-cal…”
BackgroundSchindler, F., Steinhage, V., Calkoen, S. T. S. V. B., & Heurich, M. (2024) · Applied Sciences · doi:10.3390/app14020514 · Cited by 16
“We also analyze MViT [23], because transformer architectures are the other main approach to action recognition and MViT is one of the most successful networks.”
Influential
Metadata
- Authors
- Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., & Feichtenhofer, C.
- Affiliations
- Meta (Israel) (IL)
- Meta (United States) (US)
- Venue
- 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2021-10
- Pages
- 6804-6815
- OpenAlex
- W3202698164
- Cited by
- 1,407
- References
- 111
- Citations by year
- 2026: 147 · 2025: 345 · 2024: 380 · 2023: 349 · 2022: 176 · 2021: 10
- FWCI
- 74.81
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University)
- TIB Data Manager
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- video recognition, image classification, feature pyramid, spatial resolution reduction, computational efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
