Transformers for Vision: A Survey on Innovative Methods for Computer Vision
Open accessThe DOI matches its Crossref record; 120 of the 120 references it cites were found in a registry or an index.
Abstract
Transformers have emerged as a groundbreaking architecture in the field of computer vision, offering a compelling alternative to traditional convolutional neural networks (CNNs) by enabling the modeling of long-range dependencies and global context through self-attention mechanisms. Originally developed for natural language processing, transformers have now been successfully adapted for a wide range of vision tasks, leading to significant improvements in performance and generalization. This survey provides a comprehensive overview of the fundamental principles of transformer architectures, highlighting the core mechanisms such as self-attention, multi-head attention, and positional encoding that distinguish them from CNNs. We delve into the theoretical adaptations required to apply transformers to visual data, including image tokenization and the integration of positional embeddings. A detailed analysis of key transformer-based vision architectures such as ViT, DeiT, Swin Transformer, PVT, Twins, and CrossViT are presented, alongside their practical applications in image classification, object detection, video understanding, medical imaging, and cross-modal tasks. The paper further compares the performance of vision transformers with CNNs, examining their respective strengths, limitations, and the emergence of hybrid models. Finally, current challenges in deploying ViTs, such as computational cost, data efficiency, and interpretability, and explore recent advancements and future research directions including efficient architectures, self-supervised learning, and multimodal integration are discussed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Transformer-based architectures have demonstrated remarkable performance in both vision and video tasks, thanks to their ability to model long-range dependencies and capture spatiotemporal correlations [7, 8].”
From Cross-dataset video deepfake detection using Transformer and CNN architectures (2026) · cites this paper as [7]
Citations
Cites
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
SSD: Single Shot MultiBox Detector
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., & Berg, A. C. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_2 · Cited by 22,128 · cited 1×
“YOLO [54], and SSD [55] are notable examples.”
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · arXiv · doi:10.48550/arxiv.1802.02611 · Cited by 20,063
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_49 · Cited by 20,054
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
Zhang, K., Zuo, W., Chen, Y., Meng, D., & Zhang, L. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2662206 · Cited by 8,983
Masked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489 · cited 1×
“…recent innovations, including incorporating hierarchical attention in Swin Transformers [6] and masked autoencoding in Masked AutoEncoder (MAE) [7], have substantially bridged the gaps in data efficiency and computational performance, ushering in a novel era in visual information processing.”
Analyzing and Improving the Image Quality of StyleGAN
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00813 · Cited by 6,332
Analyzing and Improving the Image Quality of StyleGAN
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. (2019) · arXiv · doi:10.48550/arxiv.1912.04958 · Cited by 6,332
Deformable DETR: Deformable Transformers for End-to-End Object Detection
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., & Dai, J. (2020) · arXiv · doi:10.48550/arxiv.2010.04159 · Cited by 6,027 · cited 1×
“Zhu et al. [57] introduced Deformable DETR, which improves efficiency by using a deformable attention module that focuses on key locations instead of all spatial features.”
Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition
Yan, S., Xiong, Y., & Lin, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12328 · Cited by 5,410
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00061 · Cited by 5,259 · cited 1×
“The Pyramid Vision Transformer (PVT) [36] is a hierarchical Vision Transformer designed to improve efficiency and scalability for dense vision tasks such as object detection and segmentation.”
SwinIR: Image Restoration Using Swin Transformer
Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., & Timofte, R. (2021) · 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw54120.2021.00210 · Cited by 4,734
Han (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P. H. S., et al. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00681 · Cited by 4,008
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P. H. S., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.15840 · Cited by 4,004
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2021) · arXiv · doi:10.48550/arxiv.2101.01169 · Cited by 3,594
BiSeNet: Bilateral Segmentation Network for Real-Time Semantic Segmentation
Yu, C., Wang, J., Peng, C., Gao, C., Yu, G., & Sang, N. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01261-8_20 · Cited by 2,968
CvT: Introducing Convolutions to Vision Transformers
Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., & Zhang, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00009 · Cited by 2,444 · cited 2×
“The Convolutional Vision Transformer (CvT) [15] is a hybrid Vision Transformer that integrates convolutional layers into the standard ViT framework to improve efficiency, feature extraction, and inductive biases.”
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F. E. H., Feng, J., & Yan, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00060 · Cited by 2,415 · cited 1×
“Several advancements have focused on reducing computational complexity while maintaining high performance [119]”
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F. E., Feng, J., & Yan, S. (2021) · arXiv · doi:10.48550/arxiv.2101.11986 · Cited by 2,415 · cited 1×
“Several advancements have focused on reducing computational complexity while maintaining high performance [119]”
Vision Transformers for Dense Prediction
Ranftl, R., Bochkovskiy, A., & Koltun, V. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01196 · Cited by 2,373
Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., & Hu, H. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00320 · Cited by 2,294 · cited 1×
“Moreover, recent innovations, including incorporating hierarchical attention in Swin Transformers [6] and masked autoencoding in Masked AutoEncoder (MAE) [7], have substantially bridged the gaps in data efficiency and computational performance, ushering in a novel era in visual information…”
Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., & Hu, H. (2021) · arXiv · doi:10.48550/arxiv.2106.13230 · Cited by 2,294 · cited 1×
“Moreover, recent innovations, including incorporating hierarchical attention in Swin Transformers [6] and masked autoencoding in Masked AutoEncoder (MAE) [7], have substantially bridged the gaps in data efficiency and computational performance, ushering in a novel era in visual information…”
Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images
Hatamizadeh, A., Nath, V., Tang, Y., Yang, D., Roth, H. R., & Xu, D. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-08999-2_22 · Cited by 2,273
Kirillov, A., He, K., Girshick, R., Rother, C., & Dollár, P. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00963 · Cited by 1,659
BEiT: BERT Pre-Training of Image Transformers
Bao, H., Dong, L., Piao, S., & Wei, F. (2021) · arXiv · doi:10.48550/arxiv.2106.08254 · Cited by 1,392
Going deeper with Image Transformers
Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., & Jegou, H. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00010 · Cited by 1,259 · cited 1×
“Below is a detailed explanation of the key architectures in this category: transformer models [40].”
Actional-Structural Graph Convolutional Networks for Skeleton-Based Action Recognition
Li, M., Chen, S., Chen, X., Zhang, Y., Wang, Y., & Tian, Q. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00371 · Cited by 1,230 · cited 1×
“Li et al. [73] extended this concept by introducing the Action Structure Graph Convolutional Network (AS-GCN) [74], which incorporated additional structural and actional connections to improve recognition accuracy.”
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
Voita, E., Talbot, D., Moiseev, F., Sennrich, R., & Titov, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1580 · Cited by 1,146 · cited 1×
“To enhance the model’s ability to capture different types of relationship, Transformers use multi-head self-attention (MHSA) [10].”
MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer
Mehta, S., & Rastegari, M. (2021) · arXiv · doi:10.48550/arxiv.2110.02178 · Cited by 1,110 · cited 1×
“Below is a detailed explanation of the key architectures in this category: The MobileViT is a lightweight Vision Transformer [35] designed for mobile and edge devices, combining the strengths of convolutions and self-attention to achieve high performance while maintaining efficiency.”
(2022) · doi:10.1007/978-3-031-20053-3_27 · Cited by 1,060
(2020) · International Conference on Machine Learning · Cited by 1,012
Stand-Alone Self-Attention in Vision Models
Ramachandran, P., Parmar, N., Vaswani, A., Bello, I., Levskaya, A., & Shlens, J. (2019) · arXiv · doi:10.48550/arxiv.1906.05909 · Cited by 977 · cited 1×
“The term ‘‘visual attention’’ was first introduced to select essential regions and locations in an image [44].”
LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference
Graham, B., El-Nouby, A., Touvron, H., Stock, P., Joulin, A., Jegou, H., & Douze, M. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01204 · Cited by 935 · cited 1×
“The LeViT (Leaked Vision Transformer) [39] is a hybrid Vision Transformer (ViT) designed to be fast, efficient, and scalable for mobile and edge computing applications.”
Zhai, X., Kolesnikov, A., Houlsby, N., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01179 · Cited by 891 · cited 1×
“…structure into the transformer design, mirroring the multi-scale feature extraction process of CNNs. PVT enabled ViTs to handle dense prediction tasks like object detection and semantic segmentation more effectively, making it more efficient for vision applications beyond classification [24].”
CNN Variants for Computer Vision: History, Architecture, Application, Challenges and Future Scope
Bhatt, D., Patel, C., Talsania, H., Patel, J., Vaghela, R., Pandya, S., Modi, K., & Ghayvat, H. (2021) · Electronics · doi:10.3390/electronics10202470 · Cited by 890 · cited 1×
“Their convolutional operations, honed by decades of research, efficiently capture hierarchical patterns (e.g., edges, textures) and benefit from robust frameworks like PyTorch [109].”
Towards Understanding Action Recognition
Jhuang, H., Gall, J., Zuffi, S., Schmid, C., & Black, M. J. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.396 · Cited by 879 · cited 1×
“Researchers have explored various strategies to improve efficiency including processing shorter video segments instead of full sequences, using skeleton-based representations (body joint positions) to reduce complexity, and designing specialized ViT architectures tailored for video tasks [67].”
Human Action Recognition and Prediction: A Survey
Kong, Y., & Fu, Y. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01594-9 · Cited by 780
Going deeper into action recognition: A survey
Herath, S., Harandi, M., & Porikli, F. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.01.010 · Cited by 634
End-to-End Dense Video Captioning with Masked Transformer
Zhou, L., Zhou, Y., Corso, J. J., Socher, R., & Xiong, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00911 · Cited by 605
A Survey of Visual Transformers
Liu, Y., Zhang, Y., Wang, Y., Hou, F., Yuan, J., Tian, J., Zhang, Y., Shi, Z., Fan, J., & He, Z. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3227717 · Cited by 549
UP-DETR: Unsupervised Pre-training for Object Detection with Transformers
Dai, Z., Cai, B., Lin, Y., & Chen, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00165 · Cited by 520 · cited 1×
“Dai et al. [61] developed UP-DETR, an unsupervised pre-training approach inspired by NLP, using random query patch detection.”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers
Wang, H., Zhu, Y., Adam, H., Yuille, A., & Chen, L.-C. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00542 · Cited by 497 · cited 2×
“It achieves advanced performance using a ‘‘dual-path transformer’’ that combines CNNs with transformers [62].”
Conditional Positional Encodings for Vision Transformers
Chu, X., Tian, Z., Zhang, B., Wang, X., & Shen, C. (2021) · arXiv · doi:10.48550/arxiv.2102.10882 · Cited by 489
Thorax disease classification with attention guided convolutional neural network
Guan, Q., Huang, Y., Zhong, Z., Zheng, Z., Zheng, L., & Yang, Y. (2019) · Pattern Recognition Letters · doi:10.1016/j.patrec.2019.11.040 · Cited by 471
A survey of the vision transformers and their CNN-transformer based variants
Khan, A., Rauf, Z., Sohail, A., Khan, A. R., Asif, H., Asif, A., & Farooq, U. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10595-0 · Cited by 451
AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition
Chen, S., Ge, C., Tong, Z., Wang, J., Tong, Z., Wang, J., & Luo, P. (2022) · neural information processing systems · doi:10.52202/068431-1212 · Cited by 451
Rethinking Transformer-based Set Prediction for Object Detection
Sun, Z., Cao, S., Yang, Y., & Kitani, K. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00359 · Cited by 356
Object Detection Using Deep Learning, CNNs and Vision Transformers: A Review
Amjoud, A. B., & Amrouch, M. (2023) · IEEE Access · doi:10.1109/access.2023.3266093 · Cited by 318
CrossFormer++: A Versatile Vision Transformer Hinging on Cross-Scale Attention
Wang, W., Chen, W., Qiu, Q., Chen, L., Wu, B., Lin, B., He, X., & Liu, W. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3341806 · Cited by 318
How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers
Steiner, A., Kolesnikov, A., Zhai, X., Wightman, R., Uszkoreit, J., & Beyer, L. (2021) · arXiv · doi:10.48550/arxiv.2106.10270 · Cited by 292
Learning Joint Spatial-Temporal Transformations for Video Inpainting
Zeng, Y., Fu, J., & Chao, H. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58517-4_31 · Cited by 281
Multimodal Token Fusion for Vision Transformers
Wang, Y., Chen, X., Cao, L., Huang, W., Sun, F., & Wang, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01187 · Cited by 272
A Comparative Study of SIFT and its Variants
Wu, J., Cui, Z., Sheng, V. S., Zhao, P., Su, D., & Gong, S. (2013) · Measurement Science Review · doi:10.2478/msr-2013-0021 · Cited by 233 · cited 1×
“Earlier techniques used handcrafted features such as SURF [70] and SIFT [71], but these struggled to capture long-range dependencies and required extensive tuning.”
Free-Form Video Inpainting With 3D Gated Convolution and Temporal PatchGAN
Chang, Y.-L., Liu, Z. Y., Lee, K.-Y., & Hsu, W. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00916 · Cited by 202 · cited 1×
“These approaches typically assume global affine transformations, which can result in inaccurate reconstructions [102].”
Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer
Xu, Y., Zhang, Z., Zhang, M., Sheng, K., Li, K., Dong, W., Zhang, L., Xu, C., & Sun, X. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20202 · Cited by 192
GANomaly: Semi-supervised Anomaly Detection via Adversarial Training
Akcay, S., Atapour-Abarghouei, A., & Breckon, T. P. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-20893-6_39 · Cited by 188
Liu, L., Zhang, J., He, R., Liu, Y., Wang, Y., Tai, Y., Luo, D., Wang, C., Li, J., & Huang, F. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00652 · Cited by 183
Training data-efficient image transformers & distillation through attention
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jégou, H. (2020) · arXiv · doi:10.48550/arxiv.2012.12877 · Cited by 182
Vision Transformers for Image Classification: A Comparative Survey
Wang, Y., Deng, Y., Zheng, Y., Chattopadhyay, P., & Wang, L. (2025) · Technologies · doi:10.3390/technologies13010032 · Cited by 174
Video Inpainting by Jointly Learning Temporal Structure and Spatial Details
Wang, C., Huang, H., Han, X., & Wang, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015232 · Cited by 169
A Comprehensive Survey of Transformers for Computer Vision
Jamil, S., Piran, M. J., & Kwon, O. (2023) · Drones · doi:10.3390/drones7050287 · Cited by 155
Spatio-temporal deformable 3D ConvNets with attention for action recognition
Li, J., Liu, X., Zhang, M., & Wang, D. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.107037 · Cited by 146 · cited 1×
“To resolve these potential challenges, Deformable VisTR uses the Spatio-Temporal Deformable Attention module [65].”
An Analysis of the SURF Method
Oyallon, E., & Rabin, J. (2015) · Image Processing On Line · doi:10.5201/ipol.2015.69 · Cited by 145 · cited 1×
“Earlier techniques used handcrafted features such as SURF [70] and SIFT [71], but these struggled to capture long-range dependencies and required extensive tuning.”
I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference
Li, Z., & Gu, Q. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01565 · Cited by 141
OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation
Wang, J., Jiang, Y., Yuan, Z., Peng, B., Wu, Z., & Jiang, Y.-G. (2024) · neural information processing systems · doi:10.52202/079017-0887 · Cited by 138
Attribute-Aware Attention Model for Fine-grained Representation Learning
Han, K., Guo, J., Zhang, C., & Zhu, M. (2018) · Proceedings of the 26th ACM international conference on Multimedia · doi:10.1145/3240508.3240550 · Cited by 132 · cited 1×
“Han et al. [48] further enhanced CNN representations by employing attribute-aware attention.”
Recent Advances in Vision Transformer: A Survey and Outlook of Recent WorkPreprint
Islam, K. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.01536 · Cited by 115
When Adversarial Training Meets Vision Transformers: Recipes from Training to Architecture
Mo, Y., Wu, D., Wang, Y., Guo, Y., & Wang, Y. (2022) · neural information processing systems · doi:10.52202/068431-1351 · Cited by 111
End-to-End Object Detection with Adaptive Clustering Transformer
Zheng, M., Gao, P., Zhang, R., Li, K., Wang, X., Li, H., & Dong, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.09315 · Cited by 110 · cited 1×
“To further reduce computational complexity, Zheng et al. [58] proposed the Adaptive Clustering Transformer (ACT), which replaces DETR’s self-attention with locality sensitivity hashing (LSH) for efficient query clustering, minimizing accuracy loss.”
On Layer Normalization in the Transformer Architecture
Xiong, R., Yang, Y., He, D., Zheng, K. W., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., & Liu, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.04745 · Cited by 99
Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations?Preprint
Fu, Y., Zhang, S., Wu, S., Wan, C., & Lin, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.08392 · Cited by 97 · cited 1×
“The following are some of the key challenges : • Susceptibility to adversarial patches: ViTs have been shown to be vulnerable to adversarial patches, which are small, specially crafted patches that can be added to an image to mislead the model [115].”
Knowledge Distillation for Multi-task Learning
Li, W., & Bilen, H. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-65414-6_13 · Cited by 94
Vision-Based Fall Detection Using ST-GCN
Keskes, O., & Noumeir, R. (2021) · IEEE Access · doi:10.1109/access.2021.3058219 · Cited by 94 · cited 1×
“ST-GCN enhances traditional GCN models by incorporating extended skeleton graphs with functional connections and a refined partitioning strategy, significantly improving action recognition performance on large-scale datasets [69], [75].”
On The Computational Complexity of Self-Attention
Keles, F. D., Wijewardena, P. M., & Hegde, C. (2022) · arXiv · doi:10.48550/arxiv.2209.04881 · Cited by 90 · cited 1×
“Standard self-attention mechanisms exhibit O ( N 2 ) complexity [19], where N represents the number of tokens.”
Wang, Q., Zhang, K., & Manjotho, A. A. (2022) · IEEE Access · doi:10.1109/access.2022.3164711 · Cited by 84 · cited 2×
“ST-GCN enhances traditional GCN models by incorporating extended skeleton graphs with functional connections and a refined partitioning strategy, significantly improving action recognition performance on large-scale datasets [69], [75].”
DeViT: Decomposing Vision Transformers for Collaborative Inference in Edge Devices
Xu, G., Hao, Z., Luo, Y., Hu, H., An, J., & Mao, S. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2023.3315138 · Cited by 83 · cited 1×
“Unlike CNNs that benefit from decades of hardware optimization, ViTs face fundamental architectural hurdles for efficient edge deployment [118].”
How Does Attention Work in Vision Transformers? A Visual Analytics Attempt
Li, Y., Wang, J., Dai, X., Wang, L., Yeh, C. M., Zheng, Y. L., Zhang, W., & Ma, K. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3261935 · Cited by 79
Towards Effective Multiple-in-One Image Restoration: A Sequential and Prompt Learning StrategyPreprint
Kong, X., Dong, C., & Zhang, L. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.03379 · Cited by 79
AS-GCN: Adaptive Semantic Architecture of Graph Convolutional Networks for Text-Rich Networks
Yu, Z., Jin, D., Liu, Z., He, D., Wang, X., Tong, H., & Han, J. (2021) · Proceedings · doi:10.1109/icdm51629.2021.00095 · Cited by 77 · cited 1×
“Li et al. [73] extended this concept by introducing the Action Structure Graph Convolutional Network (AS-GCN) [74], which incorporated additional structural and actional connections to improve recognition accuracy.”
Ashourvan, A., Telesford, Q. K., Verstynen, T., Vettel, J. M., & Bassett, D. S. (2019) · PLoS ONE · doi:10.1371/journal.pone.0215520 · Cited by 71 · cited 1×
“Hierarchical and Multi-Scale Architectures in Vision Trans-formers (ViTs) are designed to capture features at multiple scales, making them particularly effective for tasks like object detection, segmentation, and other dense prediction tasks [34].”
Li, J., Bao, Y., Liu, W., Ji, P., Wang, L., & Wang, Z. (2023) · Measurement · doi:10.1016/j.measurement.2023.113687 · Cited by 68
LeGrad: An Explainability Method for Vision Transformers via Feature Formation Sensitivity
Bousselham, W., Boggust, A., Chaybouti, S., Strobelt, H., & Kuehne, H. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01891 · Cited by 64
Pang, J., & Dong, S. X. (2024) · Energy Conversion and Management · doi:10.1016/j.enconman.2024.118343 · Cited by 60 · cited 1×
“ConvNext demonstrated that CNN-based inductive biases could still play a crucial role in visual tasks, even in the era of transformers [26].”
Reciprocal Attention Mixing Transformer for Lightweight Image Restoration
Choi, H., Na, C., Oh, J., Lee, S., Kim, J., Choe, S., Lee, J., Kim, T., & Yang, J. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00606 · Cited by 59
Variation benchmark datasets: update, criteria, quality and applications
Sarkar, A., Yang, Y., & Vihinen, M. (2019) · Database · doi:10.1093/database/baz117 · Cited by 57 · cited 1×
“The CNNs, such as EfficientNet, have been found to be more parameter-efficient when working with small datasets [107].”
StyleGAN knows Normal, Depth, Albedo, and More
Bhattad, A., McKee, D., Hoiem, D., & Forsyth, D. A. (2023) · neural information processing systems · doi:10.52202/075280-3195 · Cited by 54
Xu, R. D., Hu, S., Wan, H., Xie, Y., Cai, Y., & Wen, J. (2023) · Journal of Environmental Management · doi:10.1016/j.jenvman.2023.119894 · Cited by 51
Hybrid Design of CNN and Vision Transformer: A Review
Long, H. (2024) · International Conference on Computer Information Science and Artificial Intelligence (CISAI) · doi:10.1145/3703187.3703208 · Cited by 49 · cited 1×
“Hybrid models can learn features at multiple scales, combining the local features extracted by CNNs with the global features learned by ViTs [112].”
FPN-SE-ResNet Model for Accurate Diagnosis of Kidney Tumors Using CT Images
Abdelrahman, A., & Viriri, S. (2023) · Applied Sciences · doi:10.3390/app13179802 · Cited by 47 · cited 1×
“ViTDet outperforms CNN-based models like ResNet-FPN [43], making it a powerful choice for high-precision detection, segmentation, and dense vision applications by leveraging hierarchical feature extraction and efficient transformer-based attention.”
Aerial Image Object Detection with Vision Transformer Detector (ViTDet)
Wang, L., & Tien, A. (2023) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss52108.2023.10282836 · Cited by 44 · cited 1×
“The ViTDet (Vision Transformer for Detection) [42] is a detection-optimized Vision Transformer designed specifically for object detection and instance segmentation tasks.”
Mobile Botnet Detection: A Deep Learning Approach Using Convolutional Neural Networks
Yerima, S. Y., & Alzaylaee, M. K. (2020) · International Conference on Cyber Situational Awareness, Data Analytics and Assessment (CyberSA) · doi:10.1109/cybersa49311.2020.9139664 · Cited by 40 · cited 1×
“Hybrid approaches like Swin Transformer and BoTNet [111] combine the strengths of both paradigms, offering promising solutions across diverse domains.”
The encoding method of position embeddings in vision transformer
Jiang, K., Peng, P., Lian, Y., & Xu, W. (2022) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2022.103664 · Cited by 39 · cited 1×
“Positional embeddings (PEs) address this limitation by encoding spatial information and adding it to the patch embeddings, allowing the model to maintain spatial awareness during training and inference [21].”
Saadati, D., Manzari, O. N., & Mirzakuchaki, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.11450 · Cited by 38
Jumphoo, T., Phapatanaburi, K., Pathonsuwan, W., Anchuen, P., Uthansakul, M., & Uthansakul, P. (2024) · IEEE Access · doi:10.1109/access.2024.3357946 · Cited by 37 · cited 1×
“It comes in different variants, such as DeiT-Ti [32] (Tiny), DeiT-S (Small), and DeiT-B (Base), balancing accuracy and computational cost for various applications.”
Evaluation of the benchmark datasets for testing the efficacy of deep convolutional neural networks
Dhar, S., & Shamir, L. (2021) · Visual Informatics · doi:10.1016/j.visinf.2021.10.001 · Cited by 36 · cited 1×
“However, hybrid models that combine the strengths of both architectures, like ConvNeXt and CoAtNet, have shown great promise in bridging this gap, often outperforming both pure ViTs and CNNs [108].”
A Study on the Generality of Neural Network Structures for Monocular Depth Estimation
Bae, J., Hwang, K., & Im, S. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3332407 · Cited by 35
Into the LAION’s Den: Investigating Hate in Multimodal Datasets
Birhane, A., Prabhu, V., Han, S., Boddeti, V., & Luccioni, S. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-0930 · Cited by 33
A Road Crack Segmentation Method Based on Transformer and Multi-Scale Feature Fusion
Xu, Y., Xia, Y., Zhao, Q., Yang, K., & Li, Q. (2024) · Electronics · doi:10.3390/electronics13122257 · Cited by 30
Accelerated masked transformer for dense video captioning
Zhou, Y., & Han, N. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.03.026 · Cited by 21 · cited 1×
“The Accelerated Masked Transformer (AMT) [106] improves upon MT with enhanced efficiency.”
Yang, Z., Du, H., Niyato, D. T., Wang, X., Zhou, Y. F., Feng, L., Zhou, F., Li, W., & Qiu, X. (2025) · IEEE Wireless Communications · doi:10.1109/mwc.002.2400197 · Cited by 20 · cited 1×
“DINOV2 leveraged self-supervised learning techniques to extract meaningful visual representations, making it highly effective in data-scarce environments [27].”
Shokouhmand, A., Wen, H., Khan, S., Puma, J. A., Patel, A. J., Green, P., Ayazi, F., & Ebadi, N. (2023) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2023.3253381 · Cited by 19 · cited 1×
“This transformation converts the structured 2D image into a 1D sequence of patch embeddings, making it compatible with the Transformer framework [17].”
Image Augmentation for Object Image Classification Based On Combination of Pre-Trained CNN and SVM
Shima, Y. (2018) · Journal of Physics Conference Series · doi:10.1088/1742-6596/1004/1/012001 · Cited by 18
Visual Transformer-Based Models: A Survey
Huang, X., Bi, N., & Tan, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-09282-4_25 · Cited by 16 · cited 1×
“For applications demanding fine-grained local analysis (e.g., medical imaging) or efficient inference, CNNs retain an edge, while ViTs thrive in high-complexity domains like video analysis or multimodal systems [110].”
Esteves, C., Suhail, M., & Makadia, A. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01596 · Cited by 15
Energy Efficiency on Scalable Computing Architectures
Barrios-Hernandez, C., Sierra, D., Varrette, S., & Pacheco, D. L. (2011) · IEEE International Conference on Computer and Information Technology (CIT) · doi:10.1109/cit.2011.108 · Cited by 11 · cited 1×
“These innovations make ViTs more practical for large-scale vision tasks and real-world applications while retaining their strong performance [22].”
MobileDepth: Monocular Depth Estimation Based on Lightweight Vision Transformer
Li, Y., & Wei, X. (2024) · Applied Artificial Intelligence · doi:10.1080/08839514.2024.2364159 · Cited by 10
Dual-Encoder UNet-Based Narrowband Uncooled Infrared Imaging Denoising Network
Wang, M., Pan, Y., Qiu, S., Jin, W., Li, L., & Wang, X. (2025) · Sensors · doi:10.3390/s25051476 · Cited by 10
Anagnostopoulou, A., Gouvêa, T. S., & Sonntag, D. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.04331 · Cited by 8
Li, M. (2024) · International Conference on Cloud Computing, Big Data Application and Software Engineering (CBASE) · doi:10.1109/cbase64041.2024.10824558 · Cited by 6
Tensor-to-Image: Image-to-Image Translation with Vision TransformersPreprint
Gündüç, Y. (2021) · doi:10.36227/techrxiv.16727140 · Cited by 5
CrossFormer Embedding DeepLabv3+ for Remote Sensing Images Semantic Segmentation
Tong, Q., Zhu, Z., Zhang, M., Cao, K., & Xing, H. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.049187 · Cited by 5
Self-Supervised Pre-training of Vision Transformers for Dense Prediction TasksPreprint
Rabarisoa, J., Belissen, V., Chabot, F., & Pham, Q.-C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.15173 · Cited by 5
Attention Modules Improve Modern Image-Level Anomaly Detection: A DifferNet Case StudyPreprint
Vieira-e-Silva, A. L. B., Simões, F., Kowerko, D., Schlosser, T., Battisti, F., & Teichrieb, V. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.08686 · Cited by 2
Mask-patchcore: a robust anomaly detection model focusing on interested region
Wu, F., & Xu, S. (2025) · International Conference on Graphics and Image Processing (ICGIP) · doi:10.1117/12.3057825 · Cited by 1
DIFNet: Dual-Domain Information Fusion Network for Image Denoising
Wu, Z., Shi, W., Xu, L., Ding, Z., Liu, T., Zhang, Z., & Zheng, B. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8685-5_20 · Cited by 1
Research works printed but not linked: 9
- Keles, On the computational complexity of self-attention, 597
- Touvron, Training data-efficient image transformers & distillation through attention, 10347
- Saumya, Learn to pay attention, 1209
- Giuste, 2020, CIFAR-10 image classification using feature ensembles, arXiv:2002.03846 · arXiv
- Ge, 2021, YOLOX: Exceeding YOLO series in 2021, arXiv:2107.08430 · arXiv
- Dosovitskiy, 2020, An image is worth 16×16 words: Transformers for image recognition at scale, arXiv:2010.11929 · arXiv
- Chen, 2023, 3D TransUNet: Advancing medical image segmentation through vision transformers, arXiv:2310.07781 · arXiv
- Bianco, 2023, Improving image captioning descriptiveness by ranking and LLM-based fusion, arXiv:2306.11593 · arXiv
- Liu, 2024, Efficient partitioning vision transformer on edge devices for distributed inference, arXiv:2410.11650 · arXiv
Cited by
Kumru, E., Ugurlu, G., Sevindik, М., Ekinci, F., Güzel, M. S., Açıcı, K., & Akata, İ. (2025) · Biology · doi:10.3390/biology14070816 · Cited by 15
“In contrast, transformer-based architectures like ViT and MaxViT demonstrated diminished performance, likely due to their greater dependence on large datasets and inherently weaker inductive biases [17,18].”
An Overview Study of Deep Learning in Geophysics: Cross-Cutting Research to Advance Geoscience
Wenxue, Z., Shikun, D., Hongjun, T., Dexiang, Z., Ying, Z., & Jiang, F. (2025) · IEEE Access · doi:10.1109/access.2025.3586693 · Cited by 15
Kumar, R., Kim, Y., & Byun, Y.-C. (2025) · IEEE Access · doi:10.1109/access.2025.3600622 · Cited by 17
“Vision Transformer (ViT) heads represent a breakthrough in computer vision by adapting transformer architectures— originally designed for natural language processing—to image classification tasks [57].”
CDU-Net: An adaptive conditional denoising model for microseismic signals in complex environments
Li, J., Tang, B., Zhang, J., Zhang, S., Tang, S., Huang, X., Yi, G., & Song, C. S. (2026) · Underground Space · doi:10.1016/j.undsp.2026.02.002 · Cited by 8
Cross-dataset video deepfake detection using Transformer and CNN architectures
Petmezas, G. (2026) · Machine Vision and Applications · doi:10.1007/s00138-026-01809-w · Cited by 6
Metadata
- Authors
- Palanisamy, B., Hassija, V., Chatterjee, A., Mandal, A., Chakraborty, D., Pandey, A., Chalapathi, G. S. S., & Kumar, D.
- Affiliations
- KIIT University (IN)
- Bennett University (IN)
- Birla Institute of Technology and Science, Pilani (IN)
- ORCID
- Venue
- IEEE Access
- Published
- 2025
- Volume
- 13
- Pages
- 95496-95523
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W4410536721
- Semantic Scholar
- ac9cc0c28838a037e77f4e19433de170f47b3de9
- Cited by
- 83
- References
- 118
- Citations by year
- 2026: 64 · 2025: 19
- FWCI
- 25.24
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- IEEE Access (publishedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Birla Institute of Technology and Science, Pilani, India
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- vision transformer, self-attention, positional encoding, image tokenization, ViT, Swin Transformer, DeiT, PVT, twins, image classification, object detection, video understanding, diagnostic imaging, CNN-Transformer hybrid, self-supervised learning, interpretability, data efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.