Vision Transformers for Image Classification: A Comparative Survey
Open accessThe DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Abstract
Transformers were initially introduced for natural language processing, leveraging the self-attention mechanism. They require minimal inductive biases in their design and can function effectively as set-based architectures. Additionally, transformers excel at capturing long-range dependencies and enabling parallel processing, which allows them to outperform traditional models, such as long short-term memory (LSTM) networks, on sequence-based tasks. In recent years, transformers have been widely adopted in computer vision, driving remarkable advancements in the field. Previous surveys have provided overviews of transformer applications across various computer vision tasks, such as object detection, activity recognition, and image enhancement. In this survey, we focus specifically on image classification. We begin with an introduction to the fundamental concepts of transformers and highlight the first successful Vision Transformer (ViT). Building on the ViT, we review subsequent improvements and optimizations introduced for image classification tasks. We then compare the strengths and limitations of these transformer-based models against classic convolutional neural networks (CNNs) through experiments. Finally, we explore key challenges and potential future directions for image classification transformers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939 · cited 1×
“Some researchers propose new attention processes that apply self-attention on individual image blocks while fusing tokens [79,80] to capture multi-scale feature representations [81,82].”
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741
Identity Mappings in Deep Residual Networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · arXiv · doi:10.48550/arxiv.1603.05027 · Cited by 10,665
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595
Research works printed but not linked: 69
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. arXiv.
- Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
- Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv.
- Bello, I., Zoph, B., Vaswani, A., Shlens, J., and Le, Q.V. (November, January 27). Attention augmented convolutional networks. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
- Han, K., Wang, Y., Chen, H., Chen, X., Guo, J., Liu, Z., Tang, Y., Xiao, A., Xu, C., and Xu, Y. (2020). A Survey on Visual Transformer. arXiv.
- Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.
- Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. (2020). Training data-efficient image transformers & distillation through attention. arXiv.
- Ye, L., Rochan, M., Liu, Z., and Wang, Y. (2019, January 9–15). Cross-modal self-attention network for referring image segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA. · DOI · Paper page
- Sun, C., Myers, A., Vondrick, C., Murphy, K., and Schmid, C. (2019, January 9–15). Videobert: A joint model for video and language representation learning. Proceedings of the IEEE/CVF International Conference on Computer Vision, Long Beach, CA, USA. · DOI · Paper page
- Azad, 2024, Advances in medical image analysis with vision Transformers: A comprehensive review, Med. Image Anal., 91, 103000, 10.1016/j.media.2023.103000 · DOI · Paper page
- Liu, Z., Lv, Q., Yang, Z., Li, Y., Lee, C.H., and Shen, L. (2023). Recent progress in transformer-based medical image analysis. Comput. Biol. Med., 164. · DOI
- Maurício, J., Domingues, I., and Bernardino, J. (2023). Comparing vision transformers and convolutional neural networks for image classification: A literature review. Appl. Sci., 13. · DOI · Paper page
- Chu, X., Zhang, B., Tian, Z., Wei, X., and Xia, H. (2021). Do we really need explicit position encodings for vision transformers?. arXiv.
- Yin, W., Kann, K., Yu, M., and Schütze, H. (2017). Comparative study of CNN and RNN for natural language processing. arXiv.
- Cordonnier, J.B., Loukas, A., and Jaggi, M. (2019). On the relationship between self-attention and convolutional layers. arXiv.
- Hu, H., Zhang, Z., Xie, Z., and Lin, S. (November, January 27). Local Relation Networks for Image Recognition. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.
- Islam, M.A., Jia, S., and Bruce, N.D. (2020). How much position information do convolutional neural networks encode?. arXiv.
- Islam, M.A., Kowal, M., Jia, S., Derpanis, K.G., and Bruce, N.D. (2021). Position, padding and predictions: A deeper look at position information in cnns. arXiv.
- Zhao, H., Jia, J., and Koltun, V. (2020, January 13–19). Exploring Self-Attention for Image Recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA. · DOI · Paper page
- Ramachandran, P., Parmar, N., Vaswani, A., Bello, I., Levskaya, A., and Shlens, J. (2019). Stand-Alone Self-Attention in Vision Models. arXiv.
- Vaswani, A., Ramachandran, P., Srinivas, A., Parmar, N., Hechtman, B., and Shlens, J. (2021). Scaling local self-attention for parameter efficient visual backbones. arXiv. · DOI · Paper page
- Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., and Wei, F. (2020). Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks. arXiv. · DOI · Paper page
- Wang, S., Li, B., Khabsa, M., Fang, H., and Ma, H. (2020). Linformer: Self-attention with linear complexity. arXiv.
- Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., and Dai, J. (2019). VL-BERT: Pre-training of Generic Visual-Linguistic Representations. arXiv.
- Liu, X., Zhang, F., Hou, Z., Wang, Z., Mian, L., Zhang, J., and Tang, J. (2020). Self-supervised Learning: Generative or Contrastive. arXiv. · DOI · Paper page
- Hinton, G.E., Vinyals, O., and Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv.
- Gong, J., Qiu, X., Chen, X., Liang, D., and Huang, X. (November, January 31). Convolutional Interaction Network for Natural Language Inference. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. · DOI
- Heo, B., Yun, S., Han, D., Chun, S., Choe, J., and Oh, S.J. (2021). Rethinking spatial dimensions of vision transformers. arXiv. · DOI · Paper page
- Wu, B., Xu, C., Dai, X., Wan, A., Zhang, P., Yan, Z., Tomizuka, M., Gonzalez, J., Keutzer, K., and Vajda, P. (2020). Visual transformers: Token-based image representation and processing for computer vision. arXiv.
- Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., and Zhang, L. (2021). Cvt: Introducing convolutions to vision transformers. arXiv. · DOI · Paper page
- Zhou, D., Shi, Y., Kang, B., Yu, W., Jiang, Z., Li, Y., Jin, X., Hou, Q., and Feng, J. (2021). Refiner: Refining Self-attention for Vision Transformers. arXiv.
- Li, 2023, UniFormer: Unifying Convolution and Self-Attention for Visual Recognition, IEEE Trans. Pattern Anal. Mach. Intell., 45, 12581, 10.1109/TPAMI.2023.3282631 · DOI · Paper page
- Madan, 2024, Self-Supervised Masked Convolutional Transformer Block for Anomaly Detection, IEEE Trans. Pattern Anal. Mach. Intell., 46, 525, 10.1109/TPAMI.2023.3322604 · DOI
- Duan, 2023, Dynamic Unary Convolution in Transformers, IEEE Trans. Pattern Anal. Mach. Intell., 45, 12747
- Jaouen, 2023, Multi-modal medical Transformers: A meta-analysis for medical image segmentation in oncology, Comput. Med. Imaging Graph., 110, 102308, 10.1016/j.compmedimag.2023.102308 · DOI
- Shiri, I., Razeghi, B., Vafaei Sadr, A., Amini, M., Salimi, Y., Ferdowsi, S., Boor, P., Gündüz, D., Voloshynovskiy, S., and Zaidi, H. (2023). Multi-institutional PET/CT image segmentation using federated deep transformer learning. Comput. Methods Programs Biomed., 240. · DOI
- Ding, 2023, FTransCNN: Fusing Transformer and a CNN based on fuzzy logic for uncertain medical image segmentation, Inf. Fusion, 99, 101880, 10.1016/j.inffus.2023.101880 · DOI
- Li, 2024, UCFilTransNet: Cross-Filtering Transformer-based network for CT image segmentation, Expert Syst. Appl., 238, 121717, 10.1016/j.eswa.2023.121717 · DOI
- Karacan, 2023, Multi-image transformer for multi-focus image fusion, Signal Process. Image Commun., 119, 117058, 10.1016/j.image.2023.117058 · DOI
- Yang, 2024, Semantic perceptive infrared and visible image fusion Transformer, Pattern Recognit., 149, 110223, 10.1016/j.patcog.2023.110223 · DOI
- Mustafa, 2024, TGF: Multiscale transformer graph attention network for multi-sensor image fusion, Expert Syst. Appl., 238, 121789, 10.1016/j.eswa.2023.121789 · DOI
- Zhao, 2023, A transformer-based hierarchical registration framework for multimodality deformable image registration, Comput. Med. Imaging Graph., 108, 102286, 10.1016/j.compmedimag.2023.102286 · DOI
- Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A.L., and Zhou, Y. (2021). Transunet: Transformers make strong encoders for medical image segmentation. arXiv.
- Wu, Z., Liu, Z., Lin, J., Lin, Y., and Han, S. (2020). Lite transformer with long-short range attention. arXiv.
- Mehta, S., Koncel-Kedziorski, R., Rastegari, M., and Hajishirzi, H. (2019). Define: Deep factorized input token embeddings for neural sequence modeling. arXiv.
- Mehta, S., Ghazvininejad, M., Iyer, S., Zettlemoyer, L., and Hajishirzi, H. (2020). DeLighT: Very Deep and Light-weight Transformer. arXiv.
- Hassani, A., Walton, S., Shah, N., Abuduweili, A., Li, J., and Shi, H. (2021). Escaping the Big Data Paradigm with Compact Transformers. arXiv.
- Rao, 2023, Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networks, IEEE Trans. Pattern Anal. Mach. Intell., 45, 10883, 10.1109/TPAMI.2023.3263826 · DOI
- Wu, 2023, PSLT: A Light-Weight Vision Transformer With Ladder Self-Attention and Progressive Shift, IEEE Trans. Pattern Anal. Mach. Intell., 45, 11120, 10.1109/TPAMI.2023.3265499 · DOI
- Mou, 2023, TransCL: Transformer Makes Strong and Flexible Compressive Learning, IEEE Trans. Pattern Anal. Mach. Intell., 45, 5236
- Beltagy, I., Peters, M.E., and Cohan, A. (2020). Longformer: The long-document transformer. arXiv.
- Zhang, Z., Zhang, H., Zhao, L., Chen, T., and Pfister, T. (2021). Aggregating Nested Transformers. arXiv.
- Chen, C.F., Fan, Q., and Panda, R. (2021). Crossvit: Cross-attention multi-scale vision transformer for image classification. arXiv. · DOI · Paper page
- Chen, C.F., Fan, Q., Mallinar, N., Sercu, T., and Feris, R. (2018). Big-little net: An efficient multi-scale feature representation for visual and speech recognition. arXiv.
- Gong, C., Wang, D., Li, M., Chandra, V., and Liu, Q. (2021). Improve Vision Transformers Training by Suppressing Over-smoothing. arXiv.
- Yu, 2024, MetaFormer Baselines for Vision, IEEE Trans. Pattern Anal. Mach. Intell., 46, 896, 10.1109/TPAMI.2023.3329173 · DOI · Paper page
- Xu, 2023, TransCenter: Transformers With Dense Representations for Multiple-Object Tracking, IEEE Trans. Pattern Anal. Mach. Intell., 45, 7820, 10.1109/TPAMI.2022.3225078 · DOI · Paper page
- Murthy, S., and Krishna Prasad, P.M. (2023). Adversarial transformer network for classification of lung cancer disease from CT scan images. Biomed. Signal Process. Control, 86. · DOI
- Zhou, 2023, A superior image inpainting scheme using Transformer-based self-supervised attention GAN model, Expert Syst. Appl., 233, 120906, 10.1016/j.eswa.2023.120906 · DOI
- Ren, 2024, Pooling-based Visual Transformer with low complexity attention hashing for image retrieval, Expert Syst. Appl., 241, 122745, 10.1016/j.eswa.2023.122745 · DOI
- Sun, 2023, Patch-Based Separable Transformer for Visual Recognition, IEEE Trans. Pattern Anal. Mach. Intell., 45, 9241
- Dai, 2023, Unsupervised Pre-Training for Detection Transformers, IEEE Trans. Pattern Anal. Mach. Intell., 45, 12772
- Beyer, L., Hénaff, O.J., Kolesnikov, A., Zhai, X., and Oord, A.v.d. (2020). Are we done with ImageNet?. arXiv.
- Brock, A., De, S., Smith, S.L., and Simonyan, K. (2021). High-performance large-scale image recognition without normalization. arXiv.
- Parkhi, O.M., Vedaldi, A., Zisserman, A., and Jawahar, C. (2012, January 16–21). Cats and dogs. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA. · DOI · Paper page
- Krause, J., Stark, M., Deng, J., and Fei-Fei, L. (2013, January 2–8). 3d object representations for fine-grained categorization. Proceedings of the IEEE International Conference on Computer Vision Workshops, Sydney, Australia. · DOI · Paper page
- Shen, S., Yao, Z., Gholami, A., Mahoney, M., and Keutzer, K. (2020, January 13–18). PowerNorm: Rethinking Batch Normalization in Transformers. Proceedings of the 37th International Conference on Machine Learning, Online.
- Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., and Liu, T. (2020, January 13–18). On Layer Normalization in the Transformer Architecture. Proceedings of the 37th International Conference on Machine Learning, Online.
- Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021). Learning Transferable Visual Models From Natural Language Supervision. arXiv.
Cited by
Transformers for Vision: A Survey on Innovative Methods for Computer Vision
Palanisamy, B., Hassija, V., Chatterjee, A., Mandal, A., Chakraborty, D., Pandey, A., Chalapathi, G. S. S., & Kumar, D. (2025) · IEEE Access · doi:10.1109/access.2025.3571735 · Cited by 83
Lamaakal, I., Yahyati, C., Maleh, Y., El Makkaoui, K., Ouahbi, I., El-Latif, A. A. A., Zomorodi, M., & El-Rahiem, B. A. (2025) · Scientific Reports · doi:10.1038/s41598-025-26297-2 · Cited by 26
Conflict-aware uncertainty alignment for tool-augmented document-centric multimodal intelligence
Muthusami, R., & Saritha, K. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-02409-3
Metadata
- Authors
- Wang, Y., Deng, Y., Zheng, Y., Chattopadhyay, P., & Wang, L.
- Venue
- Technologies
- Publisher
- MDPI AG
- Published
- 2025-01-12
- Volume
- 13
- Issue
- 1
- Pages
- 32
- ISSN
- 2227-7080 (ISSN-L)
- OpenAlex
- W4406330070
- Cited by
- 174
- References
- 36
- FWCI
- 63.67
- Type
- journal article
- Language
- en
- Open access
- gold
- Funders
- Ministry of Education, Singapore (EDUN C-33-18-279-V12)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Vision Transformers for Image Classification: A Comparative Survey. https://app.cochat.ai/papers/10.3390/technologies13010032
- The paper
- Wang, Y., Deng, Y., Zheng, Y., Chattopadhyay, P., & Wang, L. (2025). Vision Transformers for Image Classification: A Comparative Survey. Technologies. https://doi.org/10.3390/technologies13010032
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.