CoChat

Vision Transformers for Image Classification: A Comparative Survey

Open access
Wang, Yaoli, Deng, Yaojun, Zheng, Yuanjin, Chattopadhyay, Pratik, Wang, Lipo · 2025 · Technologies
Cited by 174

The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.

Abstract

Transformers were initially introduced for natural language processing, leveraging the self-attention mechanism. They require minimal inductive biases in their design and can function effectively as set-based architectures. Additionally, transformers excel at capturing long-range dependencies and enabling parallel processing, which allows them to outperform traditional models, such as long short-term memory (LSTM) networks, on sequence-based tasks. In recent years, transformers have been widely adopted in computer vision, driving remarkable advancements in the field. Previous surveys have provided overviews of transformer applications across various computer vision tasks, such as object detection, activity recognition, and image enhancement. In this survey, we focus specifically on image classification. We begin with an introduction to the fundamental concepts of transformers and highlight the first successful Vision Transformer (ViT). Building on the ViT, we review subsequent improvements and optimizations introduced for image classification tasks. We then compare the strengths and limitations of these transformer-based models against classic convolutional neural networks (CNNs) through experiments. Finally, we explore key challenges and potential future directions for image classification transformers.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/technologies13010032

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 68

  1. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. arXiv.
  2. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
  3. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv.
  4. Bello, I., Zoph, B., Vaswani, A., Shlens, J., and Le, Q.V. (November, January 27). Attention augmented convolutional networks. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
  5. Han, K., Wang, Y., Chen, H., Chen, X., Guo, J., Liu, Z., Tang, Y., Xiao, A., Xu, C., and Xu, Y. (2020). A Survey on Visual Transformer. arXiv.
  6. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.
  7. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. (2020). Training data-efficient image transformers & distillation through attention. arXiv.
  8. Ye, L., Rochan, M., Liu, Z., and Wang, Y. (2019, January 9–15). Cross-modal self-attention network for referring image segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA. · DOI · Paper page
  9. Sun, C., Myers, A., Vondrick, C., Murphy, K., and Schmid, C. (2019, January 9–15). Videobert: A joint model for video and language representation learning. Proceedings of the IEEE/CVF International Conference on Computer Vision, Long Beach, CA, USA. · DOI · Paper page
  10. Azad, 2024, Advances in medical image analysis with vision Transformers: A comprehensive review, Med. Image Anal., 91, 103000, 10.1016/j.media.2023.103000 · DOI · Paper page
  11. Liu, Z., Lv, Q., Yang, Z., Li, Y., Lee, C.H., and Shen, L. (2023). Recent progress in transformer-based medical image analysis. Comput. Biol. Med., 164. · DOI
  12. Maurício, J., Domingues, I., and Bernardino, J. (2023). Comparing vision transformers and convolutional neural networks for image classification: A literature review. Appl. Sci., 13. · DOI · Paper page
  13. Chu, X., Zhang, B., Tian, Z., Wei, X., and Xia, H. (2021). Do we really need explicit position encodings for vision transformers?. arXiv.
  14. Yin, W., Kann, K., Yu, M., and Schütze, H. (2017). Comparative study of CNN and RNN for natural language processing. arXiv.
  15. Cordonnier, J.B., Loukas, A., and Jaggi, M. (2019). On the relationship between self-attention and convolutional layers. arXiv.
  16. Hu, H., Zhang, Z., Xie, Z., and Lin, S. (November, January 27). Local Relation Networks for Image Recognition. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.
  17. Islam, M.A., Jia, S., and Bruce, N.D. (2020). How much position information do convolutional neural networks encode?. arXiv.
  18. Islam, M.A., Kowal, M., Jia, S., Derpanis, K.G., and Bruce, N.D. (2021). Position, padding and predictions: A deeper look at position information in cnns. arXiv.
  19. Ramachandran, P., Parmar, N., Vaswani, A., Bello, I., Levskaya, A., and Shlens, J. (2019). Stand-Alone Self-Attention in Vision Models. arXiv.
  20. Vaswani, A., Ramachandran, P., Srinivas, A., Parmar, N., Hechtman, B., and Shlens, J. (2021). Scaling local self-attention for parameter efficient visual backbones. arXiv. · DOI · Paper page
  21. Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., and Wei, F. (2020). Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks. arXiv. · DOI · Paper page
  22. Wang, S., Li, B., Khabsa, M., Fang, H., and Ma, H. (2020). Linformer: Self-attention with linear complexity. arXiv.
  23. Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., and Dai, J. (2019). VL-BERT: Pre-training of Generic Visual-Linguistic Representations. arXiv.
  24. Liu, X., Zhang, F., Hou, Z., Wang, Z., Mian, L., Zhang, J., and Tang, J. (2020). Self-supervised Learning: Generative or Contrastive. arXiv. · DOI · Paper page
  25. Hinton, G.E., Vinyals, O., and Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv.
  26. Gong, J., Qiu, X., Chen, X., Liang, D., and Huang, X. (November, January 31). Convolutional Interaction Network for Natural Language Inference. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. · DOI
  27. Heo, B., Yun, S., Han, D., Chun, S., Choe, J., and Oh, S.J. (2021). Rethinking spatial dimensions of vision transformers. arXiv. · DOI · Paper page
  28. Wu, B., Xu, C., Dai, X., Wan, A., Zhang, P., Yan, Z., Tomizuka, M., Gonzalez, J., Keutzer, K., and Vajda, P. (2020). Visual transformers: Token-based image representation and processing for computer vision. arXiv.
  29. Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., and Zhang, L. (2021). Cvt: Introducing convolutions to vision transformers. arXiv. · DOI · Paper page
  30. Zhou, D., Shi, Y., Kang, B., Yu, W., Jiang, Z., Li, Y., Jin, X., Hou, Q., and Feng, J. (2021). Refiner: Refining Self-attention for Vision Transformers. arXiv.
  31. Li, 2023, UniFormer: Unifying Convolution and Self-Attention for Visual Recognition, IEEE Trans. Pattern Anal. Mach. Intell., 45, 12581, 10.1109/TPAMI.2023.3282631 · DOI · Paper page
  32. Madan, 2024, Self-Supervised Masked Convolutional Transformer Block for Anomaly Detection, IEEE Trans. Pattern Anal. Mach. Intell., 46, 525, 10.1109/TPAMI.2023.3322604 · DOI
  33. Duan, 2023, Dynamic Unary Convolution in Transformers, IEEE Trans. Pattern Anal. Mach. Intell., 45, 12747
  34. Jaouen, 2023, Multi-modal medical Transformers: A meta-analysis for medical image segmentation in oncology, Comput. Med. Imaging Graph., 110, 102308, 10.1016/j.compmedimag.2023.102308 · DOI
  35. Shiri, I., Razeghi, B., Vafaei Sadr, A., Amini, M., Salimi, Y., Ferdowsi, S., Boor, P., Gündüz, D., Voloshynovskiy, S., and Zaidi, H. (2023). Multi-institutional PET/CT image segmentation using federated deep transformer learning. Comput. Methods Programs Biomed., 240. · DOI
  36. Ding, 2023, FTransCNN: Fusing Transformer and a CNN based on fuzzy logic for uncertain medical image segmentation, Inf. Fusion, 99, 101880, 10.1016/j.inffus.2023.101880 · DOI
  37. Li, 2024, UCFilTransNet: Cross-Filtering Transformer-based network for CT image segmentation, Expert Syst. Appl., 238, 121717, 10.1016/j.eswa.2023.121717 · DOI
  38. Karacan, 2023, Multi-image transformer for multi-focus image fusion, Signal Process. Image Commun., 119, 117058, 10.1016/j.image.2023.117058 · DOI
  39. Yang, 2024, Semantic perceptive infrared and visible image fusion Transformer, Pattern Recognit., 149, 110223, 10.1016/j.patcog.2023.110223 · DOI
  40. Mustafa, 2024, TGF: Multiscale transformer graph attention network for multi-sensor image fusion, Expert Syst. Appl., 238, 121789, 10.1016/j.eswa.2023.121789 · DOI
  41. Zhao, 2023, A transformer-based hierarchical registration framework for multimodality deformable image registration, Comput. Med. Imaging Graph., 108, 102286, 10.1016/j.compmedimag.2023.102286 · DOI
  42. Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A.L., and Zhou, Y. (2021). Transunet: Transformers make strong encoders for medical image segmentation. arXiv.
  43. Wu, Z., Liu, Z., Lin, J., Lin, Y., and Han, S. (2020). Lite transformer with long-short range attention. arXiv.
  44. Mehta, S., Koncel-Kedziorski, R., Rastegari, M., and Hajishirzi, H. (2019). Define: Deep factorized input token embeddings for neural sequence modeling. arXiv.
  45. Mehta, S., Ghazvininejad, M., Iyer, S., Zettlemoyer, L., and Hajishirzi, H. (2020). DeLighT: Very Deep and Light-weight Transformer. arXiv.
  46. Hassani, A., Walton, S., Shah, N., Abuduweili, A., Li, J., and Shi, H. (2021). Escaping the Big Data Paradigm with Compact Transformers. arXiv.
  47. Rao, 2023, Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networks, IEEE Trans. Pattern Anal. Mach. Intell., 45, 10883, 10.1109/TPAMI.2023.3263826 · DOI
  48. Wu, 2023, PSLT: A Light-Weight Vision Transformer With Ladder Self-Attention and Progressive Shift, IEEE Trans. Pattern Anal. Mach. Intell., 45, 11120, 10.1109/TPAMI.2023.3265499 · DOI
  49. Mou, 2023, TransCL: Transformer Makes Strong and Flexible Compressive Learning, IEEE Trans. Pattern Anal. Mach. Intell., 45, 5236
  50. Beltagy, I., Peters, M.E., and Cohan, A. (2020). Longformer: The long-document transformer. arXiv.
  51. Zhang, Z., Zhang, H., Zhao, L., Chen, T., and Pfister, T. (2021). Aggregating Nested Transformers. arXiv.
  52. Chen, C.F., Fan, Q., and Panda, R. (2021). Crossvit: Cross-attention multi-scale vision transformer for image classification. arXiv. · DOI · Paper page
  53. Chen, C.F., Fan, Q., Mallinar, N., Sercu, T., and Feris, R. (2018). Big-little net: An efficient multi-scale feature representation for visual and speech recognition. arXiv.
  54. Gong, C., Wang, D., Li, M., Chandra, V., and Liu, Q. (2021). Improve Vision Transformers Training by Suppressing Over-smoothing. arXiv.
  55. Yu, 2024, MetaFormer Baselines for Vision, IEEE Trans. Pattern Anal. Mach. Intell., 46, 896, 10.1109/TPAMI.2023.3329173 · DOI · Paper page
  56. Xu, 2023, TransCenter: Transformers With Dense Representations for Multiple-Object Tracking, IEEE Trans. Pattern Anal. Mach. Intell., 45, 7820, 10.1109/TPAMI.2022.3225078 · DOI · Paper page
  57. Murthy, S., and Krishna Prasad, P.M. (2023). Adversarial transformer network for classification of lung cancer disease from CT scan images. Biomed. Signal Process. Control, 86. · DOI
  58. Zhou, 2023, A superior image inpainting scheme using Transformer-based self-supervised attention GAN model, Expert Syst. Appl., 233, 120906, 10.1016/j.eswa.2023.120906 · DOI
  59. Ren, 2024, Pooling-based Visual Transformer with low complexity attention hashing for image retrieval, Expert Syst. Appl., 241, 122745, 10.1016/j.eswa.2023.122745 · DOI
  60. Sun, 2023, Patch-Based Separable Transformer for Visual Recognition, IEEE Trans. Pattern Anal. Mach. Intell., 45, 9241
  61. Dai, 2023, Unsupervised Pre-Training for Detection Transformers, IEEE Trans. Pattern Anal. Mach. Intell., 45, 12772
  62. Beyer, L., Hénaff, O.J., Kolesnikov, A., Zhai, X., and Oord, A.v.d. (2020). Are we done with ImageNet?. arXiv.
  63. Brock, A., De, S., Smith, S.L., and Simonyan, K. (2021). High-performance large-scale image recognition without normalization. arXiv.
  64. Parkhi, O.M., Vedaldi, A., Zisserman, A., and Jawahar, C. (2012, January 16–21). Cats and dogs. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA. · DOI · Paper page
  65. Krause, J., Stark, M., Deng, J., and Fei-Fei, L. (2013, January 2–8). 3d object representations for fine-grained categorization. Proceedings of the IEEE International Conference on Computer Vision Workshops, Sydney, Australia. · DOI · Paper page
  66. Shen, S., Yao, Z., Gholami, A., Mahoney, M., and Keutzer, K. (2020, January 13–18). PowerNorm: Rethinking Batch Normalization in Transformers. Proceedings of the 37th International Conference on Machine Learning, Online.
  67. Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., and Liu, T. (2020, January 13–18). On Layer Normalization in the Transformer Architecture. Proceedings of the 37th International Conference on Machine Learning, Online.
  68. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021). Learning Transferable Visual Models From Natural Language Supervision. arXiv.

Cited by

See all 174

Metadata

Authors
Wang, Y., Deng, Y., Zheng, Y., Chattopadhyay, P., & Wang, L.
Venue
Technologies
Publisher
MDPI AG
Published
2025-01-12
Volume
13
Issue
1
Pages
32
ISSN
2227-7080 (ISSN-L)
OpenAlex
W4406330070
Cited by
174
References
37
FWCI
63.67
Type
journal article
Language
en
Open access
gold
Funders
Ministry of Education, Singapore (EDUN C-33-18-279-V12)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.