CoChat

Vision Transformers for Image Classification: A Comparative Survey

Open access
Wang, Yaoli, Deng, Yaojun, Zheng, Yuanjin, Chattopadhyay, Pratik, Wang, Lipo · 2025 · Technologies
Cited by 174

The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.

Abstract

Transformers were initially introduced for natural language processing, leveraging the self-attention mechanism. They require minimal inductive biases in their design and can function effectively as set-based architectures. Additionally, transformers excel at capturing long-range dependencies and enabling parallel processing, which allows them to outperform traditional models, such as long short-term memory (LSTM) networks, on sequence-based tasks. In recent years, transformers have been widely adopted in computer vision, driving remarkable advancements in the field. Previous surveys have provided overviews of transformer applications across various computer vision tasks, such as object detection, activity recognition, and image enhancement. In this survey, we focus specifically on image classification. We begin with an introduction to the fundamental concepts of transformers and highlight the first successful Vision Transformer (ViT). Building on the ViT, we review subsequent improvements and optimizations introduced for image classification tasks. We then compare the strengths and limitations of these transformer-based models against classic convolutional neural networks (CNNs) through experiments. Finally, we explore key challenges and potential future directions for image classification transformers.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/technologies13010032

Cites

36 of 36 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 36 references

Research works printed but not linked: 69

  1. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. arXiv.
  2. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
  3. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv.
  4. Bello, I., Zoph, B., Vaswani, A., Shlens, J., and Le, Q.V. (November, January 27). Attention augmented convolutional networks. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
  5. Han, K., Wang, Y., Chen, H., Chen, X., Guo, J., Liu, Z., Tang, Y., Xiao, A., Xu, C., and Xu, Y. (2020). A Survey on Visual Transformer. arXiv.
  6. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.
  7. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. (2020). Training data-efficient image transformers & distillation through attention. arXiv.
  8. Ye, L., Rochan, M., Liu, Z., and Wang, Y. (2019, January 9–15). Cross-modal self-attention network for referring image segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA. · DOI · Paper page
  9. Sun, C., Myers, A., Vondrick, C., Murphy, K., and Schmid, C. (2019, January 9–15). Videobert: A joint model for video and language representation learning. Proceedings of the IEEE/CVF International Conference on Computer Vision, Long Beach, CA, USA. · DOI · Paper page
  10. Azad, 2024, Advances in medical image analysis with vision Transformers: A comprehensive review, Med. Image Anal., 91, 103000, 10.1016/j.media.2023.103000 · DOI · Paper page
  11. Liu, Z., Lv, Q., Yang, Z., Li, Y., Lee, C.H., and Shen, L. (2023). Recent progress in transformer-based medical image analysis. Comput. Biol. Med., 164. · DOI
  12. Maurício, J., Domingues, I., and Bernardino, J. (2023). Comparing vision transformers and convolutional neural networks for image classification: A literature review. Appl. Sci., 13. · DOI · Paper page
  13. Chu, X., Zhang, B., Tian, Z., Wei, X., and Xia, H. (2021). Do we really need explicit position encodings for vision transformers?. arXiv.
  14. Yin, W., Kann, K., Yu, M., and Schütze, H. (2017). Comparative study of CNN and RNN for natural language processing. arXiv.
  15. Cordonnier, J.B., Loukas, A., and Jaggi, M. (2019). On the relationship between self-attention and convolutional layers. arXiv.
  16. Hu, H., Zhang, Z., Xie, Z., and Lin, S. (November, January 27). Local Relation Networks for Image Recognition. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.
  17. Islam, M.A., Jia, S., and Bruce, N.D. (2020). How much position information do convolutional neural networks encode?. arXiv.
  18. Islam, M.A., Kowal, M., Jia, S., Derpanis, K.G., and Bruce, N.D. (2021). Position, padding and predictions: A deeper look at position information in cnns. arXiv.
  19. Zhao, H., Jia, J., and Koltun, V. (2020, January 13–19). Exploring Self-Attention for Image Recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA. · DOI · Paper page
  20. Ramachandran, P., Parmar, N., Vaswani, A., Bello, I., Levskaya, A., and Shlens, J. (2019). Stand-Alone Self-Attention in Vision Models. arXiv.
  21. Vaswani, A., Ramachandran, P., Srinivas, A., Parmar, N., Hechtman, B., and Shlens, J. (2021). Scaling local self-attention for parameter efficient visual backbones. arXiv. · DOI · Paper page
  22. Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., and Wei, F. (2020). Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks. arXiv. · DOI · Paper page
  23. Wang, S., Li, B., Khabsa, M., Fang, H., and Ma, H. (2020). Linformer: Self-attention with linear complexity. arXiv.
  24. Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., and Dai, J. (2019). VL-BERT: Pre-training of Generic Visual-Linguistic Representations. arXiv.
  25. Liu, X., Zhang, F., Hou, Z., Wang, Z., Mian, L., Zhang, J., and Tang, J. (2020). Self-supervised Learning: Generative or Contrastive. arXiv. · DOI · Paper page
  26. Hinton, G.E., Vinyals, O., and Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv.
  27. Gong, J., Qiu, X., Chen, X., Liang, D., and Huang, X. (November, January 31). Convolutional Interaction Network for Natural Language Inference. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. · DOI
  28. Heo, B., Yun, S., Han, D., Chun, S., Choe, J., and Oh, S.J. (2021). Rethinking spatial dimensions of vision transformers. arXiv. · DOI · Paper page
  29. Wu, B., Xu, C., Dai, X., Wan, A., Zhang, P., Yan, Z., Tomizuka, M., Gonzalez, J., Keutzer, K., and Vajda, P. (2020). Visual transformers: Token-based image representation and processing for computer vision. arXiv.
  30. Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., and Zhang, L. (2021). Cvt: Introducing convolutions to vision transformers. arXiv. · DOI · Paper page
  31. Zhou, D., Shi, Y., Kang, B., Yu, W., Jiang, Z., Li, Y., Jin, X., Hou, Q., and Feng, J. (2021). Refiner: Refining Self-attention for Vision Transformers. arXiv.
  32. Li, 2023, UniFormer: Unifying Convolution and Self-Attention for Visual Recognition, IEEE Trans. Pattern Anal. Mach. Intell., 45, 12581, 10.1109/TPAMI.2023.3282631 · DOI · Paper page
  33. Madan, 2024, Self-Supervised Masked Convolutional Transformer Block for Anomaly Detection, IEEE Trans. Pattern Anal. Mach. Intell., 46, 525, 10.1109/TPAMI.2023.3322604 · DOI
  34. Duan, 2023, Dynamic Unary Convolution in Transformers, IEEE Trans. Pattern Anal. Mach. Intell., 45, 12747
  35. Jaouen, 2023, Multi-modal medical Transformers: A meta-analysis for medical image segmentation in oncology, Comput. Med. Imaging Graph., 110, 102308, 10.1016/j.compmedimag.2023.102308 · DOI
  36. Shiri, I., Razeghi, B., Vafaei Sadr, A., Amini, M., Salimi, Y., Ferdowsi, S., Boor, P., Gündüz, D., Voloshynovskiy, S., and Zaidi, H. (2023). Multi-institutional PET/CT image segmentation using federated deep transformer learning. Comput. Methods Programs Biomed., 240. · DOI
  37. Ding, 2023, FTransCNN: Fusing Transformer and a CNN based on fuzzy logic for uncertain medical image segmentation, Inf. Fusion, 99, 101880, 10.1016/j.inffus.2023.101880 · DOI
  38. Li, 2024, UCFilTransNet: Cross-Filtering Transformer-based network for CT image segmentation, Expert Syst. Appl., 238, 121717, 10.1016/j.eswa.2023.121717 · DOI
  39. Karacan, 2023, Multi-image transformer for multi-focus image fusion, Signal Process. Image Commun., 119, 117058, 10.1016/j.image.2023.117058 · DOI
  40. Yang, 2024, Semantic perceptive infrared and visible image fusion Transformer, Pattern Recognit., 149, 110223, 10.1016/j.patcog.2023.110223 · DOI
  41. Mustafa, 2024, TGF: Multiscale transformer graph attention network for multi-sensor image fusion, Expert Syst. Appl., 238, 121789, 10.1016/j.eswa.2023.121789 · DOI
  42. Zhao, 2023, A transformer-based hierarchical registration framework for multimodality deformable image registration, Comput. Med. Imaging Graph., 108, 102286, 10.1016/j.compmedimag.2023.102286 · DOI
  43. Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A.L., and Zhou, Y. (2021). Transunet: Transformers make strong encoders for medical image segmentation. arXiv.
  44. Wu, Z., Liu, Z., Lin, J., Lin, Y., and Han, S. (2020). Lite transformer with long-short range attention. arXiv.
  45. Mehta, S., Koncel-Kedziorski, R., Rastegari, M., and Hajishirzi, H. (2019). Define: Deep factorized input token embeddings for neural sequence modeling. arXiv.
  46. Mehta, S., Ghazvininejad, M., Iyer, S., Zettlemoyer, L., and Hajishirzi, H. (2020). DeLighT: Very Deep and Light-weight Transformer. arXiv.
  47. Hassani, A., Walton, S., Shah, N., Abuduweili, A., Li, J., and Shi, H. (2021). Escaping the Big Data Paradigm with Compact Transformers. arXiv.
  48. Rao, 2023, Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networks, IEEE Trans. Pattern Anal. Mach. Intell., 45, 10883, 10.1109/TPAMI.2023.3263826 · DOI
  49. Wu, 2023, PSLT: A Light-Weight Vision Transformer With Ladder Self-Attention and Progressive Shift, IEEE Trans. Pattern Anal. Mach. Intell., 45, 11120, 10.1109/TPAMI.2023.3265499 · DOI
  50. Mou, 2023, TransCL: Transformer Makes Strong and Flexible Compressive Learning, IEEE Trans. Pattern Anal. Mach. Intell., 45, 5236
  51. Beltagy, I., Peters, M.E., and Cohan, A. (2020). Longformer: The long-document transformer. arXiv.
  52. Zhang, Z., Zhang, H., Zhao, L., Chen, T., and Pfister, T. (2021). Aggregating Nested Transformers. arXiv.
  53. Chen, C.F., Fan, Q., and Panda, R. (2021). Crossvit: Cross-attention multi-scale vision transformer for image classification. arXiv. · DOI · Paper page
  54. Chen, C.F., Fan, Q., Mallinar, N., Sercu, T., and Feris, R. (2018). Big-little net: An efficient multi-scale feature representation for visual and speech recognition. arXiv.
  55. Gong, C., Wang, D., Li, M., Chandra, V., and Liu, Q. (2021). Improve Vision Transformers Training by Suppressing Over-smoothing. arXiv.
  56. Yu, 2024, MetaFormer Baselines for Vision, IEEE Trans. Pattern Anal. Mach. Intell., 46, 896, 10.1109/TPAMI.2023.3329173 · DOI · Paper page
  57. Xu, 2023, TransCenter: Transformers With Dense Representations for Multiple-Object Tracking, IEEE Trans. Pattern Anal. Mach. Intell., 45, 7820, 10.1109/TPAMI.2022.3225078 · DOI · Paper page
  58. Murthy, S., and Krishna Prasad, P.M. (2023). Adversarial transformer network for classification of lung cancer disease from CT scan images. Biomed. Signal Process. Control, 86. · DOI
  59. Zhou, 2023, A superior image inpainting scheme using Transformer-based self-supervised attention GAN model, Expert Syst. Appl., 233, 120906, 10.1016/j.eswa.2023.120906 · DOI
  60. Ren, 2024, Pooling-based Visual Transformer with low complexity attention hashing for image retrieval, Expert Syst. Appl., 241, 122745, 10.1016/j.eswa.2023.122745 · DOI
  61. Sun, 2023, Patch-Based Separable Transformer for Visual Recognition, IEEE Trans. Pattern Anal. Mach. Intell., 45, 9241
  62. Dai, 2023, Unsupervised Pre-Training for Detection Transformers, IEEE Trans. Pattern Anal. Mach. Intell., 45, 12772
  63. Beyer, L., Hénaff, O.J., Kolesnikov, A., Zhai, X., and Oord, A.v.d. (2020). Are we done with ImageNet?. arXiv.
  64. Brock, A., De, S., Smith, S.L., and Simonyan, K. (2021). High-performance large-scale image recognition without normalization. arXiv.
  65. Parkhi, O.M., Vedaldi, A., Zisserman, A., and Jawahar, C. (2012, January 16–21). Cats and dogs. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA. · DOI · Paper page
  66. Krause, J., Stark, M., Deng, J., and Fei-Fei, L. (2013, January 2–8). 3d object representations for fine-grained categorization. Proceedings of the IEEE International Conference on Computer Vision Workshops, Sydney, Australia. · DOI · Paper page
  67. Shen, S., Yao, Z., Gholami, A., Mahoney, M., and Keutzer, K. (2020, January 13–18). PowerNorm: Rethinking Batch Normalization in Transformers. Proceedings of the 37th International Conference on Machine Learning, Online.
  68. Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., and Liu, T. (2020, January 13–18). On Layer Normalization in the Transformer Architecture. Proceedings of the 37th International Conference on Machine Learning, Online.
  69. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021). Learning Transferable Visual Models From Natural Language Supervision. arXiv.

Cited by

See all 174

Metadata

Authors
Wang, Y., Deng, Y., Zheng, Y., Chattopadhyay, P., & Wang, L.
Venue
Technologies
Publisher
MDPI AG
Published
2025-01-12
Volume
13
Issue
1
Pages
32
ISSN
2227-7080 (ISSN-L)
OpenAlex
W4406330070
Cited by
174
References
36
FWCI
63.67
Type
journal article
Language
en
Open access
gold
Funders
Ministry of Education, Singapore (EDUN C-33-18-279-V12)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Vision Transformers for Image Classification: A Comparative Survey. https://app.cochat.ai/papers/10.3390/technologies13010032
The paper
Wang, Y., Deng, Y., Zheng, Y., Chattopadhyay, P., & Wang, L. (2025). Vision Transformers for Image Classification: A Comparative Survey. Technologies. https://doi.org/10.3390/technologies13010032

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.