The DOI matches its DataCite record, with a small difference in title or year; 56 of the 56 references it cites were found in a registry or an index.
Transformer in Transformer
Open accessAbstract
Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (e.g., 16$\times$16) as "visual sentences" and present to further divide them into smaller patches (e.g., 4$\times$4) as "visual words". The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs. Features of both words and sentences will be aggregated to enhance the representation ability. Experiments on several benchmarks demonstrate the effectiveness of the proposed TNT architecture, e.g., we achieve an 81.5% top-1 accuracy on the ImageNet, which is about 1.7% higher than that of the state-of-the-art visual transformer with similar computational cost. The PyTorch code is available at https://github.com/huawei-noah/CV-Backbones, and the MindSpore code is available at https://gitee.com/mindspore/models/tree/master/research/cv/TNT.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For example, in multi-tenant architectures, system prompts are shared across multiple requests and frequently contain detailed instructions or examples to guide the model [19].”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [19]
In the citing paper
“PromptCache [19] proposes modular reuse beyond prefixes but sacrifices accuracy, with reported performance drops of up to 43% [72], making it unsuitable for tasks requiring high precision.”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [19]
In the citing paper
“related experiments results further demonstrate ViT also has remarkable representation capacity when dealing with images and videos [19].”
From Deepfake Video Detection with Spatiotemporal Dropout Transformer (2022) · cites this paper as [19]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 6×
“Compared to CNNs, TNT can outperform the widely-used ResNet and RegNet.”
BackgroundMethodInfluentialVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3×
“The effect of #heads in standard transformer has been investigated in multiple works [21, 35] and a head width of 64 is recommended for visual tasks [9, 31].”
BackgroundMethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“In the past decade, the mainstream deep neural architectures used in the computer vision (CV) are mainly established on convolutional neural networks (CNNs) [17, 12, 11].”
BackgroundFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472 · cited 1×
“As a general backbone network, TNT can also be applied with multi-scale vision models like Faster RCNN [29].”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472 · cited 7×
“The recent transformerbased models like ViT [9] and DeiT [31] are compared.”
MethodInfluentialImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 3×
“ImageNet ILSVRC 2012 [26] is an image classification benchmark consisting of 1.”
BackgroundInfluentialSqueeze-and-Excitation NetworksPreprint
Hu, J., Shen, L., & Sun, G. (2017) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1709.01507 · Cited by 37,845 · cited 1×
“Inspired by squeeze-and-excitation (SE) network for CNNs [16], we propose to explore channel-wise attention for transformers.”
MethodPyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939 · cited 2×
“We evaluate TNT-S and DeiT-S on Faster RCNN with FPN [21].”
BackgroundMethod
Research works printed but not linked: 5
- [6][6] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML , 2020. -
- [11][11] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. arXiv preprint arXiv:2012.12556 , 2020. - · arXiv
- [23][23] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017. - · arXiv
- [26][26] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR , pages 3498-3505. IEEE, 2012. -
- [37][37] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008. -
Cited by
Sorted by citations
Items 201–300 of 1,009
Chang, B., Wang, Y., Zhao, X., Li, G., & Yuan, P. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121638 · Cited by 86
Self-citationSarkar, R., Liang, H., Fan, Z., Wang, Z., & Hao, C. (2023) · 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD) · doi:10.1109/iccad57390.2023.10323651 · Cited by 64
“…models, a new type of activation function, GELU (Gaussian Error Linear Unit) [11], has been widely used to improve training efficiency and accuracy [1], [7], [9], [27], [29]; however, its non-linearity introduces a large resource overhead and an obvious accuracy drop on FPGA (more details in Sec.”
MethodScalableViT: Rethinking the Context-oriented Generalization of Vision Transformer
Yang, R., Ma, H., Wu, J., Tang, Y.-S., Xiao, X., Zheng, M., & Li, X. (2022) · European Conference on Computer Vision · doi:10.48550/arxiv.2203.10790 · Cited by 71
“0] [ 1 64 , 1 16 , 1 4 , 1] ScalableViT-L 128 [2,6,12,4] [4,8,16,32] [0.”
CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity Prediction
Song, Z.-R., Xu, Y., He, Z.-Z., Jiang, L., Jing, N.-F., & Liang, X.-Y. (2022) · arXiv.org · doi:10.48550/arxiv.2203.04570 · Cited by 71
“In recent years, Transformer based models [10, 21, 22] have achieved high accuracy in image classification tasks.”
BackgroundS2-MLPv2: Improved Spatial-Shift MLP Architecture for VisionPreprint
Yu, T., Li, X., Cai, Y., Sun, M.-M., & Li, P. (2021) · arXiv.org · doi:10.48550/arxiv.2108.01072 · Cited by 71
“Meanwhile, we also compare with vision Transformer models including ViT (Dosovitskiy et al., 2021), DeiT (Touvron et al., 2020), PVT (Wang et al., 2021b), T2T (Yuan et al., 2021), TNT (Han et al., 2021), PiT (Heo et al., 2021), MViT (Fan et al., 2021) CaiT (Tou-vron et al., 2021b), Swin (Liu et al., 2021b), Shuffle Tra…”
BackgroundMethodTransformer-Based Named Entity Recognition in Construction Supply Chain Risk Management in Australia
Shishehgarkhaneh, M. B., Moehler, R., Fang, Y., Hijazi, A. A., & Aboutorab, H. (2024) · IEEE Access · doi:10.1109/access.2024.3377232 · Cited by 79
EGformer: Equirectangular Geometry-biased Transformer for 360 Depth Estimation
Yun, I., Shin, C., Lee, H., Lee, H.-J., & Rhee, C.-E. (2023) · IEEE International Conference on Computer Vision · doi:10.1109/iccv51070.2023.00561 · Cited by 69
“Compared to a CNN, ViT [14] possesses a global receptive field, which is highly beneficial for various vision tasks [6, 17, 52, 47, 28, 21].”
BackgroundRethinking Local Perception in Lightweight Vision Transformer
Fan, Q., Huang, H.-B., Guan, J.-Y., & He, R. (2023) · arXiv.org · doi:10.48550/arxiv.2303.17803 · Cited by 69
A review of transformer models in drug discovery and beyond
Jiang, J., Chen, L., Ke, L., Dou, B., Zhang, C., Feng, H., Zhu, Y., Qiu, H., Zhang, B., & Wei, G. (2024) · Journal of Pharmaceutical Analysis · doi:10.1016/j.jpha.2024.101081 · Cited by 87
“The transformer model adopts a novel structure that encodes input data into features with strong representation ability through the attention mechanism [36], replacing the loop layer in the traditional encoder-decoder architecture with the multi-head attention mechanism [2].”
Learning Heavily-Degraded Prior for Underwater Object Detection
Fu, C.-P., Fan, X., Xiao, J.-W., Yuan, W., Liu, R., & Luo, Z.-X. (2023) · IEEE transactions on circuits and systems for video technology (Print) · doi:10.1109/tcsvt.2023.3271644 · Cited by 68
“After DETR, some arts concentrate on improving specific parts such as transformer backbone [15], [33], [34], [35] and pre-training scheme [14], [36].”
BackgroundMVT: Multi-view Vision Transformer for 3D Object Recognition
Chen, S., Yu, T., & Li, P. (2021) · British Machine Vision Conference · doi:10.48550/arxiv.2110.13083 · Cited by 68
“T2T [40] and TNT [10] focus on improving the effectiveness of modeling local structure within patches.”
BackgroundEEG-based seizure prediction via hybrid vision transformer and data uncertainty learning
Deng, Z., Li, C., Song, R.-C., Liu, X., Qian, R.-B., & Chen, X. (2023) · Engineering applications of artificial intelligence · doi:10.1016/j.engappai.2023.106401 · Cited by 67
TFormer: A throughout fusion transformer for multi-modal skin lesion diagnosis
Zhang, Y.-L., Xie, F.-Y., Chen, J., & Liu, J. (2022) · Comput. Biol. Medicine · doi:10.48550/arxiv.2211.11393 · Cited by 67
“Recently, the follow-ups (Touvron et al., 2021; Han et al., 2021; Yuan et al., 2021; Wang et al., 2021a; Chu et al., 2021; Lee et al., 2021; Liu et al., 2021; Wang et al., 2021b) improved ViT for resolving the problems above.”
BackgroundPerLLM: Personalized Inference Scheduling with Edge-Cloud Collaboration for Diverse LLM Services
Yang, Z.-M., Yang, Y., Zhao, C., Guo, Q., He, W.-K., & Ji, W. (2024) · arXiv.org · doi:10.48550/arxiv.2405.14636 · Cited by 65
“The core architecture behind most LLMs is the Transformer [13], a type of deep learning model that uses the mechanism of self-attention [26] to weigh the influence of different words in a sentence.”
BackgroundViTA: A Vision Transformer Inference Accelerator for Edge Applications
Nag, S., Datta, G., Kundu, S., Chandrachoodan, N., & Beerel, P. A. (2023) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas46773.2023.10181988 · Cited by 76
“Although the initial works in this direction lacked scalability, subsequent works such as Vision Transformers (ViT) [1], Swin Transformers [2], TNT [3], and Data-efficient image Transformers (DeiT) [4] that adopted model architectures similar to the NLP-based Transformer [5], replacing word tokens…”
MethodEfficient self-attention mechanism and structural distilling model for Alzheimer’s disease diagnosis
Zhu, J., Tan, Y. H. V., Lin, R., Miao, J., Fan, X., Zhu, Y., Liang, P., Gong, J., & He, H. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105737 · Cited by 74
Training-free Transformer Architecture Search
Zhou, Q., Sheng, K., Zheng, X., Li, K., Sun, X., Tian, Y., Chen, J., & Ji, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01062 · Cited by 64
“Compared with other manually designed ViT architectures [6,11,14,27,37], ours TF-TAS achieves competitive results.”
BackgroundMethodM2BIST-SPNet: RUL prediction for railway signaling electromechanical devices
Hu, X.-X., Tan, L., & Tang, T. (2024) · Journal of Supercomputing · doi:10.1007/s11227-024-06111-y · Cited by 64
“Position encoding commonly employs sine and cosine functions to generate position vectors due to their periodic nature, providing a continuous and smooth representation of relationships between different positions [41, 42].”
Jin, X., Zhou, J., Rao, Y., Zhang, X.-D., Zhang, W., Ba, W., Zhou, X., & Zhang, T. (2023) · Geoderma · doi:10.1016/j.geoderma.2023.116555 · Cited by 64
LipsFormer: Introducing Lipschitz Continuity to Vision Transformers
Qi, X., Wang, J., Chen, Y.-H., Shi, Y.-K., & Zhang, L. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2304.09856 · Cited by 63
“The number of Lips-Former blocks within the four computation stages are [1, 2, 21, 1] for LipsFormer-CSwin-T, [2, 4, 32, 2] for LipsFormer-CSwin-S and LipsFormer-CSwin-B.”
MethodGenerative Adversarial Networks for Image Super-Resolution: A Survey
Wu, Z., Zhang, X.-Y., Yu, Y.-B., Zhu, Q., Lin, J. C.-W., & Tian, C.-W. (2022) · Computers, Materials & Continua · doi:10.48550/arxiv.2204.13620 · Cited by 63
“For example, using attention mechanism (i.e., Transformer [213]), residual learning operations, concatenation operations act a generator and discriminator to extract more effective features to enhance stabilities for accelerating GAN models in image super-resolution.”
TransFace: Calibrating Transformer Training for Face Recognition from a Data-Centric Perspective
Dan, J., Liu, Y., Xie, H., Deng, J., Xie, H., Xie, X., & Sun, B. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01887 · Cited by 75
“To better extract both global and local visual information, TNT [22] employs an inner transformer block to model the relationship among sub-patches.”
Patch Similarity Aware Data-Free Quantization for Vision Transformers
Li, Z., Ma, L., Chen, M., Xiao, J., & Gu, Q. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20083-0_10 · Cited by 61
“great success on various computer vision tasks [1, 6, 15, 33, 40].”
BackgroundSSTrans-Net: Smart Swin Transformer Network for medical image segmentation
Fu, L., Chen, Y., Ji, W., & Feng, Y. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106071 · Cited by 68
MLPST: MLP is All You Need for Spatio-Temporal Prediction
Zhang, Z., Huang, Z., Hu, Z., Zhao, X., Wang, W., Liu, Z., Zhang, J., Qin, S. J., & Zhao, H. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3614969 · Cited by 42
“Spatial maps are often represented and processed as image-like matrices, which has led to the widespread application of Convolutional Neural Networks (CNNs) [4, 18, 23, 33, 38] and Vision Transformers (ViT) [9, 13, 22, 30], which have been proven to be effective in capturing spatial features.”
BackgroundVDTR: Video Deblurring With Transformer
Cao, M., Fan, Y., Zhang, Y., Wang, J., & Yang, Y. (2022) · IEEE transactions on circuits and systems for video technology (Print) · doi:10.1109/tcsvt.2022.3201045 · Cited by 61
“The pure Transformer-based architecture is also designed for classification [18], [19], [35]–[38], achieving highly competitive accuracy compared to CNNs.”
BackgroundSpecTNT: a Time-Frequency Transformer for Music AudioPreprint
Lu, W., Wang, J.-C., Won, M., Choi, K., & Song, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.09127 · Cited by 38
“Recently, Transformer in Transformer (TNT), a variant of Transformer that arranges two Transformers in a hierarchical manner, was proposed [10] for image recognition.”
InfluentialCongested Crowd Instance Localization with Dilated Convolutional Swin Transformer
Gao, J., Gong, M.-G., & Li, X.-L. (2021) · Neurocomputing · doi:10.48550/arxiv.2108.00584 · Cited by 61
“[49] propose a Transformer-iN-Transformer (TNT) architecture, of which the inner extracts local features, and the outer processes patch embeddings.”
BackgroundDiagnosis of Alzheimer's disease via optimized lightweight convolution-attention and structural MRI
Khatri, U., & Kwon, G. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108116 · Cited by 67
“Swin-Transformer [22] extends vision transformer (ViT) with a local self-attention mechanism to simplify calculations, while Transformer-iN-Transformer (TNT) [23] incorporates both local and global aspects of an image to increase classification efficiency.”
Automatic Modulation Classification Based on CNN-Transformer Graph Neural Network
Wang, D., Lin, M., Zhang, X., Huang, Y., & Zhu, Y. (2023) · Sensors · doi:10.3390/s23167281 · Cited by 74
Dual Deep CNN for Tumor Brain Classification
Al-Zoghby, A. M., Al-Awadly, E., Moawad, A., Yehia, N., & Ebada, A. (2023) · Diagnostics · doi:10.3390/diagnostics13122050 · Cited by 60
“Referring to [29], this study developed a new deep-learning framework for brain tumor classification that combined ViT with a feature fusion module.”
MethodDeep Learning in Diverse Intelligent Sensor Based Systems
Zhu, Y., Wang, M., Yin, X., Zhang, J., Meijering, E. H. W., & Hu, J. (2022) · Sensors · doi:10.3390/s23010062 · Cited by 73
Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networks
Rao, Y., Liu, Z., Zhao, W., Zhou, J., & Lu, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3263826 · Cited by 75
Sun, W., Qin, Z., Deng, H., Wang, J., Zhang, Y., Zhang, K., Barnes, N. M., Birchfield, S. T., Kong, L., & Zhong, Y. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3285569 · Cited by 66
“[17], [18], [19], [20] directly combine convolution with transformers.”
BackgroundEATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithm
Zhang, J., Li, X., Wang, Y., Wang, C., Yang, Y., Liu, Y., & Tao, D. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02034-6 · Cited by 57
“The former only contains transformer module without CNN-based layers, and early works[96,132,48,97,124,32,1] follow columnar structure of original ViT.”
Self-citationDynaMixer: A Vision MLP Architecture with Dynamic Mixing
Wang, Z., Jiang, W.-H., Zhu, Y., Yuan, L., Song, Y.-B., & Liu, W. (2022) · International Conference on Machine Learning · doi:10.48550/arxiv.2201.12083 · Cited by 60
“In [22, 23, 24, 25, 26], multi-granularity was introduced into ViTs to improve the generalization ability.”
MethodScaled ReLU Matters for Training Vision Transformers
Wang, P.-C., Wang, X., Luo, H., Zhou, J., Zhou, Z.-P., Wang, F., Li, H., & Jin, R. (2021) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2109.03810 · Cited by 60
“Compared with CNNs, ViTs are sensitive to the choice of optimizer, data augmentation, learning rate, training schedule length and warmup epoch (Touvron et al. 2020, 2021; Chen, Hsieh, and Gong 2021; Xiao et al. 2021).”
BackgroundMethodInfluentialLightweight Vision Transformer with Bidirectional Interaction
Fan, Q., Huang, H.-B., Zhou, X., & He, R. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2306.00396 · Cited by 59
SpectralSWIN: a spectral-swin transformer network for hyperspectral image classification
Ayas, S., & Tunc-Gormus, E. (2022) · International Journal of Remote Sensing · doi:10.1080/01431161.2022.2105668 · Cited by 59
“Han et al. (2021) propose a Transformer-iN-Transformer model which place a sub-transformer into the architecture for extracting local features.”
MethodIdentifying Malignant Breast Ultrasound Images Using ViT-Patch
Feng, H., Yang, B., Wang, J., Liu, M.-Z., Yin, L., Zheng, W., Yin, Z.-T., & Liu, C. (2023) · Applied Sciences · doi:10.3390/app13063489 · Cited by 58
“ResNet50 [29] 83.1 48.5 - InceptionV3 [30] 86.4 54.5 - VGG16 [31] 88.1 66.7 - Swin-B [23] 86.4 60.6 88 M TNT-B [32] 83.1 63.6 65.6 M T2T-ViT-24 [22] 84.7 51.5 64.1 M PVT-Large [11] 85 Lesion localization using ViT-Patch: We also evaluated the tumor localization ability of the ViT-Patch, i.e., the…”
InfluentialLocal self-attention in transformer for visual question answering
Shen, X., Han, D., Guo, Z., Chen, C., Hua, J., & Luo, G. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04355-w · Cited by 66
Residual Mixture of ExpertsPreprint
Wu, L., Liu, M., Chen, Y., Chen, D., Dai, X., & Yuan, L. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.09636 · Cited by 45
“Triggered by ViT, a series of follow-up works [8, 9, 14, 17, 21, 25, 30, 38, 49, 50, 52, 54, 57, 59, 60] continuously refresh the image classification records and push the potential of vision transformers in computer vision to a new height.”
BackgroundBoosting Salient Object Detection With Transformer-Based Asymmetric Bilateral U-Net
Qiu, Y., Liu, Y., Zhang, L., Lu, H.-T., & Xu, J. (2021) · IEEE transactions on circuits and systems for video technology (Print) · doi:10.1109/tcsvt.2023.3307693 · Cited by 58
“Following [40], lots of studies have emerged in a short period, and much better performance than state-of-the-art CNNs has been achieved [69], [70], [71], [72], [73].”
BackgroundTransformer-Based Person Re-Identification: A Comprehensive Review
Sarker, P. K., Zhao, Q., & Uddin, M. K. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3350669 · Cited by 57
“Researchers have recently focused on increasing the modeling capability of local information [128], [129], [130].”
BackgroundMethodPPT: Token Pruning and Pooling for Efficient Vision TransformersPreprint
Wu, X.-J., Zeng, F.-H., Wang, X.-D., Wang, Y., & Chen, X.-H. (2023) · arXiv.org · doi:10.48550/arxiv.2310.01812 · Cited by 57
“Apart from the standard ViTs [8, 24], subsequent studies [5, 10–12, 17, 20, 26–28, 31, 32] further improve the performance of ViTs by varying the initial architecture or optimization strategies.”
BackgroundResultInfluentialSelf-citationSong, B., Dharma Raj, K. C., Yang, R. Y., Li, S., Zhang, C., & Liang, R. (2024) · Cancers · doi:10.3390/cancers16050987 · Cited by 60
“However, a paradigm shift has occurred with the emergence of transformer-based architectures [12].”
BackgroundChen, T., Zhang, Z., Cheng, Y., Awadallah, A. H., & Wang, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01171 · Cited by 42
“Existing works have taken two angles: ( i ) (re-)injecting locality via convolution-like structures and fusing global and local contexts, for self-attention [2,27,32,43,55,56,64,65, 67, 69, 76]; ( ii ) adopting patch-wise contrastive or mixing loss to boost diversity, for patch embeddings.”
BackgroundEfficient Adaptation of Large Vision Transformer via Adapter Re-Composing
Dong, W., Yan, D.-W., Lin, Z.-J., & Wang, P. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2310.06234 · Cited by 55
Diverse Cotraining Makes Strong Semi-Supervised Segmentor
Li, Y., Wang, X.-J., Yang, L., Feng, L.-T., Zhang, W., & Gao, Y. (2023) · IEEE International Conference on Computer Vision · doi:10.48550/arxiv.2308.09281 · Cited by 55
“Later works such as Swin Transformer [52, 51] and PVT [78, 79] proves its superiority over CNN with different inductive biases trained [28, 30, 95, 73].”
MethodLand Cover Classification for Polarimetric SAR Images Based on Vision Transformer
Wang, H., Xing, C., Yin, J., & Yang, J. (2022) · Remote Sensing · doi:10.3390/rs14184656 · Cited by 59
Fine-Grained Ship Classification by Combining CNN and Swin Transformer
Huang, L., Wang, F., Zhang, Y., & Xu, Q.-Y. (2022) · Remote Sensing · doi:10.3390/rs14133087 · Cited by 55
HiViT: Hierarchical Vision Transformer Meets Masked Image Modeling
Zhang, X., Tian, Y., Huang, W., Ye, Q.-X., Dai, Q., Xie, L.-X., & Tian, Q. (2022) · arXiv.org · doi:10.48550/arxiv.2205.14949 · Cited by 55
SepViT: Separable Vision Transformer
Li, W., Wang, X., Xia, X., Wu, J., Xiao, X., Zheng, M., & Wen, S.-P. (2022) · arXiv.org · doi:10.48550/arxiv.2203.15380 · Cited by 55
“Heads [1, 2, 4, 8] [3, 6, 12, 24] [3, 6, 12, 24] [4, 8 ,16, 32]”
BackgroundHow to Understand Masked AutoencodersPreprint
Cao, S., Xu, P., & Clifton, D. A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.03670 · Cited by 56
“Patchifying has become a standard practice in Transformer-based CV models since ViT [23], see also [30, 37].”
MethodAugmented Shortcuts for Vision TransformersPreprint
Tang, Y., Han, K., Xu, C., Xiao, A., Deng, Y., Xu, C., & Wang, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.15941 · Cited by 39
“However, in general vision transformer models [9, 13, 37, 14, 12, 40, 23], there is only a single shortcut connection with identity projection for each MSA module, which only copies the input features to the outputs.”
BackgroundSelf-citationWang, X., Jin, Y., Cen, Y.-G., Wang, T., Tang, B., & Li, Y.-D. (2022) · IEEE transactions on multimedia · doi:10.1109/tmm.2023.3318073 · Cited by 54
“Vanishing position embedding: In 2D image recognition, position encoding is an essential mechanism for preserving the local relative position of patches, conducive to improving network performance [46], [47].”
A survey of FPGA and ASIC designs for transformer inference acceleration and optimization
Kang, B. J., Lee, H. I., Yoon, S. K., Kim, Y. C., Jeong, S. B., Jeong, S. B., & Kim, H. (2024) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2024.103247 · Cited by 54
Beyond Fixation: Dynamic Window Visual Transformer
Ren, P., Li, C., Wang, G., Xiao, Y., Du, Q., Liang, X., & Chang, X. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01168 · Cited by 55
A survey of unmanned aerial vehicles and deep learning in precision agriculture
Wang, D., Zhao, M., Li, Z., Xu, S., Wu, X., Ma, X., & Liu, X. (2024) · European Journal of Agronomy · doi:10.1016/j.eja.2024.127477 · Cited by 57
Yang, H., Wang, J., Bo, Y., & Wang, J. (2025) · Neurocomputing · doi:10.1016/j.neucom.2024.129289 · Cited by 56
Glass Makes Blurs: Learning the Visual Blurriness for Glass Surface Detection
Qi, F., Tan, X., Zhang, Z., Chen, M., Xie, Y., & Ma, L. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3352232 · Cited by 56
“We refer to methods [40], [41], [42], [43] employing a Transformer-based backbone for training, setting the initial LR to 1e-4 when using SwinB as the backbone.”
MethodAyoub, M., Liao, Z., Hussain, S., Li, L., Zhang, W., & Wong, K. K. L. (2023) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2023.102294 · Cited by 58
Computation-efficient Deep Learning for Computer Vision: A Survey
Wang, Y., Han, Y., Wang, C., Song, S., Tian, Q., & Huang, G. (2023) · Cybernetics and Intelligence · doi:10.48550/arxiv.2308.13998 · Cited by 52
Self-citationA data efficient transformer based on Swin Transformer
Yao, D., & Shao, Y. (2023) · The Visual Computer · doi:10.1007/s00371-023-02939-2 · Cited by 60
Land Cover Classification of UAV Remote Sensing Based on Transformer–CNN Hybrid Architecture
Lu, T.-Y., Wan, L.-H., Qi, S.-Q., & Gao, M. (2023) · Italian National Conference on Sensors · doi:10.3390/s23115288 · Cited by 52
“TNT (Transformer in Transformer) [21] model improves ViT by dividing patch in ViT into sub-patches and representing images with finer granularity.”
MethodMulti-hypothesis representation learning for transformer-based 3D human pose estimation
Li, W.-H., Liu, H., Tang, H.-L., & Wang, P.-C. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109631 · Cited by 52
Spatiotemporal Self-attention Modeling with Temporal Patch Shift for Action Recognition
Xiang, W., Li, C., Wang, B., Wei, X., Hua, X., & Zhang, L. (2022) · European Conference on Computer Vision · doi:10.48550/arxiv.2207.13259 · Cited by 52
“Recently, with the advancement of trans-formers in 2D vision tasks [8,23,44,16,36,5], many works have been done to apply transformers on video action recognition [2,1,24,45].”
BackgroundMethodSelf-slimmed Vision Transformer
Zong, Z.-F., Li, K.-C., Song, G.-L., Wang, Y.-L., Qiao, Y., Leng, B., & Liu, Y. (2021) · European Conference on Computer Vision · doi:10.48550/arxiv.2111.12624 · Cited by 52
“Recent ViT variants mainly focus on better optimization and more powerful performance [43,41,40,34,13,4,9,35,11,7,38,15,19,18].”
BackgroundSelf-citationPeykani, P., Ramezanlou, F., Tănăsescu, C. R., & Ghanidel, S. (2025) · Applied Sciences · doi:10.3390/app15148103 · Cited by 64
Al-Hejri, A. M., Sable, A. H., Al-Tam, R. M., Al–antari, M. A., Alshamrani, S. S., Alshmrany, K. M., & Alatebi, W. (2025) · Scientific Reports · doi:10.1038/s41598-025-96527-0 · Cited by 51
“It is used to obtain a fused representation that captures both the complementary and synergistic information between diverse features 71 .”
DAT++: Spatially Dynamic Vision Transformer with Deformable Attention
Xia, Z., Pan, X., Song, S.-J., Li, E.-R. L., & Huang, G. (2023) · arXiv.org · doi:10.48550/arxiv.2309.01430 · Cited by 51
“This alternate design of attention blocks with local and global receptive fields helps the model learn strong representations, sharing similar thoughts in [35], [97], [116], [117], [118], [119], [120], which has been shown to be an effective design.”
BackgroundDeepMAD: Mathematical Architecture Design for Deep Convolutional Neural Network
Shen, X., Wang, Y., Lin, M. V., Huang, Y., Tang, H., Sun, X., & Wang, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00597 · Cited by 45
Self-citationFPT: Fine-Grained Detection of Driver Distraction Based on the Feature Pyramid Vision Transformer
Wang, H., Chen, J., Huang, Z., Li, B., Lv, J., Xi, J., Wu, B., Zhang, J., & Wu, Z. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3219676 · Cited by 48
“It has shown superior performance equal to or even better than a Convolutional Neural Networks (CNN) in optical image classification [17], [18], target detection [19], [20], semantic segmentation [21], [22], and other fields [23], [24], [25], but it has not been researched and explored in the field of driver distracti…”
BackgroundMethodSkip-Attention: Improving Vision Transformers by Paying Less Attention
Venkataramanan, S., Ghodrati, A., Asano, Y. M., Porikli, F., & Habibian, A. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2301.02240 · Cited by 51
“Token sampling improves the efficiency either by restructuring images during the tokenization step [21, 66], pruning the redundant tokens over training [26, 46] or dynamically at inference [7, 17, 43, 63].”
MethodZhao, Z., Cao, R., Un, K.-F., Yu, W.-H., Mak, P., & Martins, R. P. (2022) · IEEE Transactions on Circuits & Systems II Express Briefs · doi:10.1109/tcsii.2022.3196055 · Cited by 44
“Transformer-in-transformer (TNT) further improves the Top-1 accuracy to 81.5%, by a model with only 23.8M parameters [19].”
BackgroundMethodInfluentialReliable Few-Shot Learning Under Dual Noises
Zhang, J., Song, J., Gao, L., Sebe, N., & Shen, H. T. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3584051 · Cited by 41
“Furthermore, with the recent success in model pre-training techniques [15], [16], [17], designing effective and efficient full-finetuning based [18], [19], adapter-finetuning based [12], [13], [20], and prompt tuning based [21], [22], [23] task adaptation algorithms that can bor-row the freely…”
Shao, S., Tian, Y., & Zhang, Y.-J. (2025) · Swarm and Evolutionary Computation · doi:10.1016/j.swevo.2024.101817 · Cited by 50
Zhao, Z., Chen, S., Ge, Y., Yang, P., Wang, Y., & Song, Y.-S. (2024) · Applied Sciences · doi:10.3390/app14146287 · Cited by 50
“To address this shortcoming, this paper replaces the backbone network with one based on the Transformer framework [52].”
MethodSelf-citationGao, R., Dong, Z., Wang, Y., Cui, Z., Ye, M., Dong, B.-W., Lu, Y.-C., Wang, X., Song, Y., & Yan, S. (2024) · Agriculture · doi:10.3390/agriculture14020247 · Cited by 50
Self-citationWhen Geoscience Meets Foundation Models: Toward a general geoscience artificial intelligence system
Zhang, H., Xu, J., Cui, H., Li, L., Yang, Y., Tang, C., & Boers, N. (2024) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2024.3496478 · Cited by 59
“For instance, a weather prediction model can be realized by training on historical climate data, and this model can also be helpful for the forecast of rainfall-induced shallow landslides.”
TransRVNet: LiDAR Semantic Segmentation With Transformer
Cheng, H.-X., Han, X.-F., & Xiao, G. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3248117 · Cited by 46
“To handle these issues, many excellent works [26], [27], [28], [29], [30], [31] have been proposed.”
BackgroundWang, W., Zhong, Z., Wang, W., Chen, X., Ling, C. X., Wang, B., & Sebe, N. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02307 · Cited by 43
“In this section, we propose a Dynamically Instance-Guided Adaptation (DIGA) method for TTDA-Seg, which is backward-free and non-parametric.”
MethodVLTinT: Visual-Linguistic Transformer-in-Transformer for Coherent Video Paragraph Captioning
Yamazaki, K., Vo, K., Truong, S., Raj, B., & Le, N. T. H. (2022) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2211.15103 · Cited by 50
“(Han et al. 2021) presents TNT to further divide them into smaller 4× 4 patches.”
BackgroundBoosting vision transformers for image retrieval
Song, C. H., Yoon, J.-Y., Choi, S., & Avrithis, Y. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00019 · Cited by 59
“As a consequence, many studies [70, 66, 19, 32, 7, 43, 5] are proposed to improve ViT by bringing in locality.”
BackgroundMethodTokenMixup: Efficient Attention-guided Token-level Data Augmentation for Transformers
Choi, H. K., Choi, J., & Kim, H. J. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2210.07562 · Cited by 50
ViTKD: Practical Guidelines for ViT feature knowledge distillationPreprint
Yang, Z., Li, Z., Zeng, A., Li, Z., Yuan, C., & Li, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.02432 · Cited by 52
“Recently, Vision Transformer (ViT) (Dosovitskiy et al., 2021) has achieved great success for image classification and inspired various transformers (Yuan et al., 2021; Han et al., 2021; Touvron et al., 2021b; Liu et al., 2021).”
BackgroundPTNet3D: A 3D High-Resolution Longitudinal Infant Brain MRI Synthesizer Based on Transformers
Zhang, X., He, X.-Z., Guo, J., Ettehadi, N., Aw, N., Semanek, D., Posner, J., Laine, A., & Wang, Y. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3174827 · Cited by 50
Self-citationDriver attention prediction based on convolution and transformers
Gou, C., Zhou, Y., & Li, D. (2022) · Journal of Supercomputing · doi:10.1007/s11227-021-04151-2 · Cited by 50
Image captioning by diffusion models: a survey
Daneshfar, F., Bartani, A., & Lotfi, P. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109288 · Cited by 49
Apple recognition and picking sequence planning for harvesting robot in the complex environment
Ji, W., Zhang, T., Xu, B., & He, G. (2023) · Journal of Agricultural Engineering · doi:10.4081/jae.2023.1549 · Cited by 49
“Additionally, the transformer has the ability to gather global information, which can compensate for YOLOV5’s inade-quacies, since it is a convolutional neural network and cannot conduct global modeling (Han et al. , 2021).”
BackgroundEfficient Vision Transformer via Token Merger
Feng, Z., & Zhang, S. S. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3293763 · Cited by 37
Prototype-based Selective Knowledge Distillation for Zero-Shot Sketch Based Image Retrieval
Wang, K., Wang, Y.-F., Xu, X., Liu, X., Ou, W.-H., & Lu, H.-M. (2022) · ACM Multimedia · doi:10.1145/3503161.3548382 · Cited by 49
“The transformer architecture [2, 3, 14, 20, 33, 49] has come into prominence in a variety of deep learning tasks.”
BackgroundSelf-citationMulti-scale conv-attention U-Net for medical image segmentation
Pan, P., Zhang, C., Sun, J.-B., & Guo, L. (2025) · Scientific Reports · doi:10.1038/s41598-025-96101-8 · Cited by 48
“In deep learning image segmentation and recognition tasks, the attention mechanism has been widely used as an effective means to enhance the performance of models 28–30 .”
Isomorphic Pruning for Vision Models
Fang, G., Ma, X., Mi, M. B., & Wang, X. (2024) · European Conference on Computer Vision · doi:10.48550/arxiv.2407.04616 · Cited by 48
Cheng, L., Zhu, P.-C., Tang, K.-K., Gao, C., & Wang, Z. (2024) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i1.27755 · Cited by 48
“In recent years, the Transformer, originally proposed for Natural Language Processing (NLP), has introduced Po-sitional Encodings (PEs) for individual words (Han et al. 2021).”
Background2C2S: A two-channel and two-stream transformer based framework for offline signature verification
Ren, J.-X., Xiong, Y.-J., Zhan, H., & Huang, B. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105639 · Cited by 61
CrackViT: a unified CNN-transformer model for pixel-level crack extraction
Quan, J., Ge, B., & Wang, M. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08277-7 · Cited by 49
“Also, to improve the capacity of ViT to model the important local structure, such as edges and lines among neighboring pixels, DeiT [59], PVT [63], and TNT [20] are gradually put forward.”
BackgroundDS-DETR: A Model for Tomato Leaf Disease Segmentation and Damage Evaluation
Wu, J., Wen, C., Chen, H., Ma, Z., Zhang, T., Su, H., & Yang, C. (2022) · Agronomy · doi:10.3390/agronomy12092023 · Cited by 53
Multi-Head Self-Attention via Vision Transformer for Zero-Shot LearningPreprint
Alamri, F., & Dutta, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.00045 · Cited by 40
“In addition, TNT [Han et al., 2021] proposes transformer-backbone method modelling not only the patch-level features but also the pixel-level representations.”
MethodScattering Vision Transformer: Spectral Mixing Matters
Patro, B., & Agneeswaran, V. S. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2311.01310 · Cited by 47
Metadata
- Authors
- Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., & Wang, Y.
- Affiliations
- Huawei Technologies (China) (CN)
- University of Macau (MO)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Volume
- 34
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2103.00112
- OpenAlex
- W3133696297
- Cited by
- 1,394
- References
- 56
- Citations by year
- 2026: 14 · 2025: 276 · 2024: 457 · 2023: 387 · 2022: 186 · 2021: 74
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (0015/2019/AKP, 62072449, 61632003, 0018/2019/AKP)
- Fundo para o Desenvolvimento das Ciências e da Tecnologia (0018/2019/AKP, 0015/2019/AKP)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- vision transformer, attention mechanism, local patches, visual words, image classification, feature aggregation, computational efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
