The DOI matches its DataCite record, with a small difference in title or year; 56 of the 56 references it cites were found in a registry or an index.
Transformer in Transformer
Open accessAbstract
Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (e.g., 16$\times$16) as "visual sentences" and present to further divide them into smaller patches (e.g., 4$\times$4) as "visual words". The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs. Features of both words and sentences will be aggregated to enhance the representation ability. Experiments on several benchmarks demonstrate the effectiveness of the proposed TNT architecture, e.g., we achieve an 81.5% top-1 accuracy on the ImageNet, which is about 1.7% higher than that of the state-of-the-art visual transformer with similar computational cost. The PyTorch code is available at https://github.com/huawei-noah/CV-Backbones, and the MindSpore code is available at https://gitee.com/mindspore/models/tree/master/research/cv/TNT.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For example, in multi-tenant architectures, system prompts are shared across multiple requests and frequently contain detailed instructions or examples to guide the model [19].”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [19]
In the citing paper
“PromptCache [19] proposes modular reuse beyond prefixes but sacrifices accuracy, with reported performance drops of up to 43% [72], making it unsuitable for tasks requiring high precision.”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [19]
In the citing paper
“related experiments results further demonstrate ViT also has remarkable representation capacity when dealing with images and videos [19].”
From Deepfake Video Detection with Spatiotemporal Dropout Transformer (2022) · cites this paper as [19]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 6×
“Compared to CNNs, TNT can outperform the widely-used ResNet and RegNet.”
BackgroundMethodInfluentialVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3×
“The effect of #heads in standard transformer has been investigated in multiple works [21, 35] and a head width of 64 is recommended for visual tasks [9, 31].”
BackgroundMethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“In the past decade, the mainstream deep neural architectures used in the computer vision (CV) are mainly established on convolutional neural networks (CNNs) [17, 12, 11].”
BackgroundFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472 · cited 1×
“As a general backbone network, TNT can also be applied with multi-scale vision models like Faster RCNN [29].”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472 · cited 7×
“The recent transformerbased models like ViT [9] and DeiT [31] are compared.”
MethodInfluentialImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 3×
“ImageNet ILSVRC 2012 [26] is an image classification benchmark consisting of 1.”
BackgroundInfluentialSqueeze-and-Excitation NetworksPreprint
Hu, J., Shen, L., & Sun, G. (2017) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1709.01507 · Cited by 37,845 · cited 1×
“Inspired by squeeze-and-excitation (SE) network for CNNs [16], we propose to explore channel-wise attention for transformers.”
MethodPyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939 · cited 2×
“We evaluate TNT-S and DeiT-S on Faster RCNN with FPN [21].”
BackgroundMethod
Research works printed but not linked: 5
- [6][6] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML , 2020. -
- [11][11] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. arXiv preprint arXiv:2012.12556 , 2020. - · arXiv
- [23][23] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017. - · arXiv
- [26][26] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR , pages 3498-3505. IEEE, 2012. -
- [37][37] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008. -
Cited by
Sorted by citations
Items 301–400 of 1,009
ELSA: Enhanced Local Self-Attention for Vision TransformerPreprint
Zhou, J., Wang, P., Wang, F., Liu, Q., Li, H., & Jin, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.12786 · Cited by 39
“Lots of follow-up studies are proposed [4,5,9,12,18, 21,23,24,27–29 ,31 ,38 ,41 ,43 ,45 ,50 ,52, 56 , 76, 77, 80, 81, 84].”
COMPOSER: Compositional Reasoning of Group Activity in Videos with Keypoint-Only Modality
Zhou, H., Kadav, A., Shamsian, A., Geng, S., Lai, F., Zhao, L., Liu, T., Kapadia, M., & Graf, H. P. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19833-5_15 · Cited by 42
“Specifically, MSA splits the query, key and value for h times and performs the self-attention function in parallel, and then the output values of each head are concatenated and linearly projected to form the final output [31].”
BackgroundAdam, M., & Baroudi, U. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3506214 · Cited by 46
Ye, C., Wu, H., Oguchi, T., Tang, Y.-T., Pei, X.-J., & Wu, Y. (2025) · Remote Sensing · doi:10.3390/rs17132280 · Cited by 46
Multi-User Layer-Aware Online Container Migration in Edge-Assisted Vehicular Networks
Tang, Z., Mou, F., Lou, J., Weijia Jia, B., Wu, Y., & Zhao, W. (2023) · IEEE/ACM Transactions on Networking · doi:10.1109/tnet.2023.3330255 · Cited by 44
“For the Transformer encoder layer, the FLOPs are 4 L | E | d + 2 L 2 d , where L is the length of the input sequence, | E | is the number of edge nodes, and d is the dimension of ( query, key, value ) [46].”
BackgroundMedical image segmentation network based on multi-scale frequency domain filter
Chen, Y.-F., Zhang, X.-Q., Peng, L., He, Y., Sun, F., & Sun, H. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106280 · Cited by 46
Li, Y., Liang, M.-H., Li, H.-H., Yang, Z.-L., Du, L., & Chen, Z. (2023) · Engineering applications of artificial intelligence · doi:10.1016/j.engappai.2023.107012 · Cited by 46
“(j) Transformer (Han et al., 2021a,b): It is similar to Seq2Seq and based on the Encoder-Decoder framework to construct a training network.”
MultiCAD: Contrastive Representation Learning for Multi-modal 3D Computer-Aided Design Models
Ma, W., Xu, M., Li, X., & Zhou, X. (2023) · International Conference on Information and Knowledge Management · doi:10.1145/3583780.3614982 · Cited by 46
StARformer: Transformer with State-Action-Reward Representations for Visual Reinforcement Learning
Shang, J., Kahatapitiya, K., Li, X., & Ryoo, M. S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19842-7_27 · Cited by 25
“Recently, Transformers have been adopted in vision tasks with the key idea of breaking down images/videos into tokens [18,5,10,45,23,7,28], often outperforming convolutional networks (CNNs) in practice.”
BackgroundZig-RiR: Zigzag RWKV-in-RWKV for Efficient Medical Image Segmentation
Chen, T.-X., Zhou, X., Tan, Z., Wu, Y., Wang, Z.-Y., Ye, Z., Gong, T., Chu, Q., Yu, N.-H., & Lu, L. (2025) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3561797 · Cited by 45
“To this end, we draw inspiration from TNT [21], which effectively models local structural features with a trivial increase in computation and memory cost, and propose a novel nested architecture known as Zigzag RWKV-in-RWKV (Zig-RiR) for more efficient medical image segmentation.”
GDALR: Global Dual Attention and Local Representations in transformer for surface defect detection
Zhou, X., Zhou, S., Zhang, Y., Ren, Z., Jiang, Z., & Luo, H. (2024) · Measurement · doi:10.1016/j.measurement.2024.114398 · Cited by 50
Joint Transformer and Multi-scale CNN for DCE-MRI Breast Cancer Segmentation
Qin, C.-B., Wu, Y., Zeng, J.-Y., Tian, L.-F., Zhai, Y.-K., Li, F., & Zhang, X.-Z. (2022) · Soft Computing - A Fusion of Foundations, Methodologies and Applications · doi:10.1007/s00500-022-07235-0 · Cited by 45
“DCE-MRI is extremely sensitive in detecting early breast cancer, and thus is routinely utilized in clinical practice to assess the local extent of disease and determine the molecular type (Xiao et al. 2021).”
BackgroundApplication of belief functions to medical image segmentation: A review
Huang, L., Ruan, S., & Denœux, T. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.11.008 · Cited by 51
“More recently, the success of deep learning in the medical domain brought a lot of contributions to medical image segmentation tasks [8, 9, 10, 11, 12, 13, 14].”
BackgroundShen, Z., Liu, Z.-C., & Xing, E. (2021) · European Conference on Computer Vision · doi:10.48550/arxiv.2111.05297 · Cited by 45
“structure, unlike others [42,52,22,47,49,33,27,50], that add more parameters or involve additional fine-grained information from input [18].”
BackgroundMethodInfluentialA human digital twin approach for fatigue-aware task planning in human-robot collaborative assembly
You, Y.-C., Cai, B.-L., Pham, D., Liu, Y., & Ji, Z. (2025) · Computers & industrial engineering · doi:10.1016/j.cie.2024.110774 · Cited by 44
“Additionally, the attention-convolution BiLSTM (AC-BiLSTM) (Liu and Guo, 2019) and transformer (Han et al., 2021) methods, which are known for their proficiency in handling sequential data tasks such as text classification, are also included as suitable comparators.”
Auto-Prox: Training-Free Vision Transformer Architecture Search via Automatic Proxy Discovery
Wei, Z., Li, L., Dong, P., Hui, Z., Li, A.-G., Lu, M.-L., Pan, H., Tian, Z.-L., & Li, D.-S. (2023) · arXiv.org · doi:10.48550/arxiv.2312.09059 · Cited by 44
FIT-Net: Feature Interaction Transformer Network for Pathologic Myopia Diagnosis
Chen, S., Wu, Z., Li, M., Zhu, Y., Xie, H., Yang, P., Zhao, C., Zhang, Y., Zhang, S., Zhao, X., et al. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3260990 · Cited by 48
“Transformer layers have shown promising accuracy and efficiency in classification and segmentation tasks [15], [16], [17].”
BackgroundInfrared Image Super-Resolution: A Systematic Review and Future Trends
Huang, Y., Miyazaki, T., Liu, X.-F., & Omachi, S. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2025.3614673 · Cited by 44
“Despite the availability of datasets such as IR100 and NATO SET-140, current datasets often lack paired samples that span diverse environmental conditions or integrate multimodal information.”
Generating Transferable Adversarial Examples against Vision Transformers
Wang, Y., Wang, J., Yin, Z., Gong, R., Wang, J., Liu, A., & Liu, X. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3547989 · Cited by 33
“and are pre-trained with large image datasets, thus showing great potential and becoming prevailing among several computer vision tasks [2, 7, 24, 38, 49, 54, 66].”
BackgroundSelf-citationAggregating Global Features into Local Vision Transformer
Patel, K., Bur, A. M., Li, F., & Wang, G. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956379 · Cited by 51
“utilize both local and global features [11] [5].”
BackgroundMethodLearned Queries for Efficient Local Attention
Arar, M., Shamir, A., & Bermano, A. H. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01057 · Cited by 37
“TNT-ViT [21], on the other hand, splits coarse patches into locally attending parts.”
BackgroundLi, T., Chen, J., Liu, T., Sun, C., Zhao, Z., Chen, X., & Yan, R. (2025) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2025.111935 · Cited by 44
Block Transformer: Global-to-Local Language Modeling for Fast Inference
Bae, S., Fisch, A., Ho, N., Jo, H., Kim, T., Kim, Y., Schuster, T., Thorne, J., & Yun, S. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2406.02657 · Cited by 43
“Local modules have been incorporated in vision transformers to enhance the encoding of fine-grained local features with minimal computational overhead [31].”
BackgroundAggregating transformers and CNNs for salient object detection in optical remote sensing images
Bao, L.-X., Zhou, X.-F., Zheng, B., Yin, H., Zhu, Z.-J., Zhang, J.-Y., & Yan, C. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126560 · Cited by 43
Brain-inspired Multilayer Perceptron with Spiking Neurons
Li, W.-S., Chen, H.-T., Guo, J.-Y., Zhang, Z.-Y., & Wang, Y.-H. (2022) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52688.2022.00086 · Cited by 43
“TNT [16] proposes to model the inner information of patches by embedding Transformers into Transformers.”
MethodSelf-citationA Review of Hybrid and Ensemble in Deep Learning for Natural Language ProcessingPreprint
Jia, J., Wen, L., & Liang, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.05589 · Cited by 79
“It moved away from recurrent layers and emphasized self-attention mechanisms, allowing each word in a sequence to attend to all others [82].”
BackgroundPSAQ-ViT V2: Toward Accurate and General Data-Free Quantization for Vision Transformers
Li, Z., Chen, M., Xiao, J., & Gu, Q. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3301007 · Cited by 41
“With the success in natural language processing applications, transformers, which enjoy the global receptive field offered by the self-attention mechanism, have recently achieved significant performance gains on a range of computer vision applications [28], [29], [30], [31], [32].”
Wu, Z., Ma, N., Wang, C., Xu, C., Xu, G., & Li, M. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110427 · Cited by 39
Self-citationInvPT++: Inverted Pyramid Multi-Task Transformer for Visual Scene Understanding
Ye, H., & Xu, D. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3397031 · Cited by 30
“Initially designed for NLP tasks [33], transformers have been widely used in the study of computer problems [13], [35], [36], including multi-modal learning [37], [38], 2D [13], [39], [40], [41], [42], [43] and 3D tasks [44], [45], [46].”
MethodFarmland Obstacle Detection from the Perspective of UAVs Based on Non-local Deformable DETR
Wang, D., Li, Z., Du, X., Ma, Z., & Liu, X. (2022) · Agriculture · doi:10.3390/agriculture12121983 · Cited by 41
“…(ViT [33]), Data-efficient image Transformers (DeiT [34]), Tokens-To-Token Vision Transformer (T2T [35]), Transformer in Transformer (TNT [36]), Conditional Positional encoding Vision Transformer (CPVT [37]), Shifted Windows Transformer (Swin Trans-former [38]), object detection:…”
BackgroundRethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognition
Dooley, S., Sukthanker, R., Dickerson, J. P., White, C., Hutter, F., & Goldblum, M. (2022) · Neural Information Processing Systems · doi:10.52202/075280-3251 · Cited by 41
“68], tf_efficientnet_b7_ns’ [105], ’tnt_s_patch16_224 [43], twins_svt_large [16] , vgg19 [101], vgg19_bn [101], visformer_small [11], xception and xception65 [15].”
FedAdapter: Efficient Federated Learning for Modern NLPPreprint
Cai, D., Wu, Y., Wang, S., Lin, F. X., & Xu, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.10162 · Cited by 44
“We refer readers to recent surveys [25, 26, 41] for how those models work internally.”
MethodGLPanoDepth: Global-to-Local Panoramic Depth Estimation
Bai, J.-C., Qin, H., Lai, S.-C., Guo, J., & Guo, Y.-W. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3386403 · Cited by 41
“The limited receptive field and the limited expressiveness of an individual convolution operation necessitate sequential stacking many convolutional layers into very deep architectures.”
BackgroundSelf-citationOu, J., Jiang, L., Bai, T., Zhan, P., Liu, R., & Xiao, H. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108625 · Cited by 44
ReViT: Enhancing vision transformers feature diversity with attention residual connections
Diko, A., Avola, D., Cascio, M., & Cinque, L. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110853 · Cited by 54
“To address such an issue, Han et al. [11] introduced a new kind of neural architecture that divides image patches into sub-patches and later employs an extra transformer block to analyze their relationship.”
BackgroundMethodJamal, S., & Wimmer, H. (2023) · arXiv.org · doi:10.48550/arxiv.2311.04913 · Cited by 40
“Furthermore, attention-based mechanisms in transformer allows model interpretability and makes it easier in understanding certain classification decision [3].”
BackgroundCADTransformer: Panoptic Symbol Spotting Transformer for CAD Drawings
Fan, Z., Chen, T., Wang, P., & Wang, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01071 · Cited by 34
“Inspired by the success of transformer applied in natural language processing field, a growing interests have shown in exploring the use of transformer framework for computer vision tasks, including image generation [5, 35], image classification [5, 6, 9, 13, 21, 33, 50, 54], object detection [2, 4], semantic segmenta…”
MethodINTERN: A New Learning Paradigm Towards General VisionPreprint
Shao, J., Chen, S., Li, Y., Wang, K., Yin, Z., He, Y., Teng, J., Sun, Q., Gao, M., Liu, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.08687 · Cited by 38
“Similar to Network In Network (NIN) [90] series, Han et al. [62] leverage a Transformer-iN-Transformer (TNT) model to aggregate both patch-and pixel-level representations.”
ResultSelf-citationYi, Z., Xiao, T., & Albert, M. V. (2025) · Information · doi:10.3390/info16020136 · Cited by 43
Zhu, J., Liu, D., Chen, H., Liu, J., & Tao, Z. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112828 · Cited by 36
Uddin, M. A., Islam, M., Maglaras, L., Janicke, H., & Sarker, I. H. (2024) · Digit. Commun. Networks · doi:10.48550/arxiv.2405.08026 · Cited by 39
“Its creative architecture makes use of attention mechanisms to reliably encode input data into representations, providing unmatched performance across a range of natural language processing applications Han, Xiao, Wu, Guo, Xu, and Wang (2021).”
BackgroundLiu, Y., Min, W.-Q., Jiang, S.-Q., & Rui, Y. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3374211 · Cited by 39
“Current works mainly focused on improving the Transformer by adding convolution modules [30], [31], [32], [33] or by modifying the attention mechanism [34], [35], [36].”
MethodA new interpretable streamflow prediction approach based on SWAT-BiLSTM and SHAP
Huang, F., & Zhang, X. (2024) · Environmental Science and Pollution Research · doi:10.1007/s11356-024-32725-z · Cited by 45
“Additionally, it can produce more interpretable models (Han et al. 2021).”
Dan, J., Liu, Y., Sun, B.-G., Deng, J., & Luo, S. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3616149 · Cited by 39
Zhou, H., Dong, W. F., Liu, Y., & Chen, J. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00187 · Cited by 44
“However, these methods do not take advantage of the stateof-the art models, such as Vision Transformer [14] and its follow-ups [18, 53], Swin Transformer [30] and ConvNeXt [31, 46], which achieve superior performance compared to Res2Net on ImageNet.”
MethodPLG-ViT: Vision Transformer with Parallel Local and Global Self-Attention
Ebert, N., Stricker, D., & Wasenmüller, O. (2023) · Sensors · doi:10.3390/s23073447 · Cited by 37
Liang, S., Liu, A., Liang, J., Li, L., Bai, Y., & Cao, X. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548416 · Cited by 19
“We mainly refer to the large model combination of [13], using trained large models such as minGPT, DALL-E, and generation model VQGAN to generate object patches and randomly sample the sizes and positions to construct synthetic data.”
MethodTransformers Meet Visual Learning Understanding: A Comprehensive ReviewPreprint
Yang, Y., Jiao, L., Liu, X., Liu, F., Yang, S., Feng, Z., & Tang, X. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.12944 · Cited by 34
Wang, C., Shang, K., Zhang, H.-M., Li, Q., Hui, Y., & Zhou, S. (2021) · arXiv.org · doi:10.48550/arxiv.2111.10790 · Cited by 39
“Based on the powerful attention mechanism [4, 10, 26, 29, 36, 41] and patch-based operations, Transformer is applied to many vision tasks [9, 12, 22, 28].”
MethodSRViT: Self-Supervised Relation-Aware Vision Transformer for Hyperspectral Unmixing
Su, Y., Gao, L., Plaza, A., Sun, X., Jiang, M., & Yang, G. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3571798 · Cited by 46
“Nevertheless, most variations follow the original patch-wise scheme to divide images, which might be unable to excavate features of complex objects in di ff erent scales and locations [21].”
InfluentialHybrid Design of CNN and Vision Transformer: A Review
Long, H.-H. (2024) · Proceedings of the 2024 7th International Conference on Computer Information Science and Artificial Intelligence · doi:10.1145/3703187.3703208 · Cited by 38
A Queueing Theoretic Perspective on Low-Latency LLM Inference with Variable Token LengthPreprint
Yang, Y., Xu, Y., & Jiao, L. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.05347 · Cited by 12
“However, a large token limit can still result in significant queuing delays when inference requests arrive at the LLM platform and are processed on a first-come-first-serve (FCFS) basis.”
Ayoub, M., Liao, Z., Li, L., & Wong, K. (2023) · Comput. Medical Imaging Graph. · doi:10.1016/j.compmedimag.2023.102295 · Cited by 38
Yan, Y.-F., Liu, R., Chen, H.-B., Zhang, L.-M., & Zhang, Q. (2023) · IEEE journal of biomedical and health informatics · doi:10.1109/jbhi.2023.3289913 · Cited by 38
“The Transformer in Transformer [19] overcomes the above problems by subdividing each image block into several sub-blocks and simultaneously calculating self attention between inter sub-block and intra sub-block.”
MethodEfficient deepfake detection using shallow vision transformer
Usmani, S., Kumar, S., & Sadhya, D. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-15910-z · Cited by 44
TransCL: Transformer Makes Strong and Flexible Compressive Learning
Mou, C., & Zhang, J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3194001 · Cited by 39
“The class token is inherited from [37] and has been widely used in recent transformerbased image classification researches [49], [50], [51].”
MethodBViT: Broad Attention-Based Vision Transformer
Li, N., Chen, Y., Li, W., Ding, Z., Zhao, D., & Nie, S. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3264730 · Cited by 40
“The research about the difference between ViT and CNN [16] Following ViT [6], we randomly sample 128 images from ImageNet [22] dataset and calculate the mean distance between pixels with attention weights. demonstrates experimentally that the model with more local attention performs better on…”
MethodJiang, J., Wu, D., Deng, H., Long, Y., Tang, W., Li, X., Liu, C., Jin, Z., Zhang, W., & Qi, T. (2024) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3678518 · Cited by 36
“It excels in capturing low-level image features, such as edges and textures, in shallower layers, while deeper layers are capable of capturing higher-level semantic features, such as overall shape and structure.”
SPANet: Frequency-balancing Token Mixer using Spectral Pooling Aggregation Modulation
Yun, G., Yoo, J., Kim, K., Lee, J., & Kim, D. H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00562 · Cited by 27
“This commonly held belief has led to the development of numerous variations of MSA [16, 20, 64, 80] aimed at improving the performance of ViTs.”
BackgroundWang, Z.-H., Li, J., Tan, Z., Liu, X., & Li, M. (2023) · Electronics · doi:10.3390/electronics12051111 · Cited by 37
“Vision transformer (ViT) [31], transformer in transformer (TNT) [32], pyramid vision transformer (PVT) [33], tokens-to-token ViT (T2T-ViT) [34], and Swin transformer [35] have also gradually been proposed for extracting image features.”
BackgroundAn end‐to‐end steel surface defect detection approach via Swin transformer
Tang, B., Song, Z., Sun, W. M., & Wang, X. (2022) · IET Image Processing · doi:10.1049/ipr2.12715 · Cited by 38
Masked face recognition with convolutional visual self-attention network
Ge, Y., Liu, H., Du, J., Li, Z., & Wei, Y. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.10.025 · Cited by 44
“Compared with the ViT model, which uses patch-level feature, The Transformer iN Transformer (TNT) [37] projects pixel-level features into patch embedding space through linear transformation layers and then adds this pixel-level features to patches.”
TransMatting: Enhancing Transparent Objects Matting with Transformers
Cai, H., Xue, F., Xu, L., & Guo, L. (2022) · European Conference on Computer Vision · doi:10.48550/arxiv.2208.03007 · Cited by 37
“LocalViT [23] brings depth-wise convolutions to vision transformer to combine self-attention mechanism with locality, and shows great improvement compared to the pure Transformer, like DeiT, PVT, and TNT [14].”
MethodPoints to Patches: Enabling the Use of Self-Attention for 3D Shape Recognition
Berg, A., Oskarsson, M., & O'Connor, M. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956172 · Cited by 42
“[13] proposed the Transformer-in-Transformer architecture, which extends the baseline vision Transformer with pixel-wise attention within patches.”
MethodResultInfluentialBHViT: Binarized Hybrid Vision Transformer
Gao, T., Zhang, Y., Zhang, Z., Liu, H., Yin, K., Xu, C., & Kong, H. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00337 · Cited by 42
Self-citationZhao, J., Lv, Z., Li, D., Feng, X., Wang, Z., Wu, Y., Shi, D., Fowler, M. W., & Burke, A. F. (2024) · eTransportation · doi:10.1016/j.etran.2024.100364 · Cited by 45
LPViT: Low-Power Semi-structured Pruning for Vision Transformers
Xu, K.-X., Wang, Z., Chen, C., Geng, X., Lin, J., Aly, M. M. S., Yang, X.-L., Wu, M., Li, X.-L., & Lin, W.-S. (2024) · European Conference on Computer Vision · doi:10.48550/arxiv.2407.02068 · Cited by 36
“…challenges the prevailing CNNs with on-par or even superior performance on various image analysis and understanding tasks such as classification [9, 13, 18, 22, 44], object detection [1, 3, 61], semantic segmentation [5, 35], etc., but completely without the convolution mechanism seen in the CNNs.”
DETA: Denoised Task Adaptation for Few-Shot Learning
Zhang, J., Gao, L.-L., Luo, X., Shen, H.-T., & Song, J.-K. (2023) · IEEE International Conference on Computer Vision · doi:10.1109/iccv51070.2023.01060 · Cited by 36
“Furthermore, with the recent success in model pre-training techniques [14,16,33], designing efficient adapter-based [24, 25, 54] or finetuning-based [6, 19] task adaptation algorithms that can flexibly borrow “free” knowledge from a wide range of pre-trained models is therefore of great practical…”
TRT-ViT: TensorRT-oriented Vision Transformer
Xia, X., Li, J., Wu, J., Wang, X., Xiao, X., Zheng, M., & Wang, R. (2022) · arXiv.org · doi:10.48550/arxiv.2205.09579 · Cited by 36
A Simple Single-Scale Vision Transformer for Object Detection and Instance Segmentation
Chen, W., Du, X., Yang, F., Beyer, L., Zhai, X., Lin, T.-Y., Chen, H., Li, J., Song, X., Wang, Z., et al. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20080-9_41 · Cited by 41
“More recent trends [11,19,29,42,43,44] start following design conventions in CNNs discussed above (Section 2.”
BackgroundDong, P., Zhuang, J., Yang, Z., Ji, S., Li, Y., Xu, D., Huang, H., Hu, J., Jones, A. K., Shi, Y., et al. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3443692 · Cited by 35
“Many networks [1], [8], [9], [10] use ViTs as the backbone [8], [9] and show superior transferability to various downstream tasks with minor fine tuning.”
Self-citationCheng, Y., Cao, Y., Wang, D., Shen, W., & Li, W. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3544462 · Cited by 28
“Furthermore, large visual models such as ViT [35] and vision-language models such as CLIP [36] have also been used for anomaly reconstruction.”
MCI-GRU: Stock prediction model based on multi-head cross-attention and improved GRU
Zhu, P. F., Li, Y., Hu, Y., Xiang, S., Liu, Q., Cheng, D., & Liang, Y. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130168 · Cited by 44
“In recent years, the Transformer model [8] has demonstrated significant potential in capturing long-range dependencies and handling complex nonlinear features, owing to its architecture based on self-attention mechanisms [9].”
Latency-Aware Container Scheduling in Edge Cluster Upgrades: A Deep Reinforcement Learning Approach
Cui, H., Tang, Z., Lou, J., Weijia Jia, B., & Zhao, W. (2024) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2024.3394689 · Cited by 43
“The computational complexity of the MSA can be calculated as O ( N 2 D + ND 2 ) [40], [48].”
Akkaya, İ. B., Kathiresan, S. S., Arani, E., & Zonooz, B. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110510 · Cited by 50
ViT2EEG: Leveraging Hybrid Pretrained Vision Transformers for EEG DataPreprint
Yang, R., & Modesitt, E. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.00454 · Cited by 23
“Moreover, the transferability of pretrained ViT models [15, 24, 42], which are originally trained on vast datasets, permits their fine-tuning for subsequent tasks.”
BackgroundGPViT: A High Resolution Non-Hierarchical Vision Transformer with Group Propagation
Yang, C., Xu, J.-R., Mello, S. D., Crowley, E. J., & Wang, X. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2212.06795 · Cited by 35
“To tackle this challenge, a local attention mechanism is proposed (Liu et al., 2021; Huang et al., 2019; Dong et al., 2022; Xu et al., 2021a; Zhang et al., 2022; Han et al., 2021) to allow each image token to attend to local regions instead of the whole image.”
BackgroundConvolution Network Enlightened Transformer for Regional Crop Disease Classification
Wang, Y., Chen, Y., & Wang, D. (2022) · Electronics · doi:10.3390/electronics11193174 · Cited by 36
Self-citationLin, M., Chen, M., Zhang, Y., Shen, C., Ji, R., & Cao, L. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01861-3 · Cited by 40
“TNT (Han et al., 2021) leverages an inner block to strengthen the interactions of pixel information within each patch.”
BackgroundBOAT: Bilateral Local Attention Vision Transformer
Yu, T., Zhao, G., Li, P., & Yu, Y.-Z. (2022) · British Machine Vision Conference · doi:10.48550/arxiv.2201.13027 · Cited by 35
“TNT [10] also investigates local structure modeling.”
Transformer Uncertainty Estimation with Hierarchical Stochastic Attention
Pei, J., Wang, C., & Szarvas, G. (2021) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2112.13776 · Cited by 35
“Although transformers excel in terms of predictive performance (Tetko et al. 2020; Han et al. 2021), they do not offer the opportunity for practitioners to inspect the model confidence due to their deterministic nature, i.”
HybridHash: Hybrid Convolutional and Self-Attention Deep Hashing for Image Retrieval
He, C., & Wei, H. (2024) · International Conference on Multimedia Retrieval · doi:10.1145/3652583.3658014 · Cited by 31
“TNT [19] splits images with finer granularity to excavate features of objects at different scales and locations.”
BackgroundMutually Beneficial Transformer for Multimodal Data Fusion
Wang, J., & Tan, X. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3274545 · Cited by 52
“Afterward, ViT variants [33], such as data-efficient image transformers and transformer in transformer et al. [34], [35], [36], [37], [38], [39], demonstrated the bright prospects and powers of ViTs for image classification, object detection, video processing, etc.”
MethodHuang, W., Gao, T., Hao, Y., & Wang, X. (2023) · Energy Economics · doi:10.1016/j.eneco.2023.107106 · Cited by 43
PoSynDA: Multi-Hypothesis Pose Synthesis Domain Adaptation for Robust 3D Human Pose Estimation
Liu, H., He, J.-Y., Cheng, Z.-Q., Xiang, W., Yang, Q., Chai, W., Wang, G., Bao, X., Luo, B., Geng, Y., et al. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612368 · Cited by 31
“Even with respectable advancements, prevalent strategies [5, 14, 20, 24, 25, 67] still encounter difficulties due to the intricacies stemming from diverse scenes and the extensive range of 2D-3D human pose datasets [29].”
BackgroundAST: Adaptive Self-supervised Transformer for optical remote sensing representation
He, Q., Sun, X., Yan, Z., Wang, B., Zhu, Z., Diao, W., & Yang, M. Y. (2023) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2023.04.003 · Cited by 35
STGlow: A Flow-Based Generative Framework With Dual-Graphormer for Pedestrian Trajectory Prediction
Liang, R., Li, Y., Zhou, J., & Li, X. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3294998 · Cited by 44
Swin Transformer Assisted Prior Attention Network for Medical Image Segmentation
Liao, Z., Fan, N., & Xu, K. (2022) · International Conference on Computing and Artificial Intelligence · doi:10.1145/3532213.3532287 · Cited by 34
Object Detection Based on CNN and Vision‐Transformer: A Survey
Cao, J., Peng, B., Gao, M., Hao, H., Li, X., & Mou, H. (2025) · IET Computer Vision · doi:10.1049/cvi2.70028 · Cited by 37
Alrfou, K., Zhao, T., & Kordijazi, A. (2024) · Integrating materials and manufacturing innovation · doi:10.1007/s40192-024-00369-z · Cited by 49
“Other research with the similar objectives includes Transformer in Transformer (TNT) [44], Twins [45], and Shuffle Trans-former [46].”
Wang, Y., Lu, C., Lian, H.-L., Zhao, Y., Schuller, B. W., Zong, Y., & Zheng, W.-M. (2024) · IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp48485.2024.10447726 · Cited by 33
“…algorithms based on Transformer architecture for performance comparison, i.e., Vision Transformer (ViT) [21], Transformer in Transformer (TNT) [22], a hierarchical vision Transformer using shifted windows (Swin-Transformer) [12], local to global feature aggregation learning based on…”
MethodSelf-citationSCT: A Simple Baseline for Parameter-Efficient Fine-Tuning via Salient Channels
Zhao, H., Wang, P., Zhao, Y., Luo, H., Wang, F., & Shou, M. Z. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01918-3 · Cited by 21
“…remarkable success in computer vision tasks (Dosovitskiy et al., 2020; Liu et al., 2021; Yuan et al., 2022; Zhou et al., 2021a; Carion et al., 2020; Li et al., 2021; Strudel et al., 2021) by using large-scale training data, such as ImageNet21K and JFT-300M, to create strong representations.”
Zheng, W., Yang, H., Cai, J., Wang, P., Jiang, X., Du, S. S., Wang, Z., & Wang, Z. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101837 · Cited by 36
Self-citationZhu, S., Liao, B. Y., Hua, Y., Zhang, C., Wan, F., & Qing, X. (2022) · ISA Transactions · doi:10.1016/j.isatra.2022.07.016 · Cited by 33
VAQF: Fully Automatic Software-hardware Co-design Framework for Low-bit Vision Transformer
Sun, M., Ma, H., Kang, G., Jiang, Y., Chen, T., Ma, X., Wang, Z., & Wang, Y. (2022) · arXiv · doi:10.48550/arxiv.2201.06618 · Cited by 32
“Since then, many variants have been proposed [13], [23], [45], [42], [14], [44], [6], [37], [12], [21], [41], [2].”
Self-citationChangeViT: Unleashing plain vision transformers for change detection in remote sensing images
Zhu, D.-W., Huang, X.-H., Huang, H.-Y., Cheng, Q., Huang, Z., & Shao, Z. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.112539 · Cited by 32
Yao, Z., Tang, Z., Yang, W., & Weijia Jia, B. (2025) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2025.3562362 · Cited by 29
“This complexity is determined by the embedding dimension H and the number of patches C , which can be expressed as O ( H 2 C + HC 2 ) [55].”
Wild Mushroom Classification Based on Improved MobileViT Deep Learning
Peng, Y.-J., Xu, Y., Shi, J., & Jiang, S. (2023) · Applied Sciences · doi:10.3390/app13084680 · Cited by 32
Convolutional Embedding Makes Hierarchical Vision Transformer Stronger
Wang, C., Xu, H., Zhang, X., Wang, L., Zheng, Z., & Liu, H.-F. (2022) · European Conference on Computer Vision · doi:10.48550/arxiv.2207.13317 · Cited by 32
“The success of [19] further inspired the applications of Trans-former to various vision tasks, such as image classification[51,65,62,23,58], object detection[5,72,70,13] and semantic segmentation[54,46].”
Background(2024) · doi:10.1145/3613904.3642939 · Cited by 31
Metadata
- Authors
- Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., & Wang, Y.
- Affiliations
- Huawei Technologies (China) (CN)
- University of Macau (MO)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Volume
- 34
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2103.00112
- OpenAlex
- W3133696297
- Cited by
- 1,394
- References
- 56
- Citations by year
- 2026: 14 · 2025: 276 · 2024: 457 · 2023: 387 · 2022: 186 · 2021: 74
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (0015/2019/AKP, 62072449, 61632003, 0018/2019/AKP)
- Fundo para o Desenvolvimento das Ciências e da Tecnologia (0018/2019/AKP, 0015/2019/AKP)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- vision transformer, attention mechanism, local patches, visual words, image classification, feature aggregation, computational efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
