The DOI matches its DataCite record, with a small difference in title or year; 56 of the 56 references it cites were found in a registry or an index.
Transformer in Transformer
Open accessAbstract
Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (e.g., 16$\times$16) as "visual sentences" and present to further divide them into smaller patches (e.g., 4$\times$4) as "visual words". The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs. Features of both words and sentences will be aggregated to enhance the representation ability. Experiments on several benchmarks demonstrate the effectiveness of the proposed TNT architecture, e.g., we achieve an 81.5% top-1 accuracy on the ImageNet, which is about 1.7% higher than that of the state-of-the-art visual transformer with similar computational cost. The PyTorch code is available at https://github.com/huawei-noah/CV-Backbones, and the MindSpore code is available at https://gitee.com/mindspore/models/tree/master/research/cv/TNT.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For example, in multi-tenant architectures, system prompts are shared across multiple requests and frequently contain detailed instructions or examples to guide the model [19].”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [19]
In the citing paper
“PromptCache [19] proposes modular reuse beyond prefixes but sacrifices accuracy, with reported performance drops of up to 43% [72], making it unsuitable for tasks requiring high precision.”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [19]
In the citing paper
“related experiments results further demonstrate ViT also has remarkable representation capacity when dealing with images and videos [19].”
From Deepfake Video Detection with Spatiotemporal Dropout Transformer (2022) · cites this paper as [19]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 6×
“Compared to CNNs, TNT can outperform the widely-used ResNet and RegNet.”
BackgroundMethodInfluentialVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3×
“The effect of #heads in standard transformer has been investigated in multiple works [21, 35] and a head width of 64 is recommended for visual tasks [9, 31].”
BackgroundMethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“In the past decade, the mainstream deep neural architectures used in the computer vision (CV) are mainly established on convolutional neural networks (CNNs) [17, 12, 11].”
BackgroundFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472 · cited 1×
“As a general backbone network, TNT can also be applied with multi-scale vision models like Faster RCNN [29].”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472 · cited 7×
“The recent transformerbased models like ViT [9] and DeiT [31] are compared.”
MethodInfluentialImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 3×
“ImageNet ILSVRC 2012 [26] is an image classification benchmark consisting of 1.”
BackgroundInfluentialSqueeze-and-Excitation NetworksPreprint
Hu, J., Shen, L., & Sun, G. (2017) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1709.01507 · Cited by 37,845 · cited 1×
“Inspired by squeeze-and-excitation (SE) network for CNNs [16], we propose to explore channel-wise attention for transformers.”
MethodPyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939 · cited 2×
“We evaluate TNT-S and DeiT-S on Faster RCNN with FPN [21].”
BackgroundMethod
Research works printed but not linked: 5
- [6][6] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML , 2020. -
- [11][11] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. arXiv preprint arXiv:2012.12556 , 2020. - · arXiv
- [23][23] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017. - · arXiv
- [26][26] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR , pages 3498-3505. IEEE, 2012. -
- [37][37] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008. -
Cited by
Sorted by citations
Items 901–1,000 of 1,009
Semi-supervised intracranial aneurysm segmentation via reliable weight selection
Cao, W., Chen, X., Lv, J., Shao, L., & Si, W.-X. (2024) · The Visual Computer · doi:10.1007/s00371-024-03730-7 · Cited by 7
Chen, W.-H., Zhou, H.-T., Mao, T., Cheng, L.-D., & Xia, M. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3438236 · Cited by 7
“1 [22].”
Dai, L., Zhao, C., Zhang, X., Wang, D., & Huo, L. (2024) · Forests · doi:10.3390/f15111860 · Cited by 7
“The Transformer is an advanced deep learning architecture that leverages a self-attention mechanism, resulting in substantial breakthroughs in natural language processing (NLP) [26].”
Feasibility of Transformer Model for User Authentication Using Electromyogram Signals
Choi, H.-S. (2024) · Electronics · doi:10.3390/electronics13204134 · Cited by 7
SATA: Spatial Autocorrelation Token Analysis for Enhancing the Robustness of Vision Transformers
Nikzad, N., Liao, Y., Gao, Y., & Zhou, J. (2024) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52734.2025.00909 · Cited by 7
“For instance, models like Swin [29], PVT [50], and TNT-S [17] achieve higher standard accuracy Table 1.”
LaMamba-Diff: Linear-Time High-Fidelity Diffusion Models Based on Local Attention and MambaPreprint
Fu, Y., Chen, C., & Yu, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.02615 · Cited by 5
“Fine-grained local information within each patch is crucial for excavating features from visual data [18].”
InfluentialMeng, M., Zhao, Y., Zhang, B., Zhu, Y., Shi, W., Wen, M., & Fan, Z. (2024) · arXiv.org · doi:10.48550/arxiv.2406.10553 · Cited by 7
“The field can be broadly divided into two main approaches: video-driven methods and audio-driven methods.”
BackgroundParaLkResNet: an efficient multi-scale image classification network
Yu, T., Liu, Y., Liu, H., Chen, J., & Wang, X. (2024) · The Visual Computer · doi:10.1007/s00371-024-03508-x · Cited by 11
Revolutionizing Traffic Sign Recognition: Unveiling the Potential of Vision Transformers
Mingwin, S., Shisu, Y., Wanwag, Y., & Huing, S. (2024) · arXiv.org · doi:10.48550/arxiv.2404.19066 · Cited by 7
“BelgiumTS: To evaluate the proposed TSR model’s efficacy and robustness, it was compared to well-known models such as PVT [7], TNT [8], AlexNet [10], and GoogleNet [15] on the BelgiumTS [2] dataset.”
MethodResultVehicle Reidentification Based on Convolution and Vision Transformer Feature Fusion
Gong, R., Zhang, X., Pan, J.-N., Guo, J., & Nie, X. (2024) · IEEE Multimedia · doi:10.1109/mmul.2024.3398189 · Cited by 7
“Han et al. 13 modeled patch and pixel-level representations through inner and outer transformers to extract global and local features.”
BackgroundSelf-citationFCDS-DETR: detection transformer based on feature correction and double sampling
Wang, M., Jiao, Z., Huang, Z., & Yu, S. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09430-6 · Cited by 7
“The success of transformer in the field of NLP has laid the foundation for subsequent applications in image classifi-cation [26–29], image generation [30–33], action recognition [34–36], and fault diagnosis [37, 38].”
Pedestrian Trajectory Prediction Based on Multi-head Self-attention and Sparse Graph Convolution
Yu, C., & Wang, M. (2024) · 2024 4th International Conference on Neural Networks, Information and Communication (NNICE) · doi:10.1109/nnice61279.2024.10498338 · Cited by 7
“STAR [10] addresses the complex interactions between pedestrians by processing pedestrian trajectories through a spatial-temporal Transformer [11] to better capture the interaction information in both temporal and spatial dimensions.”
BackgroundHigh dimensional mislabeled learning
Han, H., Li, D., Liu, W., Zhang, H., & Wang, J. (2024) · Neurocomputing · doi:10.1016/j.neucom.2023.127218 · Cited by 15
A novel loss function to reproduce texture features for deep learning-based MRI-to-CT synthesis.
Yuan, S., Liu, Y.-X., Wei, R., Zhu, J., Men, K., & Dai, J.-R. (2023) · Medical Physics (Lancaster) · doi:10.1002/mp.16850 · Cited by 7
Xu, C., Yu, F., Xu, Z., Liu, C., Xiong, J., & Chen, X. (2024) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/asp-dac58780.2024.10473936 · Cited by 3
Self-citationRecursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalability
Chowdhury, J. R., & Caragea, C. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2311.04449 · Cited by 7
“RIR-related Approaches: Many approaches have used chunking for hierarchical processing [90, 98, 59, 81, 15, 79, 36, 103, 97, 83, 72, 10, 9, 43, 60] to efficiently tackle long-range inputs.”
BackgroundDeep Intrinsic Decomposition With Adversarial Learning for Hyperspectral Image Classification
Gong, Z.-Q., Qi, J.-H., Zhong, P., Zhou, X., & Yao, W. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3476932 · Cited by 7
“Vision transformer (ViT) [29], transformer in transformer (TNT) [30], and Spec-tralFormer [31] are typical transformers that can be applied for hyperspectral image classification.”
BackgroundTransformer with Hybrid Attention Mechanism for Stereo Endoscopic Video Super Resolution
Zhang, T., & Yang, J. (2023) · Symmetry · doi:10.3390/sym15101947 · Cited by 7
Dynamic Spectrum Mixer for Visual Recognition
Hu, Z.-Q., & Yu, T. (2023) · arXiv.org · doi:10.48550/arxiv.2309.06721 · Cited by 7
Sparse then Prune: Toward Efficient Vision Transformers
Prasetyo, Y., Yudistira, N., & Widodo, A. (2023) · arXiv.org · doi:10.48550/arxiv.2307.11988 · Cited by 7
“The Transformer-iN-Transformer (TNT)[12] was proposed to enhance the ability of feature representation in ViT by dividing the input images into several patches as "visual sentence" and then divide those patches into sub-patches, as the representation of "visual word".”
MethodAn interactive network based on transformer for multimodal crowd counting
Yu, Y., Zhen, C., Miao, D., Qian, J., & Tang, H. (2023) · Applied Intelligence · doi:10.1007/s10489-023-04721-2 · Cited by 8
“[27] proposed a Transformer-in-Transformer (TNT) architecture for visual recognition.”
BackgroundZheng, S., & Yuan, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01170 · Cited by 11
“Vision Transformer (ViT)[9] and its follow-ups[15, 41, 45, 57, 44] start the trend of that backbone architectures for computer vision shift from CNNs to Transformers.”
Rethinking vision transformer through human-object interaction detection
Cheng, Y.-M., Zhao, Z.-T., Wang, Z., & Duan, H. (2023) · Engineering applications of artificial intelligence · doi:10.1016/j.engappai.2023.106123 · Cited by 7
Revisit Parameter-Efficient Transfer Learning: A Two-Stage ParadigmPreprint
Zhao, H., Luo, H., Zhao, Y., Wang, P., Wang, F., & Shou, M. Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.07910 · Cited by 7
“Numerous vision transformers [5, 15, 13, 1, 17, 20, 49, 63, 51, 39, 56, 68] have been proposed following the pioneering work of ViT [14].”
BackgroundHyneter: Hybrid Network Transformer for Object Detection
Chen, D., Miao, D., & Zhao, X. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096922 · Cited by 11
“ViT and its follow-ups [21, 22, 23, 24] achieve an impressive speed-accuracy trade-off on image classification compared to convolutional networks.”
BackgroundEfficiency 360: Efficient Vision Transformers
Patro, B., & Agneeswaran, V. S. (2023) · arXiv.org · doi:10.48550/arxiv.2302.08374 · Cited by 7
“…robustness of a model, ViT [Dosovitskiy et al. , 2020] Global MSA Linear Isotropic DeiT [Touvron et al. , 2021a] Global MSA Linear Isotropic TNT [Han et al. , 2021] Global MSA Linear Isotropic T2T [Yuan et al. , 2021] Global MSA Linear Isotropic Cross-ViT [Chen et al. , 2021a] Global MSA Linear…”
BackgroundIH-ViT: Vision Transformer-based Integrated Circuit Appear-ance Defect DetectionPreprint
Wang, X., Gao, S., Zou, Y., Guo, J., & Wang, C. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.04521 · Cited by 7
“For example, Transformer-in-Transformer (TNT), Swin Transformer, regionViT, etc. TNT [28]. utilizes inner Transformer block to model the relationship between sub-patches and an outer Transformer block for patch-level information exchange.”
MethodSelf-citationA Light Touch Approach to Teaching Transformers Multi-view Geometry
Bhalgat, Y. S., Henriques, J. F., & Zisserman, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00480 · Cited by 8
“There are many such recent examples, where shedding priors in favour of learning from more data has proven to be a successful strategy, from image classification [1,12,19,28,87], action recognition [6, 22, 26, 47, 55], to text-image matching [34, 43, 59, 69] and 3D recognition [38, 88].”
BackgroundV2ReID: Vision-Outlooker-Based Vehicle Re-Identification
Qian, Y., Barthélemy, J., Iqbal, U., & Perez, P. (2022) · Sensors · doi:10.3390/s22228651 · Cited by 7
“, Swin [67], TNT [68], and VOLO [10], or using convolutions, e.”
MethodSimilarity of Neural Architectures Using Adversarial Attack Transferability
Hwang, J., Han, D., Heo, B., Park, S., Chun, S., & Lee, J.-S. (2022) · European Conference on Computer Vision · doi:10.1007/978-3-031-73113-6_7 · Cited by 7
In silico prediction methods of self-interacting proteins: an empirical and academic survey
Chen, Z., You, Z., Zhang, Q., Guo, Z.-H., Wang, S., & Wang, Y. (2022) · Frontiers of Computer Science · doi:10.1007/s11704-022-1563-1 · Cited by 7
Self-citationCSTGAN: Cycle Swin Transformer GAN for Unpaired Infrared Image Colorization
Zhao, M., Feng, G., Tan, J.-H., Zhang, N., & Lu, X. (2022) · International Conference on Control, Robotics and Intelligent System · doi:10.1145/3562007.3562053 · Cited by 7
“Transformer has made significant improvements [14], such as Vision Transformer (ViT) [4], Swin Transformer [15], Data-efficient image Transformer (DeiT) [25], Transformer-in-Transformer (TNT) [8] and Focal Transformer [29].”
MethodVisualizing and Explaining Language Models
Braşoveanu, A. M. P., & Andonie, R. (2022) · arXiv.org · doi:10.48550/arxiv.2205.10238 · Cited by 7
Wei, Z., Qi, X.-W., Bai, Z.-W., Wu, G.-Y., Nayak, S., Hao, P., Barth, M., Liu, Y.-K., & Oguchi, K. (2022) · 2022 IEEE Intelligent Vehicles Symposium (IV) · doi:10.48550/arxiv.2203.00138 · Cited by 7
“Transformer (ViT) [33] and its follow-ups [34-36].”
Global Interaction Modelling in Vision Transformer via Super TokensPreprint
Farooq, A., Awais, M., Ahmed, S. A. A., & Kittler, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.13156 · Cited by 4
“The Deit [16] introduced distillation from the CNN teacher network while the CaiT [29] proposed a layer scaling operation to aid with increasing depth of the transformers.”
BackgroundMethodDB-HDFFN: Dual Branch Hierarchical Dynamic Feature Fusion Network for medical image classification
Wen, Y., Chen, B., Shi, W.-Z., Wu, S., Yang, X., & Sheng, B. (2026) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2026.109538 · Cited by 6
The Content Moderator's Dilemma: Removal of Toxic Content and Distortions to Online Discourse
Habibi, M., Hovy, D., & Schwarz, C. R. (2026) · CESifo · doi:10.65864/zs7xnqogiy · Cited by 8
“…has received over 200,000 citations, and Transformer models have proven highly successful in numerous applications (e.g., Zhu et al., 2020; Strudel et al., 2021; Han et al., 2021; Radford et al., 2023), as they have been shown to capture text semantics more effectively than previous approaches.”
Huang, Z.-G., Chen, M.-S., & Zheng, S.-Y. (2026) · Neurocomputing · doi:10.1016/j.neucom.2025.132562 · Cited by 6
A comprehensive survey on sentiment analysis: Framework, techniques, and applications
Chandan, M. K., & Mandal, S. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100777 · Cited by 11
Yang, J., Chen, Y., Yu, T., & Zhang, Y. (2025) · Sensors · doi:10.3390/s25196065 · Cited by 4
“However, the sequential nature of LSTM structures hinders parallel execution [24].”
Performer: A High-Performance Global-Local Model-Augmented With Dual Network Interaction Mechanism
Tan, D.-Y., Hua, L.-F., Hao, R., Xu, Q., Su, Y.-S., Zheng, C., & Zhong, W.-M. (2025) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2024.3519629 · Cited by 6
“We employ several fully connected layers to match dimensions when utilizing blocks from ViT to complete the interactions [6], [34], [35].”
Zheng, Z., Luo, X., Pei-wen, L., Piao, S., Cao, X., Liu, X., Yang, L., Hu, B., Geng, Y., & Geng, D. (2025) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2025.102624 · Cited by 5
The design of consumer behavior prediction and optimization model by integrating DQN and LSTM
Liu, N., & Hu, D. (2025) · PLoS ONE · doi:10.1371/journal.pone.0327548 · Cited by 6
“The Transformer model epitomizes a deep learning paradigm, harnessing the Self-Attention mechanism to tackle sequential data, thereby adeptly addressing long-range dependency challenges and facilitating efficient parallel data processing [22].”
Li, J., Xu, H.-Y., Zhu, X.-Z., Xiong, J., & Zhang, X.-L. (2025) · Food Chemistry · doi:10.1016/j.foodchem.2025.145276 · Cited by 6
A Mamba based vision transformer for fine grained image segmentation of mural figures
Zhang, Q., Geng, G.-H., Zhou, P.-B., Yan, L.-H., Zhou, M.-Q., Li, Z., & Liu, Y.-Y. (2025) · npj Heritage Science · doi:10.1038/s40494-025-01682-2 · Cited by 6
“In computer vision, Transformers were introduced to develop the Vision Transformer (ViT) 11,12 , which effectively captures relationships between different regions of an image through its global modeling capability.”
Wang, J., Yoshie, O., & Ieiri, Y. (2025) · Neural computing & applications (Print) · doi:10.1007/s00521-025-11234-1 · Cited by 6
“The transformer-in-transformer (TNT) [18] further explores the pixel-level features within each patch.”
Jia, G., Xu, L., & Zhu, L. (2025) · International Conference on Video and Image Processing (ICVIP) · doi:10.1117/12.3059115 · Cited by 3
“Specifically, the current perception of the advantages of ViT over ConvNets mainly stems from their multi-head self-attention layers 15 , which facilitate global feature integration, whereas convolutions are inherently limited to local integration 16 .”
Leveraging deep learning for accurate and automated interpretation of molecular IR and Raman spectra
Hu, T., Zhang, Y., & Hu, W. (2025) · Progress in Natural Science Materials International · doi:10.1016/j.pnsc.2025.02.014 · Cited by 5
Transformer-based weakly supervised 3D human pose estimation
Wu, X., Xie, H.-J., Niu, X., Wang, C., Wang, Z.-L., Zhang, S., & Shan, Y. (2025) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2025.104432 · Cited by 6
Seekg: Sentiment analysis for E-Learning evaluation incorporating knowledge graphs
Yi, W., Huang, X., Kuzmin, S. A., Герасимов, И. В., & Luo, Y. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13456-1 · Cited by 6
“In text semantic mining, deep learning models have shown powerful capabilities: Liao et al. (2022) (Han et al., 2021).”
BossNAS Family: Block-Wisely Self-Supervised Neural Architecture Search
Li, C., Lin, S., Tang, T., Wang, G., Li, M., Liang, X., & Chang, X. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3529517 · Cited by 8
Transformer based deep learning accelerated numerical simulation for incompressible flow
Cui, Q., Zhang, M., Xiao, M., & Ni, G. (2024) · The Physics of Fluids · doi:10.1063/5.0243174 · Cited by 6
Fault Prediction for Rotating Mechanism of Satellite Based on SSA and Improved Informer
Lan, Q., Zhu, Y., Lin, B.-J., Zuo, Y., & Lai, Y. (2024) · Applied Sciences · doi:10.3390/app14209412 · Cited by 6
BA-Net: Bridge Attention in Deep Neural Networks
Zhang, R.-H., Zou, R., Zhao, Y., Zhang, Z.-R., Chen, J.-Z., Cao, Y., Hu, C., & Song, H. (2024) · Expert systems with applications · doi:10.48550/arxiv.2410.07860 · Cited by 6
“Subsequent modifications to ViT aimed to improve architectural design to further enhance performance in vision tasks [19], [21], [40]–[42].”
BackgroundA pyramid convolutional mixer for cervical pap-smear image classification tasks
Yang, T., Hu, H., Li, X., Qing, M., Chen, L., & Huang, Q. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106789 · Cited by 7
Learning multi-behavior user intent for session-based recommendation
Zhang, Y., Zhu, X., He, G., Li, J., & Wang, J. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125269 · Cited by 6
Xu, S., Xu, Z.-B., Zheng, J., Lin, H., Zou, L., & Lei, M. (2024) · Memetic Computing · doi:10.1007/s12293-024-00425-3 · Cited by 6
“6, GMTB is based on the Transformer architecture with improvements, including self-attention blocks and MRFB [28–31].”
CTFusion: CNN-transformer-based self-supervised learning for infrared and visible image fusion.
Du, K., Fang, L., Chen, J., Chen, D., & Lai, H. (2024) · Mathematical biosciences and engineering : MBE · doi:10.3934/mbe.2024294 · Cited by 6
“Based on ViT, a series of ViT variants were proposed to improve the performance [41, 42].”
LKCA: large kernel convolutional attention
Li, C., Shi, P., Chen, Q., Liu, J.-R., & Zhu, L. (2024) · Other Conferences · doi:10.1117/12.3031589 · Cited by 6
“Transformer in Transformer (TNT) [12] enhances the feature representation capability of local patches through internal attention.”
Tu, Y., Tu, F., Yang, Y., Qian, J., Wu, X., & Yang, S. (2024) · Science in progress · doi:10.1177/00368504241274999 · Cited by 6
Huang, Q. Y., Luo, Z., Xing, Z., Zeng, J., Chen, J., Xu, X., & Chen, Y. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3672458 · Cited by 6
“It is pre-trained on a vast corpus of text data including open-source project code, programming language specifications, etc. [53], and utilizes a transformer-based architecture [54].”
BackgroundBoostedDim attention: A novel data-driven approach to improving LiDAR-based lane detection
Patil, O., Nair, B. B., Soni, R., Thayyilravi, A., & Manoj, C. R. (2024) · Ain Shams Engineering Journal · doi:10.1016/j.asej.2024.102887 · Cited by 6
Qiu, Y., Xie, Z., Jiang, Y., & Ma, J. (2024) · Journal of Medical Imaging · doi:10.1117/1.jmi.11.3.034504 · Cited by 6
“21 TransUNet and SwinUNet combine U-Net with transformer in innovative ways, achieving high segmentation performance on large datasets.”
BackgroundCT-MVSNet: Curvature-guided multi-view stereo with transformers
Wang, L., Sun, L., & Duan, F.-Q. (2024) · Multimedia tools and applications · doi:10.1007/s11042-024-19227-3 · Cited by 6
“In the self-attention mechanism of the Transformer, when the Q and K vectors are derived from features of the same view, the Transformer’s role is to enhance the features within that view [26].”
Human Digital Twin for Real-Time Physical Fatigue Estimation in Human-Robot Collaboration
You, Y.-C., Liu, Y., & Ji, Z. (2024) · International Conference on Industrial Technology · doi:10.1109/icit58233.2024.10541029 · Cited by 6
“The AC-BiLSTM [31] and Transformer [32] methods have demonstrated their effectiveness in handling sequential data tasks, such as text classification, making them suitable comparators as well.”
MethodImproved learning efficiency of deep Monte-Carlo for complex imperfect-information card games
Luo, Q., & Tan, T.-P. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.111545 · Cited by 7
Gong, Y., Wu, P., Xu, R., Zhang, X., Wang, T., & Li, X. (2024) · The Visual Computer · doi:10.1007/s00371-024-03294-6 · Cited by 9
“TNT [42] has raised the question of whether the performance of ViTs is primarily attributed to the inherently more powerful Trans-former architecture or whether it is at least partly influenced by the use of patches as the input representation.”
Investigating Recurrent Transformers with Dynamic HaltPreprint
Chowdhury, J. R., & Caragea, C. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.00976 · Cited by 6
“Theoretical reasons (Han et al., 2021; Hao et al., 2022; Merrill et al., 2022) are also suggested for such limitations.”
BackgroundLearning Image-Adaptive Lookup Tables With Spatial Awareness for Image Harmonization
Huang, Y., Yu, D., Zhang, H., Huang, J., & Chen, S. (2023) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2023.3320662 · Cited by 5
“Subsequently, more recent methods such as CPVT [61], Swin Transformer [62], TNT [63], CrossViT [64] and LocalViT [65] introduce tailored changes to ViT to further improve image classification performance.”
MethodSun, Y., Zhang, J., & Zhang, Y.-A. (2024) · The Physics of Fluids · doi:10.1063/5.0187614 · Cited by 6
DocBinFormer: A Two-Level Transformer Network for Effective Document Image Binarization
Biswas, R., Roy, S. K., Wang, N., Pal, U., & Huang, G.-B. (2023) · arXiv.org · doi:10.48550/arxiv.2312.03568 · Cited by 6
“The presented approach diverges from TNT [10] in the way it efficiently processes and aggregates global and local information.”
MethodChai, G., Li, S., Yang, Y., Zhou, G., & Wang, Y. (2023) · Sensors · doi:10.3390/s23218793 · Cited by 6
“Transfomer, a DL network architecture based on self-attention mechanisms [16], dynamically computes attention weights for each position based on the relationships between different feature positions in the input data.”
BackgroundSelf-citationVariation-learning high-resolution network for capsulorhexis recognition of cataract surgery
Bian, G., Yue, W.-Q., Li, Z., Zhang, L., Zhang, S., Liu, W.-P., Li, S., Medeiros, E. P., Wu, W., & de Albuquerque, V. H. C. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110841 · Cited by 6
Lu, P., Creagh, A. P., Lu, H., Hai, H., Thwaites, L., & Clifton, D. A. (2023) · Italian National Conference on Sensors · doi:10.3390/s23187705 · Cited by 6
“TNT [36] processes the relationship between sub-patches via an inner transformer block and captures the interconnections among patch-level embeddings via an outer transformer block.”
Pan, L., Wang, X., Liang, Q., Shang, J., Liu, W., Xu, L., & Peng, S. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108478 · Cited by 10
Pan, L.-R., Liu, D., Dou, Y., Wang, L., Feng, Z.-C., Rong, P.-F., Xu, L.-W., & Peng, S.-L. (2023) · arXiv.org · doi:10.48550/arxiv.2307.04075 · Cited by 6
Random Position Adversarial Patch for Vision Transformers
Shao, M. (2023) · arXiv.org · doi:10.48550/arxiv.2307.04066 · Cited by 6
“Recently, vision transformers (ViTs) have garnered significant attention due to their impressive performance [3–5, 9, 11, 14, 25, 30] and their ability to surpass convolutional neural networks (CNNs) in various domains.”
BackgroundNode-Loss Detection Methods for CZ Silicon Single Crystal Based on Multimodal Data Fusion
Jiang, L., Xue, R., & Liu, D. (2023) · Sensors · doi:10.3390/s23135855 · Cited by 8
Automatic detection of COVID-19 and pneumonia from chest X-ray images using texture features
Sheikhi, F., Taghdiri, A., Moradisabzevar, D., Rezakhani, H., Daneshkia, H., & Goodarzi, M. (2023) · Journal of Supercomputing · doi:10.1007/s11227-023-05452-4 · Cited by 6
“Since ViT architectures showed superior performance to CNNs in different classification tasks [22], Jiang and Lin [31] applied Swin Transformer [39] and Transformer in Transformer [26] for the ternary classification of images into the COVID-19, pneumonia, and normal classes.”
MethodDisaster cassification net: A disaster classification algorithm on remote sensing imagery
Yuan, J., Ding, X., Liu, F., & Cai, X.-W. (2023) · Frontiers in Environmental Science · doi:10.3389/fenvs.2022.1095986 · Cited by 6
“Transformer, as a rising star of arti fi cial intelligence, emerged as a result of the application of Google ’ s BERT model (Tenney et al., 2019) to NLP (Natural Language Processing) (KangCai et al., 2020) tasks in 2019 and achieved remarkable results in machine translation tasks.”
MethodInfluentialTwinsReID: Person re-identification based on twins transformer's multi-level features
Jin, K., Zhai, J., & Gao, Y. (2022) · Mathematical Biosciences & Engineering · doi:10.3934/mbe.2023098 · Cited by 6
“TNT [10] is a Transformer-based model architecture proposed by Huawei Noah Labs in 2021.”
MethodExploring attention on faces: similarities between humans and Transformers
Cadoni, M. I., Nixon, S., Lagorio, A., & Fadda, M. (2022) · IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) · doi:10.1109/avss56176.2022.9959705 · Cited by 7
“This information is used to drive the patch-selection for a Vision Transformer improving performance and reducing compute cost.”
BackgroundInfluentialSUMLP: A Siamese U-shaped MLP-based network for change detection
Zhang, C., Wang, L., Cheng, S., & Li, Y. (2022) · Applied Soft Computing · doi:10.1016/j.asoc.2022.109766 · Cited by 5
Park, J., Kahatapitiya, K., Kim, D., Sudalairaj, S., Fan, Q., & Ryoo, M. S. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00118 · Cited by 8
“Some Transformers such as CrossViT [4], TNT [10], Re-gionViT [3] keep two branches to encode low-level semantics and high-level semantics from the early stage.”
BackgroundMethodFully-attentive and interpretable: vision and video vision transformers for pain detection
Fiorentini, G., Ertugrul, I. O., & Salah, A. A. (2022) · arXiv.org · doi:10.48550/arxiv.2210.15769 · Cited by 6
“Previous works have shown that attention maps can be used to generate visual interpretations for the predictions of vision transformers [19, 34].”
BackgroundHierarchical Walking Transformer for Object Re-Identification
Tian, X., Liu, J., Zhang, Z., Wang, C., Qu, Y., Xie, Y., & Ma, L. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548401 · Cited by 6
“It should be emphasized here that we select representative and relevant trans-former architectures (Swin Transformer [25], ViT [15], TNT-S [13], DynamicViT [31], Luna [27]) and conduct comparison on both performance and efficiency.”
BackgroundMethodInfluentialTowards Flexible Inductive Bias via Progressive Reparameterization Scheduling
Lee, Y., Lee, G., Ryoo, K.-S., Go, H., Park, J., & Kim, S. (2022) · ECCV Workshops · doi:10.48550/arxiv.2210.01370 · Cited by 6
“Other approaches use the local attention mechanisms for introducing locality to ViTs [13, 22], which attend to the neighbor elements and improve the local extraction ability of global attention mechanisms.”
MethodZhao, L.-J., Liu, C., & Qu, H. (2022) · Italian National Conference on Sensors · doi:10.3390/s22186855 · Cited by 6
“…focus on improving the ability of global feature extraction and strengthening the modeling of local information, such as transformer in transformer [29], Swin transformer [30], Regionvit [31], etc. Transformer in transformer [29] constructs a sequence of image blocks and a sequence of super-pixels…”
MethodUCViT: Hardware-Friendly Vision Transformer via Unified Compression
Song, H., Wang, Y., Wang, M.-Q., & Wang, Z.-F. (2022) · International Symposium on Circuits and Systems · doi:10.1109/iscas48785.2022.9937660 · Cited by 6
Self-citationSSA-SL Transformer for Bearing Fault Diagnosis under Noisy Factory Environments
Lee, S., & Jeong, J. (2022) · Electronics · doi:10.3390/electronics11091504 · Cited by 6
CD-Net: Histopathology Representation Learning using Pyramidal Context-Detail Network
Kapse, S., Das, S., & Prasanna, P. (2022) · arXiv.org · doi:10.48550/arxiv.2203.15078 · Cited by 6
“For baseline, we adapt ViT-S [28] and TNT-S [16] architecture on TCGA dataset.”
BackgroundMethodInfluentialHuang, B., Liu, Y., Tang, B., & Fang, G. (2026) · Neural Processing Letters · doi:10.1007/s11063-025-11823-0 · Cited by 5
A General Framework for Activation Function Optimization Based on Mollification Theory
Zhang, W., Zhang, Y., Zheng, Y., & Mo, W. (2025) · Mathematics · doi:10.3390/math14010072 · Cited by 5
Li, H.-X., Liu, Y., & Liu, L.-P. (2025) · Engineering applications of artificial intelligence · doi:10.1016/j.engappai.2025.112150 · Cited by 5
Financial Time Series Prediction With Multi-Granularity Graph Augmented Learning
Zhu, P., Li, Y.-T., Liu, Q.-Y., Cheng, D.-W., & Jiang, C. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3607005 · Cited by 5
“Time series models like long short-term memory (LSTM) networks mitigate gradient vanishing and explosion problems through gating mechanisms, while Transformer-based models [5] utilize self-attention to capture long-range dependencies with strong performance on sequential data.”
RLTNT: An explainable residual learning-based transformer model for kidney disease classification
Firos, V. M., Alphonse, P. J. A., Fiore, U., & Gangadharan, G. R. (2025) · Image and Vision Computing · doi:10.1016/j.imavis.2025.105781 · Cited by 5
Du, X.-K., Cao, J.-F., Jiang, X.-Y., Wang, Q., Xu, B., Liu, Z.-Y., Chen, Y.-K., & Yuan, C.-H. (2025) · Scientific Reports · doi:10.1038/s41598-025-22926-y · Cited by 5
“Attention mechanism models : Transformer 9 as the pioneering work of attention mechanisms, achieves direct modeling of relationships between arbitrary positions in sequences through global self-attention mechanisms.”
Deo, B. S., Pal, M., K.Panigrahi, P., & Pradhan, A. (2025) · Discover Electronics · doi:10.1007/s44291-025-00112-8 · Cited by 5
“Despite the disruption of certain aspects of the internal image framework [41], the non-overlapping patch strategy is employed to feed patches into the Trans-former model, primarily because it efficiently avoids computational redundancy.”
Transformer in civil engineering defect detection: A survey
Zhang, J., 郑智能, & Ling, T. (2025) · Journal of Traffic and Transportation Engineering (English Edition) · doi:10.1016/j.jtte.2024.11.008 · Cited by 2
Cao, H., Lu, H., Wang, X.-S., & Wang, X. (2025) · IEEE International Conference on Computer Vision · doi:10.1109/iccv51701.2025.00194 · Cited by 5
“For the first setting, the target ViT models contain four normally trained ViTs: CaiT-S/24 [34], TNT-S [13], LeViT-256 [11], ConViT-B [6], three robust ViTs: RVT-S ∗ [25], Drvit [23], Vit+Dat [24], and an adversarially trained ViT: ViT-B/16 AT [27].”
Metadata
- Authors
- Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., & Wang, Y.
- Affiliations
- Huawei Technologies (China) (CN)
- University of Macau (MO)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Volume
- 34
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2103.00112
- OpenAlex
- W3133696297
- Cited by
- 1,394
- References
- 56
- Citations by year
- 2026: 14 · 2025: 276 · 2024: 457 · 2023: 387 · 2022: 186 · 2021: 74
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (0015/2019/AKP, 62072449, 61632003, 0018/2019/AKP)
- Fundo para o Desenvolvimento das Ciências e da Tecnologia (0018/2019/AKP, 0015/2019/AKP)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- vision transformer, attention mechanism, local patches, visual words, image classification, feature aggregation, computational efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
