The DOI matches its DataCite record, with a small difference in title or year; 56 of the 56 references it cites were found in a registry or an index.
Transformer in Transformer
Open accessAbstract
Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (e.g., 16$\times$16) as "visual sentences" and present to further divide them into smaller patches (e.g., 4$\times$4) as "visual words". The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs. Features of both words and sentences will be aggregated to enhance the representation ability. Experiments on several benchmarks demonstrate the effectiveness of the proposed TNT architecture, e.g., we achieve an 81.5% top-1 accuracy on the ImageNet, which is about 1.7% higher than that of the state-of-the-art visual transformer with similar computational cost. The PyTorch code is available at https://github.com/huawei-noah/CV-Backbones, and the MindSpore code is available at https://gitee.com/mindspore/models/tree/master/research/cv/TNT.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For example, in multi-tenant architectures, system prompts are shared across multiple requests and frequently contain detailed instructions or examples to guide the model [19].”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [19]
In the citing paper
“PromptCache [19] proposes modular reuse beyond prefixes but sacrifices accuracy, with reported performance drops of up to 43% [72], making it unsuitable for tasks requiring high precision.”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [19]
In the citing paper
“related experiments results further demonstrate ViT also has remarkable representation capacity when dealing with images and videos [19].”
From Deepfake Video Detection with Spatiotemporal Dropout Transformer (2022) · cites this paper as [19]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 6×
“Compared to CNNs, TNT can outperform the widely-used ResNet and RegNet.”
BackgroundMethodInfluentialVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3×
“The effect of #heads in standard transformer has been investigated in multiple works [21, 35] and a head width of 64 is recommended for visual tasks [9, 31].”
BackgroundMethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“In the past decade, the mainstream deep neural architectures used in the computer vision (CV) are mainly established on convolutional neural networks (CNNs) [17, 12, 11].”
BackgroundFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472 · cited 1×
“As a general backbone network, TNT can also be applied with multi-scale vision models like Faster RCNN [29].”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472 · cited 7×
“The recent transformerbased models like ViT [9] and DeiT [31] are compared.”
MethodInfluentialImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 3×
“ImageNet ILSVRC 2012 [26] is an image classification benchmark consisting of 1.”
BackgroundInfluentialSqueeze-and-Excitation NetworksPreprint
Hu, J., Shen, L., & Sun, G. (2017) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1709.01507 · Cited by 37,845 · cited 1×
“Inspired by squeeze-and-excitation (SE) network for CNNs [16], we propose to explore channel-wise attention for transformers.”
MethodPyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939 · cited 2×
“We evaluate TNT-S and DeiT-S on Faster RCNN with FPN [21].”
BackgroundMethod
Research works printed but not linked: 5
- [6][6] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML , 2020. -
- [11][11] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. arXiv preprint arXiv:2012.12556 , 2020. - · arXiv
- [23][23] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017. - · arXiv
- [26][26] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR , pages 3498-3505. IEEE, 2012. -
- [37][37] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008. -
Cited by
Sorted by citations
Items 801–900 of 1,009
Transformer in Transformer as Backbone for Deep Reinforcement Learning
Mao, H., Zhao, R., Chen, H., Hao, J.-Y., Chen, Y., Li, D., Zhang, J., & Xiao, Z. (2022) · arXiv.org · doi:10.48550/arxiv.2212.14538 · Cited by 10
“…Transformer (Vaswani et al., 2017; Kenton & Toutanova, 2019; Brown et al., 2020) and Vision Transformer (Dosovitskiy et al., 2020; Liu et al., 2021; Han et al., 2021) are the most outstanding breakthroughs in natural language processing (NLP) and computer vision (CV), and they have demonstrated…”
BackgroundMethodInfluentialLi, W., Cao, D., & Xiang, M. (2022) · Remote Sensing · doi:10.3390/rs14184544 · Cited by 10
Alrfou, K., Kordijazi, A., & Zhao, T. (2022) · arXiv.org · doi:10.48550/arxiv.2208.04149 · Cited by 10
“Recent research have focused on improving the transformer models to capture local information and resulted in new variations of the transformer-based models such as Transformer in Transformer (TNT) [44], Twins [45], Swin-Transformer [46], and Shuffle Transformer [47].”
MethodZhang, Z., Miao, C., Liu, C., Tian, Q., & Zhou, Y. (2022) · Mathematics · doi:10.3390/math10111915 · Cited by 10
“TNT [30] enhances local attention by more fine-grained patch division.”
BackgroundGroup-in-Group Relation-Based Transformer for 3D Point Cloud Learning
Liu, S., Fu, K., Wang, M.-N., & Song, Z.-J. (2022) · Remote Sensing · doi:10.3390/rs14071563 · Cited by 10
HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems
Li, Y., Gu, R., Huan, C., Wang, Z., Yao, R., Tian, C., & Chen, G. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3749168 · Cited by 17
“These models are predominantly based on the Transformer [20, 55] architecture, which has established itself as the de facto standard for LLMs such as GPT [44] and LLaMA [53, 54] due to its scalability and its ability to effectively model complex sequence dependencies.”
ShiftAddNAS: Hardware-inspired search for more accurate and efficient neural networks
You, H., Li, B., Shi, H., Fu, Y., & Lin, Y. C. (2022) · arXiv · doi:10.48550/arxiv.2205.08119 · Cited by 9
“Its translation invariant and weight sharing ability leads to various SOTA CNNs (He et al., 2016) or hybrid CNN-Transformer models (Xiao et al., 2021).”
Tracing unknown tumor origins with a biological-pathway-based transformer model
Xie, J., Chen, Y., Luo, S., Yang, W., Lin, Y., Wang, L., Ding, X., Tong, M., & Yu, R. (2024) · Cell Reports Methods · doi:10.1016/j.crmeth.2024.100797 · Cited by 9
“In recent years, transformer architectures 20 have surfaced as powerful tools across an array of artificial intelligence tasks such as large language models, 21,22 image classification, 23 and bioinformatics for protein structure prediction.”
Sinno, A., Baghdadi, R., Narch, R., Rayes, S. E., Tokajian, S. T., & Khoury, C. A. (2025) · Journal of Virology · doi:10.1128/jvi.01554-25 · Cited by 8
“A representative example is LucaProt, which has demonstrated the capacity to recover highly divergent RNA viruses from complex environmental metagenomes, revealing lineages inaccessible to conventional work flows (12, 88).”
Usman, M. T., Khan, H., Khan, H. A., Rida, I., Zhu, X., & Koo, J. (2025) · Image and Vision Computing · doi:10.1016/j.imavis.2025.105777 · Cited by 9
Uncertainty Estimation with Self-Distillation for semi-supervised few-shot classification
Chen, J., Li, P., & Gu, R. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113948 · Cited by 7
Wang, Y., Xu, C., Na, W., Liu, D., Yan, J., Yao, S., & Wu, Q. (2025) · IEEE Sensors Journal · doi:10.1109/jsen.2025.3587271 · Cited by 10
“…established transformer models as baselines to evaluate the performance of our model, including vision transformer (ViT) [52], transformer in trans-former (TNT) [56], time series transformer (TST) [57], shifted windows transformer (SwinT) [58], and patch time series transformer (PatchTST) [59].”
Self-citationCross-Domain Continual Learning for Edge Intelligence in Wireless ISAC Networks
Hu, J., Li, X., Su, Z., & Luo, J. (2025) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2025.3588914 · Cited by 9
“The complexity of an FC layer is proportional to the product of its input and output dimensions, while an MHSA layer processing a sequence of length N and element dimension L has a complexity of O ( N 2 L + NL 2 ) [54].”
A survey of sentiment analysis methods based on graph neural network
Rad, R. A., Yamaghani, M. R., & Nourbakhsh, A. (2025) · International Journal of Data Science and Analytics · doi:10.1007/s41060-025-00714-1 · Cited by 11
“Most researchers have used it as an alternative to the LSTM and Bi-LSTM models [149].”
Enhancing facial age estimation with local and global multi-attention mechanisms
Liu, X.-L., Qiu, M.-Y., Zhang, Z.-Q., Shi, Y., Li, Z., Chen, X., Liu, Y.-L., Chen, X.-R., & Yu, H.-M. (2025) · Pattern Recognition Letters · doi:10.1016/j.patrec.2025.01.005 · Cited by 9
Ren, W.-J., Li, B., Peng, H., & Wang, J. (2024) · Signal, Image and Video Processing · doi:10.1007/s11760-024-03731-y · Cited by 9
“Dosovitskiy et al. [12] introduced Transformer [13], originally designed for tasks in natural language processing, into image segmentation and achieved great success.”
Heterogeneous Dual-Attentional Network for WiFi and Video-Fused Multi-Modal Crowd Counting
Hao, L., Huang, B., Jia, B., & Mao, G. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3444469 · Cited by 11
“Second, we divide the feature plane in each channel into S 2D patches, and then re-assemble [51] them into embedding vectors Q F , K F and F F with the dimension of S × ˆ C , where ˆ C = HW S C j ( j ∈ [ Q, K, V ]) , such that each .”
MethodDistribution-agnostic landslide hazard modelling via Graph Transformers
Belvederesi, G., Tanyaș, H., Lipani, A., Dahal, A., & Lombardo, L. (2024) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2024.106231 · Cited by 9
Chen, H., Tan, W.-Q., Guo, J.-J., & Yang, F.-R. (2024) · Physical Communication · doi:10.1016/j.phycom.2024.102516 · Cited by 9
Self-citationTerzi, D. S. (2024) · Plant Pathology · doi:10.1111/ppa.13997 · Cited by 9
Liu, Z.-Y., Zhang, W.-R., Pan, M.-Y., Loo, C., & Pasupa, K. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.112055 · Cited by 9
SAKD: Sparse attention knowledge distillation
Guo, Z., Zhang, P.-Z., & Liang, P. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.105020 · Cited by 9
Segmentation of ethnic clothing patterns with fusion of multiple attention mechanisms
Ning, T., Gao, Y., & Han, Y. (2024) · Complex & Intelligent Systems · doi:10.1007/s40747-024-01457-5 · Cited by 9
Joint Caching, Communication, Computation Resource Management in Mobile-Edge Computing Networks
Han, M., Sun, X.-H., Wang, X.-J., Zhan, W., & Chen, X. (2024) · IEEE Wireless Communications and Networking Conference · doi:10.1109/wcnc57260.2024.10571326 · Cited by 9
“To address it, we propose the Transformer-based Actor Critic (TAC) by combing the transformer and the actor-critic architecture [7].”
MethodSlowFastFormer for 3D human pose estimation
Zhou, L., Chen, Y., & Wang, J. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.103992 · Cited by 13
MetaSwin: a unified meta vision transformer model for medical image segmentation
Lee, S., & Lee, M. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.1762 · Cited by 9
“Recently, attention mechanisms have gained significant attention in the deep learning community, and many efforts have been made to design attention-based token mixer components ( D’Ascoli et al., 2021 ; Han et al., 2021 ).”
BackgroundImage Transformers for Diabetic Retinopathy Detection from Fundus Datasets
Kumar, N. S., Balasubramanian, R., & Phirke, M. (2023) · Revue d'Intelligence Artificielle · doi:10.18280/ria.370626 · Cited by 9
“Hence, architectures like Vision Outlooker (VOLO) [35] and Transformer in Transformer (TNT) [36] have been proposed, where the patches are divided into smaller patches and these sub-patches are streamed to a stack of sub-transformers for further processing.”
MethodSequential architecture-agnostic black-box attack design and analysis
Mumcu, F., & Yilmaz, Y. (2023) · Pattern Recognition · doi:10.2139/ssrn.4339656 · Cited by 9
Computer vision-enriched discrete choice models, with an application to residential location choicePreprint
van Cranenburgh, S., & Garrido-Valenzuela, F. (2023) · Transportation Research Part A: Policy and Practice · doi:10.48550/arxiv.2308.08276 · Cited by 9
“One possible way to circumvent this technical barrier is using more lightweight feature extractors, such as TNT-Ti (Han et al. 2021), DeiT-Tiny (Touvron et al. 2021), or Mixer-S/16 (Tolstikhin et al. 2021).”
ViLP: Knowledge Exploration using Vision, Language, and Pose Embeddings for Video Action Recognition
Chaudhuri, S., & Bhattacharya, S. (2023) · Indian Conference on Computer Vision, Graphics and Image Processing · doi:10.1145/3627631.3627637 · Cited by 9
“Our approach out-performs regular video recognition [4, 38, 43] by a substantial margin by employing only base Vision Transformers (ViTs) as backbone.”
BackgroundA bilateral employment situation prediction model for college students using GCN and LSTM
Shen, J. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1494 · Cited by 9
“For college students’ comprehensive quality classification model, I select some excellent classification models: ResNet (Wu, Shen & Van Den Hengel, 2019), GRU (Dey & Salem, 2017), transformer (Han et al., 2021), Decision Tree (Song & Ying, 2015), XGBoost (Chen et al.”
MethodTransformer for multiple object tracking: Exploring locality to visionPreprint
Wu, S., Hadachi, A., Lu, C.-R., & Vivet, D. (2023) · Pattern Recognition Letters · doi:10.2139/ssrn.4143319 · Cited by 9
DBT-UNETR: Double Branch Transformer with Cross Fusion for 3D Medical Image Segmentation
Tao, H., Mao, K.-M., & Zhao, Y.-H. (2022) · IEEE International Conference on Bioinformatics and Biomedicine · doi:10.1109/bibm55620.2022.9995409 · Cited by 9
“…and global information representation, Transformer-based models have been adopted for many computer vision tasks, i.e, image classification [6]- [9], image segmentation [10]- [12], object detection [13]- [15], etc. Inspired by attention mechanisms, there is a series of models like TransUNet…”
MethodMonocular BEV Perception of Road Scenes via Front-to-Top View Projection
Liu, W.-X., Li, Q., Yang, W.-X., Cai, J.-X., Yu, Y.-L., Ma, Y.-X., He, S., & Pan, J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3377812 · Cited by 9
“TNT [67] jointly leverages the inner transformer block and outer transformer block to enhance information exchange.”
MethodLearning modality-fused representation based on transformer for emotion analysis
Shi, P., Hu, M. Q., Ren, F., Shi, X., & Xu, L. (2022) · Journal of Electronic Imaging · doi:10.1117/1.jei.31.6.063032 · Cited by 7
“Inspired by the channel-wise attention for TE, 48 we embed SEnet 47 in different locations of the multilayer perceptron (MLP) block to enhance the feature extraction ability.”
MethodTFormer: 3D Tooth Segmentation in Mesh Scans with Geometry Guided Transformer
Xiong, H., Li, K., Tan, K.-Q., Feng, Y., Zhou, J. T., Hao, J., & Liu, Z.-Q. (2022) · arXiv.org · doi:10.48550/arxiv.2210.16627 · Cited by 9
“…by the success of Transformers in various computer vision and natural language processing tasksVaswani et al. (2017); Dosovit-skiy et al. (2021); Han et al. (2021); Liu et al. (2021); Guo et al. (2021); Zhao et al. (2021), we propose a novel 3D Transformer-based framework, named TFormer, for…”
BackgroundRegularizing self-attention on vision transformers with 2D spatial distance loss
Mormille, L. H., Broni-Bediako, C., & Atsumi, M. (2022) · Artificial Life and Robotics · doi:10.1007/s10015-022-00774-7 · Cited by 8
“As is well known, the most common approach is to combine self-attention layers with convolution layer [7, 11–13], with a variety of architectures proposed in the literature [14–17].”
BackgroundPointDet++: an object detection framework based on human local features with transformer encoder
Tang, Y., Wang, B., He, W., & Qian, F. (2022) · Neural computing & applications (Print) · doi:10.1007/s00521-022-06938-7 · Cited by 9
“The level design of swintransformer [10] and Transformer In Transformer [7] structure has become the current mainstream feature extraction backbone.”
BackgroundMethodSWAT: Spatial Structure Within and Among Tokens
Kahatapitiya, K., & Ryoo, M. (2021) · International Joint Conference on Artificial Intelligence · doi:10.48550/arxiv.2111.13677 · Cited by 9
“Transformer-inTransformer [31] and Nested Transformers [86] both have a separate token structure within tokens.”
BackgroundToward Unified Fine-Grained Vehicle Classification and Automatic License Plate Recognition
Lima, G. E., Nascimento, V., Santos, E., Nascimento, E., Jr., Laroca, R., & Menotti, D. (2026) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2026.5899 · Cited by 8
“[2024] developed a method for satellite imagery, combining DenseNet [Huang et al ., 2017] and Transformer-in-Transformer [Han et al ., 2021] layers to extract fine-grained spatial features.”
Shan, D., Yang, M., Huang, L., Panduo, D., & Qu, B. (2025) · Medical Physics (Lancaster) · doi:10.1002/mp.70003 · Cited by 8
Local feature enhancement transformer for image super-resolution
Weijie, H., & Detian, H. (2025) · Scientific Reports · doi:10.1038/s41598-025-07650-x · Cited by 8
“Recent studies have focused on improving the modeling of local information 32–34 .”
A Review Toward Deep Learning for High Dynamic Range Reconstruction
Martins, G. D. L., Lopez-Cabrejos, J., Martins, J., Leher, Q. O., Ferreti, G. D. S., Carvalho, L. H. C., Lima, F. B., Paixão, T., & Alvarez, A. B. (2025) · Applied Sciences · doi:10.3390/app15105339 · Cited by 10
Automated analysis system for micro-defects in 3D printed concrete
Zhao, H., Wang, X., Sun, J., Wu, F., Liu, X., Chen, Z., & Wang, Y. (2025) · Automation in Construction · doi:10.1016/j.autcon.2025.106105 · Cited by 8
Self-citationMultivariate Time-Series Missing Data Imputation with Convolutional Transformer Model
Wang, Y.-X., Ding, H., & Li, H.-D. (2025) · Symmetry · doi:10.3390/sym17050686 · Cited by 8
Self-citationCorn pose estimation using 3D object detection and stereo images
Gao, Y.-L., Li, Z., Hong, Q.-Q., Li, B., & Zhang, L.-F. (2025) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2025.110016 · Cited by 8
CPAT: cross-patch aggregated transformer for time series forecasting
Liu, B., Wu, L., Wang, X., Huang, J., & Zhang, G. (2025) · Machine Learning · doi:10.1007/s10994-025-06758-5 · Cited by 8
“…et al., 2019), computer vision (CV) (Dosovitskiy et al., 2020; Carion et al., 2020; Wang et al., 2021; Han et al., 2022; Touvron et al., 2021; Han et al., 2021), and speech processing tasks (Dong et al., 2018; Povey et al., 2018), Transformer-based architectures have been actively explored…”
A Multi-Modal Knowledge-Enhanced Framework for Vessel Trajectory Prediction
Yu, H., Li, T., Torp, K., & Jensen, C. S. (2025) · International Symposium on Spatial and Temporal Data (SSTD) · doi:10.1145/3748777.3748784 · Cited by 7
“• Transformer [12] is utilized widely across various domains due to its powerful attention architecture and ability to capture complex patterns in sequential data.”
Mixed-Granularity Implicit Representation for Continuous Hyperspectral Compressive Reconstruction
Li, J.-N., Chen, H., Zhao, W., Chen, R., & Xu, T.-F. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3551891 · Cited by 8
“Transformers, celebrated for their ability to handle long-range spatial interactions, have seen widespread use across various disciplines [20].”
EDA: Energy-Efficient Inter-Layer Model Compilation for Edge DNN Inference Acceleration
Pao, B.-R., Chen, I.-C., Chang, E., & Yeh, T.-T. (2025) · International Symposium on High-Performance Computer Architecture · doi:10.1109/hpca61900.2025.00050 · Cited by 8
“Natural language applications often use transformer models [22], [27] to generate texts and translate languages.”
Yan, Z., Yue, L., Luo, W., & Sun, J. (2024) · Alexandria Engineering Journal · doi:10.1016/j.aej.2024.11.003 · Cited by 10
Toward Large-Scale Non-Motorized Vehicle Helmet Wearing Detection: A New Benchmark and Beyond
Jing, W., Zhu, Z., Chen, H., Wang, H., & Shao, F. (2025) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2025.3527678 · Cited by 11
“Restrictions apply. regions and channels, and integrates the Transformer for capturing global features with self-attention [46], [47].”
I-DINO: High-Quality Object Detection for Indoor Scenes
Fan, Z., Mei, W., Liu, W., Chen, M., & Qiu, Z. (2024) · Electronics · doi:10.3390/electronics13224419 · Cited by 8
SW-YOLOX: A YOLOX-based real-time pedestrian detector with shift window-mixed attention mechanism
Tsai, C., Wang, R.-Y., & Chiu, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128357 · Cited by 7
Wang, T.-T., Hu, Y.-F., Fang, Q., He, B., Gong, X., & Wang, P. (2024) · IEEE transactions on intelligent transportation systems (Print) · doi:10.1109/tits.2024.3436911 · Cited by 8
“Various vision trans-former (ViT) [17] variants, such as DeepViT [18], TNT [19], Sinkhorn Transformer [20], and Nystromformer [21], have been explored, each offering different advantages in terms of attention mechanisms and computational efficiency.”
BackgroundAn Overview of Methods of Industrial Anomaly Detection
Yang, S., Liu, S., Shang, P., & Wang, H. (2024) · International Conference on Robotics, Control and Automation Engineering (RCAE) · doi:10.1109/rcae62637.2024.10834253 · Cited by 10
“Transformer[8] exhibits greater capability in capturing comprehensive global information, thereby possessing the potential to outperform AE and establish itself as a novel framework for anomaly detection in reconstruction networks.”
Cherakhloo, M., Ranjbar, M., Mohhebbi, M., & Vahdat, B. V. (2024) · National and International Iranian Conference on Biomedical Engineering (ICBME) · doi:10.1109/icbme64381.2024.10895619 · Cited by 4
“In recent years, ViTs [5], have emerged as a powerful alternative to traditional Convolutional Neural Networks (CNNs) in computer vision tasks such as image classification [6][7], object detection [8][9], and image segmentation [10][11].”
Zheng, Z., Ge, Z., Tian, Z., Yang, X., & Zhou, Y. (2024) · Journal of Real-Time Image Processing · doi:10.1007/s11554-024-01521-w · Cited by 8
“Finally, The integration of local and global information is achieved through both structural design and attention mechanisms [29, 30].”
Fine-grained personalized federated learning via transformer in the transformer framework
Li, Y., Ge, L., & Jiang, M. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112276 · Cited by 11
Augmented Self-Mask Attention Transformer for Naturalistic Driving Action Recognition
Zhang, T., Wang, Q., Dong, X., Yu, W., Sun, H. S., Zhou, X., Zhen, A., Cui, S., Wu, D., & He, Z. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00705 · Cited by 8
“Recently, transformer has shown remarkable performance in TAL [1, 10, 14, 26], which replaces global self-attention with local self-attention to decrease computational complexity.”
BackgroundA dual data stream hybrid neural network for classifying pathological images of lung adenocarcinoma
Li, L., Mei, Z., Li, Y., Yu, Y., & Liu, M. (2024) · Comput. Biol. Medicine · doi:10.1016/j.compbiomed.2024.108519 · Cited by 8
An effective graph embedded YOLOv5 model for forest fire detection
Yuan, H., Lu, Z.-M., Zhang, R.-Z., Li, J.-S., Wang, S., & Fan, J. (2024) · International Conference on Climate Informatics · doi:10.1111/coin.12640 · Cited by 8
“37,38 Transformer-based object detection algorithms can be divided into two categories according to network structure: Transformer neck, a set prediction method based on Transformer, and Transformer backbone, a network structure based on CNN extraction features and Transformer implementation…”
MethodTransformers: Statistical interpretation, architectures and applications
Meng, F., & Wang, Y. (2024) · Applied and Computational Engineering · doi:10.54254/2755-2721/43/20230832 · Cited by 5
“To be extended, Facebook AI team [5], [6], [7], , [8], , [9], [10] illustrates that some heads are less useful in model performance.”
BackgroundSelf-citationExpediting In-Network Federated Learning by Voting-Based Consensus Model Compression
Su, X., Zhou, Y.-P., Cui, L.-Z., & Guo, S. (2024) · IEEE Conference on Computer Communications · doi:10.1109/infocom52122.2024.10621376 · Cited by 8
“With the wide deployment of smart devices, an enormous amount of data can be collected for training advanced large models to empower intelligent services such as face recognition and natural language processing [1], which however gives rise to privacy leakage concerns.”
BackgroundImproving Robustness for Vision Transformer with a Simple Dynamic Scanning Augmentation
Kotyan, S., & Vargas, D. V. (2023) · Neurocomputing · doi:10.2139/ssrn.4374280 · Cited by 8
“One example is Transformer-in-Transformer (TNT) Han et al. (2021) architecture, in which an extra transformer module is used to process pixel-level embeddings.”
Yang, L., Fu, Y.-X., Lu, S.-N., Sun, Z.-H., Mei, J.-L., Zhao, W.-X., & Hu, Y. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.09.012 · Cited by 8
Low-Dimensional Text Representations for Sentiment Analysis NLP Tasks
Akritidis, L., & Bozanis, P. (2023) · SN Computer Science · doi:10.1007/s42979-023-01913-y · Cited by 8
“In accordance to its name, BERT trains a Transformer on a bidirectional fashion [14, 49].”
BackgroundX-MLP: A Patch Embedding-Free MLP Architecture for Vision
Wang, X., Cai, Z., & Peng, C.-L. (2023) · IEEE International Joint Conference on Neural Network · doi:10.1109/ijcnn54540.2023.10191954 · Cited by 8
LCANet: A Lightweight Context-Aware Network for Bladder Tumor Segmentation in MRI Images
Wang, Y.-X., Li, X., & Ye, X. (2023) · Mathematics · doi:10.3390/math11102357 · Cited by 8
Self-citationResearch on Automatic Classification and Detection of Mutton Multi-Parts Based on Swin-Transformer
Zhao, S., Bai, Z.-C., Wang, S.-C., & Gu, Y. (2023) · Foods · doi:10.3390/foods12081642 · Cited by 8
“These networks have been applied to various fields, such as object detection [27–29], semantic segmentation [30–32], image classification [33–35], and image generation [36].”
MethodTowards private stylists via personalized compatibility learning
Mo, D., Zou, X., Pang, K., & Wong, W. K. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.119632 · Cited by 8
Adversarial Transformers for Weakly Supervised Object Localization
Meng, M., Zhang, T., Zhang, Z., Zhang, Y., & Wu, F. H. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3220055 · Cited by 7
“[43], [44], [45], object detection [46], [47], video processing”
BackgroundTowards More Accurate and Complete Iris Segmentation Using Hybrid Transformer U-Net
Sun, Y., Lu, Y., Liu, Y.-N., & Zhu, X.-D. (2022) · 2022 IEEE International Joint Conference on Biometrics (IJCB) · doi:10.1109/ijcb54206.2022.10007944 · Cited by 8
“Although preliminary studies have shown the promising application of image-based Transformer in classification [26, 15, 14], segmentation [41, 32, 49, 23] or detection [7, 50], transformers do not have inductive bias for images which means that they can not perform well on a small-scale dataset [8].”
BackgroundEffective Vision Transformer Training: A Data-Centric Perspective
Zhou, B.-J., Wang, P.-C., Wan, J., Liang, Y.-N., & Wang, F. (2022) · arXiv.org · doi:10.48550/arxiv.2209.15006 · Cited by 8
“Although various architectures [19, 9, 44, 43, 4, 16, 26, 36, 18, 14, 30, 13, 3, 23, 28, 11, 46, 50] have been continuously suggested to improve the ViTs, these models still suffer from suboptimal performance under the limited training data.”
BackgroundHan, M., Zhan, Y., Luo, Y., Du, B., Hu, H., Wen, Y., & Tao, D. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3204684 · Cited by 11
“In addition, there are other forms of attention mechanisms, such as graph attention [55] and self-attention [56], [57].”
BackgroundSIL-Net: A Semi-Isotropic L-shaped network for dermoscopic image segmentation
Zhang, Z., Jiang, Y., Qiao, H., Wang, M., Yan, W., & Chen, J. (2022) · Comput. Biol. Medicine · doi:10.1016/j.compbiomed.2022.106146 · Cited by 8
EINet: camouflaged object detection with pyramid vision transformer
Li, C., & Jiao, G. (2022) · J. Electronic Imaging · doi:10.1117/1.jei.31.5.053002 · Cited by 8
TR-Net: A Transformer-Based Neural Network for Point Cloud Processing
Liu, L., Chen, E.-Q., & Ding, Y. (2022) · Machines · doi:10.3390/machines10070517 · Cited by 8
“In more detail, each layer of TNT is comprised of two successive blocks, an inner block models the pixel-wise interaction within each patch, and an outer block extracts the global information from patch embeddings.”
MethodVision Transformer with Convolutions Architecture SearchPreprint
Zhang, H., Hao, K., Witold Pedrycz, C., Gao, L., Tang, X., & Wei, B. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.10435 · Cited by 7
“Transformer-iN-Transformer [21] utilizes external Transformer blocks to deal with the embedding of the patch and the relationship between the pixels, respectively.”
MethodMontrezol, J., Oliveira, H. S., & Oliveira, H. P. (2026) · Machine Learning with Applications · doi:10.1016/j.mlwa.2026.100844 · Cited by 7
Effah, D., Zia, A., Awrangjeb, M., Gao, Y.-S., & Sarpong, K. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11413-5 · Cited by 7
“Recently, vision transformers paved the way for their use in computer vision tasks (Han et al. 2021).”
TFMR-RAG: a knowledge base for traffic facility management regulations
Gan, L., Yan, C., Wang, T., & Zhang, W. (2025) · Transportmetrica A Transport Science · doi:10.1080/23249935.2025.2581158 · Cited by 7
Gao, X., & Fang, Q. (2025) · Discover Artificial Intelligence · doi:10.1007/s44163-025-00459-7 · Cited by 4
“Han et al. further extended these findings by introducing hierarchical Transformer models that refine contextual embeddings, offering improved performance in multi-level sentiment analysis [10].”
Gao, X., Dong, H., Zhang, L.-J., Wang, Y., Yang, X., & Li, Z. (2025) · Applied Sciences · doi:10.3390/app15168928 · Cited by 7
Self-citationA Symmetric Multi-Scale Convolutional Transformer Network for Plant Disease Image Classification
Xu, C., & Yang, T. (2025) · Symmetry · doi:10.3390/sym17081232 · Cited by 7
Self-citationLiu, J., Song, X., Zhang, D., Thomale, J., He, D., & Hong, L. (2025) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.1269 · Cited by 6
Poster: Towards Privacy-Preserving and Personalized Smart Homes via Tailored Small Language Models
Huang, X.-Y., Shen, L., Ma, Z.-J., & Zheng, Y.-Q. (2025) · ACM/IEEE International Conference on Mobile Computing and Networking · doi:10.1145/3680207.3765677 · Cited by 7
“Recognizing the strong generalizability and language processing capabilities of LLMs [32], [33], [34], [35], researchers are attempting to integrate them with smart homes for enhanced user experiences.”
C4D-YOLOv8: improved YOLOv8 for object detection on drone-captured images
Zeng, W., Wu, P., Wang, J., Hu, G., & Zhao, J. (2025) · Signal Image and Video Processing · doi:10.1007/s11760-025-04457-1 · Cited by 8
“We also add a small target detection layer on top of the existing three detection heads.”
Low-latency vision transformers via large-scale multi-head attention
Gross, R. D., Halevi, T., Koresh, E., Tzach, Y., & Kanter, I. (2025) · Physica A: Statistical Mechanics and its Applications · doi:10.48550/arxiv.2506.23832 · Cited by 7
Infrared Ship Detection in Complex Nearshore Scenes Based on Improved YOLOv5s
Liu, X.-W., Liu, M., & Yin, Y. (2025) · Italian National Conference on Sensors · doi:10.3390/s25133979 · Cited by 7
“In recent years, the emergence of Swin Transformer [34], which follows Vision Transformer (ViT) [35], is another important breakthrough in the field of computer vision.”
A Neuromorphic Transformer Architecture Enabling Hardware-Friendly Edge Computing
Zhou, P., C., R., Chen, Y., Liu, Z. T., Liu, Y., Meng, L., Qiao, G., Liu, Y., Yu, Q., & Hu, S. G. (2025) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2025.3557955 · Cited by 8
“Therefore, its deployment and application typically depend on high-performance hardware platforms like GPUs and TPUs [7], [8], [9], [10].”
Leaner Transformers: More Heads, Less DepthPreprint
Saratchandran, H., Teney, D., & Lucey, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.20802 · Cited by 6
“We apply our strategy to a variety of alternative transformer-based architectures in the 60–90 M parameter range: DeiT [34], XCiT [2], TNT [14], VOLO [42], and DaViT [7], all pretrained on ImageNet-1k.”
FastViT: Real-Time Linear Attention Accelerator for Dense Predictions of Vision Transformer (ViT)
Ran, Z., Ye, Z.-W., Wu, C., Cheung, R. C. C., & Yan, H. (2025) · International Symposium on Circuits and Systems · doi:10.1109/iscas56072.2025.11043624 · Cited by 7
GRU–Transformer Hybrid Model for GNSS/INS Integration in Orchard Environments
Gao, P., Fang, J., He, J.-L., Ma, S., Wen, G., & Li, Z. (2025) · Agriculture · doi:10.3390/agriculture15111135 · Cited by 7
“Since a single head can only attend to local information features, a multi-head attention mechanism is used in the Transformer [32].”
Unveiling drug-induced osteotoxicity: A machine learning approach and webserver
Cui, H., He, Y., Wang, Z., Liu, K., Li, W., & Han, W. (2025) · Journal of Hazardous Materials · doi:10.1016/j.jhazmat.2025.138044 · Cited by 9
XSS Attack Detection Based on Multisource Semantic Feature Fusion
Hu, Z., Zhang, J., & Yang, H.-Y. (2025) · Electronics · doi:10.3390/electronics14061174 · Cited by 7
Huang, J.-W., Wang, A.-M., Sun, G., Li, J.-H., Wang, J.-C., Niyato, D., & Leung, V. C. M. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3631861 · Cited by 7
Zhang, G., Liu, H., Guo, J., & Guo, T. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125314 · Cited by 8
Self-citationJing, J., Hu, J.-F., Zhao, Z.-P., & Liu, Y. (2025) · IET Image Processing · doi:10.1049/ipr2.13320 · Cited by 7
RT-DETRmg: a lightweight real-time detection model for small traffic signs
Wang, Y., Chen, J., Yang, B., Chen, Y., Su, Y., & Liu, R. (2024) · Journal of Supercomputing · doi:10.1007/s11227-024-06800-8 · Cited by 7
“Currently, object detection architectures can be broadly categorized into two main types: one based on convolutional neural networks (CNNs) [11–14] and the other based on transformer [15–18] architectures.”
Self-citation
Metadata
- Authors
- Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., & Wang, Y.
- Affiliations
- Huawei Technologies (China) (CN)
- University of Macau (MO)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Volume
- 34
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2103.00112
- OpenAlex
- W3133696297
- Cited by
- 1,394
- References
- 56
- Citations by year
- 2026: 14 · 2025: 276 · 2024: 457 · 2023: 387 · 2022: 186 · 2021: 74
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (0015/2019/AKP, 62072449, 61632003, 0018/2019/AKP)
- Fundo para o Desenvolvimento das Ciências e da Tecnologia (0018/2019/AKP, 0015/2019/AKP)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- vision transformer, attention mechanism, local patches, visual words, image classification, feature aggregation, computational efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
