The DOI matches its DataCite record, with a small difference in title or year; 56 of the 56 references it cites were found in a registry or an index.
Transformer in Transformer
Open accessAbstract
Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (e.g., 16$\times$16) as "visual sentences" and present to further divide them into smaller patches (e.g., 4$\times$4) as "visual words". The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs. Features of both words and sentences will be aggregated to enhance the representation ability. Experiments on several benchmarks demonstrate the effectiveness of the proposed TNT architecture, e.g., we achieve an 81.5% top-1 accuracy on the ImageNet, which is about 1.7% higher than that of the state-of-the-art visual transformer with similar computational cost. The PyTorch code is available at https://github.com/huawei-noah/CV-Backbones, and the MindSpore code is available at https://gitee.com/mindspore/models/tree/master/research/cv/TNT.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For example, in multi-tenant architectures, system prompts are shared across multiple requests and frequently contain detailed instructions or examples to guide the model [19].”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [19]
In the citing paper
“PromptCache [19] proposes modular reuse beyond prefixes but sacrifices accuracy, with reported performance drops of up to 43% [72], making it unsuitable for tasks requiring high precision.”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [19]
In the citing paper
“related experiments results further demonstrate ViT also has remarkable representation capacity when dealing with images and videos [19].”
From Deepfake Video Detection with Spatiotemporal Dropout Transformer (2022) · cites this paper as [19]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 6×
“Compared to CNNs, TNT can outperform the widely-used ResNet and RegNet.”
BackgroundMethodInfluentialVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3×
“The effect of #heads in standard transformer has been investigated in multiple works [21, 35] and a head width of 64 is recommended for visual tasks [9, 31].”
BackgroundMethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“In the past decade, the mainstream deep neural architectures used in the computer vision (CV) are mainly established on convolutional neural networks (CNNs) [17, 12, 11].”
BackgroundFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472 · cited 1×
“As a general backbone network, TNT can also be applied with multi-scale vision models like Faster RCNN [29].”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472 · cited 7×
“The recent transformerbased models like ViT [9] and DeiT [31] are compared.”
MethodInfluentialImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 3×
“ImageNet ILSVRC 2012 [26] is an image classification benchmark consisting of 1.”
BackgroundInfluentialSqueeze-and-Excitation NetworksPreprint
Hu, J., Shen, L., & Sun, G. (2017) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1709.01507 · Cited by 37,845 · cited 1×
“Inspired by squeeze-and-excitation (SE) network for CNNs [16], we propose to explore channel-wise attention for transformers.”
MethodPyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939 · cited 2×
“We evaluate TNT-S and DeiT-S on Faster RCNN with FPN [21].”
BackgroundMethod
Research works printed but not linked: 5
- [6][6] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML , 2020. -
- [11][11] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. arXiv preprint arXiv:2012.12556 , 2020. - · arXiv
- [23][23] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017. - · arXiv
- [26][26] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR , pages 3498-3505. IEEE, 2012. -
- [37][37] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008. -
Cited by
Sorted by citations
Items 601–700 of 1,009
3D Human Mesh Reconstruction by Learning to Sample Joint Adaptive Tokens for Transformers
Xue, Y., Chen, J.-S., Zhang, Y., Yu, C., Ma, H., & Ma, H.-B. (2022) · ACM Multimedia · doi:10.1145/3503161.3548133 · Cited by 17
“To reduce the computational burden, recently proposed transformers [23] [8] [39] usually split the whole image into regularly shaped windows, and apply self-attention calculation”
BackgroundMethodInfluentialSundaram, D. D. M., Kasukurthi, Y., & Jayachandran, S. (2025) · 2025 5th International Conference on Soft Computing for Security Applications (ICSCSA) · doi:10.1109/icscsa66339.2025.11170958 · Cited by 16
“In this work, we investigate three prominent transformer architectures designed for computer vision tasks: the Vision Transformer (ViT) [3], Swin Transformer [4], and Transformer in Transformer (TNT) [5].”
Chen, X., Xu, H., Yu, S., Hu, W., Zhang, Z., Wang, X., Yuan, Y., Wang, M., Chen, L., Lin, X., et al. (2025) · Biology · doi:10.3390/biology14060651 · Cited by 21
“This eliminates the reliance on prior biological knowledge for feature design and enables the modeling of non-linear, high-dimensional interactions (e.g., capturing cell-type-specific gene co-expression dynamics), which are often intractable for shallow machine learning models [74].”
Applications of Large Models in Medicine
Su, Y., Lu, Z., Liu, J., Pang, K., Dai, H., Liu, S., Jia, Y., Ge, L.-J., & Yang, J.-M. (2025) · AI Med · doi:10.71423/aimed.20250105 · Cited by 16
“Generally, the architectures of these foundation models are based on Transformer[42].”
Liu, Q., Li, Z., Zhang, L., & Deng, J. (2025) · Sensors · doi:10.3390/s25020438 · Cited by 17
UAV-based person re-identification: A survey of UAV datasets, approaches, and challenges
Albaluchi, Y., Fu, B., Damer, N., Ramachandra, R., & Raja, K. B. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104261 · Cited by 16
Wang, B., Hu, X., Ge, R., Xu, C., Zhang, J., Gao, Z., Shu, Z., & Polat, K. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102932 · Cited by 15
Self-citationAdaptive Masked Autoencoder Transformer for image classification
Chen, X., Liu, C., Hu, P., Lin, J., Gong, Y., Chen, Y., Peng, D., & Geng, X. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.111958 · Cited by 14
Attention-Based Multi-Kernelized and Boundary-Aware Network for image semantic segmentation
Zhou, X., Wu, G., Sun, X., Hu, P., & Liu, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127988 · Cited by 20
Multi-View Attentive Contextualization for Multi-View 3D Object Detection
Liu, X., Zheng, C., Qian, M., Xue, N., Chen, C., Zhang, Z.-B., Li, C., & Wu, T.-F. (2024) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52733.2024.01579 · Cited by 16
“Concurrently, models like TNT [13], ViL [63], PVTv2-linear [55], POTTER [66], and PaCa [12] explore the integration of local and global contexts.”
MethodOnce for Both: Single Stage of Importance and Sparsity Search for Vision Transformer Compression
Ye, H., Yu, C., Ye, P., Xia, R.-Q., Tang, Y.-S., Lu, J.-W., Chen, T., & Zhang, B. (2024) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52733.2024.00533 · Cited by 16
Bai, J., Li, B., Wang, X., Wang, H., & Guo, Y. (2024) · Journal of Bionic Engineering · doi:10.1007/s42235-023-00472-5 · Cited by 13
“For the RL policy networks in this study, three strategy networks were designed: MLP strategy [39], LSTM strategy [40], and Transformer strategy [41].”
Attention-guided and fine-grained feature extraction from face images for gaze estimation
Wu, C., Hu, H., Lin, K., Wang, Q., Liu, T., & Chen, G. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106994 · Cited by 14
FlameTransNet: Advancing Forest Flame Segmentation with Fusion and Augmentation Techniques
Chen, B., Bai, D., Lin, H., & Jiao, W. (2023) · Forests · doi:10.3390/f14091887 · Cited by 16
An Efficient FPGA-Based Accelerator for Swin Transformer
Liu, Z., Yin, P., & Ren, Z. (2023) · arXiv.org · doi:10.48550/arxiv.2308.13922 · Cited by 16
“Some models based on the Transformer architecture, such as VIT[2], TNT[3], and DeiT[4], segment images into a sequence of image patches and process them using self-attention mechanisms.”
MethodRolling Bearing Fault Diagnosis Based on SVD-GST Combined with Vision Transformer
Xie, F.-Y., Wang, G., Zhu, H., Sun, E., Fan, Q., & Wang, Y. (2023) · Electronics · doi:10.3390/electronics12163515 · Cited by 16
Self-citationCross-Modal Transformer for RGB-D semantic segmentation of production workshop objects
Ru, Q., Chen, G., Zuo, T., & Liao, X. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109862 · Cited by 16
DUFormer: Solving Power Line Detection Task in Aerial Images Using Semantic Segmentation
An, D., Zhang, Q., Chao, J., Li, T., Qiao, F., Bian, Z.-P., & Deng, Y. (2023) · Chinese Conference on Pattern Recognition and Computer Vision · doi:10.1007/978-981-99-8543-2_5 · Cited by 16
“TNT [11] further divides patches into multiple subpatches and introduces a new structure, Transformer-iN-Transformer, which uses internal Transformer blocks to model the relationship between patches and external Transformer blocks to achieve patch-level information exchange.”
MethodA Close Look at Spatial Modeling: From Attention to Convolution
Ma, X., Wang, H., Qin, C., Li, K., Zhao, X., Fu, J., & Fu, Y. (2022) · arXiv.org · doi:10.48550/arxiv.2212.12552 · Cited by 16
Full Contextual Attention for Multi-resolution Transformers in Semantic Segmentation
Themyr, L., Rambour, C., Thome, N., Collins, T., & Hostettler, A. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00324 · Cited by 13
A novel hybrid transformer-CNN architecture for environmental microorganism classification
Shao, R., Bi, X.-M., & Chen, Z. (2022) · PLoS ONE · doi:10.1371/journal.pone.0277557 · Cited by 16
“Subsequently, many models have been proposed in order to obtain better results in image classification tasks [34, 53, 54].”
BackgroundStARformer: Transformer With State-Action-Reward Representations for Robot Learning
Shang, J., Li, X., Kahatapitiya, K., Lee, Y., & Ryoo, M. S. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3204708 · Cited by 18
Exploring Adversarial Robustness of Vision Transformers in the Spectral Perspective
Kim, G., Kim, J., & Lee, J. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00393 · Cited by 18
“Other variants include token-to-token ViT [15], pyramid ViT [16], Trans-former in Transformer [17], cross-covariance image Trans-former [18], etc.”
BackgroundHaloAE: An HaloNet based Local Transformer Auto-Encoder for Anomaly Detection and Localization
Mathian, É., Liu, H.-D., Fernandez-Cuesta, L., Samaras, D., Foll, M., & Chen, L.-M. (2022) · VISIGRAPP · doi:10.5220/0011865900003417 · Cited by 16
“This study therefore implies that Transformer-based vision models could be improved by simultaneously applying the self-attention operation between and within patches [37] [33] [23].”
MethodLearning Multiscale Transformer Models for Sequence Generation
Li, B., Zheng, T., Jing, Y., Jiao, C.-B., Xiao, T., & Zhu, J.-B. (2022) · International Conference on Machine Learning · doi:10.48550/arxiv.2206.09337 · Cited by 16
“The details are as follows: TNT TNT is an improved variant of Vision Transformer (Dosovitskiy et al., 2021).”
BackgroundResultInfluentialImage transformers for classifying acute lymphoblastic leukemia
Cho, P., Dash, S., Tsaris, A., & Yoon, H.-J. (2022) · Medical Imaging · doi:10.1117/12.2611496 · Cited by 16
“Since the convolutions are locally sensitive but lack a global understanding of the images, there have been several studies focused on the applicability of the Transformer models to computer vision.(23,24)”
BackgroundMixing and Shifting: Exploiting Global and Local Dependencies in Vision MLPs
Zheng, H., He, P., Chen, W.-Z., & Zhou, M.-Y. (2022) · arXiv.org · doi:10.48550/arxiv.2202.06510 · Cited by 16
“TNT (Han et al., 2021) takes small Transformer blocks to capture local information.”
MethodWhat Makes for Hierarchical Vision Transformer?
Fang, Y. K., Wang, X., Wu, R., & Liu, W. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3282019 · Cited by 20
Towards Secure Code Generation With LLMs: A Study on Common Weakness Enumeration
Zhao, J., Sun, Y., Huang, C., Liu, C., Guan, Y., Zeng, Y., & Liu, Y. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3619281 · Cited by 17
“CodeT5 is a pre-trained encoder-decoder Transformer [46] model, which leverages the code semantics conveyed from the developer-assigned identifiers and supports multiple programming languages.”
Predicting cardiotoxicity in drug development: A deep learning approach
Liu, K., Cui, H., Yu, X., Li, W.-N., & Han, W.-W. (2025) · Journal of Pharmaceutical Analysis · doi:10.1016/j.jpha.2025.101263 · Cited by 15
“The core of the model consists of a module comprising four Transformer encoder layers [50].”
Qi, X.-H., Xu, Z.-Y., & Wang, F.-H. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.112600 · Cited by 15
Adaptive class token knowledge distillation for efficient vision transformer
Kang, M., Son, S., & Kim, D. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112531 · Cited by 15
GhostViT: Expediting Vision Transformers via Cheap Operations
Cao, H., Qu, Z., Chen, G., Li, X., Thiele, L., & Knoll, A. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3326795 · Cited by 15
“…1) using a hierarchical pyramid structure to reduce the spatial resolution of the input (the token sequence length) across the stages [7], [13]; 2) restricting the attention computation in a small local window [9], [14]; 3) reducing the spatial structure of key and value via down-sampling [7], [8].”
MethodData-independent Module-aware Pruning for Hierarchical Vision Transformers
He, Y., & Zhou, J. T. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2404.13648 · Cited by 15
A Spatial-Channel Feature-Enriched Module Based on Multicontext Statistics Attention
Tao, H., & Duan, Q. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3339722 · Cited by 17
Towards transferable adversarial attacks on vision transformers for image classification
Guo, X., Chen, P., Lu, Z.-H., Chai, H.-F., Du, X., & Wu, X. (2024) · Journal of systems architecture · doi:10.1016/j.sysarc.2024.103155 · Cited by 15
Rethinking Position Embedding Methods in the Transformer Architecture
Zhou, X., Ren, Z., Zhou, S., Jiang, Z., Yu, T., & Luo, H. (2024) · Neural Processing Letters · doi:10.1007/s11063-024-11539-7 · Cited by 18
“Furthermore, we observed that the concatenation method decreases the top-1 accuracy of the TnT series [39] by − 0.1%.”
InfluentialPrompt-based for Low-Resource Tibetan Text Classification
An, B. (2023) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3603168 · Cited by 10
Jafari, M., Shoeibi, A., Ghassemi, N., Heras, J., Khosravi, A., Ling, S.-G., Alizadehsani, R., Beheshti, A., Zhang, Y.-D., Wang, S.-H., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2210.14611 · Cited by 15
“Transformer models used in this article include Beit [42], Cait [43], Coat [44], Deit [45], Pit [46], Swin [47], TNT [48], Visformer [49], and ViT [50].”
MethodCOVID-19 diagnosis via chest X-ray image classification based on multiscale class residual attention
Liu, S.-W., Cai, T., Tang, X., Zhang, Y., & Wang, C. (2022) · Comput. Biol. Medicine · doi:10.1016/j.compbiomed.2022.106065 · Cited by 15
“…classification methods, including COVID-Net [54], VGG [102], DenseNet [103], Xception [104], Inception [105], ResNet [92], RegNet [106], Effi-cientNet [107], ViT [100], TNT [108], DeiT [109], Swin [101], VOLO [110], ConvNeXt [111], ConvMixer [112], CSWin [113] and Pool-Former [114].”
MethodLan, G., Xiao, S., Wen, J.-B., Yang, J.-C., Lu, W., Li, B.-H., Meng, Q.-G., & Gao, X.-B. (2026) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2026.3678220 · Cited by 14
“For instance, the Transformer architecture leverages self-attention to identify the most relevant parts of input sequences for target tasks, resulting in breakthroughs in natural language processing and computer vision [35], [36].”
Non-intrusive reduced order modeling of fluid flows via finite element inspired graph neural network
Lu, Y., Xiao, D.-H., Fu, R., Xu, X., & Sun, S. (2026) · Journal of Computational Physics · doi:10.1016/j.jcp.2026.114727 · Cited by 14
PDAViT: Pyramid dual-attention vision transformer
Zhou, X., Jiang, Z.-Y., Ren, Z.-H., Zhang, Y.-C., Yu, T.-Z., Ji, W.-Y., & Liu, Z. (2026) · Neurocomputing · doi:10.1016/j.neucom.2025.131966 · Cited by 14
Antonesi, G., Cioara, T., Anghel, I., Michalakopoulos, V., Sarmas, E., & Toderean, L. (2025) · arXiv.org · doi:10.48550/arxiv.2506.06359 · Cited by 14
Wu, M., Yang, S.-K., Li, D., Liu, L., & Bian, C. (2025) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2025.111380 · Cited by 14
Visual object tracking: Review and challenges
Chen, Z., Peng, C., Liu, S., & Ding, W. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113140 · Cited by 14
Mugisha, S., Kisitu, R., & Tushabe, F. (2025) · arXiv.org · doi:10.48550/arxiv.2504.16128 · Cited by 14
“Their validation is often generic, missing real-world on-device benchmarks critical for farming applications, hence leaving a gap in practical validation for real-time agricultural applications [16], [32], [37], [44].”
InfluentialWang, Z., Fu, S.-S., Fu, S., Li, D.-B., Liu, D.-D., Yao, Y.-X., Yin, H., & Bai, L. (2025) · Scientific Reports · doi:10.1038/s41598-025-90151-8 · Cited by 14
PVT-MA: pyramid vision transformers with multi-attention fusion mechanism for polyp segmentation
Shang, X., Wu, S., Liu, Y.-H., Zhao, Z.-F., & Wang, S. (2024) · Applied intelligence (Boston) · doi:10.1007/s10489-024-06041-5 · Cited by 14
“Transformer in Transformer(TiT) [30] addresses the problem of ViT focusing solely on relations between patches, neglecting intra-patch relationships.”
Han, J., Zhou, Y., Gao, X., & Zhao, Y. (2024) · Remote Sensing · doi:10.3390/rs16193658 · Cited by 14
Enhanced Trajectory Forecasting for Hypersonic Glide Vehicle via Physics-Embedded Neural ODE
Lu, S., & Qian, Y. (2024) · Drones · doi:10.3390/drones8080377 · Cited by 14
Wang, F., Yu, D., Huang, L., Zhang, Y., Chen, Y., & Wang, Z. (2024) · Geo-spatial Information Science · doi:10.1080/10095020.2024.2331552 · Cited by 14
“Escorcia-Gutierrez et al. (2022) put forth an optimal Mask-CNN methodology, dubbed OMRCNN-SHD, which targets small ship detection in autonomous shipping technologies.”
An Empirical Study on Code Review Activity Prediction and Its Impact in Practice
Olewicki, D., Habchi, S., & Adams, B. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660806 · Cited by 11
Huang, C., Jiang, Y., Yang, X., Wei, C., Chen, H., Xiong, W., Lin, H., Wang, X., Tian, T., & Tan, H. (2024) · Translational Vision Science & Technology · doi:10.1167/tvst.13.4.8 · Cited by 16
“…capabilities in capturing spatial relationships compared to CNNs. Transformer networks, including initial and visual transformers based on self-attention mechanisms, 18 have been proposed to grade retinal fundus images with RGB, 19,20 owing to their capacity to capture long-distance dependencies.”
BackgroundDeepFake Video Detection using Vision Transformer
Hussien, S. A., & Mohamed, S. (2024) · International journal of intelligent computing and information sciences/International Journal of Intelligent Computing and Information Sciences · doi:10.21608/ijicis.2024.272675.1324 · Cited by 13
“ViT [22] represents an exciting new direction in the development of deep learning models for image processing, and may have important applications in areas such as computer vision.”
BackgroundA Hybrid-Model-Based CNC Machining Trajectory Error Prediction and Compensation Method
He, W., Zhang, L., Hu, Y., Zhou, Z., Qiao, Y., & Yu, D. M. (2024) · Electronics · doi:10.3390/electronics13061143 · Cited by 17
Wan, C., Fang, J., Li, K., Zhang, Q., Zhang, S., & Yang, W. (2024) · Comput. Biol. Medicine · doi:10.1016/j.compbiomed.2024.108281 · Cited by 14
Yuan, L., Song, J., & Fan, Y. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.1798 · Cited by 14
Predicting consumer choice from raw eye-movement data using the RETINA deep learning architecture
Unger, M., Wedel, M., & Tuzhilin, A. (2023) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-023-00989-7 · Cited by 16
“…in this paper we also present the novel RETINA (Raw Eye Tracking and Image Ncoder Architecture) approach that combines a transformer-based method (Han et al. 2021) with metric learning (Kulis et al. 2013) into one integrated approach that allows us to analyze consumers’ entire raw eye-gaze…”
Residual Swin transformer-based weld crack leakage monitoring of pressure pipeline
Huang, J., Zhang, Z., Qin, R., Yu, Y., Li, Y., Wen, G.-R., Cheng, W., & Chen, X.-F. (2023) · Welding in the World · doi:10.1007/s40194-023-01632-1 · Cited by 14
“As a new neural architecture to replace CNN, transformer architecture can encode input data into powerful nonlinear features based on the attention mechanism [26].”
Chiang, Y.-H., Tseng, B.-Y., Wang, J.-P., Chen, Y.-W., Tung, C.-C., Yu, C.-H., Chen, P.-Y., & Chen, C.-S. (2023) · Journal of Materials Research and Technology · doi:10.1016/j.jmrt.2023.10.200 · Cited by 14
Exploring Driving Behavior for Autonomous Vehicles Based on Gramian Angular Field Vision Transformer
You, J., Chen, Y., Jiang, Z., Liu, Z., Huang, Z., Ding, Y., & Ran, B. (2024) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2024.3445710 · Cited by 15
“In recent years, Transformer models, particularly those based on self-attention mechanisms, have emerged as the foremost choice for Natural Language Processing (NLP) tasks [57], [58], [59], [60] due to their exceptional performance.”
BackgroundMulti-Attention Multi-Image Super-Resolution Transformer (MAST) for Remote Sensing
Li, J., Lv, Q., Zhang, W., Zhu, B., Zhang, G., & Tan, Z. (2023) · Remote Sensing · doi:10.3390/rs15174183 · Cited by 14
Ma, M., Zhang, X., Li, Y., Wang, X., Zhang, R., Wang, Y., Sun, P., Wang, X., & Sun, X. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107313 · Cited by 11
Self-citationLagunas, M., Impata, B., Martinez, V., Fernandez, V., Georgakis, C., Braun, S., & Bertrand, F. (2023) · LatinX in AI at Computer Vision and Pattern Recognition Conference 2023 · doi:10.48550/arxiv.2305.10018 · Cited by 14
“ViTs have been extended with knowledge distillation [29], using token-level and patch-level transformer layers [8], or progressively downsampling the image [37].”
MethodAkkaya, I. B., Kathiresan, S., Arani, E., & Zonooz, B. (2023) · arXiv.org · doi:10.48550/arxiv.2305.08551 · Cited by 14
“Following the introduction of the original ViT model [2], numerous studies have focused on improving classification performance through architectural modifications, knowledge distillation, or advanced data augmentation techniques [11, 15, 8, 16, 17, 3, 18].”
YOLF-ShipPnet: Improved RetinaNet with Pyramid Vision Transformer
Qiu, Z.-R., Rong, S.-Y., & Ye, L.-K. (2023) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-023-00235-4 · Cited by 14
“Han et al. (2021) issued a brand-new vision architecture of a transformer called Transformer iN Trans-former (TNT), which divides the local patch into sub-patch.”
BackgroundMethodLightweight and Optimization Acceleration Methods for Vision Transformer: A Review
Chen, M., Jun, G., & Yu, W. (2022) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc55140.2022.9921989 · Cited by 16
“A major difference from TNT [35], Swin [37] and Twins [43] is that the local self-attention in [38] includes an additional regional token for exchanging global information with the local token.”
BackgroundERNIE-mmLayout: Multi-grained MultiModal Transformer for Document Understanding
Wang, W., Huang, Z.-J., Luo, B., Chen, Q., Peng, Q.-M., Pan, Y.-X., Yin, W., Feng, S., Sun, Y., Yu, D.-H., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2209.08569 · Cited by 14
“TNT [13] divides the input images into several patches as “visual sentence” and then further divides them into sub-patches as “visual words”.”
MethodCui, Y., Liu, F., Liu, X., Li, L., & Qian, X. (2022) · Remote Sensing · doi:10.3390/rs14102451 · Cited by 15
“Han et al. [29] proposed a novel Transformer-iN-Tran-sformer (TNT) model to abstract both patch-level and pixel-level representation.”
An Efficient Intrusion Detection Model Based on Convolutional Neural Network and Transformer
Zhang, Z.-Y., & Wang, L. (2022) · International Conference on Advanced Cloud and Big Data · doi:10.1109/cbd54617.2021.00050 · Cited by 14
Implicit and Explicit Attention for Zero-Shot Learning
Alamri, F., & Dutta, A. (2021) · German Conference on Pattern Recognition · doi:10.48550/arxiv.2110.00860 · Cited by 14
“For example, CaiT [40] introduces deeper transformer networks, Swin Transformer [31] proposes a hierarchical Transformer capturing visual representation by computing self-attention via shifted windows, and TNT [21] applies the Transformer to compute the visual representations using both patch-level and pixel-level inf…”
MethodSeeking an Optimal Approach for Computer-Aided Pulmonary Embolism DetectionPreprint
Islam, N., Gehlot, S., Zhou, Z.-W., Gotway, M., & Liang, J.-M. (2021) · MLMI@MICCAI · doi:10.48550/arxiv.2109.07029 · Cited by 14
DiffNMR: diffusion models for nuclear magnetic resonance spectra elucidation
Yang, Q., Wu, B., Liu, X., Chen, B., Li, W., Long, G., Chen, X., & Xiao, M. (2025) · Materials Futures · doi:10.1088/2752-5724/ae301f · Cited by 12
“Transformers are sequence-to-sequence models adept at capturing long-range dependencies, which makes them well-suited to interpret the complex patterns in spectra and directly output a molecular structure.”
Using large language models for semantic interoperability: A systematic literature review
Abu-Salih, B., Alotaibi, S., Alanazi, A. L., Abukhurma, R., Al-Shboul, B. A., Khoury, A., & Aljaafari, M. A. H. (2025) · ICT Express · doi:10.1016/j.icte.2025.06.011 · Cited by 15
Multi-Modal Brain Tumor Segmentation via 3D Multi-Scale Self-attention and Cross-attention
Huang, Y., Chen, L.-T., & Zhou, C. (2025) · arXiv.org · doi:10.48550/arxiv.2504.09088 · Cited by 13
“Thanks to the long-range dependency modeling ability of transformers, transformer-based frameworks have gained popularity and achieved performance improvement on various computer vision tasks, including classification [12], [13], [19], segmentation [20], [21], and detection [22], [23].”
HGFormer: Topology-Aware Vision Transformer With HyperGraph Learning
Wang, H., Zhang, S., & Leng, B. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3542958 · Cited by 9
“Previous works include local attention [14], convolution enhancement [15], multi-scale resolutions [16] and hierarchical architecture [17].”
Wu, Q., Ji, W., Zhou, G., & Yang, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113281 · Cited by 14
Adversarial Backdoor Defense in CLIP
Kuang, J., Liang, S.-Y., Liang, J.-W., Liu, K., & Cao, X.-C. (2024) · arXiv.org · doi:10.48550/arxiv.2409.15968 · Cited by 13
“And the text encoder is transformer [38].”
CapsLoc3D: Point Cloud Retrieval for Large-Scale Place Recognition Based on 3D Capsule Networks
Zhang, J., Zhang, Y.-Z., Liao, M., Tian, R., Coleman, S. A., & Kerr, D. (2024) · IEEE transactions on intelligent transportation systems (Print) · doi:10.1109/tits.2023.3346953 · Cited by 13
“Several works exploit the Transformer [28], the most representative one being the NDT-Transformer [29], which uses a 3D Normal Distribution Transform (NDT) representation to condense the original point clouds into probability NDT cells, implicitly learning the probability distribution and geometric…”
MethodMobile Transformer Accelerator Exploiting Various Line Sparsity and Tile-Based Dynamic Quantization
Kwon, E., Yoon, J., & Kang, S. (2023) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2023.3347291 · Cited by 16
“Transformers for computer vision are classified according to whether only the transformer is utilized [14], [19], [20] or both CNN and transformer are utilized together [1], [21], [22].”
BackgroundDSTN: Dynamic Spatio-Temporal Network for Early Fault Warning in Chemical Processes
Duan, C., Wu, Z., Zhu, L., Xu, X.-R., Zhu, J.-M., Wei, Z., & Yang, X. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111892 · Cited by 13
Bai, Y., Sun, M., Zhang, L., Wang, Y., Liu, S., Liu, Y., Tan, J., Yang, Y., & Lv, C. (2024) · Applied Sciences · doi:10.3390/app14093829 · Cited by 11
Self-citationLiu, X., Zhao, Y., Wang, S.-G., & Wei, J. (2024) · Applied intelligence (Boston) · doi:10.1007/s10489-024-05496-w · Cited by 13
“For instance, TNT [42] effectively preserved and extracted local information and has undergone extensive verification through large-scale experiments on ImageNet and downstream tasks.”
CPFTransformer: transformer fusion context pyramid medical image segmentation network
Li, J., Ye, J., Zhang, R., Wu, Y., Berhane, G., Deng, H., & Shi, H. (2023) · Frontiers in Neuroscience · doi:10.3389/fnins.2023.1288366 · Cited by 13
“(Han et al., 2021; Liu et al., 2021; Valanarasu et al., 2021; Wang et al., 2021; Xie et al., 2021; Zhang et al., 2021, 2022; Guo et al., 2022) leverage features from Transformers and CNNs to improve segmentation models.”
BackgroundImproved deep learning image classification algorithm based on Swin Transformer V2
Wei, J., Chen, J.-R., Wang, Y., Luo, H., & Li, W. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1665 · Cited by 13
“ViT has surpassed state-of-the-art performance on various image classi fi cation datasets and has dramatically inspired subsequent research ( Chen, Fan & Panda, 2021 ; Han et al., 2021 ).”
BackgroundSelf-citationXu, C., Du, Y., Wang, J., Zheng, W., Li, T., & Yuan, Z.-S. (2023) · Computational Intelligence · doi:10.1111/coin.12607 · Cited by 9
Self-citationKarim, M., Deng, J.-Q., Ayoub, M., Dong, W.-Z., Zhang, B.-Y., Yousaf, M. S., Bhutto, Y. A., & Ishfaque, M. (2023) · Land · doi:10.3390/land12101926 · Cited by 13
Learnable Hierarchical Label Embedding and Grouping for Visual Intention Understanding
Shi, Q., Ye, M., Zhang, Z., & Du, B. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3247876 · Cited by 14
“CvT, PiT and TNT [47], [48], [49], [50], [51], [52] have performed well in image classification.”
MethodXiao, M., Wu, M., Qiao, Z., Fu, Y., Ning, Z., Du, Y., & Zhou, Y. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3671149 · Cited by 8
“We develop a 2-level hierarchical transformer architecture to advance this feature.”
MethodKey information-guided networks for medical image segmentation in medical systems
Liu, C., Cheng, Y.-S., & Tamura, S. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121851 · Cited by 11
Yu, S., Dou, Z., & Wang, S. (2023) · Tsinghua Science & Technology · doi:10.26599/tst.2022.9010044 · Cited by 17
Yang, F., Li, W., Liang, B.-B., & Zhang, J. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3250652 · Cited by 13
“[27] proposed a transformer-iN-transformer (TNT) to explore the pixel-wise interaction between each patch and extract the global information simultaneously.”
BackgroundNetwork Expansion For Practical Training Acceleration
Ding, N., Tang, Y., Han, K., Xu, C., & Wang, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01941 · Cited by 7
“Recently, a new type of architecture named vision transformers (ViTs) have emerged and soon achieved state-of-the-art performances on multiple computer vision tasks [16, 48, 52, 57].”
BackgroundSelf-citationGSB: Group Superposition Binarization for Vision Transformer with Limited Training Samples
Gao, T., Xu, C.-Z., Zhang, L., & Kong, H. (2023) · Neural Networks · doi:10.48550/arxiv.2305.07931 · Cited by 13
“Computational Complexity Analysis Following the TNT [48], we can calculate the FLOPs of a full-precision transformer block by the Eq.”
MethodSelf-citationSEM-POS: Grammatically and Semantically Correct Video Captioning
Nadeem, A., Hilton, A., Dawes, R., Thomas, G., & Mustafa, A. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00259 · Cited by 12
“Transformer for Captioning: Transformer [48, 20] is increasingly being used for object detection [7], classification [14], scene understanding [63] and image captioning [30] tasks.”
MethodPointPatchMix: Point Cloud Mixing with Patch Scoring
Wang, Y., Wang, J., Li, J., Zhao, Z., Chen, G.-Y., Liu, A.-F., & Heng, P.-A. (2023) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2303.06678 · Cited by 13
“Many of the previous works [25, 26, 27, 12] have mostly applied simple data augmentation at the point level, such as random jitter, rotation, and scaling.”
BackgroundSelf-citationToken Selection is a Simple Booster for Vision Transformers
Zhou, D., Hou, Q., Yang, L., Jin, X., & Feng, J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3208922 · Cited by 15
Li, B.-N., Hu, Y.-H., Nie, X.-C., Han, C.-Y., Jiang, X.-J., Guo, T.-D., & Liu, L.-Q. (2022) · doi:10.48550/arxiv.2208.02646 · Cited by 13
“Vision Transformer (ViT) [6] has achieved great success for various vision tasks, e.g. , image recognition [31; 32; 19; 7; 12], object detection [1], human body shape estimation [17], etc. Prior works mainly focus on researches of patch division, architecture design and task extension.”
BackgroundSolving ImageNet: a Unified Scheme for Training any Backbone to Top Results
Ridnik, T., Lawen, H., Ben-Baruch, E., & Noy, A. (2022) · arXiv.org · doi:10.48550/arxiv.2204.03475 · Cited by 13
Metadata
- Authors
- Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., & Wang, Y.
- Affiliations
- Huawei Technologies (China) (CN)
- University of Macau (MO)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Volume
- 34
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2103.00112
- OpenAlex
- W3133696297
- Cited by
- 1,394
- References
- 56
- Citations by year
- 2026: 14 · 2025: 276 · 2024: 457 · 2023: 387 · 2022: 186 · 2021: 74
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (0015/2019/AKP, 62072449, 61632003, 0018/2019/AKP)
- Fundo para o Desenvolvimento das Ciências e da Tecnologia (0018/2019/AKP, 0015/2019/AKP)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- vision transformer, attention mechanism, local patches, visual words, image classification, feature aggregation, computational efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
