The DOI matches its DataCite record, with a small difference in title or year; 56 of the 56 references it cites were found in a registry or an index.
Transformer in Transformer
Open accessAbstract
Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (e.g., 16$\times$16) as "visual sentences" and present to further divide them into smaller patches (e.g., 4$\times$4) as "visual words". The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs. Features of both words and sentences will be aggregated to enhance the representation ability. Experiments on several benchmarks demonstrate the effectiveness of the proposed TNT architecture, e.g., we achieve an 81.5% top-1 accuracy on the ImageNet, which is about 1.7% higher than that of the state-of-the-art visual transformer with similar computational cost. The PyTorch code is available at https://github.com/huawei-noah/CV-Backbones, and the MindSpore code is available at https://gitee.com/mindspore/models/tree/master/research/cv/TNT.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For example, in multi-tenant architectures, system prompts are shared across multiple requests and frequently contain detailed instructions or examples to guide the model [19].”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [19]
In the citing paper
“PromptCache [19] proposes modular reuse beyond prefixes but sacrifices accuracy, with reported performance drops of up to 43% [72], making it unsuitable for tasks requiring high precision.”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [19]
In the citing paper
“related experiments results further demonstrate ViT also has remarkable representation capacity when dealing with images and videos [19].”
From Deepfake Video Detection with Spatiotemporal Dropout Transformer (2022) · cites this paper as [19]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 6×
“Compared to CNNs, TNT can outperform the widely-used ResNet and RegNet.”
BackgroundMethodInfluentialVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3×
“The effect of #heads in standard transformer has been investigated in multiple works [21, 35] and a head width of 64 is recommended for visual tasks [9, 31].”
BackgroundMethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“In the past decade, the mainstream deep neural architectures used in the computer vision (CV) are mainly established on convolutional neural networks (CNNs) [17, 12, 11].”
BackgroundFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472 · cited 1×
“As a general backbone network, TNT can also be applied with multi-scale vision models like Faster RCNN [29].”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472 · cited 7×
“The recent transformerbased models like ViT [9] and DeiT [31] are compared.”
MethodInfluentialImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 3×
“ImageNet ILSVRC 2012 [26] is an image classification benchmark consisting of 1.”
BackgroundInfluentialSqueeze-and-Excitation NetworksPreprint
Hu, J., Shen, L., & Sun, G. (2017) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1709.01507 · Cited by 37,845 · cited 1×
“Inspired by squeeze-and-excitation (SE) network for CNNs [16], we propose to explore channel-wise attention for transformers.”
MethodPyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939 · cited 2×
“We evaluate TNT-S and DeiT-S on Faster RCNN with FPN [21].”
BackgroundMethod
Research works printed but not linked: 5
- [6][6] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML , 2020. -
- [11][11] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. arXiv preprint arXiv:2012.12556 , 2020. - · arXiv
- [23][23] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017. - · arXiv
- [26][26] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR , pages 3498-3505. IEEE, 2012. -
- [37][37] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008. -
Cited by
Sorted by citations
Items 401–500 of 1,009
WeakSAM: Segment Anything Meets Weakly-supervised Instance-level Recognition
Zhu, L., Zhou, J., Liu, Y., Hao, X., Liu, W., & Wang, X. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680960 · Cited by 28
“As for the WSIS, we use SAM-ViT-H to generate pseudo instance labels and train the R-CNN-based and query-based methods, i.e., Mask R-CNN [24] and Mask2former [10], respectively.”
Gao, L., Niu, K., Jiao, B., Wang, P., & Zhang, Y. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3273719 · Cited by 35
“For example, ViT [75] and its variants [76], [77] employ the transformer encoder architecture for image classification and achieve promising performance.”
Limited Data, Unlimited Potential: A Study on ViTs Augmented by Masked Autoencoders
Das, S., Jain, T., Reilly, D., Balaji, P., Karmakar, S., Marjit, S., Li, X., Das, A., & Ryoo, M. S. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00673 · Cited by 23
PreNAS: Preferred One-Shot Learning Towards Efficient Neural Architecture Search
Wang, H., Ge, C., Chen, H., & Sun, X.-Y. (2023) · International Conference on Machine Learning · doi:10.48550/arxiv.2304.14636 · Cited by 31
“…ViT models that were elaborately designed by humans are also included, among which DeiT (Touvron et al., 2021), ConViT (d’Ascoli et al., 2021), TNT (Han et al., 2021) and 2021) are pure ViT architectures, while LVT (Yang et al., 2022) and BoTNet (Srinivas et al., 2021) are hybrid architectures…”
A Global-Information-Constrained Deep Learning Network for Digital Elevation Model Super-Resolution
Han, X., Ma, X., Li, H., & Chen, Z. (2023) · Remote Sensing · doi:10.3390/rs15020305 · Cited by 35
PaCa-ViT: Learning Patch-to-Cluster Attention in Vision Transformers
Grainger, R., Paniagua, T., Song, X., Cuntoor, N. P., Lee, M. W., & Wu, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01781 · Cited by 39
“DeiT-S/16 [43] 22.1 4.6 79.9 T2T-ViT t -14 [58] 22.0 6.1 80.7 PVT-S [48] 24.5 3.8 79.8 TNT-S [20] 23.8 5.2 81.3 SWin-T [32] 29.0 4.5 81.3 CvT-13 [51] 20.0 4.5 81.6 Twins-SVT-S [9] 24.0 2.8 81.3 FocalAtt-Tiny [56] 28.9 4.9 82.2 PVTv2-B2 [47] 25.4 3.9 82.0 PVTv2-B2-li [47] 22 posed PaCa ViT obtains…”
InfluentialDomain Adaptation via Bidirectional Cross-Attention TransformerPreprint
Wang, X., Guo, P., & Zhang, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.05887 · Cited by 22
“Then, many variants of ViT [Touvron et al., 2021; Han et al., 2021; Ranftl et al., 2021; Liu et al., 2021] are proposed to achieve promising performance on computer vision tasks when compared with their CNN counterparts.”
BackgroundResultWang, Z., Peng, D., Shang, Y., & Gao, J. (2021) · Frontiers in Neuroscience · doi:10.3389/fnins.2021.756868 · Cited by 31
“As the basis for powerful architectures, the self-attention mechanism in transformer has displaced CNN and RNN across a variety of tasks (Vaswani et al., 2017; Zhao et al., 2020; Han et al., 2021; Liu et al., 2021; Radford et al., 2021; Touvron et al., 2021; Wang et al., 2021; Yuan et al., 2021).”
BackgroundPixArt-$α$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis
Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wu, Y., Wang, Z., Kwok, J., Luo, P., Lu, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.00426 · Cited by 30
“…also a promising architecture on many computer vision tasks like image classification (Touvron et al., 2021; Zhou et al., 2021; Yuan et al., 2021; Han et al., 2021), object detection (Liu et al., 2021; Wang et al., 2021; 2022; Ge et al., 2023; Carion et al., 2020), semantic segmentation (Zheng…”
BackgroundLi, J., Zhao, F., Zhao, H., Zhou, G., Xu, J., Gao, M., Li, X., Dai, W., Zhou, H., Hu, Y., et al. (2024) · Plant Phenomics · doi:10.34133/plantphenomics.0220 · Cited by 33
VELO: A Vector Database-Assisted Cloud-Edge Collaborative LLM QoS Optimization Framework
Yao, Z., Tang, Z., Lou, J., Shen, P., & Weijia Jia, B. (2024) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws62655.2024.00105 · Cited by 25
“This complexity is determined by the embedding dimension H and the number of patches C , which can be expressed as O ( H 2 C + HC 2 ) [40].”
BackgroundAnsari, S. A., Agrawal, A. P., Wajid, M. A., Wajid, M. S., & Zafar, A. (2024) · Interdisciplinary Sciences Computational Life Sciences · doi:10.1007/s12539-024-00630-1 · Cited by 30
“Prominent vision transformer models include SWIN [1], SLIDE [2], Vit (Vision transformer) [3], DE it (Data Efficient image transformer) [4], alongside hybrid transformers incorporating CNN architecture [5].”
Saranya, T., Deisy, C., & Sridevi, S. (2024) · Comput. Biol. Medicine · doi:10.1016/j.compbiomed.2024.108584 · Cited by 30
MF-Net: Multiple-feature extraction network for breast lesion segmentation in ultrasound images
Wang, J., Liu, G.-Q., Liu, D., & Chang, B. (2024) · Expert systems with applications · doi:10.1016/j.eswa.2024.123798 · Cited by 30
TSegFormer: 3D Tooth Segmentation in Intraoral Scans with Geometry Guided Transformer
Xiong, H., Li, K., Tan, K., Feng, Y., Zhou, J. T., Hao, J., Ying, H., Wu, J., & Liu, Z.-Q. (2023) · International Conference on Medical Image Computing and Computer-Assisted Intervention · doi:10.1007/978-3-031-43987-2_41 · Cited by 30
“Inspired by the success of transformers in various tasks [19,2,5,13,4,28], we propose a novel 3D transformer framework, named TSegFormer, to address the aforementioned challenges.”
MethodMulti-Stage Spatio-Temporal Aggregation Transformer for Video Person Re-Identification
Tang, Z., Zhang, R., Peng, Z., Chen, J., & Lin, L. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3231103 · Cited by 34
“Recently, the Transformer architecture [22], [30], [53], [89] has attracted much attention in the computer vision area because of its excellent context modeling ability.”
BackgroundMulti-granularity siamese transformer-based change detection in remote sensing imagery
Song, L., Xia, M., Yao, X., Weng, L., Hu, K., Lin, H., & Qian, M. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108960 · Cited by 27
Prashanthi, B., Krishna, A. V. P., & Rao, C. M. (2024) · Multimedia tools and applications · doi:10.1007/s11042-024-19866-6 · Cited by 29
“The trade-off for these efficiency gains is a larger model (more network parameters) and slower processing times [27–29].”
An ensemble neural network approach for space–time landslide predictive modelling
Lim, J., Santinelli, G., Dahal, A., Vrieling, A., & Lombardo, L. (2024) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2024.104037 · Cited by 12
“Therefore, alternatives such as Transformer Neural Networks (Han et al., 2021), where an attention mechanism (Vaswani et al., 2017) flexibly recognises important segments of a time series independently of the relative positions, might be more suitable for further studies.”
MethodUnderstanding the architecture of vision transformer and its variants: A review
Neha, F., & Bansal, A. K. (2024) · 2024 1st International Conference on Innovative Engineering Sciences and Technological Research (ICIESTR) · doi:10.1109/iciestr60916.2024.10798341 · Cited by 29
Wang, X., Gao, S., Guo, J., Wang, C., Xiong, L., & Zou, Y. (2024) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-024-00423-w · Cited by 30
“Recently, researchers have shifted focus towards enhancing local information modeling capabilities, as seen in Transformer-in-Transformer (TNT), Swin Transformer, and regionViT [32–34].”
Self-citationPanGu-π: Enhancing Language Model Architectures via Nonlinearity Compensation
Wang, Y.-H., Chen, H., Tang, Y.-H., Guo, T.-Y., Han, K., Nie, Y., Wang, X.-T., Hu, H., Bai, Z., Wang, Y., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2312.17276 · Cited by 29
“Ongoing research focuses on refining local information processing [56], [57], [58], [59], [60], [61], [62], simplifying attention mechanisms [63], [64], [65], and exploring alternative modules [66], [67], [68], [69], [70], [71].”
BackgroundMethodSelf-citationYu, X.-Y., Zhao, Y.-Q., Yin, X., & Lindemann, L. (2023) · at - Automatisierungstechnik · doi:10.48550/arxiv.2312.04242 · Cited by 29
“For instance, we can use recurrent neural networks (RNNs) [59], long short term memory (LSTM) networks [60], or transformers [61].”
Fu, X., Ren, Q., Wu, H., Xiang, F., Luo, Q., Yue, J., Chen, Y., & Zhang, F. (2023) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2023.3315060 · Cited by 30
“Recently, the transformer model has achieved SOTA records on computer vision tasks [4], [5], [6], [7], [8], [9], and it has been demonstrated that variants of the transformer model can also be alternatives to CNNs. Vision transformer (ViT) is proposed to apply the transformer model to image…”
MethodA Full Transformer-based Framework for Automatic Pain Estimation using Videos
Gkikas, S., & Tsiknakis, M. (2023) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc40787.2023.10340872 · Cited by 24
“Such an approach is the Transformer in Transformer (TNT) [13], which enhances the local feature representation by the further division of the patches into sub-patches.”
MethodMeta-Polyp: A Baseline for Efficient Polyp Segmentation
Trinh, Q.-H. (2023) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms58004.2023.00312 · Cited by 39
“In late 2022, a new approach called Meta-former was proposed as a baseline for combining CNN [9] and Transformer models [8].”
BackgroundMethodInfluentialSemantically Structured Image Compression via Irregular Group-Based Decoupling
Feng, R., Gao, Y.-X., Jin, X., Feng, R., & Chen, Z.-B. (2023) · IEEE International Conference on Computer Vision · doi:10.1109/iccv51070.2023.01581 · Cited by 29
Dynamic Temporal Filtering in Video Models
Long, F.-C., Qiu, Z.-F., Pan, Y.-W., Yao, T., Ngo, C., & Mei, T. (2022) · European Conference on Computer Vision · doi:10.1007/978-3-031-19833-5_28 · Cited by 29
“Various variants [14,25,28,43,49,63] validated the power of self-attention for image feature learning.”
BackgroundConv2NeXt: Reconsidering Conv NeXt Network Design for Image Recognition
Feng, J., Tan, H., Li, W., & Xie, M. (2022) · International Conference on Computers and Artificial Intelligence Technology (CAIT) · doi:10.1109/cait56099.2022.10072172 · Cited by 31
“In later work, [22] divided those patches (16×16) into smaller patches (4×4).”
BackgroundMethodInfluentialAuto-scaling Vision Transformers without TrainingPreprint
Chen, W., Huang, W. F., Du, X., Song, X., Wang, Z., & Zhou, D. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.11921 · Cited by 16
“…the visual representation learning, recently works try to train deeper ViTs (Touvron et al., 2021; Zhou et al., 2021), incorporate convolutions (Wu et al., 2021; d’Ascoli et al., 2021; Yuan et al., 2021a), and design multi-scale feature extractions (Chen et al., 2021b; Zhang et al., 2021; Wang…”
BackgroundVisual Parser: Representing Part-whole Hierarchies with TransformersPreprint
Sun, S., Yue, X., Bai, S., & Torr, P. H. S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.05790 · Cited by 30
“3% top-1 accuracy, outperforming TNT-B [23], T2T-ViT-24 [71], PVT-Large [64] by 0.”
TCFormer: Visual Recognition via Token Clustering Transformer
Zeng, W., Jin, S., Xu, L., Liu, W., Qian, C., Ouyang, W., Luo, P., & Wang, X. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3425768 · Cited by 24
Resource-Efficient Generative Mobile Edge Networks in 6G Era: Fundamentals, Framework and Case Study
Lai, B., Wen, J.-B., Kang, J., Du, H.-Y., Nie, J.-T., Yi, C.-Y., Kim, D. I., & Xie, S. (2023) · IEEE wireless communications · doi:10.1109/mwc.007.2300582 · Cited by 28
“The relationship between the amount of computation and the number of layers can be approximated as a linear relationship [15], therefore the amount of computation can be expressed as ( AL n + B ) , where A > 0 and B > 0 are the corresponding parameter.”
BackgroundMaxViT-UNet: Multi-Axis Attention for Medical Image SegmentationPreprint
Khan, A. R., & Khan, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.08396 · Cited by 22
“The CNNs are famous for their location equivariance inductive bias, a property lacking by vanilla self-attention mechanism [23, 59].”
BackgroundRethinking Token-Mixing MLP for MLP-based Vision BackbonePreprint
Yu, T., Li, X., Cai, Y., Sun, M.-M., & Li, P. (2021) · British Machine Vision Conference · doi:10.48550/arxiv.2106.14882 · Cited by 28
“The following works including TNT [13] and T2T [42] enhance the effectiveness of modeling local structure, achieving better performance.”
MethodInfluentialElectric vehicle range prediction considering real-time driving factors and battery capacity index
Ma, X., Li, J.-A., Cui, H.-J., Cheng, L., Ji, Y., & Wang, J. (2025) · Transportation Research Part D: Transport and Environment · doi:10.1016/j.trd.2025.104795 · Cited by 27
A Survey of the Self Supervised Learning Mechanisms for Vision TransformersPreprint
Khan, A., Sohail, A., Fiaz, M., Hassan, M., Afridi, T. H., Marwat, S. U., Munir, F., Ali, S., Naseem, H., Zaheer, M. Z., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.17059 · Cited by 26
“These architectures have obtained state-of-the-art performance in different computer vision based tasks, including image classification, object segmentation and object detection.”
Jiang, H., Li, Z.-Y., Wei, H., Xiong, X., Ruan, J., Lu, J.-M., Mao, H.-Y., & Zhao, R. (2024) · International Joint Conference on Artificial Intelligence · doi:10.48550/arxiv.2404.12090 · Cited by 27
“In computer vision, some methods apply two Transformers to handle tasks like image classification and person search, e.g., TNT [Han et al. , 2021]., ViViT [Arnab et al. , 2021], DualFormer [Liang et al. , 2022] and COAT [Yu et al. , 2022b].”
MethodImproving the Adversarial Transferability of Vision Transformers with Virtual Dense Connection
Zhang, J., Huang, Y., Xu, Z.-E., Wu, W., & Lyu, M. R. (2024) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i7.28541 · Cited by 27
VPCFormer: A transformer-based multi-view finger vein recognition model and a new benchmark
Zhao, P., Song, Y., Wang, S., Xue, J., Zhao, S., Liao, Q., & Yang, W. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.110170 · Cited by 27
“Alternatively, the Trans-former iN Transformer (TNT) [27] combined local and global Transformers for enhanced feature extraction and context modeling.”
Li, D., Wang, X., Zhang, J., & Ji, Z. (2023) · Global Energy Interconnection · doi:10.1016/j.gloei.2023.10.008 · Cited by 31
Deep learning methods applied to digital elevation models: state of the art
Ruíz, J. J. R., López, F. J. A., Reinoso-Gordo, J. F., Ureña-Cámara, M. A., & Quesada, F. J. (2023) · Geocarto International · doi:10.1080/10106049.2023.2252389 · Cited by 29
“…due to the limitation of convolution kernel size the probability of capturing and aggregating low-level features (global information) into high-level information is very low (Vaswani et al. 2017; Han et al. 2021), which explains the loss of global information in traditional networks such as CNNs.”
BackgroundSemisupervised Machine Fault Diagnosis Fusing Unsupervised Graph Contrastive Learning
Yang, C., Liu, J., Zhou, K.-N., & Jiang, X. (2023) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3220847 · Cited by 27
“Multihead attention is the key technique in the transformer, which is powerful in nature language processing [25], [26].”
MethodMLP-based classification of COVID-19 and skin diseases
Zhang, R., Wang, L., Cheng, S., & Song, S. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120389 · Cited by 38
MSTFDN: Multi-scale transformer fusion dehazing network
Yang, Y., Zhang, H.-W., Wu, X., & Liang, X.-Z. (2022) · Applied intelligence (Boston) · doi:10.1007/s10489-022-03674-2 · Cited by 27
“The later proposed TNT model [12] solves the problem of lack of contact within the ViT patches by performing an internal Transformer on all patches.”
MethodBaek, S., Jung, E.-Y., Pyo, J., Pachepsky, Y., Son, H., & Cho, K. (2022) · Water Research · doi:10.1016/j.watres.2022.118494 · Cited by 27
GroupTransNet: Group Transformer Network for RGB-D Salient Object Detection
Fang, X., Jiang, M.-F., Zhu, J.-C., Shao, X.-L., & Wang, H.-P. (2022) · Neurocomputing · doi:10.48550/arxiv.2203.10785 · Cited by 27
“Han et al [26] regards local patches as visual sentences and further divided them into smaller patches as visual words.”
BackgroundYin, H.-C., Bista, P., Ghimire, R., Yang, H., & Carroll, K. (2026) · Vadose Zone Journal · doi:10.1002/vzj2.70077 · Cited by 26
PainFormer: A Vision Foundation Model for Automatic Pain Assessment
Gkikas, S., Rojas, R. F., & Tsiknakis, M. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3605475 · Cited by 26
“1) PainFormer: Vision Transformers (ViT) have been successfully implemented across various image analysis tasks, showcasing the effectiveness of their core self-attention mechanism [60].”
Setting the Record Straight on Transformer Oversmoothing
Dovonon, G., Bronstein, M., & Kusner, M. J. (2024) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2401.04301 · Cited by 26
“Many works around this time found that it was possible to improve vision Trans-formers by replacing self-attention layers with convolutional layers (Han et al., 2021; Liu et al., 2021b; Jiang et al., 2021; Touvron et al., 2021b; Yuan et al., 2021; Park & Kim, 2022).”
BackgroundA Survey on Open-Set Image Recognition
Sun, J., & Dong, Q. (2023) · arXiv.org · doi:10.48550/arxiv.2312.15571 · Cited by 26
Factorization Vision Transformer: Modeling Long-Range Dependency With Local Window Cost
Qin, H., Zhou, D., Xu, T., Bian, Z., & Li, J. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3342172 · Cited by 24
Resizer Swin Transformer-Based Classification Using sMRI for Alzheimer’s Disease
Huang, Y., & Li, W. (2023) · Applied Sciences · doi:10.3390/app13169310 · Cited by 35
“One of the strongest deep-learning models available today is the Transformer, and a key component is the attention mechanism [23].”
BackgroundYang, H., Zhang, Z., Liu, X., & Jing, P. (2023) · Environmental Research · doi:10.1016/j.envres.2023.116821 · Cited by 26
Jackson, I., Sáenz, M. J., Ivanov, D., & Ma, B. (2026) · Journal of the Operational Research Society · doi:10.1080/01605682.2025.2608868 · Cited by 25
“Causal Attention (Sui et al., 2022): Attention ensures y reflects dependencies within the output and the input x (Han et al., 2021).”
Li, H., Xiao, L., Cao, L.-H., Wu, D., Liu, Y., Li, Y., Zhang, Y., & Bao, H.-Y. (2025) · Expert systems with applications · doi:10.1016/j.eswa.2025.129677 · Cited by 25
Liu, J., & Xie, Y. (2025) · Results in Engineering · doi:10.1016/j.rineng.2025.105930 · Cited by 21
DAMPNN: Dynamic Adaptive Message Passing Neural Network for Industrial Soft Sensor
Feng, Y., Yang, C., Kong, L., & Yang, C. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3475419 · Cited by 26
“Motivated by the strong feature extraction ability of Transformer [20], we apply a single self-attention module to learn the relationships between features.”
Enhancing furcation involvement classification on panoramic radiographs with vision transformers
Zhang, X., Guo, E., Liu, X., Zhao, H., Yang, J., Li, W., Wu, W.-L., & Sun, W.-B. (2025) · BMC Oral Health · doi:10.1186/s12903-025-05431-6 · Cited by 25
“Transformer-based architectures, such as the Vision Transformer (ViT), offer an alternative approach by dividing images into patches and processing them sequentially [18].”
Improving learning-based birdsong classification by utilizing combined audio augmentation strategies
Kumar, A. S., Schlosser, T., Kahl, S., & Kowerko, D. (2024) · Ecological Informatics · doi:10.1016/j.ecoinf.2024.102699 · Cited by 21
Prediction model of measurement errors in current transformers based on deep learning.
Li, Z.-H., Cui, J.-X., Lu, H., Zhou, F., Diao, Y.-L., & Li, Z.-X. (2024) · Review of Scientific Instruments · doi:10.1063/5.0190206 · Cited by 25
A Semantic and Motion-Aware Spatiotemporal Transformer Network for Action Detection
Korban, M., Youngs, P. A., & Acton, S. T. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3377192 · Cited by 25
“Currently, the most common method for encoding spatial positions in images is to segment them into patches [8], [28], [29].”
MethodMarine target detection for PPI images based on YOLO-SWFormer
Zhang, Q., Li, Y., Zhang, Z., Yin, S.-B., & Ma, L. (2023) · Alexandria Engineering Journal · doi:10.1016/j.aej.2023.10.014 · Cited by 25
Wang, Z., Dong, N.-Q., & Voiculescu, I. (2022) · International Conference on Information Photonics · doi:10.1109/icip46576.2022.9897482 · Cited by 25
“Variants such as the Swin-Transformer [9] use a shift window and patch merging to detect large variations in the scale of visual entities, with efficient computational cost for high resolution images; DieT [10] introduces a new distillation procedure to train with lower computation resources that ensure a ‘student’ le…”
MethodText Sentiment Analysis Based on Transformer and Augmentation
Gong, X., Ying, W., Zhong, S., & Gong, S. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.906061 · Cited by 35
“Transformer serving as the core architecture of the attention mechanism has been widely used for language models (Han et al., 2021) to capture complex linguistic patterns.”
Make A Long Image Short: Adaptive Token Length for Vision TransformersPreprint
Zhu, Y., Zhu, Y., Du, J., Wang, Y., Ou, Z., Feng, F., & Tang, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.01686 · Cited by 9
“LV-ViT [21], CoaT [45], Swin [25], Twins [9], Visformer [7], ConViT [43], TNT [17], and EfficientNet [36].”
MethodSelf-citationLiu, W., Li, Q., Lin, X., Yang, W., He, S., & Yu, Y. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02045-3 · Cited by 28
Brain Stroke Classification Using CT Scans with Transformer-Based Models and Explainable AI
Qari, S., & Thafar, M. A. (2025) · Diagnostics · doi:10.3390/diagnostics15192486 · Cited by 22
“The Transformer-in-Transformer (TNT) [37] model, specifically the tnt_s_patch16_224 variant from the Timm library, extends ViT by introducing a nested attention structure.”
WCG-VMamba: A multi-modal classification model for corn disease
Wang, H., He, M.-F., Zhu, M., & Liu, G.-H. (2025) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2024.109835 · Cited by 24
Fairness Definitions in Language Models Explained
Yin, Z., Wang, Z., Palikhe, A., & Zhang, W. (2026) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70063 · Cited by 21
“The SEAT test statistic is computed for the two sets of embeddings, which are represented by the [CLS] [63] token of the last layer from BERT [45].”
Photovoltaic Panel Fault Detection and Diagnosis Based on a Targeted Transformer-Style Model
Li, S.-X., Chen, H., Zhang, A., Gong, C.-Y., Menhas, M. I., Liang, W.-B., Wang, Z., & Yang, N. (2024) · IEEE transactions on industry applications · doi:10.1109/tia.2023.3322688 · Cited by 24
“Swin-transformer [20], TNT [21], and ViT [22] are transformer-style models that do not rely on convolutional blocks at all.”
MethodViT-MVT: A Unified Vision Transformer Network for Multiple Vision Tasks
Xie, T., Dai, K., Jiang, Z., Li, R., Mao, S., Wang, K., & Zhao, L. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3342141 · Cited by 26
“After that, numerous researchers explore to employ the transformer into computer visual tasks [2], [22], [23].”
Microbial Colony Detection Based on Deep Learning
Yang, F., Zhong, Y., Yang, H., Wan, Y., Hu, Z., & Peng, S. (2023) · Applied Sciences · doi:10.3390/app131910568 · Cited by 20
What Can Simple Arithmetic Operations Do for Temporal Modeling?
Wu, W., Song, Y., Sun, Z., Wang, J., Xu, C., & Ouyang, W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01261 · Cited by 19
“Image Transformers [9, 16, 31]”
Self-citationPrompt-ICM: A Unified Framework towards Image Coding for Machines with Task-driven Prompts
Feng, R.-Y., Liu, J., Jin, X., Pan, X., Sun, H.-M., & Chen, Z.-B. (2023) · arXiv.org · doi:10.48550/arxiv.2305.02578 · Cited by 24
“Image coding for machines (ICM) targets at compressing and transmitting source images to support downstream intelligent tasks, such as image classification [22,29,32,36, 52], object detection [47,48,67–69], instance segmentation [7,31,51], and semantic segmentation [3,12,13,53,81,86].”
BackgroundJu, R.-K., Chen, C.-C., Chiang, J.-S., Lin, Y.-S., Chen, W.-H., & Chien, C.-T. (2023) · Multimedia tools and applications · doi:10.1007/s11042-023-16088-0 · Cited by 24
“…for image super-resolution have started to develop by leaps and bounds [8, 45, 51]. convolutional neural networks (CNNs) [35, 36, 36–38 , 49 In recent years, Transformers [15, 41] have achieved outstanding success in CV by employing the self-attention mechanism to gather context information.”
GMILT: A Novel Transformer Network That Can Noninvasively Predict EGFR Mutation Status
Zhao, W., Chen, W., Li, G., Lei, D., Yang, J., Chen, Y., Jiang, Y., Wu, J., Ni, B., Sun, Y., et al. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3190671 · Cited by 26
“Recently, a new kind of encoder–decoder neural architecture, transformer, is proposed, which can effectively extract and utilize the relational features between different input data or feature representations [15], [16].”
BackgroundVision Transformers: From Semantic Segmentation to Dense Prediction
Zhang, L., Lu, J., Zheng, S., Zhao, X., Zhu, X., Fu, Y., Xiang, T., Feng, J., & Torr, P. H. S. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02173-w · Cited by 23
“First, though convolution is completely removed in [18] as in our SETR, their model still follows the conventional FCN design in that spatial resolution of feature maps is reduced progressively.”
Yang, M., Guo, Z., Wang, D., Wang, B., Wang, Z., & Huang, T. (2026) · Renewable Energy · doi:10.1016/j.renene.2025.123933 · Cited by 23
Lv, N.-Z., Xu, L., Chen, Y.-L., Sun, W., Tian, J.-Y., & Zhang, S. (2024) · Scientific Reports · doi:10.1038/s41598-024-77464-w · Cited by 23
“Similarly, there are also methods inspired by the transformer, such as 35,36 .”
MethodSwin transformer-based fork architecture for automated breast tumor classification
Üzen, H., Fırat, H., Atila, O., & Şengür, A. (2024) · Expert systems with applications · doi:10.1016/j.eswa.2024.125009 · Cited by 23
Face anti-spoofing with cross-stage relation enhancement and spoof material perception
Li, D.-Y., Chen, G., Wu, X.-X., Yu, Z.-T., & Tan, M.-K. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106275 · Cited by 23
He, J., Zhang, S., Yang, C., Wang, H., Gao, J., Huang, W., Wang, Q., Wang, X., Yuan, W., Wu, Y., et al. (2024) · Frontiers in Plant Science · doi:10.3389/fpls.2024.1327237 · Cited by 21
“Attention is a fundamental element of the visual converter and a crucial tool for capturing long-term dependencies (Han et al., 2021; Zhou et al., 2021).”
MethodSupremacy of attention-based transformer in oral cancer classification using histopathology images
Deo, B., Pal, M., Panigrahi, P., & Pradhan, A. (2024) · International Journal of Data Science and Analysis · doi:10.1007/s41060-023-00502-9 · Cited by 23
“However, non-overlapping style partly breaks the internal framework of an image [44].”
EViT: An Eagle Vision Transformer With Bi-Fovea Self-Attention
Shi, Y., Sun, M., Wang, Y., Ma, J., & Chen, Z. (2025) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2025.3532282 · Cited by 27
Self-citationYe, Z., Li, S., Mi, X., Shao, B., Dai, Z., Ding, B., Feng, S.-W., Sun, B., Shen, Y., & Xiao, Z.-D. (2023) · Briefings Bioinform. · doi:10.1093/bib/bbad164 · Cited by 23
Diverse features discovery transformer for pedestrian attribute recognition
Zheng, A.-H., Wang, H.-M., Wang, J.-X., Huang, H.-B., He, R., & Hussain, A. (2023) · Engineering applications of artificial intelligence · doi:10.1016/j.engappai.2022.105708 · Cited by 23
WAMF-FPI: A Weight-Adaptive Multi-Feature Fusion Network for UAV Localization
Wang, G., Chen, J., Dai, M. J., & Zheng, E. (2023) · Remote Sensing · doi:10.3390/rs15040910 · Cited by 29
Task-Agnostic Vision Transformer for Distributed Learning of Image Processing
Kim, B., Kim, J., & Ye, J. C. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3226892 · Cited by 20
“learn the large receptive field of images [36], [37], [38], [39].”
BackgroundLu, P., Wang, C., Hagenah, J., Ghiasi, S., Vital Consortium, Zhu, T.-T., Thwaites, L., & Clifton, D. (2022) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2022.3216383 · Cited by 23
“TNT [35] chooses an inner transformer block to process the relationship between sub-patches and an outer transformer block to model the relationship among patch-level embeddings.”
MethodMulti-Manifold Attention for Vision Transformers
Konstantinidis, D., Papastratis, I., Dimitropoulos, K., & Daras, P. (2023) · IEEE Access · doi:10.1109/access.2023.3329952 · Cited by 26
Exploring Advances in Transformers and CNN for Skin Lesion Diagnosis on Small Datasets
de Lima, L. M., & Krohling, R. A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-21689-3_21 · Cited by 20
“Inspired by the advances achieved by the ViT model, various models and modifications (e.g. TNT [24], Swin [25] / SwinV2 [26], CrossViT [27], XCiT [28], PiT [29], CaiT [30] ) were proposed and presented promising results.”
MethodRow-wise Accelerator for Vision Transformer
Wang, H.-Y., & Chang, T.-S. (2022) · International Conference on Artificial Intelligence Circuits and Systems · doi:10.1109/aicas54282.2022.9869928 · Cited by 23
“For the classification task on ImageNet, lots of vision transformers such as ViT [3], TNT [5], Swin [4] and PVT [6] use the encoder of Transformer.”
MethodTan, X., Gao, K., Liu, B., Fu, Y., & Kang, L. (2021) · Journal of Applied Remote Sensing · doi:10.1117/1.jrs.15.038509 · Cited by 23
“To make full use of the patch-level and pixel-level features, the deep network model is constructed with transformer-iN-transformer (TNT) 33 structure as the basic module to fully model the global and local information in the input EMP cubes.”
MethodTrade-off between Robustness and Accuracy of Vision Transformers
Li, Y., & Xu, C. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00730 · Cited by 23
“Off-the-shelf vision transformers [6,11,29,56] are often well trained to pursue the natural accuracy through Eq.”
BackgroundSelf-citationFrom multi-omics to deep learning: advances in cfDNA-based liquid biopsy for multi-cancer screening
Luo, X., Xie, S., Hong, F., Li, X., Wei, Y., Zhou, Y., Su, W., Yang, Y. R., Tang, L., Dao, F., et al. (2025) · Biomarker Research · doi:10.1186/s40364-025-00874-z · Cited by 24
“Compared with CNNs or RNNs, transformers model long range structure without a fixed receptive field and train efficiently in parallel [85, 88].”
Multi-axis vision transformer for medical image segmentation
Khan, A. R., & Khan, A. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111251 · Cited by 22
AI-driven U.S. drought prediction using machine learning and deep learning
Tanriverdi, I., & Batmaz, İ. (2025) · Climate Dynamics · doi:10.1007/s00382-025-07720-w · Cited by 33
“This is achieved through the Multi-Head Attention layers, which allow the model to focus on different parts of the sequence at once, improving its ability to understand context and dependencies across the entire sequence (Han et al. 2021).”
An overview of transformers for video anomaly detection
Dilek, E., & Dener, M. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11218-1 · Cited by 28
“TNT [19], Swin transformer [20], Cswin transformer [21], Twins [22], CAT [23], Shuffle transformer [24], Regionvit [25], and tokens-to-token vit [26] serve as examples of how vision transformer takes advantage of both local and global information exchange.”
A multi-scenario data-driven approach for anomaly detection in electric vehicle battery systems
Jia, Z., Wang, Z., Sun, Z., Sun, X., Liu, P., & Ruzzenenti, F. (2025) · eTransportation · doi:10.1016/j.etran.2025.100418 · Cited by 23
Metadata
- Authors
- Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., & Wang, Y.
- Affiliations
- Huawei Technologies (China) (CN)
- University of Macau (MO)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Volume
- 34
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2103.00112
- OpenAlex
- W3133696297
- Cited by
- 1,394
- References
- 56
- Citations by year
- 2026: 14 · 2025: 276 · 2024: 457 · 2023: 387 · 2022: 186 · 2021: 74
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (0015/2019/AKP, 62072449, 61632003, 0018/2019/AKP)
- Fundo para o Desenvolvimento das Ciências e da Tecnologia (0018/2019/AKP, 0015/2019/AKP)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- vision transformer, attention mechanism, local patches, visual words, image classification, feature aggregation, computational efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
