The DOI matches its DataCite record, with a small difference in title or year; 56 of the 56 references it cites were found in a registry or an index.
Transformer in Transformer
Open accessAbstract
Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (e.g., 16$\times$16) as "visual sentences" and present to further divide them into smaller patches (e.g., 4$\times$4) as "visual words". The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs. Features of both words and sentences will be aggregated to enhance the representation ability. Experiments on several benchmarks demonstrate the effectiveness of the proposed TNT architecture, e.g., we achieve an 81.5% top-1 accuracy on the ImageNet, which is about 1.7% higher than that of the state-of-the-art visual transformer with similar computational cost. The PyTorch code is available at https://github.com/huawei-noah/CV-Backbones, and the MindSpore code is available at https://gitee.com/mindspore/models/tree/master/research/cv/TNT.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For example, in multi-tenant architectures, system prompts are shared across multiple requests and frequently contain detailed instructions or examples to guide the model [19].”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [19]
In the citing paper
“PromptCache [19] proposes modular reuse beyond prefixes but sacrifices accuracy, with reported performance drops of up to 43% [72], making it unsuitable for tasks requiring high precision.”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [19]
In the citing paper
“related experiments results further demonstrate ViT also has remarkable representation capacity when dealing with images and videos [19].”
From Deepfake Video Detection with Spatiotemporal Dropout Transformer (2022) · cites this paper as [19]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 6×
“Compared to CNNs, TNT can outperform the widely-used ResNet and RegNet.”
BackgroundMethodInfluentialVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3×
“The effect of #heads in standard transformer has been investigated in multiple works [21, 35] and a head width of 64 is recommended for visual tasks [9, 31].”
BackgroundMethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“In the past decade, the mainstream deep neural architectures used in the computer vision (CV) are mainly established on convolutional neural networks (CNNs) [17, 12, 11].”
BackgroundFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472 · cited 1×
“As a general backbone network, TNT can also be applied with multi-scale vision models like Faster RCNN [29].”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472 · cited 7×
“The recent transformerbased models like ViT [9] and DeiT [31] are compared.”
MethodInfluentialImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 3×
“ImageNet ILSVRC 2012 [26] is an image classification benchmark consisting of 1.”
BackgroundInfluentialSqueeze-and-Excitation NetworksPreprint
Hu, J., Shen, L., & Sun, G. (2017) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1709.01507 · Cited by 37,845 · cited 1×
“Inspired by squeeze-and-excitation (SE) network for CNNs [16], we propose to explore channel-wise attention for transformers.”
MethodPyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939 · cited 2×
“We evaluate TNT-S and DeiT-S on Faster RCNN with FPN [21].”
BackgroundMethod
Research works printed but not linked: 5
- [6][6] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML , 2020. -
- [11][11] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. arXiv preprint arXiv:2012.12556 , 2020. - · arXiv
- [23][23] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017. - · arXiv
- [26][26] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR , pages 3498-3505. IEEE, 2012. -
- [37][37] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008. -
Cited by
Sorted by citations
Items 701–800 of 1,009
Zheng, W., Guo, Q., Yang, H., Wang, P., & Wang, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.15926 · Cited by 9
“Transformers have witnessed great success in the application of natural language processing [24, 18], and recently in computer vision too [21, 45, 46].”
BackgroundRaftMLP: How Much Can Be Done Without Attention and with Less Spatial Locality?
Tatsunami, Y., & Taki, M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-26351-4_28 · Cited by 20
“Immediately after the ViT paper appeared, various related works [1,4,12,14,15,33,49,53,56,58] have been done.”
BackgroundA comprehensive review of facial beauty prediction using deep learning techniques
Boukhari, D. E., Dornaika, F., Chemsa, A., & Ahmed, A. T. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112009 · Cited by 9
He, Y., & Shen, W. (2025) · Computers in Industry · doi:10.1016/j.compind.2025.104359 · Cited by 12
Praveena, S., & Devi, S. P. (2025) · Cybernetics & Systems · doi:10.1080/01969722.2025.2583968 · Cited by 11
“Transformers have become popular due to their remarkable performance in sequence-to-sequence tasks (Han et al. 2021).”
Liao, L., Xie, Y.-Z., Ning, C.-X., & Wu, S. (2025) · Mechanical systems and signal processing · doi:10.1016/j.ymssp.2025.113305 · Cited by 12
Zhang, P., Hua, Z., Qiu, Q., & Ding, J. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3572772 · Cited by 8
“The discriminator is based on a single-layer Transformer Encoder [29].”
A spatially aware global and local perspective approach for few-shot incremental learning
Wu, H., Zheng, Z.-J., Lv, L.-S., Xu, Y.-F., Bardou, D., Niu, S., Yu, G.-H., & Wang, Y. (2025) · Scientific Reports · doi:10.1038/s41598-025-08323-5 · Cited by 12
“Recently, with the popularity of self-attention structures, various architectures have been proposed in succession such as multi-head attention 56 , transformer 57 , and graph attention network 58 .”
Self-citationContext transformer with multiscale fusion for robust facial emotion recognition
Gan, Y.-L., Xu, L.-H., Song, S., & Tao, X.-M. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111720 · Cited by 12
Bukhari, S. M., Zafar, M. H., Moosavi, S. K. R., & Sanfilippo, F. (2025) · Array · doi:10.1016/j.array.2025.100401 · Cited by 12
Wang, Z., Fu, S., Zhang, H.-G., Wang, C., Xia, C., Hou, P., Shun, C., & Ge, S.-W. (2025) · Scientific Reports · doi:10.1038/s41598-025-92715-0 · Cited by 12
“In addition, Transformer, from the field of natural language processing, has excellent performance by using self-attention mechanism to obtain data features 19 .”
Dyantyi, N., & Mkabile-Masebe, B. (2025) · Research in Social Sciences and Technology · doi:10.46303/ressat.2025.22 · Cited by 12
“People who are easily reachable and eager to engage in the study are selected in this type of sampling (Han et al., 2021).”
Temporal fusion transformer model for predicting differential pressure in reverse osmosis process
Lee, S., Shim, J., Lee, J., Sung, H.-C., Lee, C., & Cho, K. H. (2025) · Journal of Water Process Engineering · doi:10.1016/j.jwpe.2024.106914 · Cited by 12
Koresh, E., Gross, R. D., Meir, Y., Tzach, Y., Halevi, T., & Kanter, I. (2025) · Physica A: Statistical Mechanics and its Applications · doi:10.1016/j.physa.2025.130529 · Cited by 12
Zheng, H.-X., Hou, H.-F., & Qin, Z. (2024) · Sustainability · doi:10.3390/su16219185 · Cited by 12
A photovoltaic cell defect detection model capable of topological knowledge extraction
Qu, Z., Li, L., Zang, J., Xu, Q., Xu, X., Dong, Y., & Fu, K. (2024) · Scientific Reports · doi:10.1038/s41598-024-72717-0 · Cited by 11
“This approach, however, constrains the model’s ability to selectively extract features specific to the target 18 .”
BackgroundCotton Disease Recognition Method in Natural Environment Based on Convolutional Neural Network
Shao, Y., Yang, W., Wang, J., Lu, Z., Zhang, M., & Chen, D. (2024) · Agriculture · doi:10.3390/agriculture14091577 · Cited by 11
“Pooling-based vision Transformer (PiT) [36], CF-ViT [37], Transformer in Transformer (TNT) [38], LeVit [39], CoAtNet [40], Vision Transformer(ViT) [41], and Swin Transformer [42] based on Transformer.”
Mamba-UIE: Enhancing Underwater Images with Physical Model Constraint
Zhang, S., Duan, Y., Li, D., & Zhao, R. (2024) · arXiv.org · doi:10.48550/arxiv.2407.19248 · Cited by 12
Xiong, Y., Shi, Q., Shen, L., Chen, C., Lu, W., & Xu, C. (2024) · Frontiers in Energy Research · doi:10.3389/fenrg.2024.1424204 · Cited by 12
Self-citationChen, S., Zhang, Z., Zhang, L., He, R., Li, Z., Xu, M.-B., & Ma, H. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3367899 · Cited by 12
“To tackle these challenges, several enhanced frameworks based on VIT [33], [34], [35], [36] have emerged.”
BackgroundMethodHD-Former: A hierarchical dependency Transformer for medical image segmentation
Wu, H., Min, W.-D., Gai, D., Huang, Z., Geng, Y., Wang, Q., & Chen, R. (2024) · Comput. Biol. Medicine · doi:10.1016/j.compbiomed.2024.108671 · Cited by 12
Transformer-Based Efficient CSI Feedback for THz Band FDD MIMO Systems
Ji, D.-J., & Chung, B.-C. (2024) · IEEE Wireless Communications Letters · doi:10.1109/lwc.2023.3329019 · Cited by 12
“Each transformer encoder layer, assuming a fourfold dimension expansion ratio, computes 2 N c D model (6 D model + N c ) FLOPs and 12 D 2 model trainable parameters [15].”
MethodChen, Z., Tang, J.-X., Qiu, H., & Chen, M.-S. (2023) · Frontiers in Marine Science · doi:10.3389/fmars.2023.1306229 · Cited by 12
MTU2-Net: Extracting Internal Solitary Waves from SAR Images
Saheya, B., An, Z., Jin, Q., Chen, X., Gong, M., & Zeng, T. (2023) · Remote Sensing · doi:10.3390/rs15235441 · Cited by 11
Res-ViT: Residual Vision Transformers for Image Recognition Tasks.
Elmi, S., & Morris, B. (2023) · IEEE International Conference on Tools with Artificial Intelligence · doi:10.1109/ictai59109.2023.00052 · Cited by 12
“TNT [18]: is a computer vision model that extends the transformer architecture to multi-level feature learning.”
MultiScale Spectral-Spatial Convolutional Transformer for Hyperspectral Image Classification
Gong, Z.-Q., Zhou, X., & Yao, W. (2023) · IET Image Processing · doi:10.48550/arxiv.2310.18550 · Cited by 12
“Some variants have also been developed to expand the application domains and further enhance its ability to extract multi-scale information, hierarchical information, and other capabilities, such as Vision Transformer [13], Transformer in Transformer [14], swin transformer [15], DeiT [16].”
BackgroundKumar, A. S., Schlosser, T., Langner, H., Ritter, M., & Kowerko, D. (2023) · Bioengineering · doi:10.3390/bioengineering10101177 · Cited by 12
“To explore novel approaches to image segmentation and classification, different transformer-based models [57] with improved prediction scores, encompassing, i.a., data-efficient image transformers (DeiTs) as well as hybrid transformers [80–82], will be further assessed.”
BackgroundPT-Informer: A Deep Learning Framework for Nuclear Steam Turbine Fault Diagnosis and Prediction
Zhou, J., An, Z., Yang, Z.-L., Zhang, Y.-H., Chen, H.-L., Chen, W., Luo, Y., & Guo, Y. (2023) · Machines · doi:10.3390/machines11080846 · Cited by 12
Rao, Z., Wang, H., Chen, L., Lian, Y., Zhong, Y.-L., Liu, Z., & Cai, Y.-F. (2023) · IEEE transactions on intelligent transportation systems (Print) · doi:10.1109/tits.2023.3253554 · Cited by 12
“Trans-former and self-attention models present the ability to capture long-distance features in both the Natural Language Processing (NLP) and Computer Vision (CV) fields [20], [21], [22], [23], [24].”
BackgroundMDL-NAS: A Joint Multi-domain Learning Framework for Vision Transformer
Wang, S., Xie, T., Cheng, J., Zhang, X., & Liu, H. (2023) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52729.2023.01924 · Cited by 12
“…[13] as a starting point, various variants of vision transformers have lately been proposed to resolve the inherent challenges of ViT, such as data-efficient training [40], position embedding [11], effective tokenization [16, 54], multi-scale processing [10, 48] and hierarchical design [27, 44].”
BackgroundMethodLiu, D., Chen, P., Yao, M., Lu, Y.-J., Cai, Z., & Tian, Y. (2023) · arXiv.org · doi:10.48550/arxiv.2304.11631 · Cited by 12
“TNT [11] create the Transformer-iN-Transformer for pixel-level modeling.”
Huang, Y., Chen, L., Zhou, C., Yan, N., Qiao, L., Lan, S., & Wen, Y. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110544 · Cited by 12
Region-adaptive Concept Aggregation for Few-shot Visual Recognition
Han, M., Zhan, Y.-B., Yu, B.-S., Luo, Y., Hu, H., Du, B., Wen, Y., & Tao, D.-C. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1358-8 · Cited by 12
“Typical attention mechanisms include normal attention networks[44, 45], self-attention networks[41, 46, 47], and graph attention networks[48].”
BackgroundBiTSRS: A Bi-Decoder Transformer Segmentor for High-Spatial-Resolution Remote Sensing Images
Liu, Y., Zhang, Y., Wang, Y., & Mei, S.-H. (2023) · Remote Sensing · doi:10.3390/rs15030840 · Cited by 12
Self-citationmmLayout: Multi-grained MultiModal Transformer for Document Understanding
Wang, W., Huang, Z., Luo, B., Chen, Q., Peng, Q., Pan, Y., Yin, W., Feng, S., Sun, Y., Yu, D., et al. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548406 · Cited by 23
“TNT [13] divides the input images into several patches as “visual sentence” and then further divides them into sub-patches as “visual words”.”
MethodImproving Road Extraction for Autonomous Driving Using Swin Transformer Unet
Ge, C., Nie, Y., Kong, F., & Xu, X. (2022) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc55140.2022.9922395 · Cited by 23
“Other approaches [24, 25] have attempted to leverage the power of ViTs as a stand-alone block in their architectures which otherwise consist of CNN-based components.”
BackgroundMulti-granularity vision transformer via semantic token for hyperspectral image classification
Li, B., Ouyang, E., Hu, W., Zhang, G., Zhao, L., & Wu, J. (2022) · International Journal of Remote Sensing · doi:10.1080/01431161.2022.2142078 · Cited by 14
“Such an exquisite paradigm can effectively capture the sequential visual information and estimate the attention between different sub-patches (Han et al. 2021).”
BackgroundGSWIN: Gated MLP Vision Model with Hierarchical Structure of Shifted Window
Go, M., & Tachibana, H. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096453 · Cited by 9
“, 2021), TNT (alternating local/global self-attention) (Han et al., 2021), MViT (fine-to-coarse aggregation using multi-head pooling attention) (Fan et al.”
MAN and CAT: mix attention to nn and concatenate attention to YOLO
Guan, R., Man, K. L., Zhao, H., Zhang, R.-X., Yao, S., Smith, J. S., Lim, E. G., & Yue, Y. (2022) · The Journal of Supercomputing · doi:10.1007/s11227-022-04726-7 · Cited by 11
“ViT-based classification models include ViT [10], Swin [11], Twins [12], DeiT [13], CeiT [14] and TNT [15], etc.”
MethodAdaptive Split-Fusion Transformer
Su, Z., Chen, J., Pang, L., Ngo, C., & Jiang, Y. (2023) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme55011.2023.00204 · Cited by 13
“Type Model Image Params MACs ImageNetSize (M) (G) Top-1 Top-5 C N N (S m al l) ResNet-50 [25] 224 25.6 7.6 76.7 93.3 RegNetY-4G [26] 224 21.0 4.0 80.0 - ConvNeXt-T [27] 224 29.0 4.5 82.1 - Tr an sf or m er (S m al l) PVT-S [16] 224 24.5 7.6 79.8 - DeiT-S [15] 224 22.0 4.6 79.9 95.0 Swin-T [17] 224 28.0 4.5 81.2 95.5 T…”
MethodInfluentialSpikeFormer: Image Reconstruction from the Sequence of Spike Camera Based on Transformer
She, C., & Qing, L.-Y. (2022) · International Conference on Image and Graphics Processing · doi:10.1145/3512388.3512399 · Cited by 12
KUŞ, İ., Koçak, C., & Keleş, A. (2025) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2025.200615 · Cited by 11
Xu, X., Yang, J., Hu, D., Khan, M. A., Por, L. Y., & Li, C.-S. (2025) · IEEE journal of biomedical and health informatics · doi:10.1109/jbhi.2025.3607446 · Cited by 11
“Finally, we compare four common attention mechanisms, i.e., channel attention (CA) [68], spatial attention (SA) [69], convolutional block attention module (CBAM) [70], and self-attention (Self-A) [71], against peer methods.”
Contrastive learning-based drug screening model for GluN1/GluN3A inhibitors
Li, K., Zeng, Y., Xiong, Y.-D., Wu, H.-C., Fang, S., Qu, Z.-Y., Zhu, Y., Du, B., Gao, Z.-B., & Hu, W.-B. (2025) · Acta Pharmacologica Sinica · doi:10.1038/s41401-025-01580-0 · Cited by 11
“For target feature extraction, we employ a Transformer encoder model [50], which processes the target sequence as input to generate a feature representation Φ target for the target: where, Φ target is the feature vector representing the target protein, and Transformer is a function that applies the…”
Singh, S., Bansal, S., Saddik, A. E., & Saini, M. (2025) · arXiv.org · doi:10.48550/arxiv.2504.03219 · Cited by 11
“The introduction of transformer models[36][37] has revolutionized the field of NLP[38][39].”
Chetoui, M., & Akhloufi, M. (2025) · Biomedicines · doi:10.3390/biomedicines13010141 · Cited by 11
“To solve the problems of traditional convolutional methods for identifying long-range dependencies, we integrate a Transformer block [24] into the U-Net architecture, particularly during the bottleneck stage.”
Bouaziz, S., Hafiane, A., Canals, R., & Nedjai, R. (2024) · Remote Sensing · doi:10.48550/arxiv.2412.16631 · Cited by 11
“They incorporate feedback connections that allow information from previous time steps to influence the current output, enabling the network to retain temporal dependencies [175,176].”
Transformer-empowered receiver design of OFDM communication systems
Yue, B., Qiu, S., Yang, C. X., Peng, L., & Zhang, Y. (2024) · Computer Communications · doi:10.1016/j.comcom.2024.107960 · Cited by 13
Garg, A., Gupta, S., Vats, S., Handa, P., & Goel, N. (2024) · Expert Systems · doi:10.1111/exsy.13697 · Cited by 14
“A pivotal moment in LLM evolution transpired in 2017 with the introduction of transformer models (Han et al., 2021).”
Zhao, R., Lei, Z.-H., & Zhao, Z.-Y. (2024) · Frontiers in Energy Research · doi:10.3389/fenrg.2024.1376677 · Cited by 11
RetractedSemantic segmentation using cross-stage feature reweighting and efficient self-attention
Ma, Y., & Lan, X. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.104996 · Cited by 10
Composite attention mechanism network for deep contrastive multi-view clustering
Du, T., Zheng, W., & Xu, X. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106361 · Cited by 11
Masked LoGoNet: Fast and Accurate 3D Image Analysis for Medical Domain
Monsefi, A. K., Karisani, P., Zhou, M., Choi, S., Doble, N., Ji, H., Parthasarathy, S., & Ramnath, R. (2024) · Knowledge Discovery and Data Mining · doi:10.1145/3637528.3672069 · Cited by 11
“To model long-range dependencies in images, existing studies mostly use vision transformers [2, 7, 14, 18, 19, 22, 31, 45, 48], and draw ideas from sequence modeling in Natural Language Processing (NLP).”
Liu, X., Ding, Y., Zhang, Y., & Tang, J. (2024) · Journal of Applied Biomedicine · doi:10.1016/j.bbe.2024.02.001 · Cited by 10
Chao, P., Chang, C.-H., Wu, J.-J., Liu, Y.-H., Shiau, J., Shih, H.-H., Lin, G.-Z., Lee, S., & Lee, T.-F. (2024) · Cancer Control · doi:10.1177/10732748241286749 · Cited by 11
PDiT: Interleaving Perception and Decision-making Transformers for Deep Reinforcement Learning
Mao, H.-Y., Zhao, R., Li, Z.-Y., Xu, Z., Chen, H., Chen, Y., Zhang, B., Xiao, Z., Zhang, J.-G., & Yin, J. (2023) · Adaptive Agents and Multi-Agent Systems · doi:10.48550/arxiv.2312.15863 · Cited by 11
“In computer vision, some methods apply two Transformers to handle tasks like image classification and person search, e.g., TNT [16]., ViViT [2], DualFormer [29] and COAT [45].”
MethodDouble Transformer Super-Resolution for Breast Cancer ADC Images
Yang, Y., Xiang, T., Lv, X., Li, L.-H., Lok, M.-L., & Zeng, T.-Y. (2023) · IEEE journal of biomedical and health informatics · doi:10.1109/jbhi.2023.3341250 · Cited by 11
“Subsequently,variousno-table advancements have been made based on ViT [58], [59], [60].”
MethodSong, Y.-L., Sun, Z.-Z., Zhang, R., Min, H.-J., Li, Q., Cai, J., Wang, X., Zhou, Q., & Jiang, D. (2023) · Plants · doi:10.3390/plants12234043 · Cited by 11
TCRec: A novel paper recommendation method based on ternary coauthor interaction
Xiao, X., Xu, J., Huang, J., Zhang, C., & Chen, X. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111065 · Cited by 11
Xu, Y., Wang, X., Zhang, H., & Lin, H. (2023) · IET Image Processing · doi:10.1049/ipr2.12929 · Cited by 11
AttMOT: Improving Multiple-Object Tracking by Introducing Auxiliary Pedestrian Attributes
Li, Y., Xiao, Z., Yang, L., Meng, D., Zhou, X., Fan, H., & Zhang, L. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3384446 · Cited by 11
“It is also worth mentioning that with the spread of Transformer [52] in computer vision, the attention mechanism [31] is now widely used in MOT and its subtasks like person Re-ID [53], [54].”
MobileFaceFormer: a lightweight face recognition model against face variations
Li, J., Zhou, L., & Chen, J. (2023) · Multimedia tools and applications · doi:10.1007/s11042-023-15954-1 · Cited by 11
“Recently, ViT [7] and its follow-ups [1, 2, 12, 18, 26, 33, 37] show great performance over computer vision tasks.”
Pyramid feature fusion through shifted window self-attention for tobacco leaf classification
He, Z.-F., Chen, G.-C., Zhang, Y.-H., Zhao, C., He, P.-F., & Shi, B. (2023) · Expert systems with applications · doi:10.2139/ssrn.4227571 · Cited by 11
Fang, N., Qiu, L.-M., Zhang, S.-Y., Wang, Z.-L., Hu, K., & Wang, K. (2023) · IEEE transactions on intelligent transportation systems (Print) · doi:10.1109/tits.2023.3348795 · Cited by 11
“As described in [45], the FLOPs of standard multi-head attention (MA) is shown in Equ.”
BackgroundDing, C., Tang, D., Zheng, X., Wang, Q., He, Y., & Zhang, L. (2023) · IEEE Instrumentation & Measurement Magazine · doi:10.1109/mim.2023.10083000 · Cited by 17
“Transformer in trans-former (TNT) [12] encodes the input data into powerful features by means of an attention mechanism.”
BackgroundTask-Aware Weakly Supervised Object Localization With Transformer
Meng, M., Zhang, T., Zhang, Z., Zhang, Y., & Wu, F. H. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3230902 · Cited by 7
“Since transformers have achieved remarkable success in NLP tasks [55], [56], many efforts have been made to introduce transformers to vision tasks including image classification [28], [57], [58], [59], [60], object detection [29], [30], object tracking [54], [61] and video processing [62], [63], [64].”
BackgroundCPAM: Cross Patch Attention Module for Complex Texture Tile Block Defect Detection
Zhu, W., Wang, Q., Luo, L., Zhang, Y.-Z., Lu, Q., Yeh, W.-C., & Liang, J. (2022) · Applied Sciences · doi:10.3390/app122311959 · Cited by 11
Fcaformer: Forward Cross Attention in Hybrid Vision Transformer
Zhang, H., Hu, W., & Wang, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00557 · Cited by 10
FocusFormer: Focusing on What We Need via Architecture SamplerPreprint
Liu, J., Cai, J.-F., & Zhuang, B.-H. (2022) · arXiv.org · doi:10.48550/arxiv.2208.10861 · Cited by 11
End-to-End Video Violence Detection with Transformer
Zhou, L. (2022) · 2022 5th International Conference on Pattern Recognition and Artificial Intelligence (PRAI) · doi:10.1109/prai55851.2022.9904115 · Cited by 11
Multimodal fault detection model for photovoltaic modules
Yu, S.-S., Gan, F.-B., Han, T., Hou, S.-N., Feng, X., & Chen, K. (2026) · Scientific Reports · doi:10.1038/s41598-025-28603-4 · Cited by 10
“RT-DETR is the first Transformer framework that supports real-time end-to-end object detection 23 , eliminating the reliance on traditional non-maximum suppression (NMS) used in object detection, thus avoiding the negative impact of NMS on inference speed and detection accuracy.”
Xiao, Y., Zheng, S., Feng, H., Huang, Y., Leng, J.-W., & Hong, J. (2025) · Journal of manufacturing systems · doi:10.1016/j.jmsy.2025.08.002 · Cited by 10
Wu, R., Abedin, M., Zeng, H.-J., & Lucey, B. M. (2025) · Journal of Forecasting · doi:10.1002/for.70024 · Cited by 10
SEFRQO: A Self-Evolving Fine-Tuned RAG-Based Query Optimizer
Liu, H., Zhang, Q., Marcus, R., & Sabek, I. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3769826 · Cited by 5
“LLMs often adopt an auto-regressive approach to generate response sequences [16].”
Continual Learning for Wireless Channel Prediction
Mohsin, M., Umer, M., Bilal, A., Jamshed, M. A., & Cioffi, J. M. (2025) · arXiv.org · doi:10.48550/arxiv.2506.22471 · Cited by 10
“Gated recurrent unit (GRU) (Dey & Salem, 2017) and Trans-former (Han et al., 2021) chitectural diagrams and hyperparameter settings for the LSTM, GRU and Transformer backbones appear in Appendix A.”
AnchorFormer: Differentiable Anchor Attention for Efficient Vision Transformer
Shan, J., Wang, J., Zhao, L.-F., Cai, L., Zhang, H.-Y., & Liritzis, I. (2025) · Pattern Recognition Letters · doi:10.48550/arxiv.2505.16463 · Cited by 10
“2a is the image classification results on the ImageNet dataset.”
A dual-branch model combining convolution and vision transformer for crop disease classification
Meng, Q., Guo, J., Zhang, H., Zhou, Y., & Zhang, X. (2025) · PLoS ONE · doi:10.1371/journal.pone.0321753 · Cited by 12
“ViT, LeViT [40], MobileViT-v2 [32], PVTv2 [41] and TNT [42] were chosen and represented the ViT-based models.”
Self-citationHuang, X., Lang, X., Guo, T., & Yu, L. (2025) · Electric power systems research · doi:10.1016/j.epsr.2024.111360 · Cited by 10
Ravula, J., & Singh, N. (2025) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2025.200524 · Cited by 10
Nunnari, G., & Calvari, S. (2025) · Applied Sciences · doi:10.3390/app15052604 · Cited by 10
A multidimensional adaptive transformer network for fatigue detection
Wu, D.-M., Deng, L., Lu, Q.-P., & Liu, S.-H. (2025) · Cognitive Neurodynamics · doi:10.1007/s11571-025-10224-2 · Cited by 10
“Unlike traditional models such as recurrent neural networks (RNNs), which process data sequentially (Han et al. 2021), Transformer models employ an attention mechanism that enables them to concentrate on significant aspects of the data at any given moment, thereby facilitating greater flexibility and…”
Meta-LSTR: Meta-Learning with Long Short-Term Transformer for futures volatility prediction
Chen, Y.-Z., Ye, N., Zhang, W.-Y., Fan, J., Mumtaz, S., & Li, X.-M. (2024) · Expert systems with applications · doi:10.1016/j.eswa.2024.125926 · Cited by 10
Wang, S., Zhao, Y., Li, J.-Y., Yi, Z., Li, J., Zuo, C., Yao, Y., & Liu, A. (2024) · Comput. Medical Imaging Graph. · doi:10.1016/j.compmedimag.2024.102457 · Cited by 10
BUZZ: Beehive-structured Sparse KV Cache with Segmented Heavy Hitters for Efficient LLM Inference
Zhao, J., Fang, Z., Li, S., Yang, S., & He, S. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3810942 · Cited by 14
“With the introduction of the attention mechanism [3], transformer-based network structures have been proven to perform well in various fields, such as personalized content recommendation on streaming platforms, drug discovery through molecular property prediction, and enhancing”
Mixed Prototype Correction for Causal Inference in Medical Image Classification
Zhang, Y., Huang, Z.-A., Hong, Z., Wu, S., Wu, J., & Tan, K. C. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681395 · Cited by 5
Cao, B., Ye, J., Wei, Y., & Shan, H. (2025) · neural information processing systems · doi:10.52202/085713-2049 · Cited by 3
“We observe that the self-attention mechanism presents powerful global spatial modeling capability [5, 13, 29, 45], and this capability is parameter-agnostic.”
A dual encoder LDCT image denoising model based on cross-scale skip connections
Wang, L., Wang, Y.-L., Ren, W., Yu, J., Chang, X., Guo, X.-D., & Hu, L. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128741 · Cited by 10
Self-citationST2SI: Image Style Transfer via Vision Transformer using Spatial Interaction
Li, W., Chen, Y., Guo, X., & He, X. (2024) · Computers & Graphics · doi:10.1016/j.cag.2024.104084 · Cited by 9
Hyneter:Hybrid Network Transformer for Multiple Computer Vision Tasks
Chen, D., Miao, D., & Zhao, X. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3367043 · Cited by 13
“ViT and its follow-ups [12], [13], [14], [15] achieve an impressive speed-accuracy tradeoff on image classification compared with convolution networks.”
BackgroundGELT: A graph embeddings based lite-transformer for knowledge tracing
Liang, Z., Wu, R., Liang, Z., Yang, J., Wang, L., & Su, J. (2024) · PLoS ONE · doi:10.1371/journal.pone.0301714 · Cited by 13
“The Transformer model [16], incorporating an attention mechanism [17], has gained popularity in recent years for its ability to greatly enhance prediction accuracy in deep learning models.”
BackgroundMethodFAM: Improving columnar vision transformer with feature attention mechanism
Huang, L., Bai, X., Zeng, J., Yu, M., Pang, W., & Wang, K. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.103981 · Cited by 11
Tang, J., Gong, Z., Tao, B., & Yin, Z. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102390 · Cited by 7
Marrying Global–Local Spatial Context for Image Patches in Computer-Aided Assessment
Yu, J., Ma, T., Chen, H., Lai, M., Ju, Z., & Xu, Y. (2023) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2023.3290205 · Cited by 13
Masked image modeling-based boundary reconstruction for 3D medical image segmentation
Liu, C., Cheng, Y., & Tamura, S. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107526 · Cited by 10
Novel Physics-Based Machine-Learning Models for Indoor Air Quality Approximations
Mohammadshirazi, A., Nadafian, A., Monsefi, A. K., Rafiei, M., & Ramnath, R. (2023) · arXiv.org · doi:10.48550/arxiv.2308.01438 · Cited by 10
“The performance of the proposed models is compared with each other and to that of state-of-the-art non-physics-based models that are Transformers [17, 43], FEDformer [41], and Informer [43].”
MethodInfluentialVisual Prompt Flexible-Modal Face Anti-Spoofing
Yu, Z., Cai, R., Cui, Y., Liu, A., & Chen, C. (2023) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2024.3520534 · Cited by 10
“[51], [52], [53], [54] are proposed from then, which achieve promising performance compared with its counterpart CNNs for image analysis tasks [55], [56], [57].”
Multi-Attention Transformer for Naturalistic Driving Action Recognition
Dong, X., Zhao, R., Sun, H., Wu, D., Wang, J., Zhou, X., Liu, J., Cui, S., & He, Z. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00574 · Cited by 11
“Kai Han et al. [8] proposed the Transformer in Transformer (TNT) model, which fused the features of the outer patches and the inner patches, enriching feature information and improving feature expression.”
MethodTranSFormer: Slow-Fast Transformer for Machine Translation
Li, B., Jing, Y., Tan, X., Xing, Z., Xiao, T., & Zhu, J.-B. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2305.16982 · Cited by 10
Said, Y., Atri, M., Albahar, M., Atitallah, A. B., & Alsariera, Y. (2023) · Mathematics · doi:10.3390/math11051127 · Cited by 10
Automatic Sleep Stage Classification Method based on Transformer-in-Transformer
Kim, M., Jung, K., & Chung, W. (2023) · Balkan Conference in Informatics · doi:10.1109/bci57258.2023.10078607 · Cited by 10
Metadata
- Authors
- Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., & Wang, Y.
- Affiliations
- Huawei Technologies (China) (CN)
- University of Macau (MO)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Volume
- 34
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2103.00112
- OpenAlex
- W3133696297
- Cited by
- 1,394
- References
- 56
- Citations by year
- 2026: 14 · 2025: 276 · 2024: 457 · 2023: 387 · 2022: 186 · 2021: 74
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (0015/2019/AKP, 62072449, 61632003, 0018/2019/AKP)
- Fundo para o Desenvolvimento das Ciências e da Tecnologia (0018/2019/AKP, 0015/2019/AKP)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- vision transformer, attention mechanism, local patches, visual words, image classification, feature aggregation, computational efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
