Transformer in Transformer
Open accessAbstract
Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (e.g., 16$\times$16) as "visual sentences" and present to further divide them into smaller patches (e.g., 4$\times$4) as "visual words". The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs. Features of both words and sentences will be aggregated to enhance the representation ability. Experiments on several benchmarks demonstrate the effectiveness of the proposed TNT architecture, e.g., we achieve an 81.5% top-1 accuracy on the ImageNet, which is about 1.7% higher than that of the state-of-the-art visual transformer with similar computational cost. The PyTorch code is available at https://github.com/huawei-noah/CV-Backbones, and the MindSpore code is available at https://gitee.com/mindspore/models/tree/master/research/cv/TNT.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In multi-tenant architectures, system prompts are shared across multiple requests and frequently contain detailed instructions or examples to guide the model.
“For example, in multi-tenant architectures, system prompts are shared across multiple requests and frequently contain detailed instructions or examples to guide the model [19].”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [19]
Vision transformers have remarkable representation capacity when dealing with images and videos.
“related experiments results further demonstrate ViT also has remarkable representation capacity when dealing with images and videos [19].”
From Deepfake Video Detection with Spatiotemporal Dropout Transformer (2022) · cites this paper as [19]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 6× · see citing works
“Compared to CNNs, TNT can outperform the widely-used ResNet and RegNet.”
BackgroundMethodInfluentialVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3× · see citing works
“The effect of #heads in standard transformer has been investigated in multiple works [21, 35] and a head width of 64 is recommended for visual tasks [9, 31].”
BackgroundMethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1× · see citing works
“In the past decade, the mainstream deep neural architectures used in the computer vision (CV) are mainly established on convolutional neural networks (CNNs) [17, 12, 11].”
BackgroundFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · cited 1× · see citing works
“As a general backbone network, TNT can also be applied with multi-scale vision models like Faster RCNN [29].”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 3× · see citing works
“ImageNet ILSVRC 2012 [26] is an image classification benchmark consisting of 1.”
BackgroundInfluentialSqueeze-and-Excitation NetworksPreprint
Hu, J., Shen, L., & Sun, G. (2017) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1709.01507 · Cited by 37,845 · cited 1× · see citing works
“Inspired by squeeze-and-excitation (SE) network for CNNs [16], we propose to explore channel-wise attention for transformers.”
MethodPyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · see citing works
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939 · cited 2× · see citing works
“We evaluate TNT-S and DeiT-S on Faster RCNN with FPN [21].”
BackgroundMethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2× · see citing works
“, the famous BERT [8] and GPT-3 [2] models.”
MethodAn Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · cited 7× · see citing works
“The recent transformerbased models like ViT [9] and DeiT [31] are compared.”
MethodInfluentialEnd-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285 · see citing works
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253 · cited 4×
“We construct a pure transformer object detection pipeline by combining our TNT and DETR [3].”
MethodInfluentialEfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072 · cited 2× · see citing works
“Note that all the transformer-based models are still inferior to EfficientNet which utilizes special depth-wise convolutions, so it is yet a challenge of how to beat EfficientNet using pure transformer.”
MethodObject recognition from local scale-invariant features
Lowe, D. G. (1999) · Proceedings of the Seventh IEEE International Conference on Computer Vision · doi:10.1109/iccv.1999.790410 · Cited by 16,472 · cited 1× · see citing works
“visual transformers ignore the local relation and structure information inside the patch which is important for visual recognition [22, 3].”
BackgroundLayer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,745 · cited 1× · see citing works
“LN. Layer normalization [1] is a key part in transformer for stable training and faster convergence.”
BackgroundWang, X., Girshick, R., Gupta, A., & He, K. (2017) · arXiv · Cited by 11,689 · cited 1× · see citing works
“explore self-attention mechanism in non-local networks [37] for capturing long-range dependencies in video and image recognition.”
BackgroundWang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596 · cited 1× · see citing works
“explore self-attention mechanism in non-local networks [37] for capturing long-range dependencies in video and image recognition.”
Backgroundmixup: Beyond Empirical Risk Minimization
Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2017) · arXiv · doi:10.48550/arxiv.1710.09412 · Cited by 5,831 · cited 1× · see citing works
“We adopt the same data augmentation strategy as that in DeiT [31] including random crop, random clip, Rand-Augment [7], Random Erasing [44], Mixup [42] and CutMix [41].”
MethodCutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features
Yun, S., Han, D., Chun, S., Oh, S. J., Yoo, Y., & Choe, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00612 · Cited by 5,400 · cited 1× · see citing works
“We adopt the same data augmentation strategy as that in DeiT [31] including random crop, random clip, Rand-Augment [7], Random Erasing [44], Mixup [42] and CutMix [41].”
MethodPyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00061 · Cited by 5,259 · cited 4× · see citing works
“For fair comparison, we adopt the training and testing settings in PVT [36] and add a 2×2 average pooling to make the output size of TNT backbone the same as that of PVT and ResNet.”
BackgroundMethodInfluentialHan (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682 · cited 1× · see citing works
“Nowadays, transformer architectures have been used in a growing number of computer vision tasks [10] such as image recognition [6, 40, 29], object detection [46], and segmentation [43, 38].”
BackgroundSelf-citationGhostNet: More Features From Cheap OperationsPreprint
Han, K., Wang, Y.-H., Tian, Q., Guo, J.-Y., Xu, C.-J., & Xu, C. (2019) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1911.11907 · Cited by 4,540 · see citing works
Self-citationRethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P. H. S., et al. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00681 · Cited by 4,008 · cited 1× · see citing works
“Nowadays, transformer architectures have been used in a growing number of computer vision tasks [10] such as image recognition [6, 40, 29], object detection [46], and segmentation [43, 38].”
BackgroundSelf-citationRandom Erasing Data Augmentation
Zhong, Z., Zheng, L., Kang, G., Li, S., & Yang, Y. (2017) · arXiv · doi:10.48550/arxiv.1708.04896 · Cited by 3,617 · cited 1× · see citing works
“We adopt the same data augmentation strategy as that in DeiT [31] including random crop, random clip, Rand-Augment [7], Random Erasing [44], Mixup [42] and CutMix [41].”
MethodRandom Erasing Data Augmentation
Zhong, Z., Zheng, L., Kang, G., Li, S., & Yang, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.7000 · Cited by 3,617 · see citing works
Gaussian Error Linear Units (GELUs)
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537 · cited 2× · see citing works
“The MLP is applied between self-attention layers for feature transformation and non-linearity: MLP(X) = FC(σ(FC(X))), FC(X) = XW + b, (2) where W and b are the weight and bias term of fully-connected layer respectively, and σ(·) is the activation function such as GELU [13].”
BackgroundAutomated Flower Classification over a Large Number of Classes
Nilsback, M.-E., & Zisserman, A. (2008) · 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing · doi:10.1109/icvgip.2008.47 · Cited by 3,442 · cited 1× · see citing works
“These datasets include superordinate-level object classification (CIFAR-10 [16], CIFAR-100 [16]) and fine-grained object classification (Oxford-IIIT Pets [23], Oxford 102 Flowers [22] and iNaturalist 2019 [34]), shown in Table 2.”
MethodScene Parsing through ADE20K Dataset
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., & Torralba, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.544 · Cited by 3,326 · cited 3× · see citing works
“ADE20K [45] Segmentation 20,210 2,000 150”
Randaugment: Practical automated data augmentation with a reduced search space
Cubuk, E. D., Zoph, B., Shlens, J., & Le, Q. V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00359 · Cited by 3,223 · cited 1× · see citing works
“We adopt the same data augmentation strategy as that in DeiT [31] including random crop, random clip, Rand-Augment [7], Random Erasing [44], Mixup [42] and CutMix [41].”
MethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L., & Dollár, P. (2014) · arXiv · doi:10.48550/arxiv.1405.0312 · Cited by 2,579 · cited 1× · see citing works
“COCO2017 [19] Detection 118,287 5,000 80”
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F. E. H., Feng, J., & Yan, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00060 · Cited by 2,415 · cited 1× · see citing works
“Nowadays, transformer architectures have been used in a growing number of computer vision tasks [10] such as image recognition [6, 40, 29], object detection [46], and segmentation [43, 38].”
BackgroundDesigning Network Design Spaces
Radosavovic, I., Kosaraju, R. P., Girshick, R. B., He, K., & Dollár, P. (2020) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.2003.13678 · Cited by 2,313 · cited 2×
“To have a better understanding of current progress of visual transformers, we also include the representative CNN-based models such as ResNet [12], RegNet [25] and EfficientNet [28].”
MethodPre-Trained Image Processing Transformer
Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., Ma, S., Xu, C., Xu, C., & Gao, W. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01212 · Cited by 2,308 · cited 1× · see citing works
“, denosing and super-resolution) as a series of translations and develop the IPT model for handling multiple low-level computer vision problems [4].”
Self-citationMMDetection: Open MMLab Detection Toolbox and Benchmark
Chen, K., Wang, J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Xu, J., et al. (2019) · arXiv · doi:10.48550/arxiv.1906.07155 · Cited by 2,227 · cited 1× · see citing works
“Backbone Params (M) Epochs AP AP50 AP75 APS APM APL ResNet-50 [21, 5] 41.”
BackgroundDeformable DETR: Deformable Transformers for End-to-End Object Detection
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., & Dai, J. (2020) · arXiv · doi:10.48550/arxiv.2010.04159 · Cited by 1,837 · cited 1× · see citing works
“Nowadays, transformer architectures have been used in a growing number of computer vision tasks [10] such as image recognition [6, 40, 29], object detection [46], and segmentation [43, 38].”
BackgroundGooch, P. (1969) · doi:10.1136/bmj.4.5674.53-c · Cited by 1,834 · cited 2×
“Classification 2,040 6,149 102 Oxford-IIIT Pets [23] 3,680 3,669 37 iNaturalist 2019 [34] 265,240 3,003 1,010 CIFAR-10 [16] 50,000 10,000 10 CIFAR-100 [16] 50,000 10,000 100”
MethodThe iNaturalist Species Classification and Detection Dataset
Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., & Belongie, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00914 · Cited by 1,451 · cited 2× · see citing works
“Classification 2,040 6,149 102 Oxford-IIIT Pets [23] 3,680 3,669 37 iNaturalist 2019 [34] 265,240 3,003 1,010 CIFAR-10 [16] 50,000 10,000 10 CIFAR-100 [16] 50,000 10,000 100”
MethodFractalNet: Ultra-Deep Neural Networks without ResidualsPreprint
Larsson, G., Maire, M., & Shakhnarovich, G. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07648 · Cited by 810 · cited 1× · see citing works
“The main advanced technologies apart from common settings [12] include AdamW [20], label smoothing [27], DropPath [18], and repeated augmentation [14].”
BackgroundRethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2015) · arXiv · doi:10.48550/arxiv.1512.00567 · Cited by 741 · cited 1× · see citing works
“The main advanced technologies apart from common settings [12] include AdamW [20], label smoothing [27], DropPath [18], and repeated augmentation [14].”
BackgroundConditional Positional Encodings for Vision Transformers
Chu, X., Tian, Z., Zhang, B., Wang, X., & Shen, C. (2021) · arXiv · doi:10.48550/arxiv.2102.10882 · Cited by 489 · cited 1× · see citing works
“Nowadays, transformer architectures have been used in a growing number of computer vision tasks [10] such as image recognition [6, 40, 29], object detection [46], and segmentation [43, 38].”
BackgroundAre Sixteen Heads Really Better than One?
Michel, P., Levy, O., & Neubig, G. (2019) · arXiv · doi:10.48550/arxiv.1905.10650 · Cited by 404 · cited 1× · see citing works
“The effect of #heads in standard transformer has been investigated in multiple works [21, 35] and a head width of 64 is recommended for visual tasks [9, 31].”
MethodEnd-to-end Lane Shape Prediction with Transformers
Liu, R., Yuan, Z.-J., Liu, T., & Xiong, Z. (2020) · IEEE Workshop/Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00374 · Cited by 369 · see citing works
Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
Brendel, W., & Bethge, M. (2019) · arXiv · doi:10.48550/arxiv.1904.00760 · Cited by 323 · cited 1× · see citing works
“visual transformers ignore the local relation and structure information inside the patch which is important for visual recognition [22, 3].”
BackgroundToward Transformer-Based Object DetectionPreprint
Beal, J., Kim, E. S., Tzeng, E., Park, D. H., Zhai, A., & Kislyuk, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.09958 · Cited by 255 · cited 1× · see citing works
“This design also affects the subsequent works including DeiT [31], ViT-FRCNN [2], IPT [6] and SETR [38].”
MethodLin, M., Chen, Q., & Yan, S. (2013) · arXiv · doi:10.48550/arxiv.1312.4400 · Cited by 252 · see citing works
Training data-efficient image transformers & distillation through attention
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jégou, H. (2020) · arXiv · doi:10.48550/arxiv.2012.12877 · Cited by 182 · cited 9× · see citing works
“Transformer-based DeiT-Ti [31] 224×224 5.”
MethodInfluentialAugment your batch: Improving generalization through instance repetition
Hoffer, E., Ben-Nun, T., Hubara, I., Giladi, N., Hoefler, T., & Soudry, D. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00815 · Cited by 166 · cited 1× · see citing works
“The main advanced technologies apart from common settings [12] include AdamW [20], label smoothing [27], DropPath [18], and repeated augmentation [14].”
BackgroundManifold Regularized Dynamic Network Pruning
Tang, Y.-H., Wang, Y.-H., Xu, Y., Deng, Y., Xu, C., Tao, D., & Xu, C. (2021) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr46437.2021.00498 · Cited by 115 · see citing works
Self-citationReNAS: Relativistic Evaluation of Neural Architecture Search
Xu, Y., Wang, Y.-H., Han, K., Tang, Y.-H., Jui, S.-L., Xu, C.-J., & Xu, C. (2019) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr46437.2021.00439 · Cited by 112 · see citing works
Self-citationSegmenting Transparent Objects in the Wild with Transformer
Xie, E., Wang, W., Wang, W., Sun, P., Xu, H., Liang, D., & Luo, P. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/165 · Cited by 111 · see citing works
Grafit: Learning fine-grained image representations with coarse labels
Touvron, H., Sablayrolles, A., Douze, M., Cord, M., & Jeǵou, H. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00091 · Cited by 77 · cited 1× · see citing works
“Model Params (M) ImageNet CIFAR10 CIFAR100 Flowers Pets iNat-19 CNN-based Grafit ResNet-50 [32] 25.”
MethodModel Rubik's Cube: Twisting Resolution, Depth and Width for TinyNetsPreprint
Han, K., Wang, Y., Zhang, Q., Zhang, W., Xu, C., & Zhang, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.14819 · Cited by 63 · see citing works
Self-citationAugmented Shortcuts for Vision TransformersPreprint
Tang, Y.-H., Han, K., Xu, C., Xiao, A., Deng, Y., Xu, C., & Wang, Y.-H. (2021) · Neural Information Processing Systems · doi:10.48550/arxiv.2106.15941 · Cited by 55 · cited 1× · see citing works
“Nowadays, transformer architectures have been used in a growing number of computer vision tasks [10] such as image recognition [6, 40, 29], object detection [46], and segmentation [43, 38].”
BackgroundSelf-citationTrans2Seg: Transparent Object Segmentation with TransformerPreprint
Xie, E.-Z., Wang, W., Wang, W., Sun, P., Xu, H., Liang, D., & Luo, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.08461 · Cited by 27 · cited 3× · see citing works
“We adopt the segmentation framework of Trans2Seg [38] to build the pure transformer semantic segmentation based on TNT backbone.”
BackgroundMethodInfluential
Flagged first, then by citations
1 of the 56 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · see citing works
“Concurrent to our work are some that modify the ViT architecture [66, 15, 25] for better image classification.”
MethodPyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00061 · Cited by 5,259 · see citing works
Attention mechanisms in computer vision: A survey
Guo, M.-H., Xu, T.-X., Liu, J.-J., Liu, Z.-N., Jiang, P.-T., Mu, T.-J., Zhang, S.-H., Martin, R. R., Cheng, M.-M., & Hu, S.-M. (2022) · Computational Visual Media · doi:10.1007/s41095-022-0271-y · Cited by 2,566 · see citing works
“[8], [9], [15], [21], [31], [32], [34], [35] , [22], [26], [62], [63], [64], [65], [66], [67] , [41], [68], [69], [70], [71], [72], [73], [74] , [8], [34], [42], [43], [75], [76], [77], [78] , [27], [44], [45], [46], [79], [80], [81], [82] , [61], [83], [84], [85], [86], [87], [88], [89] , [47], [90], [91], [92], [93]…”
Pvt v2: Improved baselines with pyramid vision transformer
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2022) · Computational Visual Media · doi:10.1007/s41095-022-0274-8 · Cited by 2,439 · see citing works
MetaFormer is Actually What You Need for Vision
Yu, W., Luo, M., Zhou, P., Si, C., Zhou, Y., Wang, X., Feng, J., & Yan, S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01055 · Cited by 1,380 · see citing works
“Based on this common belief, many variants of the attention modules [13, 21, 55, 66] have been developed to improve the vision transformer.”
BackgroundCSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows
Dong, X., Bao, J., Chen, D., Zhang, W., Yu, N., Yuan, L., Chen, D., & Guo, B. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01181 · Cited by 1,458 · see citing works
A survey on sentiment analysis methods, applications, and challenges
Wankhade, M., Rao, A. C. S., & Kulkarni, C. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10144-1 · Cited by 1,728 · see citing works
“In the work of Han et al. (2021) Transformer network revolutionized the area of NLP and replaced the usage of LSTM and Bi-LSTM.”
MethodTransformers in Medical Imaging: A SurveyPreprint
Shamshad, F., Khan, S. H., Zamir, S. W., Khan, M. H., Hayat, M., Khan, F., & Fu, H. (2022) · Medical Image Anal. · doi:10.48550/arxiv.2201.09873 · Cited by 1,386 · see citing works
“Similarly, Jiang et al. [215] leverage Swin Transformer [126] and Transformer-in-Transformer [216] to classify COVID-19 images from Pneumonia and normal images.”
MethodLin, T., Wang, Y., Liu, X., & Qiu, X. (2022) · AI Open · doi:10.1016/j.aiopen.2022.10.001 · Cited by 1,422 · see citing works
Self-citationMaxViT: Multi-Axis Vision Transformer
Tu, Z., Talebi, H., Zhang, H., Yang, F., Milanfar, P., Bovik, A., & Li, Y.-X. (2022) · European Conference on Computer Vision · doi:10.48550/arxiv.2204.01697 · Cited by 1,218
“More recent works have been focused on improving model and data efficiency, including sparse attention [1,18,62,84,93], improved locality [24, 95], pyramidal designs [21, 85], improved training strategies [79, 80, 99], etc.”
BackgroundGuo, M.-H., Lu, C.-Z., Liu, Z.-N., Cheng, M., & Hu, S. (2023) · Computational Visual Media · doi:10.1007/s41095-023-0364-2 · Cited by 1,163 · see citing works
InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions
Wang, W., Dai, J., Chen, Z., Huang, Z., Li, Z., Zhu, X., Hu, X., Lu, T., Lu, L., Li, H., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01385 · Cited by 1,080 · see citing works
“With the remarkable success of transformers in large-scale language models [3–8], vision transformers (ViTs) [2, 9–15] have also swept the computer vision field and are becoming the primary choice for the research and practice of large-scale vision foundation models.”
BackgroundMViTv2: Improved Multiscale Vision Transformers for Classification and Detection
Li, Y., Wu, C.-Y., Fan, H., Mangalam, K., Xiong, B., Malik, J., & Feichtenhofer, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00476 · Cited by 837 · see citing works
CMT: Convolutional Neural Networks Meet Vision Transformers
Guo, J., Han, K., Wu, H., Tang, Y., Chen, X., Wang, Y., & Xu, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01186 · Cited by 1,005 · see citing works
“Since transformers achieved remarkable success in natural language processing (NLP) [9,58], many attempts [6,10,12,14,33,36,55–57,60,63,67] have been made to introduce transformer-like architectures to vision tasks.”
BackgroundMethodSelf-citationVision Transformer Adapter for Dense PredictionsPreprint
Chen, Z., Duan, Y., Wang, W., He, J.-J., Lu, T., Dai, J.-F., & Qiao, Y. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2205.08534 · Cited by 959 · see citing works
“Benefiting from the dynamic feature extraction capability and the long-term dependence of the attention mechanism, Vision Transformer (ViT) [22] and its variants [1,11,21,26,44,52,75] soon rose in many computer vision tasks such as object detection and semantic segmentation, surpassing CNN models and reaching state-of…”
BackgroundSegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J. M., & Luo, P. (2021) · arXiv · doi:10.48550/arxiv.2105.15203 · Cited by 5,090 · see citing works
Vision Transformer with Deformable Attention
Xia, Z., Pan, X., Song, S.-J., Li, L.-E.-R., & Huang, G. (2022) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52688.2022.00475 · Cited by 902 · see citing works
“This alternate design of attention blocks with local and global receptive fields helps the model learn strong representations, sharing a similar pattern in GLiT [5], TNT [15] and Point-DAT”
MethodFeature dimensionality reduction: a review
Jia, W., Sun, M., Lian, J., & Hou, S. (2022) · Complex & Intelligent Systems · doi:10.1007/s40747-021-00637-x · Cited by 797 · see citing works
“Other methods are designed such as Transformer in Transformer (TNT) [152], bottleneck Transformer (BoTNet) [153].”
Gold-YOLO: Efficient Object Detector via Gather-and-Distribute Mechanism
Wang, C.-C., He, W., Nie, Y., Guo, J.-Y., Liu, C., Han, K., & Wang, Y.-H. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2309.11331 · Cited by 779
“In the field of object detection, by using the encoder-decoder structure with the transformer, researchers have constructed a series of DETR-like object detection models, such as DETR [3] and DINO [43].”
MethodSelf-citationReview of Image Classification Algorithms Based on Convolutional Neural Networks
Chen, L., Li, S., Bai, Q., Yang, J., Jiang, S., & Miao, Y. (2021) · Remote Sensing · doi:10.3390/rs13224712 · Cited by 845 · see citing works
“Recently, more variant ViT models, e.g., DeiT [220], PVT [221], TNT [222], and Swin [223], have been proposed for the pursuit of stronger performance.”
BackgroundTouvron, H., Cord, M., & Jégou, H. (2022) · European Conference on Computer Vision · doi:10.48550/arxiv.2204.07118 · Cited by 733
Contextual Transformer Networks for Visual Recognition
Li, Y., Yao, T., Pan, Y., & Mei, T. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3164083 · Cited by 771 · see citing works
“Different from the conventional representation scheme adopted in ViT [14] that solely divides input image into patches, [32] first divides the inputs into several patches as “visual sentences”, and then divides them into sub-patches as “visual words”.”
MethodVision GNN: An Image is Worth Graph of NodesPreprint
Han, K., Wang, Y.-H., Guo, J.-Y., Tang, Y.-H., & Wu, E. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2206.00272 · Cited by 689 · see citing works
“The main improvements include pyramid architecture [54, 33], local attention [14, 33] and position encoding [58].”
BackgroundSelf-citationGhostNetV2: Enhance Cheap Operation with Long-Range AttentionPreprint
Tang, Y.-H., Han, K., Guo, J.-Y., Xu, C., Xu, C., & Wang, Y.-H. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2211.12905 · Cited by 660 · see citing works
“Originating from the NLP field [36], attention-based models are introduced to computer vision recently [6, 9, 34, 7].”
MethodSelf-citationPolyp-PVT: Polyp Segmentation with Pyramid Vision Transformers
Dong, B., Wang, W., Fan, D.-P., Li, J., Fu, H., & Shao, L. (2021) · CAAI Artificial Intelligence Research · doi:10.26599/air.2023.9150015 · Cited by 643 · see citing works
“TNT [43] employs a transformer suitable for fine-grained image tasks, which further divides the original image patch and conducts self-attention mechanism calculations in smaller units.”
MethodScaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning
Chen, R. J., Chen, C., Li, Y., Chen, T. Y., Trister, A. D., Krishnan, R. G., & Mahmood, F. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01567 · Cited by 621 · see citing works
“- Swin, ViL, TNT, PVT, MViT) in addressing the varying scale / aspect ratios of visual tokens [27, 31, 51, 72, 81].”
BackgroundMambaVision: A Hybrid Mamba-Transformer Vision Backbone
Hatamizadeh, A., & Kautz, J. (2024) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52734.2025.02352 · Cited by 590 · see citing works
A Survey of Visual Transformers
Liu, Y., Zhang, Y., Wang, Y., Hou, F., Yuan, J. S., Tian, J., Zhang, Y. S., Shi, Z., Jianping Fan, B., & He, Z. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3227717 · Cited by 614 · see citing works
“Twins [60] employs a spatially separable self-attention mechanism, similar to depthwise convolution [172] or windowwise TNT [59], to model the local–global representation.”
BackgroundMethodInfluentialSelf-citationFocal Self-attention for Local-Global Interactions in Vision TransformersPreprint
Yang, J., Li, C., Zhang, P., Dai, X., Xiao, B., Yuan, L., & Gao, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.00641 · Cited by 389 · see citing works
A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks
Islam, S., Elmekki, H., Elsebai, A., Bentahar, J., Drawel, N., Rjoub, G., & Pedrycz, W. (2023) · Expert systems with applications · doi:10.48550/arxiv.2306.07303 · Cited by 543
“• TNT: Transformer in Transformers (TNT) is a transformer-based computer vision model that uses a transformer model inside another transformer model to capture features inside local patches of an image (Han et al., 2021).”
BackgroundMethodInfluentialDaViT: Dual Attention Vision Transformers
Ding, M., Xiao, B., Codella, N., Luo, P., Wang, J.-D., & Yuan, L. (2022) · European Conference on Computer Vision · doi:10.48550/arxiv.2204.03645 · Cited by 440
MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation
Li, W., Liu, H., Tang, H., Wang, P., & Gool, L. V. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01280 · Cited by 465 · see citing works
“Recently, Transformer [39] equipped with the powerfully global self-attention mechanism has received increasingly research interest in the computer vision community [10, 11, 23, 28, 46].”
BackgroundEarthformer: Exploring Space-Time Transformers for Earth System ForecastingPreprint
Gao, Z., Li, M., Shi, X., Wang, H., Wang, Y., Yeung, D.-Y., & Zhu, Y. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2207.05833 · Cited by 431 · see citing works
“Alarms (truth=0, pred=1), the prediction and the ground-truth are rescaled back to the range 0-255 and binarized at thresholds [16 , 74 , 133 , 160 , 181 , 219] .”
MethodSelf-citationInceptionNeXt: When Inception Meets ConvNeXt
Yu, W., Zhou, P., Yan, S., & Wang, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00542 · Cited by 475 · see citing works
“Follow-up works [20, 52, 53, 64, 67, 69, 79] like Swin [39] continually improve model performance, achieving new state-of-the-art on various vision tasks.”
BackgroundCrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention
Wang, W.-X., Yao, L., Chen, L., Lin, B., Cai, D., He, X.-F., & Liu, W. (2021) · International Conference on Learning Representations · doi:10.48550/arxiv.2108.00154 · Cited by 398
“…are all competitive vision transformers, including DeiT (Touvron et al., 2021), PVT (Wang et al., 2021), T2T-ViT (Yuan et al., 2021), TNT (Han et al., 2021), CViT (Chen et al., 2021a), Twins (Chu et al., 2021), Swin (Liu et al., 2021b), NesT (Zhang et al., 2021b), CvT (Wu et al., 2021),…”
BackgroundMetaFormer Baselines for Vision
Yu, W., Si, C., Zhou, P., Luo, M., Zhou, Y., Feng, J., Yan, S., & Wang, X. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3329173 · Cited by 355 · see citing works
“As such, many attention-based token mixers [82, 21, 73, 45, 88] have been proposed in the aim to strengthen the Vision Transformers (ViTs) [17].”
BackgroundMPViT: Multi-Path Vision Transformer for Dense Prediction
Lee, Y., Kim, J., Willette, J., & Hwang, S. J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00714 · Cited by 388 · see citing works
“We provide a full summary of comparisons on ImageNet-1K classification in Table 8 by adding more recent Vision Transformers including ViL [71], TnT [21], ViTAE [66], HRFormer [69], and Twins [10].”
MethodDilateFormer: Multi-Scale Dilated Transformer for Visual Recognition
Jiao, J., Tang, Y.-M., Lin, K.-Y., Gao, Y., Ma, A. J., Wang, Y., & Zheng, W. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3243616 · Cited by 412 · see citing works
Self-citationSemi-Supervised Medical Image Segmentation via Cross Teaching between CNN and Transformer
Luo, X.-D., Hu, M., Song, T., Wang, G.-T., & Zhang, S.-T. (2021) · International Conference on Medical Imaging with Deep Learning · doi:10.48550/arxiv.2112.04894 · Cited by 349
“After that, many variants of vision transformers achieved great success in natural image recognition tasks, like Swin-Transformer (Liu et al., 2021), DieT (Touvron et al., 2021), PVT (Wang et al., 2021b), TiT (Han et al., 2021), etc. Benefiting from the great representation capacity of transformers, several works attem…”
BackgroundPhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer
Yu, Z., Shen, Y., Shi, J.-G., Zhao, H., Torr, P., & Zhao, G.-Y. (2021) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52688.2022.00415 · Cited by 349 · see citing works
StyleSwin: Transformer-based GAN for High-resolution Image Generation
Zhang, B., Gu, S., Zhang, B., Bao, J., Chen, D., Wen, F., Wang, Y., & Guo, B. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01102 · Cited by 264 · see citing works
“Later, transformers dominate the benchmarks in a broad of discriminative tasks [10,17,43,53,56,59,60,64].”
BackgroundSelf-citationShunted Self-Attention via Multi-Scale Token Aggregation
Ren, S., Zhou, D., He, S., Feng, J., & Wang, X. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01058 · Cited by 361 · see citing works
EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformers
Pan, J.-T., Bulat, A., Tan, F.-W., Zhu, X.-T., Dudziak, L., Li, H., Tzimiropoulos, G., & Martínez, B. (2022) · European Conference on Computer Vision · doi:10.48550/arxiv.2205.03436 · Cited by 328
“, the token sequence length) progressively down-sampled across the stages [12,55,7], (2) the locally-grouped self-attention mechanisms for controlling the length of input token sequences and parameter sharing [31,16], (3) the pooling attention schemes to subsample the key and value by a factor [54,35,60,55].”
BackgroundMethodFast Vision Transformers with HiLo AttentionPreprint
Pan, Z.-Z., Cai, J.-F., & Zhuang, B.-H. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2205.13213 · Cited by 327 · see citing works
“To address this problem, TNT [21] introduced additional global tokens and MixFormer [6] mixed local window attention with depthwise convolutional layers.”
MethodViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond
Zhang, Q., Xu, Y., Zhang, J., & Tao, D. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-022-01739-w · Cited by 465 · see citing works
“Recently, some works try to introduce the intrinsic IB of CNNs into vision transformers explicitly [26,58,25,43,18,86, 80,91,9,49].”
BackgroundDynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
Rao, Y., Zhao, W., Liu, B., Lu, J., Zhou, J., & Hsieh, C.-J. (2021) · arXiv · doi:10.48550/arxiv.2106.02034 · Cited by 448 · see citing works
CDTrans: Cross-domain Transformer for Unsupervised Domain AdaptationPreprint
Xu, T., Chen, W., Wang, P., Wang, F., Li, H., & Jin, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.06165 · Cited by 195 · see citing works
“…that introduces a distillation strategy for transformers to help with ViT training; many other ViT variants (Yuan et al., 2021a; Wang et al., 2021; Han et al., 2021a; Chen et al., 2021a; Ranftl et al., 2021; Liu et al., 2021) are proposed from then, which achieve promising performance compared…”
BackgroundAdaptive Token Sampling for Efficient Vision Transformers
Fayyaz, M., Koohpayegani, S. A., Jafari, F., Sengupta, S., Joze, H. R. V., Sommerlade, E., Pirsiavash, H., & Gall, J. (2021) · European Conference on Computer Vision · doi:10.1007/978-3-031-20083-0_24 · Cited by 307 · see citing works
More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using SparsityPreprint
Liu, S., Chen, T.-L., Chen, X.-H., Chen, X.-X., Xiao, Q., Wu, B.-Q., Kärkkäinen, T., Pechenizkiy, M., Mocanu, D., & Wang, Z.-Y. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2207.03620 · Cited by 303 · see citing works
“Middle: ERF of ConvNeXt-T constructed with kernel sizes of [31, 29, 27, 13] trained with the structural re-parameterization used in RepLKNet [17].”
MethodConv2Former: A Simple Transformer-Style ConvNet for Visual Recognition
Hou, Q., Lu, C.-Z., Cheng, M., & Feng, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3401450 · Cited by 256 · see citing works
“Some works [39], [40], [41], [42], [43], [44] propose to introduce local dependencies into ViTs, showing great performance in visual recognition.”
BackgroundLarge Language Models in Law: A Survey
Lai, J., Gan, W.-S., Wu, J.-Y., Qi, Z.-L., & Yu, P. S. (2023) · AI Open · doi:10.48550/arxiv.2312.03718 · Cited by 280
SwinJSCC: Taming Swin Transformer for Deep Joint Source-Channel Coding
Yang, K., Wang, S., Dai, J., Qin, X., Niu, K., & Zhang, P. (2024) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2024.3424842 · Cited by 227 · see citing works
“Our work on deep JSCC using Transformer-based architectures is most closely related to the Vision Transformer (ViT) [15] and its derivatives [34], [35].”
RMT: Retentive Networks Meet Vision Transformers
Fan, Q., Huang, H., Chen, M., Liu, H., & He, R. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00539 · Cited by 276 · see citing works
“By splitting the image into small, non-overlapped patches sequence, Vision Transformer (ViTs) [12] also have attracted great attention and become widely used on vision tasks [5, 16, 20, 42, 58, 66].”
BackgroundWave-ViT: Unifying Wavelet and Transformers for Visual Representation Learning
Yao, T., Pan, Y.-W., Li, Y.-H., Ngo, C., & Mei, T. (2022) · European Conference on Computer Vision · doi:10.48550/arxiv.2207.04978 · Cited by 273
“Different from ViT that solely divides input image into patches, TNT [19] first decomposes the inputs into several patches as “visual sentences”, and then divides them into sub-patches as “visual words”.”
MethodAdaViT: Adaptive Vision Transformers for Efficient Image Recognition
Meng, L., Li, H., Chen, B.-C., Lan, S., Wu, Z., Jiang, Y.-G., & Lim, S.-N. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01199 · Cited by 262 · see citing works
“While obtaining superior performance, the computational cost of vision transformers is still intensive and scales up quickly as the numbers of patches, self-attention heads and transformer blocks increase.”
BackgroundPark, C., Jeong, Y., Cho, M., & Park, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01644 · Cited by 240 · see citing works
“Moreover, recent vision work [3, 9, 12, 43] has attempted to exploit the advantages of attention-based models.”
BackgroundReview of Large Vision Models and Visual Prompt EngineeringPreprint
Wang, J., Liu, Z., Zhao, L., Wu, Z., Ma, C., Yu, S., Dai, H.-X., Yang, Q., Liu, Y.-H., Zhang, S.-Y., et al. (2023) · Meta-Radiology · doi:10.48550/arxiv.2307.00855 · Cited by 252 · see citing works
“Subsequent endeavors in computer vision began to improve and extend ViT model, such as DeiT [73], Swin Transformer [74], TNT [75], MAE [76], MoCo-v3 [77], BeiT [78], etc.”
MethodVision-Language Models in Remote Sensing: Current progress and future trends
Xiang, L., Wen, C., Hu, Y., Yuan, Z., & Zhu, X. X. (2024) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2024.3383473 · Cited by 199 · see citing works
“Several variants of the ViT architecture have been proposed, including DeiT [145], TNT [44], and PVT [156], which further improved performance by incorporating techniques such as distillation, token mixing, and pyramid vision transformer.”
MethodClassification of Brain Tumor from Magnetic Resonance Imaging Using Vision Transformers Ensembling
Tummala, S., Kadry, S., Bukhari, S. A. C., & Rauf, H. T. (2022) · Current Oncology · doi:10.3390/curroncol29100590 · Cited by 244 · see citing works
“In addition, in a future study, it could be interesting to investigate the ability of other vision transformer variants, such as swin vision transformers [45], data-efficient vision transformers [46], and transformer in transformer models [47], for the brain tumor classification from MRI.”
BackgroundCTformer: convolution-free Token2Token dilated vision transformer for low-dose CT denoising
Wang, D., Fan, F.-L., Wu, Z., Liu, R., Wang, F., & Yu, H. (2022) · Physics in Medicine and Biology · doi:10.1088/1361-6560/acc000 · Cited by 241 · see citing works
MiM-ISTD: Mamba-in-Mamba for Efficient Infrared Small-Target Detection
Chen, T.-X., Ye, Z., Tan, Z., Gong, T., Wu, Y., Chu, Q., Liu, B., Yu, N.-H., & Ye, J.-P. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3485721 · Cited by 237 · see citing works
“To improve network efficiency while maintaining high accuracy, we draw inspiration from both Mamba [30] and TNT [35] and propose a Mamba-in-Mamba (MiM) architecture for ISTD, which contains an MiM hierarchical encoder that implements efficient local and global feature extraction.”
Large AI Models in Health Informatics: Applications, Challenges, and the Future
Qiu, J., Li, L., Sun, J., Peng, J., Shi, P., Zhang, R., Dong, Y., Lam, K., Lo, F. P.-W., Xiao, B. J., et al. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3316750 · Cited by 246 · see citing works
“Later, works like TNT [41] and Swin Transformer [42] were proposed to better adapt transformers to visual data.”
MethodQuadTree Attention for Vision TransformersPreprint
Tang, S., Zhang, J., Zhu, S., & Tan, P. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.02767 · Cited by 121 · see citing works
“…(Radosavovic et al., 2020) 21.0 4.0 80.0 DeiT-Small/16 (Touvron et al., 2021) 22.1 4.6 79.9 Swin-T (Liu et al., 2021) 29.0 4.5 81.3 TNT-S (Han et al., 2021) 23.8 5.2 81.3 CeiT (Yuan et al., 2021a) 24.2 4.5 82.0 PVTv2-b2 (Wang et al., 2021c) 25.4 4.0 82.0 Focal-T (Yang et al., 2021) 29.1…”
InfluentialA Comprehensive Survey of Convolutions in Deep Learning: Applications, Challenges, and Future Trends
Younesi, A., Ansari, M., Fazli, M., Ejlali, A., Shafique, M., & Henkel, J. (2024) · IEEE Access · doi:10.1109/access.2024.3376441 · Cited by 228 · see citing works
“…long-range dependencies and multi-scale contextual information more effectively than CNNs. Models like ViT, DeiT, and Visual BERT demonstrate state-of-the-art results in tasks like image classification when pre-trained on large datasets [347], [348], [349], [350], [351], [352], [353], [354].”
BackgroundA systematic review for transformer-based long-term series forecasting
Su, L., Zuo, X., Li, R., Wang, X., Zhao, H., & Huang, B. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-024-11044-2 · Cited by 219 · see citing works
“In the context of modeling text-related data, the initial step involves its vectorization.”
AS-MLP: An Axial Shifted MLP Architecture for VisionPreprint
Lian, D., Yu, Z., Sun, X., & Gao, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.08391 · Cited by 172 · see citing works
Inferring super-resolution tissue architecture by integrating spatial transcriptomics with histology
Zhang, D., Schroeder, A., Yan, H., Yang, H., Hu, J., Lee, M. Y. Y., Cho, K. S., Susztak, K., Xu, G. X., Feldman, M. D., et al. (2024) · Nature Biotechnology · doi:10.1038/s41587-023-02019-9 · Cited by 219 · see citing works
“23 and 24).”
Gao, L., Liu, H., Yang, M.-H., Chen, L., Wan, Y., Xiao, Z., & Qian, Y. (2021) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.36227/techrxiv.14866185 · Cited by 219
“Due to the limited amount of remote sensing image data, the TNT model [34] has poor experimental results.”
MethodSHViT: Single-Head Vision Transformer with Memory Efficient Macro Design
Yun, S., & Ro, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00550 · Cited by 208 · see citing works
One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation
Hao, Z.-W., Guo, J.-Y., Han, K., Tang, Y.-H., Hu, H., Wang, Y.-H., & Xu, C. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2310.19444 · Cited by 213
“Vision transformers [11, 21, 47] inherited from the NLP field are attention-based models.”
BackgroundSelf-citationA survey: object detection methods from CNN to transformer
Arkin, E., Yadikar, N., Xu, X., Aysa, A., & Ubul, K. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13801-3 · Cited by 232 · see citing works
“Therefore, some researchers have proposed new solutions [27] to this problem.”
BackgroundMethodCoarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction
Lin, J., Cai, Y.-Y., Hu, X., Wang, H.-Q., Yuan, X., Zhang, Y.-L., Timofte, R., & Gool, L. V. (2022) · European Conference on Computer Vision · doi:10.48550/arxiv.2203.04845 · Cited by 209
“Vision Transformer has been widely applied in image classification [2,13,21,22,31,42,72,85,86], object detection [11,18,19,24,67,68,92,107], semantic segmentation [16,51,54,74,87,91,96,106], human pose estimation [7,38,43,45,48,55,59,105], and so on.”
MethodCamoFormer: Masked Separable Attention for Camouflaged Object Detection
Yin, B., Zhang, X.-Y., Fan, D.-P., Jiao, S.-H., Cheng, M.-M., Gool, L. V., & Hou, Q. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3438565 · Cited by 208 · see citing works
“Transformer architectures are initially designed for nat-ural language processing [6, 52] and then get popular in computer vision [4,16,47,56,59,61].”
YOLO-SE: Improved YOLOv8 for Remote Sensing Object Detection and Recognition
Wu, T.-Y., & Dong, Y. (2023) · Applied Sciences · doi:10.3390/app132412977 · Cited by 204 · see citing works
“TNT [34], for instance, divides patches into multiple sub-patches and introduces a novel transformer-in-transformer architecture.”
Patch Slimming for Efficient Vision Transformers
Tang, Y., Han, K., Wang, Y., Xu, C., Guo, J., Xu, C., & Tao, D. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01185 · Cited by 201 · see citing works
Self-citationVision Transformers for Image Classification: A Comparative Survey
Wang, Y., Deng, Y., Zheng, Y.-J., Chattopadhyay, P., & Wang, L.-P. (2025) · Technologies · doi:10.3390/technologies13010032 · Cited by 202 · see citing works
“This approach led to the development of complex transformer models and mechanisms, including the TNT [76], HaloNets [31], and Swin Transformer [77].”
Self-citationJoint Spatio-Temporal Modeling for Semantic Change Detection in Remote Sensing Images
Ding, L., Zhang, J., Guo, H.-T., Zhang, K., Liu, B., & Bruzzone, L. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3362795 · Cited by 192 · see citing works
“The vision transformers have been first used in high-level recognition tasks such as image classification [22], [23] and object detection [24].”
MethodMiniViT: Compressing Vision Transformers with Weight Multiplexing
Zhang, J., Peng, H., Wu, K., Liu, M., Xiao, B., Fu, J., & Yuan, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01183 · Cited by 130 · see citing works
SpectFormer: Frequency and Attention is what you need in a Vision Transformer
Patro, B. N., Namboodiri, V. P., & Agneeswaran, V. S. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00924 · Cited by 171 · see citing works
“Method GFL [10] 44.5 63.0 48.3 Swin-T [13] 47”
InfluentialMogaNet: Multi-order Gated Aggregation Network
Li, S.-Y., Wang, Z., Liu, Z.-C., Tan, C., Lin, H., Wu, D., Chen, Z., Zheng, J.-B., & Li, S. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2211.03295 · Cited by 186
“To resolve this problem, endeavors have been made by reintroducing locality priors (Wu et al., 2021a; Dai et al., 2021; Han et al., 2021a; Li et al., 2022a; Chen et al., 2022) and pyramid-like hier-∗ archical layouts (Liu et al., 2021; Fan et al., 2021; Wang et al., 2021b) to ViTs, albeit at the…”
Three things everyone should know about Vision Transformers
Touvron, H., Cord, M., El-Nouby, A., Verbeek, J., & Jégou, H. (2022) · European Conference on Computer Vision · doi:10.48550/arxiv.2203.09795 · Cited by 182
“Some recent hybrid architectures [18, 21, 23] preprocess their input images with a convolutional stem, to improve accuracy and training stability [71].”
BackgroundTowards Transferable Adversarial Attacks on Vision Transformers
Wei, Z.-P., Chen, J.-J., Goldblum, M., Wu, Z.-X., Goldstein, T., & Jiang, Y.-G. (2021) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2109.04176 · Cited by 182
“We average the attack success rate over numerous black-box models: DeiT-B, LeViT-256, CaiT-S-24, PiT-B, ConViT-B, TNT-S and Visformer-S. and SGM (Wu et al. 2020a”
MethodInfluentialD-former: a U-shaped Dilated Transformer for 3D medical image segmentation
Wu, Y., Liao, K.-Y., Chen, J.-T., Wang, J., Chen, D., Gao, H., & Wu, J. (2022) · Neural computing & applications (Print) · doi:10.1007/s00521-022-07859-1 · Cited by 179 · see citing works
“There were different kinds of adaptations for vanilla Transformer, such as Swin Transformer [46], pyramid vision Transformer [45], Transformer in Transformer [71], and aggregating nested Transformers [15].”
MethodHRFormer: High-Resolution Transformer for Dense Prediction
Yuan, Y., Fu, R., Huang, L., Lin, W., Zhang, C., Chen, X., & Wang, J. (2021) · arXiv · doi:10.48550/arxiv.2110.09408 · Cited by 178 · see citing works
“They have shown that increasing the spatial resolution of the representations output by the Vision Transformer is important for semantic segmen- tation.”
MethodInfluentialLiang, H.-X., Fan, Z.-W., Sarkar, R., Jiang, Z.-Y., Chen, T.-L., Zou, K., Cheng, Y., Hao, C., & Wang, Z.-Y. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2210.14793 · Cited by 178 · see citing works
“Vision Transformer There are growing interests in exploring the use of transformers [73, 3] for computer vision tasks since its success in the natural language processing [73, 74, 75], including image generation [76, 77], generative adversarial networks [78, 79], image classification [76, 3, 80, 81, 82, 83, 81, 84], s…”
BackgroundAn Image Patch is a Wave: Phase-Aware Vision MLP
Tang, Y.-H., Han, K., Guo, J.-Y., Xu, C., Li, Y., Xu, C., & Wang, Y.-H. (2021) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52688.2022.01066 · Cited by 177 · see citing works
“[17] present a nested transformer architecture to capture global and local information simultaneously.”
BackgroundSelf-citationDuAT: Dual-Aggregation Transformer Network for Medical Image Segmentation
Tang, F., Xu, Z.-X., Huang, Q., Wang, J., Hou, X.-X., Su, J., & Liu, J.-X. (2022) · Chinese Conference on Pattern Recognition and Computer Vision · doi:10.48550/arxiv.2212.11677 · Cited by 175
“Vision Transformer-based methods [1, 8, 9, 19, 40, 58, 46] have also been proposed and demonstrate promising performance.”
MethodSPViT: Enabling Faster Vision Transformers via Latency-Aware Soft Token Pruning
Kong, Z., Dong, P., Ma, X., Meng, X., Niu, W., Sun, M., Shen, X., Yuan, G., Ren, B., Tang, H., et al. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20083-0_37 · Cited by 169 · see citing works
“We compare our method with several representative methods including DynamicViT [69], IA-RED2 [64], RegNetY [67], CrossViT [7], VTP [112], ATS [28], CvT [85], PVT [83], T2T-ViT [104], UP-DeiT [95], PS-ViT [78], Evo-ViT [89], TNT [34], HVT [65], Swin [49], CoaT [88], CPVT [18], EViT [46], UVC [97], MD-DeiT [39],and S2Vi…”
BackgroundMethodInfluentialSelf-citationDeep learning for accelerated and robust MRI reconstruction
Heckel, R., Jacob, M., Chaudhari, A. S., Perlman, O., & Shimron, E. (2024) · Magnetic Resonance Materials in Physics Biology and Medicine · doi:10.1007/s10334-024-01173-8 · Cited by 144 · see citing works
“Transformers [149, 150] have powerful computational capabilities due to their use of an attention mechanism [151] that makes it possible to weigh the importance of different parts of the input data and capture long-range dependencies.”
BackgroundFasterViT: Fast Vision Transformers with Hierarchical Attention
Hatamizadeh, A., Heinrich, G., Yin, H., Tao, A., Álvarez, J., Kautz, J., & Molchanov, P. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2306.06189 · Cited by 167
Lite Vision Transformer with Enhanced Self-Attention
Yang, C., Wang, Y., Zhang, J., Zhang, H., Wei, Z., Lin, Z., & Yuille, A. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01169 · Cited by 164 · see citing works
“TNT [20] further splits tokens into smaller ones, extracts their features and integrates them with normal token features.”
BackgroundInfluentialSelf-citationConvMAE: Masked Convolution Meets Masked AutoencodersPreprint
Gao, P., Ma, T., Li, H., Lin, Z.-Y., Dai, J.-F., & Qiao, Y. (2022) · arXiv.org · doi:10.48550/arxiv.2205.03892 · Cited by 164 · see citing works
“Such successes have also been observed in vision backbone design [25, 63, 53, 17, 58, 24].”
BackgroundYu, Z., Shen, Y., Shi, J.-G., Zhao, H., Cui, Y., Zhang, J., Torr, P., & Zhao, G.-Y. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01758-1 · Cited by 161 · see citing works
“Many other ViT variants [Han et al., 2020, Khan et al., 2021, Touvron et al., 2021, Liu et al., 2021e, Yuan et al., 2021, Wang et al., 2021b, Han et al., 2021, Chen et al., 2021a, Ding et al., 2021] are proposed from then, which achieve promising performance compared with its counterpart CNNs for…”
BackgroundImDiffusion: Imputed Diffusion Models for Multivariate Time Series Anomaly DetectionPreprint
Chen, Y.-H., Zhang, C., Ma, M.-H., Liu, Y., Ding, R., Li, B., He, S., Rajmohan, S., Lin, Q., & Zhang, D.-M. (2023) · Proceedings of the VLDB Endowment · doi:10.48550/arxiv.2307.00754 · Cited by 160 · see citing works
Ying, X., Liu, L., Wang, Y., Li, R., Chen, N., Lin, Z., Sheng, W., & Zhou, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01490 · Cited by 140 · see citing works
“Zhang et al. [52, 53] explored cross-level correlation and transformer-based method [18] to predict accurate target mask.”
MethodSelf-citationShen, W., Peng, Z., Wang, X., Wang, H., Cen, J., Jiang, D., Xie, L., Yang, X., & Tian, Q. C. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3246102 · Cited by 153 · see citing works
“The emergence of vision transformer (ViT) [167] and its variants [168], [169], [170], [171]”
BackgroundRevisiting Classifier: Transferring Vision-Language Models for Video Recognition
Wu, W., Sun, Z., & Ouyang, W. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i3.25386 · Cited by 112 · see citing works
MixFormer: Mixing Features across Windows and Dimensions
Chen, Q., Wu, Q., Wang, J., Hu, Q., Hu, T., Ding, E., Cheng, J., & Wang, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00518 · Cited by 165 · see citing works
“After that, various methods [2, 14, 25, 36, 51, 56] are proposed to improve the performance of vision transformers, demonstrating competitive results on the image classification task.”
MethodJoint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers
Wei, S., Ye, T., Zhang, S., Tang, Y., & Liang, J. (2023) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52729.2023.00208 · Cited by 152 · see citing works
“Comparison between DeiT & LV-ViT with our TPS applied and other transformer methods, including token pruning [8,9,16,20, 20,25,35], vanilla ViTs [2,13,23,28,38,39] and hybrid ViTs [17,31,34,40].”
MethodPMVT: a lightweight vision transformer for plant disease identification on mobile devices
Li, G., Wang, Y., Zhao, Q. Q., Yuan, P., & Chang, B. (2023) · Frontiers in Plant Science · doi:10.3389/fpls.2023.1256773 · Cited by 139 · see citing works
“We also chose many heavyweight networks such as PoolFormer (Yu et al., 2022b), CVT (Wu et al., 2021), TNT (Han et al., 2021), and ResNet (He et al., 2016) for comparison.”
BackgroundMethodSelf-citation
- Authors
- Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., & Wang, Y.
- Affiliations
- Huawei Technologies (China) (CN)
- University of Macau (MO)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Volume
- 34
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2103.00112
- OpenAlex
- W3133696297
- Cited by
- 1,394
- References
- 56
- Citations by year
- 2026: 14 · 2025: 276 · 2024: 457 · 2023: 387 · 2022: 186 · 2021: 74
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (0015/2019/AKP, 62072449, 61632003, 0018/2019/AKP)
- Fundo para o Desenvolvimento das Ciências e da Tecnologia (0018/2019/AKP, 0015/2019/AKP)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- vision transformer, attention mechanism, local patches, visual words, image classification, feature aggregation, computational efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
