The DOI matches its DataCite record, with a small difference in title or year; 56 of the 56 references it cites were found in a registry or an index.
Transformer in Transformer
Open accessAbstract
Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (e.g., 16$\times$16) as "visual sentences" and present to further divide them into smaller patches (e.g., 4$\times$4) as "visual words". The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs. Features of both words and sentences will be aggregated to enhance the representation ability. Experiments on several benchmarks demonstrate the effectiveness of the proposed TNT architecture, e.g., we achieve an 81.5% top-1 accuracy on the ImageNet, which is about 1.7% higher than that of the state-of-the-art visual transformer with similar computational cost. The PyTorch code is available at https://github.com/huawei-noah/CV-Backbones, and the MindSpore code is available at https://gitee.com/mindspore/models/tree/master/research/cv/TNT.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For example, in multi-tenant architectures, system prompts are shared across multiple requests and frequently contain detailed instructions or examples to guide the model [19].”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [19]
In the citing paper
“PromptCache [19] proposes modular reuse beyond prefixes but sacrifices accuracy, with reported performance drops of up to 43% [72], making it unsuitable for tasks requiring high precision.”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [19]
In the citing paper
“related experiments results further demonstrate ViT also has remarkable representation capacity when dealing with images and videos [19].”
From Deepfake Video Detection with Spatiotemporal Dropout Transformer (2022) · cites this paper as [19]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 6×
“Compared to CNNs, TNT can outperform the widely-used ResNet and RegNet.”
BackgroundMethodInfluentialVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3×
“The effect of #heads in standard transformer has been investigated in multiple works [21, 35] and a head width of 64 is recommended for visual tasks [9, 31].”
BackgroundMethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“In the past decade, the mainstream deep neural architectures used in the computer vision (CV) are mainly established on convolutional neural networks (CNNs) [17, 12, 11].”
BackgroundFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472 · cited 1×
“As a general backbone network, TNT can also be applied with multi-scale vision models like Faster RCNN [29].”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472 · cited 7×
“The recent transformerbased models like ViT [9] and DeiT [31] are compared.”
MethodInfluentialImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 3×
“ImageNet ILSVRC 2012 [26] is an image classification benchmark consisting of 1.”
BackgroundInfluentialSqueeze-and-Excitation NetworksPreprint
Hu, J., Shen, L., & Sun, G. (2017) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1709.01507 · Cited by 37,845 · cited 1×
“Inspired by squeeze-and-excitation (SE) network for CNNs [16], we propose to explore channel-wise attention for transformers.”
MethodPyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939 · cited 2×
“We evaluate TNT-S and DeiT-S on Faster RCNN with FPN [21].”
BackgroundMethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2×
“, the famous BERT [8] and GPT-3 [2] models.”
MethodEnd-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253 · cited 4×
“We construct a pure transformer object detection pipeline by combining our TNT and DETR [3].”
MethodInfluentialEfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072 · cited 2×
“Note that all the transformer-based models are still inferior to EfficientNet which utilizes special depth-wise convolutions, so it is yet a challenge of how to beat EfficientNet using pure transformer.”
MethodObject recognition from local scale-invariant features
Lowe, D. G. (1999) · Proceedings of the Seventh IEEE International Conference on Computer Vision · doi:10.1109/iccv.1999.790410 · Cited by 16,472 · cited 1×
“visual transformers ignore the local relation and structure information inside the patch which is important for visual recognition [22, 3].”
BackgroundLayer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762 · cited 1×
“LN. Layer normalization [1] is a key part in transformer for stable training and faster convergence.”
BackgroundSqueeze-and-Excitation Networks
Hu, J., Shen, L., Albanie, S., Sun, G., & Wu, E. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2913372 · Cited by 12,157
Self-citationWang, X., Girshick, R., Gupta, A., & He, K. (2017) · arXiv · Cited by 11,689 · cited 1×
“explore self-attention mechanism in non-local networks [37] for capturing long-range dependencies in video and image recognition.”
BackgroundWang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596 · cited 1×
“explore self-attention mechanism in non-local networks [37] for capturing long-range dependencies in video and image recognition.”
BackgroundDeformable DETR: Deformable Transformers for End-to-End Object Detection
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., & Dai, J. (2020) · arXiv · doi:10.48550/arxiv.2010.04159 · Cited by 6,027 · cited 1×
“Nowadays, transformer architectures have been used in a growing number of computer vision tasks [10] such as image recognition [6, 40, 29], object detection [46], and segmentation [43, 38].”
Backgroundmixup: Beyond Empirical Risk Minimization
Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2017) · arXiv · doi:10.48550/arxiv.1710.09412 · Cited by 5,831 · cited 1×
“We adopt the same data augmentation strategy as that in DeiT [31] including random crop, random clip, Rand-Augment [7], Random Erasing [44], Mixup [42] and CutMix [41].”
MethodCutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features
Yun, S., Han, D., Chun, S., Oh, S. J., Yoo, Y., & Choe, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00612 · Cited by 5,400 · cited 1×
“We adopt the same data augmentation strategy as that in DeiT [31] including random crop, random clip, Rand-Augment [7], Random Erasing [44], Mixup [42] and CutMix [41].”
MethodPyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00061 · Cited by 5,259 · cited 4×
“For fair comparison, we adopt the training and testing settings in PVT [36] and add a 2×2 average pooling to make the output size of TNT backbone the same as that of PVT and ResNet.”
BackgroundMethodInfluentialHan (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682 · cited 1×
“Nowadays, transformer architectures have been used in a growing number of computer vision tasks [10] such as image recognition [6, 40, 29], object detection [46], and segmentation [43, 38].”
BackgroundSelf-citationGhostNet: More Features From Cheap OperationsPreprint
Han, K., Wang, Y.-H., Tian, Q., Guo, J.-Y., Xu, C.-J., & Xu, C. (2019) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1911.11907 · Cited by 4,540
Self-citationRethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P. H. S., et al. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00681 · Cited by 4,008 · cited 1×
“Nowadays, transformer architectures have been used in a growing number of computer vision tasks [10] such as image recognition [6, 40, 29], object detection [46], and segmentation [43, 38].”
BackgroundSelf-citationRandom Erasing Data Augmentation
Zhong, Z., Zheng, L., Kang, G., Li, S., & Yang, Y. (2017) · arXiv · doi:10.48550/arxiv.1708.04896 · Cited by 3,617 · cited 1×
“We adopt the same data augmentation strategy as that in DeiT [31] including random crop, random clip, Rand-Augment [7], Random Erasing [44], Mixup [42] and CutMix [41].”
MethodRandom Erasing Data Augmentation
Zhong, Z., Zheng, L., Kang, G., Li, S., & Yang, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.7000 · Cited by 3,617
Gaussian Error Linear Units (GELUs)
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537 · cited 2×
“The MLP is applied between self-attention layers for feature transformation and non-linearity: MLP(X) = FC(σ(FC(X))), FC(X) = XW + b, (2) where W and b are the weight and bias term of fully-connected layer respectively, and σ(·) is the activation function such as GELU [13].”
BackgroundAutomated Flower Classification over a Large Number of Classes
Nilsback, M.-E., & Zisserman, A. (2008) · 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing · doi:10.1109/icvgip.2008.47 · Cited by 3,442 · cited 1×
“These datasets include superordinate-level object classification (CIFAR-10 [16], CIFAR-100 [16]) and fine-grained object classification (Oxford-IIIT Pets [23], Oxford 102 Flowers [22] and iNaturalist 2019 [34]), shown in Table 2.”
MethodScene Parsing through ADE20K Dataset
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., & Torralba, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.544 · Cited by 3,326 · cited 3×
“ADE20K [45] Segmentation 20,210 2,000 150”
Randaugment: Practical automated data augmentation with a reduced search space
Cubuk, E. D., Zoph, B., Shlens, J., & Le, Q. V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00359 · Cited by 3,223 · cited 1×
“We adopt the same data augmentation strategy as that in DeiT [31] including random crop, random clip, Rand-Augment [7], Random Erasing [44], Mixup [42] and CutMix [41].”
MethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L., & Dollár, P. (2014) · arXiv · doi:10.48550/arxiv.1405.0312 · Cited by 2,579 · cited 1×
“COCO2017 [19] Detection 118,287 5,000 80”
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F. E. H., Feng, J., & Yan, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00060 · Cited by 2,415 · cited 1×
“Nowadays, transformer architectures have been used in a growing number of computer vision tasks [10] such as image recognition [6, 40, 29], object detection [46], and segmentation [43, 38].”
BackgroundDesigning Network Design Spaces
Radosavovic, I., Kosaraju, R. P., Girshick, R. B., He, K., & Dollár, P. (2020) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.2003.13678 · Cited by 2,313 · cited 2×
“To have a better understanding of current progress of visual transformers, we also include the representative CNN-based models such as ResNet [12], RegNet [25] and EfficientNet [28].”
MethodPre-Trained Image Processing Transformer
Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., Ma, S., Xu, C., Xu, C., & Gao, W. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01212 · Cited by 2,308 · cited 1×
“, denosing and super-resolution) as a series of translations and develop the IPT model for handling multiple low-level computer vision problems [4].”
Self-citationMMDetection: Open MMLab Detection Toolbox and Benchmark
Chen, K., Wang, J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Xu, J., et al. (2019) · arXiv · doi:10.48550/arxiv.1906.07155 · Cited by 2,227 · cited 1×
“Backbone Params (M) Epochs AP AP50 AP75 APS APM APL ResNet-50 [21, 5] 41.”
BackgroundGooch, P. (1969) · doi:10.1136/bmj.4.5674.53-c · Cited by 1,834 · cited 2×
“Classification 2,040 6,149 102 Oxford-IIIT Pets [23] 3,680 3,669 37 iNaturalist 2019 [34] 265,240 3,003 1,010 CIFAR-10 [16] 50,000 10,000 10 CIFAR-100 [16] 50,000 10,000 100”
MethodThe iNaturalist Species Classification and Detection Dataset
Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., & Belongie, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00914 · Cited by 1,451 · cited 2×
“Classification 2,040 6,149 102 Oxford-IIIT Pets [23] 3,680 3,669 37 iNaturalist 2019 [34] 265,240 3,003 1,010 CIFAR-10 [16] 50,000 10,000 10 CIFAR-100 [16] 50,000 10,000 100”
MethodFractalNet: Ultra-Deep Neural Networks without ResidualsPreprint
Larsson, G., Maire, M., & Shakhnarovich, G. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07648 · Cited by 931 · cited 1×
“The main advanced technologies apart from common settings [12] include AdamW [20], label smoothing [27], DropPath [18], and repeated augmentation [14].”
BackgroundRethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2015) · arXiv · doi:10.48550/arxiv.1512.00567 · Cited by 741 · cited 1×
“The main advanced technologies apart from common settings [12] include AdamW [20], label smoothing [27], DropPath [18], and repeated augmentation [14].”
BackgroundConditional Positional Encodings for Vision Transformers
Chu, X., Tian, Z., Zhang, B., Wang, X., & Shen, C. (2021) · arXiv · doi:10.48550/arxiv.2102.10882 · Cited by 489 · cited 1×
“Nowadays, transformer architectures have been used in a growing number of computer vision tasks [10] such as image recognition [6, 40, 29], object detection [46], and segmentation [43, 38].”
BackgroundAre Sixteen Heads Really Better than One?
Michel, P., Levy, O., & Neubig, G. (2019) · arXiv · doi:10.48550/arxiv.1905.10650 · Cited by 404 · cited 1×
“The effect of #heads in standard transformer has been investigated in multiple works [21, 35] and a head width of 64 is recommended for visual tasks [9, 31].”
MethodEnd-to-end Lane Shape Prediction with Transformers
Liu, R., Yuan, Z.-J., Liu, T., & Xiong, Z. (2020) · IEEE Workshop/Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00374 · Cited by 369
Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
Brendel, W., & Bethge, M. (2019) · arXiv · doi:10.48550/arxiv.1904.00760 · Cited by 323 · cited 1×
“visual transformers ignore the local relation and structure information inside the patch which is important for visual recognition [22, 3].”
BackgroundToward Transformer-Based Object DetectionPreprint
Beal, J., Kim, E. S., Tzeng, E., Park, D. H., Zhai, A., & Kislyuk, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.09958 · Cited by 286 · cited 1×
“This design also affects the subsequent works including DeiT [31], ViT-FRCNN [2], IPT [6] and SETR [38].”
MethodLin, M., Chen, Q., & Yan, S. (2013) · arXiv · doi:10.48550/arxiv.1312.4400 · Cited by 252
Training data-efficient image transformers & distillation through attention
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jégou, H. (2020) · arXiv · doi:10.48550/arxiv.2012.12877 · Cited by 182 · cited 9×
“Transformer-based DeiT-Ti [31] 224×224 5.”
MethodInfluentialAugment your batch: Improving generalization through instance repetition
Hoffer, E., Ben-Nun, T., Hubara, I., Giladi, N., Hoefler, T., & Soudry, D. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00815 · Cited by 166 · cited 1×
“The main advanced technologies apart from common settings [12] include AdamW [20], label smoothing [27], DropPath [18], and repeated augmentation [14].”
BackgroundManifold Regularized Dynamic Network Pruning
Tang, Y., Wang, Y., Xu, Y., Deng, Y., Xu, C., Tao, D., & Xu, C. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00498 · Cited by 122
Self-citationReNAS: Relativistic Evaluation of Neural Architecture Search
Xu, Y., Wang, Y.-H., Han, K., Tang, Y.-H., Jui, S.-L., Xu, C.-J., & Xu, C. (2019) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr46437.2021.00439 · Cited by 112
Self-citationSegmenting Transparent Objects in the Wild with Transformer
Xie, E., Wang, W., Wang, W., Sun, P., Xu, H., Liang, D., & Luo, P. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/165 · Cited by 111
Grafit: Learning fine-grained image representations with coarse labels
Touvron, H., Sablayrolles, A., Douze, M., Cord, M., & Jeǵou, H. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00091 · Cited by 77 · cited 1×
“Model Params (M) ImageNet CIFAR10 CIFAR100 Flowers Pets iNat-19 CNN-based Grafit ResNet-50 [32] 25.”
MethodModel Rubik's Cube: Twisting Resolution, Depth and Width for TinyNetsPreprint
Han, K., Wang, Y., Zhang, Q., Zhang, W., Xu, C., & Zhang, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.14819 · Cited by 64
Self-citationAugmented Shortcuts for Vision TransformersPreprint
Tang, Y., Han, K., Xu, C., Xiao, A., Deng, Y., Xu, C., & Wang, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.15941 · Cited by 39 · cited 1×
“Nowadays, transformer architectures have been used in a growing number of computer vision tasks [10] such as image recognition [6, 40, 29], object detection [46], and segmentation [43, 38].”
BackgroundSelf-citationTrans2Seg: Transparent Object Segmentation with TransformerPreprint
Xie, E.-Z., Wang, W., Wang, W., Sun, P., Xu, H., Liang, D., & Luo, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.08461 · Cited by 27 · cited 3×
“We adopt the segmentation framework of Trans2Seg [38] to build the pure transformer semantic segmentation based on TNT backbone.”
BackgroundMethodInfluential
Research works printed but not linked: 5
- [6][6] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML , 2020. -
- [11][11] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. arXiv preprint arXiv:2012.12556 , 2020. - · arXiv
- [23][23] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017. - · arXiv
- [26][26] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR , pages 3498-3505. IEEE, 2012. -
- [37][37] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008. -
Cited by
Sorted by citations
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
“Concurrent to our work are some that modify the ViT architecture [66, 15, 25] for better image classification.”
MethodPyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00061 · Cited by 5,259
Attention mechanisms in computer vision: A survey
Guo, M.-H., Xu, T.-X., Liu, J.-J., Liu, Z.-N., Jiang, P.-T., Mu, T.-J., Zhang, S.-H., Martin, R. R., Cheng, M.-M., & Hu, S.-M. (2022) · Computational Visual Media · doi:10.1007/s41095-022-0271-y · Cited by 2,566
“[8], [9], [15], [21], [31], [32], [34], [35] , [22], [26], [62], [63], [64], [65], [66], [67] , [41], [68], [69], [70], [71], [72], [73], [74] , [8], [34], [42], [43], [75], [76], [77], [78] , [27], [44], [45], [46], [79], [80], [81], [82] , [61], [83], [84], [85], [86], [87], [88], [89] , [47], [90], [91], [92], [93]…”
Pvt v2: Improved baselines with pyramid vision transformer
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2022) · Computational Visual Media · doi:10.1007/s41095-022-0274-8 · Cited by 2,439
MetaFormer is Actually What You Need for Vision
Yu, W., Luo, M., Zhou, P., Si, C., Zhou, Y., Wang, X., Feng, J., & Yan, S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01055 · Cited by 1,449
“Based on this common belief, many variants of the attention modules [13, 21, 55, 66] have been developed to improve the vision transformer.”
Background
Metadata
- Authors
- Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., & Wang, Y.
- Affiliations
- Huawei Technologies (China) (CN)
- University of Macau (MO)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Volume
- 34
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2103.00112
- OpenAlex
- W3133696297
- Cited by
- 1,394
- References
- 56
- Citations by year
- 2026: 14 · 2025: 276 · 2024: 457 · 2023: 387 · 2022: 186 · 2021: 74
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (0015/2019/AKP, 62072449, 61632003, 0018/2019/AKP)
- Fundo para o Desenvolvimento das Ciências e da Tecnologia (0018/2019/AKP, 0015/2019/AKP)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- vision transformer, attention mechanism, local patches, visual words, image classification, feature aggregation, computational efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
