Patch Slimming for Efficient Vision Transformers
Open accessAbstract
This paper studies the efficiency problem for visual transformers by excavating redundant calculation in given networks. The recent transformer architecture has demonstrated its effectiveness for achieving excellent performance on a series of computer vision tasks. However, similar to that of convolutional neural networks, the huge computational cost of vision transformers is still a severe issue. Considering that the attention mechanism aggregates different patches layer-by-layer, we present a novel patch slimming approach that discards useless patches in a topdown paradigm. We first identify the effective patches in the last layer and then use them to guide the patch selection process of previous layers. For each layer, the impact of a patch on the final output feature is approximated and patches with less impacts will be removed. Experimental results on benchmark datasets demonstrate that the proposed method can significantly reduce the computational costs of vision transformers without affecting their performances. For example, over 45% FLOPs of the ViT-Ti model can be reduced with only 0.2% top-1 accuracy drop on the ImageNet dataset.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Vision Transformers suffer from higher memory footprints, computational overheads, and power consumption than convolutional neural networks, hindering their deployment and real-time inference on resource-constrained edge devices.
“Nevertheless, as compared to convolutional neural networks (CNNs), ViTs suffer from higher memory footprints, computational overheads, and power consumption, hindering their deployment and real-time inference on resourceconstrained edge devices [25, 17, 15, 37].”
From I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference (2023) · cites this paper as [37]
The Introduction says vision transformers have high computational cost that blocks deployment on resource-limited devices such as phones and IoT devices. It says this is similar to CNNs, not that ViTs have higher costs than CNNs. The cited passages do not establish higher memory use, power consumption, or a specific obstacle to real-time inference. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · see citing works
(2017) · Cited by 11,625 · see citing works
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00061 · Cited by 5,259 · cited 1× · see citing works
“DeiT-S [38] 4.6 79.8 DeiT-B [38] 17.5 81.8 PVT-Small [40] 3.8 79.8 PVT-Medium [40] 6.7 81.2 PVT-Large [40] 9.8 81.7 T2T-ViT-14 [43] 5.2 81.5 T2T-ViT-19 [43] 8.9 81.9 T2T-ViT-24 [43] 14.1 82.3 TNT-S [12] 5.2 81.5 TNT-B [12] 14.1 82.9 Swin-T [26] 4.5 81.3 Swin-S [26] 8.7 83.0 Swin-B [26] 15.4 83.5…”
InfluentialHan, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662 · see citing works
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_32 · Cited by 4,408 · see citing works
Pruning Filters for Efficient ConvNets
Li, H., Kadav, A., Durdanovic, I., Samet, H., & Graf, H. P. (2016) · arXiv · doi:10.48550/arxiv.1608.08710 · Cited by 2,928 · see citing works
Learning Efficient Convolutional Networks through Network Slimming
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., & Zhang, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.298 · Cited by 2,761 · see citing works
Channel Pruning for Accelerating Very Deep Neural Networks
He, Y., Zhang, X., & Sun, J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.155 · Cited by 2,647 · see citing works
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F. E. H., Feng, J., & Yan, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00060 · Cited by 2,415 · see citing works
Pre-Trained Image Processing Transformer
Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., Ma, S., Xu, C., Xu, C., & Gao, W. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01212 · Cited by 2,308 · see citing works
Self-citationLearning Structured Sparsity in Deep Neural NetworksPreprint
Wen, W. X., Wu, C., Wang, Y., Chen, Y., & Li, H. H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.03665 · Cited by 1,861 · see citing works
Self-citationLearning both Weights and Connections for Efficient Neural Networks
Han, S., Pool, J., Tran, J., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1506.02626 · Cited by 1,481 · see citing works
Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., & Wang, Y. (2021) · arXiv · doi:10.48550/arxiv.2103.00112 · Cited by 1,394 · see citing works
Self-citationFilter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration
He, Y., Liu, P., Wang, Z., Hu, Z., & Yang, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00447 · Cited by 1,280 · see citing works
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
Voita, E., Talbot, D., Moiseev, F., Sennrich, R., & Titov, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1580 · Cited by 1,146 · see citing works
Approximation of dynamical systems by continuous time recurrent neural networks
Funahashi, K.-I., & Nakamura, Y. (1993) · Neural Networks · doi:10.1016/s0893-6080(05)80125-x · Cited by 980 · see citing works
Are Sixteen Heads Really Better than One?
Michel, P., Levy, O., & Neubig, G. (2019) · arXiv · doi:10.48550/arxiv.1905.10650 · Cited by 404 · see citing works
Fast ConvNets Using Group-Wise Brain Damage
Lebedev, V., & Lempitsky, V. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.280 · Cited by 397 · see citing works
Knowledge Distillation by On-the-Fly Native EnsemblePreprint
Xu, L., Zhu, X., & Gong, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.04606 · Cited by 363 · see citing works
Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
Ye, J., Lu, X., Lin, Z., & Wang, J. Z. (2018) · arXiv · doi:10.48550/arxiv.1802.00124 · Cited by 328 · see citing works
On lipschitz continuity of the solution mapping to the skorokhod problem, with applications
Dupuis, P. G., & Ishii, H. (1991) · Stochastics and stochastics reports · doi:10.1080/17442509108833688 · Cited by 255 · see citing works
Dynamic Channel Pruning: Feature Boosting and SuppressionPreprint
Gao, X., Zhao, Y., Dudziak, Ł., Mullins, R., & Xu, C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.05331 · Cited by 244 · see citing works
Self-citationPost-Training Quantization for Vision Transformer
Liu, Z., Wang, Y., Han, K., Ma, S., & Gao, W. (2021) · arXiv · doi:10.48550/arxiv.2106.14156 · Cited by 192 · see citing works
Self-citationScalable Vision Transformers with Hierarchical Pooling
Pan, Z., Zhuang, B., Liu, J., He, H., & Cai, J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00043 · Cited by 168 · see citing works
Spatio-Temporal Transformer Network for Video Restoration
Kim, T. H., Sajjadi, M. S. M., Hirsch, M., & Schölkopf, B. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01219-9_7 · Cited by 165 · see citing works
Channel Gating Neural NetworksPreprint
Hua, W., Zhou, Y. J., De, C., Zhang, Z., & Suh, G. E. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.12549 · Cited by 162 · see citing works
All Tokens Matter: Token Labeling for Training Better Vision Transformers
Jiang, Z., Hou, Q., Yuan, L., Zhou, D., Shi, Y., Jin, X., Wang, A., & Feng, J. (2021) · arXiv · doi:10.48550/arxiv.2104.10858 · Cited by 158 · see citing works
SCOP: Scientific Control for Reliable Neural Network PruningPreprint
Tang, Y., Wang, Y., Xu, Y., Tao, D., Xu, C., Xu, C., & Xu, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.10732 · Cited by 116 · see citing works
Self-citationManifold Regularized Dynamic Network Pruning
Tang, Y.-H., Wang, Y.-H., Xu, Y., Deng, Y., Xu, C., Tao, D., & Xu, C. (2021) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr46437.2021.00498 · Cited by 115 · see citing works
Self-citationPoWER-BERT: Accelerating BERT Inference via Progressive Word-vector EliminationPreprint
Goyal, S., Choudhury, A. R., Raje, S., Chakaravarthy, V. T., Sabharwal, Y., & Verma, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.08950 · Cited by 102 · see citing works
Augmented Shortcuts for Vision TransformersPreprint
Tang, Y.-H., Han, K., Xu, C., Xiao, A., Deng, Y., Xu, C., & Wang, Y.-H. (2021) · Neural Information Processing Systems · doi:10.48550/arxiv.2106.15941 · Cited by 55 · see citing works
Self-citationBCNet: Searching for Network Width with Bilaterally Coupled Network
Su, X. Q., You, S., Wang, F., Qian, C., Zhang, C., & Xu, C. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00221 · Cited by 37 · see citing works
Self-citationLocally Free Weight Sharing for Network Width SearchPreprint
Su, X. Q., You, S., Huang, T., Wang, F., Qian, C., Zhang, C., & Xu, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.05258 · Cited by 28 · see citing works
Self-citationVisual Transformer Pruning.Preprint
Zhu, M., Han, K., Tang, Y., & Wang, Y. (2021) · arXiv (Cornell University) · Cited by 24 · see citing works
Self-citationPyramidTNT: Improved Transformer-in-Transformer Baselines with Pyramid ArchitecturePreprint
Han, K., Guo, J.-Y., Tang, Y.-H., & Wang, Y.-H. (2022) · arXiv.org · doi:10.48550/arxiv.2201.00978 · Cited by 22 · see citing works
Self-citationBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
2 of the 42 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830, 2016. · arXiv
- Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 1, 7 · arXiv
- Zhenhua Liu, Jizheng Xu, Xiulian Peng, and Ruiqin Xiong. Frequency-domain dynamic pruning for convolutional neu- ral networks. Advances in neural information processing systems, 31, 2018. 1
- Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through at- tention. arXiv preprint arXiv:2012.12877, 2020. 1, 7 · arXiv
- Mingjian Zhu, Kai Han, Yehui Tang, and Yunhe Wang. Vi- sual transformer pruning. arXiv preprint arXiv:2104.08500, 2021. 2, 6, 7 · arXiv
Cited by
Sorted by citations
Han (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682 · see citing works
Self-citationCMT: Convolutional Neural Networks Meet Vision Transformers
Guo, J., Han, K., Wu, H., Tang, Y., Chen, X., Wang, Y., & Xu, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01186 · Cited by 1,005 · see citing works
Self-citationTopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation
Zhang, W., Huang, Z., Luo, G., Chen, T., Wang, X., Liu, W., Yu, G., & Shen, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01177 · Cited by 374 · see citing works
A Comprehensive Survey of Convolutions in Deep Learning: Applications, Challenges, and Future Trends
Younesi, A., Ansari, M., Fazli, M., Ejlali, A., Shafique, M., & Henkel, J. (2024) · IEEE Access · doi:10.1109/access.2024.3376441 · Cited by 228 · see citing works
Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer
Xu, Y., Zhang, Z., Zhang, M., Sheng, K., Li, K., Dong, W., Zhang, L., Xu, C., & Sun, X. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20202 · Cited by 202 · see citing works
Self-citationEfficient Visual Tracking with Exemplar Transformers
Blatter, P., Kanakis, M., Danelljan, M., & Gool, L. V. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00162 · Cited by 171 · see citing works
SPViT: Enabling Faster Vision Transformers via Latency-Aware Soft Token Pruning
Kong, Z., Dong, P., Ma, X., Meng, X., Niu, W., Sun, M., Shen, X., Yuan, G., Ren, B., Tang, H., et al. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20083-0_37 · Cited by 169 · see citing works
Self-citationA Survey on Efficient Vision Transformers: Algorithms, Techniques, and Performance Benchmarking
Papa, L., Russo, P., Amerini, I., & Zhou, L. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3392941 · Cited by 177 · see citing works
An Image Patch is a Wave: Phase-Aware Vision MLP
Tang, Y.-H., Han, K., Guo, J.-Y., Xu, C., Li, Y., Xu, C., & Wang, Y.-H. (2021) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52688.2022.01066 · Cited by 177 · see citing works
Self-citationA Survey of Techniques for Optimizing Transformer Inference
Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., & Somani, A. K. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102990 · Cited by 154 · see citing works
Less is More: Focus Attention for Efficient DETR
Zheng, D., Dong, W., Hu, H., Chen, X., & Wang, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00614 · Cited by 152 · see citing works
Self-citationI-ViT: Integer-only Quantization for Efficient Vision Transformer Inference
Li, Z., & Gu, Q. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01565 · Cited by 141 · see citing works
“Nevertheless, as compared to convolutional neural networks (CNNs), ViTs suffer from higher memory footprints, computational overheads, and power consumption, hindering their deployment and real-time inference on resourceconstrained edge devices [16,18,25,35].”
BackgroundHeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformers
Dong, P., Sun, M., Lu, A., Xie, Y., Liu, K., Kong, Z., Meng, X., Li, Z., Lin, X., Fang, Z., et al. (2023) · 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca56546.2023.10071047 · Cited by 128 · see citing works
Self-citationRepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformers
Li, Z., Xiao, J., Yang, L., & Gu, Q. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01580 · Cited by 127 · see citing works
Chasing Sparsity in Vision Transformers: An End-to-End Exploration
Chen, T., Cheng, Y., Gan, Z., Yuan, L., Zhang, L., & Wang, Z. (2021) · arXiv · doi:10.48550/arxiv.2106.04533 · Cited by 111 · see citing works
FlexiViT: One Model for All Patch Sizes
Beyer, L., Izmailov, P., Kolesnikov, A., Caron, M., Kornblith, S., Zhai, X., Minderer, M., Tschannen, M., Alabdulmohsin, I., & Pavetic, F. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01393 · Cited by 106 · see citing works
Hire-MLP: Vision MLP via Hierarchical Rearrangement
Guo, J., Tang, Y., Han, K., Chen, X., Wu, H., Xu, C., Xu, C., & Wang, Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00090 · Cited by 100 · see citing works
Self-citationWidth & Depth Pruning for Vision Transformers
Yu, F., Huang, K., Wang, M., Cheng, Y., Chu, W., & Cui, L. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20222 · Cited by 95 · see citing works
Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers
Wei, S., Ye, T., Zhang, S., Tang, Y., & Liang, J. (2023) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52729.2023.00208 · Cited by 152 · see citing works
Self-citationNot All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image RecognitionPreprint
Wang, Y., Huang, R., Song, S., Huang, Z., & Huang, G. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.15075 · Cited by 101 · see citing works
Self-citationCF-ViT: A General Coarse-to-Fine Method for Vision Transformer
Chen, M., Lin, M., Li, K., Shen, Y.-H., Wu, Y.-J., Chao, F., & Ji, R.-R. (2022) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i6.25860 · Cited by 122 · see citing works
Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space
Chavan, A., Shen, Z., Liu, Z., Liu, Z., Cheng, K.-T., & Xing, E. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00488 · Cited by 72 · see citing works
SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformer
Chen, X., Liu, Z., Tang, H., Yi, L., Zhao, H., & Han, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00205 · Cited by 69 · see citing works
Adaptive and Background-Aware Vision Transformer for Real-Time UAV Tracking
Li, S., Yang, Y., Zeng, D., & Wang, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01286 · Cited by 76 · see citing works
Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networks
Rao, Y., Liu, Z.-Y., Zhao, W.-L., Zhou, J., & Lu, J.-W. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3263826 · Cited by 60 · see citing works
Similarity-Guided Layer-Adaptive Vision Transformer for UAV Tracking
Xue, C., Zhong, B., Liang, Q., Zheng, Y., Li, N., Xue, Y., & Song, S. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00631 · Cited by 68 · see citing works
Patch Similarity Aware Data-Free Quantization for Vision Transformers
Li, Z., Ma, L., Chen, M., Xiao, J., & Gu, Q. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20083-0_10 · Cited by 61 · see citing works
Vision-Language Models for Edge Networks: A Comprehensive Survey
Sharshar, A., Khan, L. U., Ullah, W., & Guizani, M. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3579032 · Cited by 74 · see citing works
ViTs as backbones: Leveraging vision transformers for feature extraction
Elharrouss, O., Himeur, Y., Mahmood, Y., Alrabaee, S., Ouamane, A., Bensaali, F., Bechqito, Y., & Chouchane, A. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102951 · Cited by 59 · see citing works
Saha, S., & Xu, L. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130417 · Cited by 66 · see citing works
Yao, S., Sun, H., Xiang, T.-Z., Wang, X., & Cao, X. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3475219 · Cited by 54 · see citing works
Attention Map Guided Transformer Pruning for Occluded Person Re-Identification on Edge Device
Mao, J., Yao, Y., Sun, Z., Huang, X., Shen, F., & Shen, H. T. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3265159 · Cited by 49 · see citing works
Which Tokens to Use? Investigating Token Reduction in Vision Transformers
Haurum, J. B., Escalera, S., Taylor, G. W., & Moeslund, T. B. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00085 · Cited by 47 · see citing works
Long, S., Zhao, Z., Pi, J., Wang, S., & Wang, J.-D. (2022) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52729.2023.00996 · Cited by 79 · see citing works
Dynamic Neural Network Structure: A Review for its Theories and Applications
Guo, J., Chen, C. L. P., Liu, Z., & Yang, X. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3377194 · Cited by 47 · see citing works
Self-citationMonoATT: Online Monocular 3D Object Detection with Adaptive Token Transformer
Zhou, Y., Zhu, H., Liu, Q., Chang, S., & Guo, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01678 · Cited by 45 · see citing works
Making Vision Transformers Efficient from A Token Sparsification View
Chang, S., Wang, P., Lin, M., Wang, F., Zhang, D. J., Jin, R., & Shou, M. Z. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00600 · Cited by 43 · see citing works
DiffRate : Differentiable Compression Rate for Efficient Vision Transformers
Chen, M., Shao, W., Xu, P., Lin, M., Zhang, K., Chao, F., Ji, R., Qiao, Y., & Luo, P. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01574 · Cited by 43 · see citing works
PSAQ-ViT V2: Toward Accurate and General Data-Free Quantization for Vision Transformers
Li, Z., Chen, M., Xiao, J., & Gu, Q. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3301007 · Cited by 41 · see citing works
Self-slimmed Vision Transformer
Zong, Z., Li, K., Song, G., Wang, Y., Qiao, Y., Leng, B., & Liu, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20083-0_26 · Cited by 39 · see citing works
Self-citationVision transformer models for mobile/edge devices: a survey
Lee, S. I., Koo, K., Lee, J. H., Lee, G., Jeong, S., Seongjun, O., & Kim, H. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01312-0 · Cited by 39 · see citing works
Content-aware Token Sharing for Efficient Semantic Segmentation with Vision Transformers
Lu, C., de Geus, D., & Dubbelman, G. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02263 · Cited by 36 · see citing works
Lin, M., Chen, M., Zhang, Y.-X., Shen, C.-H., Ji, R.-R., & Cao, L.-J. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-023-01861-3 · Cited by 35 · see citing works
LGViT: Dynamic Early Exiting for Accelerating Vision Transformer
Xu, G., Hao, J., Shen, L., Hu, H., Luo, Y., Lin, H., & Shen, J. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611762 · Cited by 35 · see citing works
Multi-Aspect Convolutional-Transformer Network for SAR Automatic Target Recognition
Li, S., Pan, Z., & Hu, Y. (2022) · Remote Sensing · doi:10.3390/rs14163924 · Cited by 35 · see citing works
PatchDropout: Economizing Vision Transformers Using Patch Dropout
Liu, Y., Matsoukas, C., Strand, F., Azizpour, H., & Smith, K. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00394 · Cited by 36 · see citing works
Boost Vision Transformer with GPU-Friendly Sparsity and Quantization
Yu, C., Chen, T., Gan, Z., & Fan, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02170 · Cited by 33 · see citing works
Dynamic Perceiver for Efficient Visual Recognition
Han, Y., Han, D., Liu, Z., Wang, Y., Pan, X., Pu, Y., Deng, C., Feng, J., Song, S., & Huang, G. M. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00551 · Cited by 32 · see citing works
Self-citationSparsifiner: Learning Sparse Instance-Dependent Attention for Efficient Vision Transformers
Wei, C., Duke, B., Jiang, R., Aarabi, P., Taylor, G. W., & Shkurti, F. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02172 · Cited by 30 · see citing works
Guo, R., Chen, X. L., Wang, L., Wang, Y., Sun, H., Wei, J., Han, H., Liu, L., Wei, S., Hu, Y., et al. (2024) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2024.3402174 · Cited by 28 · see citing works
Self-citationTFormer: A Transmission-Friendly ViT Model for IoT Devices
Lu, Z., Ding, C., Juefei-Xu, F., Boddeti, V. N., Wang, S., & Yang, Y. (2023) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2022.3222765 · Cited by 24 · see citing works
Cao, J., Ye, P., Li, S., Yu, C., Tang, Y., Lu, J., & Chen, T. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01487 · Cited by 23 · see citing works
Self-citationA survey of visual neural networks: current trends, challenges and opportunities
Feng, P., & Tang, Z. (2022) · Multimedia Systems · doi:10.1007/s00530-022-01003-8 · Cited by 23 · see citing works
Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attention
Liu, X., Wu, T., & Guo, G. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/136 · Cited by 23 · see citing works
MLP Can Be a Good Transformer Learner
Lin, S., Lyu, P., Liu, D., Tang, T., Liang, X., Song, A., & Chang, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01843 · Cited by 22 · see citing works
A Decade of Video Analytics at Edge: Training, Deployment, Orchestration, and Platforms
Bai, T., Zhao, H., Huang, L., Wang, Z., Kim, D. I., & Nallanathan, A. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2025.3563377 · Cited by 22 · see citing works
Explainability of Vision Transformers: a Comprehensive Review and New PerspectivesPreprint
Kashefi, R., Barekatain, L., Sabokrou, M., & Aghaeipoor, F. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5055345 · Cited by 22 · see citing works
Multi-Dimensional Vision Transformer Compression via Dependency Guided Gaussian Process Search
Hou, Z., & Kung, S.-Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00411 · Cited by 20 · see citing works
Homogeneous tokenizer matters: Homogeneous visual tokenizer for remote sensing image understanding
Shao, R., Zhang, Z., Tao, C., Zhang, Y., Peng, C., & Li, H. (2024) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2024.09.009 · Cited by 19 · see citing works
EvolutionViT: Multi-objective evolutionary vision transformer pruning under resource constraints
Liu, L., Yen, G. G., & He, Z. (2024) · Information Sciences · doi:10.1016/j.ins.2024.121406 · Cited by 18 · see citing works
Efficient Vision Transformer for Human-Centric AIoT Applications Through Token Tracking Assignment
Li, C., Peng, Y., Liu, G., Li, Y., Yang, X., & Chen, C. (2023) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2023.3323373 · Cited by 18 · see citing works
UPDP: A Unified Progressive Depth Pruner for CNN and Vision Transformer
Liu, J., Tang, D., Huang, Y., Zhang, L., Zeng, X. C., Li, D., Lu, M., Peng, J., Wang, Y., Jiang, F., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i12.29296 · Cited by 18 · see citing works
Self-citationLu, W. S., Zhang, Z., & Nguyen, M. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3385318 · Cited by 18 · see citing works
Towards Accurate Post-Training Quantization of Vision Transformers via Error Reduction
Zhong, Y., You, H., Hu, J., Zhang, Y., & Ji, R. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3528042 · Cited by 19 · see citing works
Efficient token pruning in Vision Transformers using an attention-based Multilayer Network
Marchetti, M., Traini, D., Domenico Ursino, A., & Virgili, L. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127449 · Cited by 17 · see citing works
An Efficient Window-Based Vision Transformer Accelerator via Mixed-Granularity Sparsity
Dong, Q., Zhang, S., & Wang, Z. (2025) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2025.3527541 · Cited by 16 · see citing works
Luo, Z., Yan, H., & Pan, X. (2023) · Journal of Computational Methods in Engineering Applications · doi:10.62836/jcmea.v3i1.030107 · Cited by 36 · see citing works
GhostViT: Expediting Vision Transformers via Cheap Operations
Cao, H., Qu, Z., Chen, G., Li, X.-Y., Thiele, L., & Knoll, A. C. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3326795 · Cited by 15 · see citing works
FDViT: Improve the Hierarchical Architecture of Vision Transformer
Xu, Y.-X., Li, C., Li, D., Sheng, X., Jiang, F., Tian, L., & Sirasao, A. (2023) · IEEE International Conference on Computer Vision · doi:10.1109/iccv51070.2023.00547 · Cited by 21 · see citing works
Mao, J., Shen, Y., Guo, J., Yao, Y., Hua, X., & Shen, H. T. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3535405 · Cited by 15 · see citing works
Self-citationLightweight and Optimization Acceleration Methods for Vision Transformer: A Review
Chen, M., Gao, J., & Yu, W. (2022) · 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc55140.2022.9921989 · Cited by 14 · see citing works
Accelerating ViT Inference on FPGA through Static and Dynamic Pruning
Parikh, D., Li, S., Zhang, B., Kannan, R., Busart, C. E., & Prasanna, V. K. (2024) · IEEE Symposium on Field-Programmable Custom Computing Machines · doi:10.1109/fccm60383.2024.00018 · Cited by 17 · see citing works
Bi, Y.-D., Abrol, A., Jia, S., Sui, J., & Calhoun, V. (2024) · NeuroImage · doi:10.1016/j.neuroimage.2024.120674 · Cited by 15 · see citing works
Multi-Dimensional Model Compression of Vision Transformer
Hou, Z., & Kung, S. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859786 · Cited by 14 · see citing works
Efficient Transformer-based 3D Object Detection with Dynamic Token Halting
Ye, M., Meyer, G. P., Chai, Y., & Liu, Q. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00775 · Cited by 11 · see citing works
Mask-Guided Vision Transformer for Few-Shot Learning
Chen, Y., Xiao, Z., Pan, Y., Zhao, L., Dai, H., Wu, Z., Li, C., Zhang, T., Li, C., Zhu, D., et al. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3418527 · Cited by 27 · see citing works
MULTIFLOW: Shifting Towards Task-Agnostic Vision-Language Pruning
Farina, M., Mancini, M., Cunegatti, E., Cunegatti, E., Iacca, G., & Ricci, E. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01532 · Cited by 10 · see citing works
DiVIT: Algorithm and architecture co-design of differential attention in vision transformer
Li, Y., Hu, Y., Wu, F., & Li, K. (2022) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2022.102520 · Cited by 10 · see citing works
Multi-Dimension Compression of Feed-Forward Network in Vision Transformers
Bai, S., Chen, J., Yang, Y., & Liu, Y. (2023) · Pattern Recognition Letters · doi:10.1016/j.patrec.2023.10.014 · Cited by 10 · see citing works
Darvishnezhad, M., & Sebt, M. A. (2023) · International Journal of Remote Sensing · doi:10.1080/01431161.2023.2176722 · Cited by 10 · see citing works
Adaptive patch selection to improve Vision Transformers through Reinforcement Learning
Cauteruccio, F., Marchetti, M., Traini, D., Domenico Ursino, A., & Virgili, L. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06516-z · Cited by 9 · see citing works
DTSNet: Dynamic Transformer Slimming for Efficient Vision Recognition
Xiao, W., Li, X., Hu, L., Hao, Y., & Chen, M. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3607796 · Cited by 9 · see citing works
Multi-tailed vision transformer for efficient inference
Wang, Y., Du, B., Wang, W., & Xu, C. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106235 · Cited by 10 · see citing works
Self-citationZhu, L., Chen, T., Ji, D., Ye, J., & Liu, J. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3531299 · Cited by 8 · see citing works
Efficient feature selection for pre-trained vision transformers
Huang, L., Zeng, J., Yu, M., Ding, W., Bai, X., & Wang, K. (2025) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2025.104326 · Cited by 8 · see citing works
TFRNet: Semantic Segmentation Network with Token Filtration and Refinement Method
Ma, Y., & Hu, X. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3378465 · Cited by 8 · see citing works
Feature fusion method based on local binary graph for PolSAR image classification
Sebt, M. A., & Darvishnezhad, M. (2023) · IET Radar Sonar & Navigation · doi:10.1049/rsn2.12389 · Cited by 8 · see citing works
Adversarial Robust Modulation Recognition Guided by Attention Mechanisms
Zhan, Q., Zhang, X., Sun, M., Song, L., & Zhou, Z. (2025) · IEEE Open Journal of Signal Processing · doi:10.1109/ojsp.2025.3526577 · Cited by 7 · see citing works
Explainability of vision transformers: a comprehensive review and new perspectives
Kashefi, R., Barekatain, L., Sabokrou, M., & Aghaeipoor, F. (2026) · Multimedia Tools and Applications · doi:10.1007/s11042-026-21313-7 · Cited by 7 · see citing works
Make a Long Image Short: Adaptive Token Length for Vision Transformers
Zhou, Q., & Zhu, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43415-0_5 · Cited by 7 · see citing works
FLORA: Fine-grained Low-Rank Architecture Search for Vision Transformer
Chang, C.-C., Sung, Y.-Y., Yu, S., Huang, N.-C., Marculescu, D., & Wu, K. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00247 · Cited by 7 · see citing works
Wei, Y., Feng, Y., Zu, D., & Zhang, X. (2024) · Crop Protection · doi:10.1016/j.cropro.2024.107018 · Cited by 7 · see citing works
RDA-MTE: an innovative model for emotion recognition in sports behavior decision-making
Zhang, S. (2024) · Frontiers in Neuroscience · doi:10.3389/fnins.2024.1466013 · Cited by 6 · see citing works
Network Expansion For Practical Training Acceleration
Ding, N., Tang, Y.-H., Han, K., Xu, C., & Wang, Y.-H. (2023) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52729.2023.01941 · Cited by 13 · see citing works
Self-citationATFTrans: attention-weighted token fusion transformer for robust and efficient object tracking
Xu, L., Wang, L., & Guo, Z. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09444-0 · Cited by 6 · see citing works
ViT- ToGo: Vision Transformer Accelerator with Grouped Token Pruning
Lee, S., Cho, K.-M., Kwon, E., Park, S., Kim, S., & Kang, S. (2024) · Design, Automation, and Test in Europe · doi:10.23919/date58400.2024.10546804 · Cited by 6 · see citing works
Human Activity Recognition Based on Feature Fusion of Millimeter Wave Radar and Inertial Navigation
Shi, J., Zhu, Y., He, J., Xu, Z., Chu, L., Braun, R., & Shi, Q. (2025) · IEEE Journal of Microwaves · doi:10.1109/jmw.2025.3539957 · Cited by 8 · see citing works
Data Level Lottery Ticket Hypothesis for Vision Transformers
Shen, X., Kong, Z., Qin, M., Dong, P., Yuan, G., Meng, X., Tang, H., Ma, X., & Wang, Y. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/153 · Cited by 6 · see citing works
Self-citationUnleashing Knowledge Potential of Source Hypothesis for Source-Free Domain Adaptation
Hu, B., Liu, J., Zheng, K., & Zha, Z.-J. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3333190 · Cited by 5 · see citing works
UCC: A unified cascade compression framework for vision transformer models
Chen, D., Lin, K., & Deng, Q. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128747 · Cited by 5 · see citing works
- Authors
- Tang, Y., Han, K., Wang, Y., Xu, C., Guo, J., Xu, C., & Tao, D.
- Affiliations
- Peking University (CN)
- Huawei Technologies (China) (CN)
- The University of Sydney (AU)
- Jingdong (China) (CN)
- Venue
- 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2022-06
- Pages
- 12155-12164
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4312340826
- Cited by
- 201
- References
- 42
- Citations by year
- 2026: 22 · 2025: 52 · 2024: 49 · 2023: 54 · 2022: 19 · 2021: 5
- FWCI
- 15.78
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (61876007)
- Australian Research Council (DP210101859)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- vision transformer, computational efficiency, attention mechanism, patch selection, FLOPs reduction, ViT
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
