Width & Depth Pruning for Vision Transformers
Open accessAbstract
Transformer models have demonstrated their promising potential and achieved excellent performance on a series of computer vision tasks. However, the huge computational cost of vision transformers hinders their deployment and application to edge devices. Recent works have proposed to find and remove the unimportant units of vision transformers. Despite achieving remarkable results, these methods take one dimension of network width into consideration and ignore network depth, which is another important dimension for pruning vision transformers. Therefore, we propose a Width & Depth Pruning (WDPruning) framework that reduces both width and depth dimensions simultaneously. Specifically, for width pruning, a set of learnable pruning-related parameters is used to adaptively adjust the width of transformer. For depth pruning, we introduce several shallow classifiers by using the intermediate information of the transformer blocks, which allows images to be classified by shallow classifiers instead of the deeper classifiers. In the inference period, all of the blocks after shallow classifiers can be dropped so they don’t bring additional parameters and computation. Experimental results on benchmark datasets demonstrate that the proposed method can significantly reduce the computational costs of mainstream vision transformers such as DeiT and Swin Transformer with a minor accuracy drop. In particular, on ILSVRC-12, we achieve over 22% pruning ratio of FLOPs by compressing DeiT-Base, even with an increase of 0.14% Top-1 accuracy.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
WDPruning on DeiT-base improves throughput by 15% for an accuracy drop of 1%.
“The pruning results on DeiT-base [52] show that the throughput can be improved by 15% for an accuracy drop of 1%.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [99]
WDPruning, S2ViTE, and SAViT methods increase the accuracy of the compressed model with fewer model parameters on the DeiT-base model.
“For example, WDPruning [99], S 2 ViTE [98], SAViT [65] methods increase the accuracy of the compressed model with less number of model parameters on DeiT-base model.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [99]
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · see citing works
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285 · see citing works
mixup: Beyond Empirical Risk Minimization
Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2017) · arXiv · doi:10.48550/arxiv.1710.09412 · Cited by 5,831 · see citing works
CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features
Yun, S., Han, D., Chun, S., Oh, S. J., Yoo, Y., & Choe, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00612 · Cited by 5,400 · see citing works
Scalable Person Re-identification: A Benchmark
Zheng, L., Shen, L., Tian, L., Wang, S., Wang, J., & Tian, Q. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.133 · Cited by 4,728 · see citing works
Randaugment: Practical automated data augmentation with a reduced search space
Cubuk, E. D., Zoph, B., Shlens, J., & Le, Q. V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00359 · Cited by 3,223 · see citing works
Learning Efficient Convolutional Networks through Network Slimming
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., & Zhang, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.298 · Cited by 2,761 · see citing works
Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., & Hu, H. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00320 · Cited by 2,294 · see citing works
Deformable DETR: Deformable Transformers for End-to-End Object Detection
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., & Dai, J. (2020) · arXiv · doi:10.48550/arxiv.2010.04159 · Cited by 1,837 · see citing works
BranchyNet: Fast inference via early exiting from deep neural networks
Teerapittayanon, S., McDanel, B., & Kung, H. T. (2016) · 2016 23rd International Conference on Pattern Recognition (ICPR) · doi:10.1109/icpr.2016.7900006 · Cited by 1,326 · see citing works
Quantifying Attention Flow in Transformers
Abnar, S., & Zuidema, W. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.385 · Cited by 995 · see citing works
End-to-End Video Instance Segmentation with Transformers
Wang, Y., Xu, Z., Wang, X., Shen, C., Cheng, B., Shen, H., & Xia, H. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00863 · Cited by 706 · see citing works
Neimark, D., Bar, O., Zohar, M., & Asselmann, D. (2021) · 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw54120.2021.00355 · Cited by 551 · see citing works
Escaping the Big Data Paradigm with Compact TransformersPreprint
Hassani, A., Walton, S., Shah, N., Abuduweili, A., Li, J., & Shi, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.05704 · Cited by 473 · see citing works
DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
Rao, Y., Zhao, W., Liu, B., Lu, J., Zhou, J., & Hsieh, C.-J. (2021) · arXiv · doi:10.48550/arxiv.2106.02034 · Cited by 448 · cited 1× · see citing works
“Similarly, DynamicViT (Rao et al. 2021) also prunes the less informative tokens by training a prediction module.”
Are Sixteen Heads Really Better than One?
Michel, P., Levy, O., & Neubig, G. (2019) · arXiv · doi:10.48550/arxiv.1905.10650 · Cited by 404 · see citing works
Reducing Transformer Depth on Demand with Structured Dropout
Fan, A., Grave, E., & Joulin, A. (2019) · arXiv · doi:10.48550/arxiv.1909.11556 · Cited by 387 · see citing works
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
Xin, J., Tang, R., Lee, J., Yu, Y., & Lin, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.204 · Cited by 353 · see citing works
VOLO: Vision Outlooker for Visual Recognition
Yuan, L., Hou, Q., Jiang, Z., Feng, J., & Yan, S. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3206108 · Cited by 309 · see citing works
What’s Hidden in a Randomly Weighted Neural Network?
Ramanujan, V., Wortsman, M., Kembhavi, A., Farhadi, A., & Rastegari, M. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01191 · Cited by 241 · see citing works
BERT Loses Patience: Fast and Robust Inference with Early ExitPreprint
Zhou, W., Xu, C., Ge, T., McAuley, J., Xu, K., & Wei, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.04152 · Cited by 207 · see citing works
Patch Slimming for Efficient Vision Transformers
Tang, Y., Han, K., Wang, Y., Xu, C., Guo, J., Xu, C., & Tao, D. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01185 · Cited by 201 · see citing works
Training data-efficient image transformers & distillation through attention
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jégou, H. (2020) · arXiv · doi:10.48550/arxiv.2012.12877 · Cited by 182 · see citing works
SCOP: Scientific Control for Reliable Neural Network PruningPreprint
Tang, Y., Wang, Y., Xu, Y., Tao, D., Xu, C., Xu, C., & Xu, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.10732 · Cited by 116 · see citing works
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector EliminationPreprint
Goyal, S., Choudhury, A. R., Raje, S., Chakaravarthy, V. T., Sabharwal, Y., & Verma, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.08950 · Cited by 102 · see citing works
Zhu, M., Tang, Y., & Han, K. (2021) · arXiv · doi:10.48550/arxiv.2104.08500 · Cited by 63 · see citing works
BranchyNet: Fast Inference via Early Exiting from Deep Neural NetworksPreprint
Teerapittayanon, S., McDanel, B., & Kung, H. T. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.01686 · Cited by 58 · see citing works
Token Labeling: Training a 85.4% Top-1 Accuracy Vision Transformer with 56M Parameters on ImageNet
Jiang, Z., Hou, Q., Yuan, L., Zhou, D., Jin, X., Wang, A., & Feng, J. (2021) · Cited by 32 · see citing works
Scalable Visual Transformers with Hierarchical PoolingPreprint
Pan, Z., Zhuang, B., Liu, J., He, H., & Cai, J. (2021) · arXiv (Cornell University) · Cited by 22 · see citing works
MLPruning: A Multilevel Structured Pruning Framework for Transformer-based ModelsPreprint
Yao, Z., Ma, L., Shen, S., Keutzer, K., & Mahoney, M. W. (2021) · arXiv (Cornell University) · Cited by 6 · see citing works
Armour: Generalizable Compact Self-Attention for Vision TransformersPreprint
Meng, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.01778 · Cited by 3 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
4 of the 39 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations
Cheng, H., Zhang, M., & Shi, Q. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3447085 · Cited by 554 · see citing works
A Survey on Efficient Vision Transformers: Algorithms, Techniques, and Performance Benchmarking
Papa, L., Russo, P., Amerini, I., & Zhou, L. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3392941 · Cited by 177 · see citing works
A Survey of Techniques for Optimizing Transformer Inference
Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., & Somani, A. K. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102990 · Cited by 154 · see citing works
X-Pruner: eXplainable Pruning for Vision Transformers
Lu, Y., & Xiang, W. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02333 · Cited by 80 · see citing works
Saha, S., & Xu, L. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130417 · Cited by 66 · see citing works
Attention Map Guided Transformer Pruning for Occluded Person Re-Identification on Edge Device
Mao, J., Yao, Y., Sun, Z., Huang, X., Shen, F., & Shen, H. T. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3265159 · Cited by 49 · see citing works
Cross-Modality Distillation for Multi-Modal Tracking
Zhang, T., Zhang, Q., Debattista, K., & Han, J. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3555485 · Cited by 46 · see citing works
Vision transformer models for mobile/edge devices: a survey
Lee, S. I., Koo, K., Lee, J. H., Lee, G., Jeong, S., Seongjun, O., & Kim, H. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01312-0 · Cited by 39 · see citing works
A survey on deep learning fundamentals
Tian, C., Cheng, T., Peng, Z., Zuo, W., Tian, Y., Zhang, Q., Wang, F., & Zhang, D. D. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11368-7 · Cited by 38 · see citing works
Contemporary Advances in Neural Network Quantization: A Survey
Li, M., Huang, Z., Chen, L., Ren, J., Jiang, M., Li, F., Fu, J., & Gao, C. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650109 · Cited by 33 · see citing works
Yu, H., Chen, S., Chu, Y., Li, M., Ding, Y., Cui, R., & Zhao, X. T. (2024) · Applied Energy · doi:10.1016/j.apenergy.2024.124007 · Cited by 25 · see citing works
Isomorphic Pruning for Vision Models
Fang, G., Ma, X., Mi, M. B., & Wang, X. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73404-5_14 · Cited by 24 · see citing works
TFormer: A Transmission-Friendly ViT Model for IoT Devices
Lu, Z., Ding, C., Juefei-Xu, F., Boddeti, V. N., Wang, S., & Yang, Y. (2023) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2022.3222765 · Cited by 24 · see citing works
EvolutionViT: Multi-objective evolutionary vision transformer pruning under resource constraints
Liu, L., Yen, G. G., & He, Z. (2024) · Information Sciences · doi:10.1016/j.ins.2024.121406 · Cited by 18 · see citing works
ELSA: Exploiting Layer-wise N:M Sparsity for Vision Transformer Acceleration
Huang, N.-C., Chang, C.-C., Lin, W., Taka, E., Marculescu, D., & Wu, K. H. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00799 · Cited by 18 · see citing works
UPDP: A Unified Progressive Depth Pruner for CNN and Vision Transformer
Liu, J., Tang, D., Huang, Y., Zhang, L., Zeng, X. C., Li, D., Lu, M., Peng, J., Wang, Y., Jiang, F., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i12.29296 · Cited by 18 · see citing works
Efficient token pruning in Vision Transformers using an attention-based Multilayer Network
Marchetti, M., Traini, D., Domenico Ursino, A., & Virgili, L. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127449 · Cited by 17 · see citing works
Wang, D., Zhang, J., Liu, S. B., Lyu, F., Huang, W., & Bing Hong, X. U. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3384610 · Cited by 17 · see citing works
An Efficient Window-Based Vision Transformer Accelerator via Mixed-Granularity Sparsity
Dong, Q., Zhang, S., & Wang, Z. (2025) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2025.3527541 · Cited by 16 · see citing works
OnDev-LCT: On-Device Lightweight Convolutional Transformers towards federated learning
Thwal, C. M., Nguyen, M. N. H., Tun, Y. L., Kim, S. T., Thai, M. T., & Hong, C. S. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.11.044 · Cited by 18 · see citing works
Developmental Plasticity-Inspired Adaptive Pruning for Deep Spiking and Artificial Neural Networks
Han, B., Zhao, F., Zeng, Y., & Shen, G. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3467268 · Cited by 16 · see citing works
DCT-ViT: High-Frequency Pruned Vision Transformer With Discrete Cosine Transform
Lee, J., & Kim, H. (2024) · IEEE Access · doi:10.1109/access.2024.3410231 · Cited by 14 · see citing works
Accelerating ViT Inference on FPGA through Static and Dynamic Pruning
Parikh, D., Li, S., Zhang, B., Kannan, R., Busart, C. E., & Prasanna, V. K. (2024) · IEEE Symposium on Field-Programmable Custom Computing Machines · doi:10.1109/fccm60383.2024.00018 · Cited by 17 · see citing works
Retracted: DSiV: Data Science for Intelligent Vehicles
Zhang, J., Pu, J., Chen, J., Fu, H., Tao, Y., Wang, S., Chen, Q., Xiao, Y., Chen, S., Cheng, Y., et al. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3264601 · Cited by 12 · see citing works
Self-citationRetractedYu, Z. H., Huang, Z., Hou, M., Yan, Y., & Liu, Y.-S. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126921 · Cited by 10 · see citing works
PQ-SAM: Post-training Quantization for Segment Anything Model
Liu, X., Ding, X., Yu, L., Xi, Y., Li, W., Tu, Z., Hu, J., Chen, H., Yin, B., & Xiong, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72684-2_24 · Cited by 10 · see citing works
Long-range zero-shot generative deep network quantization
Luo, Y., Gao, Y., Zhang, Z., Fan, J., Zhang, H., & Xu, M. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.07.042 · Cited by 10 · see citing works
A missing multimodal imputation diffusion model for 2D X-ray and 3D CT in COVID-19 diagnosis
Gao, Z., Chen, D., & Shen, Y. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127367 · Cited by 9 · see citing works
Adaptive patch selection to improve Vision Transformers through Reinforcement Learning
Cauteruccio, F., Marchetti, M., Traini, D., Domenico Ursino, A., & Virgili, L. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06516-z · Cited by 9 · see citing works
GTCO: Graph and Tensor Co-Design for Transformer-Based Image Recognition on Tensor Cores
Bai, Y., Yao, X., Sun, Q. L., Zhao, W., Chen, S., Wang, Z., & Yu, B. (2023) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2023.3317169 · Cited by 9 · see citing works
Markov-PQ: Joint Pruning-Quantization via Learnable Markov Chain
Li, Y., Zhang, X., Xie, W., Zhang, J., Fang, L., & Du, J. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3437182 · Cited by 8 · see citing works
BinaryViT: Toward Efficient and Accurate Binary Vision Transformers
Xiao, J., Li, Z., Li, J., Yang, L., & Gu, Q. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3457610 · Cited by 11 · see citing works
Ahmed, S., Arafat, A. A., Najafi, D., Mahmood, A. U., Rizve, M. N., Nahian, M. A., Zhou, R., Angizi, S., & Rakin, A. S. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02806 · Cited by 8 · see citing works
A General and Efficient Training for Transformer via Token Expansion
Huang, W., Shen, Y., Xie, J., Zhang, B., He, G., Li, K., Sun, X., & Lin, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01494 · Cited by 8 · see citing works
LRP-QViT: Mixed-Precision Vision Transformer Quantization Using Layer Importance Score
Ranjan, N., & Savakis, A. E. (2025) · IEEE International Conference on Digital Signal Processing (DSP) · doi:10.1109/dsp65409.2025.11075195 · Cited by 8 · see citing works
FLORA: Fine-grained Low-Rank Architecture Search for Vision Transformer
Chang, C.-C., Sung, Y.-Y., Yu, S., Huang, N.-C., Marculescu, D., & Wu, K. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00247 · Cited by 7 · see citing works
Efficient Visual Transformer by Learnable Token Merging
Wang, Y., & Yang, Y. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3588186 · Cited by 6 · see citing works
Channel Pruning Method Based on Decoupling Feature Scale Distribution in Batch Normalization Layers
Qiu, Z., Wei, P., Yao, M., Zhang, R., & Kuang, Y. (2024) · IEEE Access · doi:10.1109/access.2024.3382994 · Cited by 6 · see citing works
ChatGPT – Technical Research Model, Capability Analysis, and Application Prospects
Wang, T., & Zhu, Q. (2024) · IEEE Advanced Information Technology, Electronic and Automation Control Conference (IAEAC) · doi:10.1109/iaeac59436.2024.10503756 · Cited by 6 · see citing works
All-in-One Hardware-Oriented Model Compression for Efficient Multi-Hardware Deployment
Wang, H., Ling, P., Fan, X. N., Tu, T., Zheng, J., Chen, H., Jin, Y., & Chen, E. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3434626 · Cited by 6 · see citing works
ViT- ToGo: Vision Transformer Accelerator with Grouped Token Pruning
Lee, S., Cho, K.-M., Kwon, E., Park, S., Kim, S., & Kang, S. (2024) · Design, Automation, and Test in Europe · doi:10.23919/date58400.2024.10546804 · Cited by 6 · see citing works
Life regression based patch slimming for vision transformers
Chen, J., Chen, L., Jiang, Y., Shi, T., Cheng, L., Feng, Z., & Song, M. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106340 · Cited by 6 · see citing works
An Information Theory-Inspired Strategy for Automated Network Pruning
Zheng, X., Ma, Y., Xi, T., Zhang, G., Ding, E., Li, Y., Chen, J., Tian, Y., & Ji, R. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02437-z · Cited by 5 · see citing works
EasyViT: An Adaptive Collaborative Edge Computing Framework for Vision Transformer
Wen, D., Liang, G., Li, T., Chen, L., Li, J., & Li, T. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3578605 · Cited by 5 · see citing works
PRANCE: Joint Token-Optimization and Structural Channel-Pruning for Adaptive ViT Inference
Li, Y., Tang, C., Meng, Y., Fan, J., Chai, Z., Ma, X., Wang, Z., & Zhu, W. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3605239 · Cited by 4 · see citing works
SI-BiViT: Binarizing Vision Transformers with Spatial Interaction
Yin, P., Zhu, X., Song, J., Gao, L., & Shen, H. T. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680872 · Cited by 4 · see citing works
Energy-Efficient RISC-V-Based Vector Processor for Cache-Aware Structurally-Pruned Transformers
Min, J. G., Kam, D., Byun, Y., Park, G., & Lee, Y. (2023) · International Symposium on Low Power Electronics and Design (ISLPED) · doi:10.1109/islped58423.2023.10244508 · Cited by 4 · see citing works
Compressing Vision Transformer from the View of Model Property in Frequency Domain
Wang, Z., Xie, X., Luo, H., Huang, T., Dong, W., Xiong, K., Liu, Y., Li, X., Wang, F., & Shi, G. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02561-w · Cited by 3 · see citing works
SNP: Structured Neuron-Level Pruning to Preserve Attention Scores
Shim, K.-H., Yun, J., & Choi, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73007-8_6 · Cited by 3 · see citing works
POA: Pre-training Once for Models of All Sizes
Zhang, Y., Guo, X., Lao, J., Yu, L., Ru, L., Wang, J., Ye, G., He, H., Chen, J., & Yang, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72646-0_8 · Cited by 3 · see citing works
Energy-Aware Vision Model Partitioning for Edge AI
Katare, D., Zhou, M., Chen, Y., Janssen, M., & Ding, A. Y. (2025) · ACM Symposium on Applied Computing · doi:10.1145/3672608.3707792 · Cited by 3 · see citing works
Mix-QSAM: Mixed-Precision Quantization of the Segment Anything Model
Ranjan, N., & Savakis, A. E. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00310 · Cited by 4 · see citing works
Chen, C., Xie, S., Liu, J., Cheng, Z., Zeng, Z., & Zhuang, H. (2024) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2024.3364677 · Cited by 3 · see citing works
Vu, D., Dang, T., Nguyen, Q.-T., & Pham, T. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-2074-6_5 · Cited by 2 · see citing works
Importance-Guided Interpretability and Pruning for Video Transformers in Driver Action Recognition
Panadero, R., Schörkhuber, D., & Gelautz, M. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00517 · Cited by 2 · see citing works
Towards Faster and More Compact Foundation Models for Molecular Property Prediction
Ghunaim, Y., Villa, A., Ignácz, G., Székely, G., Alfarra, M., & Ghanem, B. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00010 · Cited by 2 · see citing works
AMAP: Automatic Multihead Attention Pruning by Similarity-Based Pruning Indicator
Lee, E., & Hwang, Y. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3606750 · Cited by 2 · see citing works
Back to fundamentals: Low-level visual features guided progressive token pruning
Ouyang, Y., Liang, Y., Li, Q., Guo, X., Luo, Y., Wu, D., Wang, H. E., & Pan, Y. (2025) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2025.103579 · Cited by 2 · see citing works
Attacking Compressed Vision Transformers
Parekh, S., Shukla, P., & Shah, D. P. (2023) · Lecture notes in networks and systems · doi:10.1007/978-3-031-28073-3_50 · Cited by 2 · see citing works
Sparse Double Descent in Vision Transformers: Real or Phantom Threat?
Quétu, V., Milovanović, M., & Tartaglione, E. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43153-1_41 · Cited by 2 · see citing works
Similarity-Aware Fast Low-Rank Decomposition Framework for Vision Transformers
Luo, Y.-J., Tai, Y.-S., Lin, M.-G., & Wu, A.-Y. A. (2024) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas58744.2024.10557934 · Cited by 2 · see citing works
DARIO: Differentiable Vision Transformer Pruning With Low-Cost Proxies
Sun, H., Heuillet, A., Mohr, F., & Tabia, H. (2024) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2024.3501685 · Cited by 2 · see citing works
Huang, H., Wang, M., Li, X., Zhong, B., Yang, Z., Lu, T., Zhang, Y., & Yu, Z. (2025) · IEEE Transactions on Circuits & Systems II Express Briefs · doi:10.1109/tcsii.2025.3590432 · Cited by 2 · see citing works
Self-citationFinding Optimal Numerical Format for Sub-8-Bit Post-Training Quantization of Vision Transformers
Lee, J., Hwang, Y., & Choi, J. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096798 · Cited by 2 · see citing works
An Automatic Vision Transformer Pruning Method Based on Binary Particle Swarm Optimization
Tmamna, J., Ayed, E. B., Fourati, R., & Ayed, M. B. (2023) · IEEE Symposium on Computers and Communications (ISCC) · doi:10.1109/iscc58397.2023.10218144 · Cited by 2 · see citing works
Shieh, Y.-D., Lin, M.-G., Wang, H., & Wu, A.-Y. A. (2025) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas56072.2025.11043915 · Cited by 1 · see citing works
Revisiting Reconfigurable Acceleration of Vision Transformer with Patch Pruning
Chen, H., Ni, Y., Huang, W., Oh, H., Das, T., Wen, F., & Imani, M. (2025) · International Symposium on Low Power Electronics and Design (ISLPED) · doi:10.1109/islped65674.2025.11261783 · Cited by 1 · see citing works
TransFusion: End-to-End Transformer Acceleration via Graph Fusion and Pipelining
Zhang, L., Amaral, J. N., & Niu, D. (2025) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3725843.3756105 · Cited by 1 · see citing works
INSPIRE: In-Sensor Compressed Weight Retrieval for Enhancing ViT Efficiency at Edge
Ahmed, S., Najafi, D., Nahian, M. A., Khoshavi, N., Arafat, A. A., Rizve, M. N., Nikdast, M., Rakin, A. S., & Angizi, S. (2026) · Design, Automation, and Test in Europe · doi:10.23919/date69613.2026.11539411 · Cited by 1 · see citing works
Voting from Nearest Tasks: Meta-Vote Pruning of Pre-trained Models for Downstream Tasks
Zhao, H., Zhou, T., Long, G., Jiang, J. B., & Zhang, C. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43415-0_4 · Cited by 1 · see citing works
Neural-Based Video Compression on Solar Dynamics Observatory Images
Khoshkhahtinat, A., Zafari, A., Mehta, P. M., Nasrabadi, N. M., Thompson, B. J., Kirk, M. S., & da Silva, D. E. (2024) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2024.3409524 · Cited by 1 · see citing works
RobSparse: Automatic Search for GPU-Friendly Robust and Sparse Vision Transformers
Su, Y., Zhang, S., Wang, Y., Wang, X., Zhao, L., Meng, D., & Hou, R. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-2064-7_23 · Cited by 1 · see citing works
Learning to Prune and Low-Rank Adaptation for Compact Language Model Deployment
Ali, A. H., Zhang, F., Yang, L., & Fan, D. (2025) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1145/3658617.3697648 · Cited by 1 · see citing works
Memory-Efficient Vision Transformers: An Activation-Aware Mixed-Rank Compression Strategy
Azizi, S., Nazemi, M., & Pedram, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-91979-4_6 · Cited by 1 · see citing works
A Lightweight Dynamic Hierarchical Neural Network Model and Learning Paradigm
Liao, L., Lin, J., Zhang, W., & Cai, J. (2025) · International Journal of Intelligent Systems · doi:10.1155/int/6833629
Intra-head pruning for vision transformers via inter-layer dimension relationship modeling
Zhang, P., Tian, C., Zhao, L., & Duan, Z. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107656
DPP-ViT: Dynamic Patch Pruning for Low Complexity Vision Transformer Accelerator
Cho, H., Jo, J., Hwang, S.-E., & Park, J. (2025) · IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS) · doi:10.1109/aicas64808.2025.11173125
S-HUB: Scalable Deep Neural Network Fusion for Smart Home Hubs
Yao, Y., Zhao, D. Y., Xu, N., & Zhang, Z. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3626540
Gupta, S. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-032-03072-6_17
Kurniawan, G. W., Hsieh, J.-W., & Sun, C. (2025) · IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr67560.2025.00091
Token Compression Meets Compact Vision Transformers: a Survey and Comparative Evaluation for Edge AI
Nguyen, P., & Cheung, N. M. (2025) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc65261.2025.11249396
Width & Depth Pruning for Vision Transformers
Association for Artificial Intelligence 2022, Cheng, Y., Chu, W., Cui, L., Huang, K., Wang, M., & Yu, F. (2022) · Underline Science Inc. · doi:10.48448/kds7-5y12
Self-citationQuery-Key Joint Decomposition for Efficient Compression of TransformersPreprint
Zhou, Y., Zhao, S., Tang, W., Ma, A., & Kung, S. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5875209
Entropy-Guided Condensing for Vision Transformer
Lin, S., Lyu, P., Liu, D., Li, Z., Wang, W., Chang, X., & Zheng, Y. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-026-02753-y
MO-SAE:Multi-Objective Stacked Autoencoders Optimization for Edge Anomaly Detection
Zhang, L., Kong, S., Guo, T., Li, S., & Ji, Z. (2025) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc58881.2025.11343286
Glandorf, P., & Rosenhahn, B. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00316
Violence detection in public places based on YOLOv8 algorithm
Jin, Z. (2026) · AIP conference proceedings · doi:10.1063/12.0043524
ABC-Pruning of Convolutional Neural Networks and Vision TransformersPreprint
Dorighello, R. S., Lüders, R., & Delgado, M. B. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6538786
Tan, Z., Karkehabadi, A., Homayoun, H., Nikoubin, T., & Sasan, A. (2026) · ACM Great Lakes Symposium on VLSI (GLSVLSI) · doi:10.1145/3787109.3816377
HypTM: Efficient Vision Transformers With Hyperbolic Token Merging
Chen, X. W., Zhang, X., Xie, W., Ma, J., Li, Y., & Fang, L. (2026) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2026.3709249
Jin, W., Yu, N., Zhao, Q., & Wu, J. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2759-4_1
Compression towards the minimal performance-guaranteed neural network
Tan, Z., He, J., & HuaZhen, L. I. N. (2026) · Scientia Sinica Mathematica · doi:10.1360/ssm-2026-0089
LayerNorm-Aware Compression of Vision Transformers via Quantization and Pruning
Sharma, S., Archita, & Kancharla, P. (2026) · Proceedings - International Conference on Image Processing · doi:10.1109/icip61757.2026.11630285
Single domain generalized polyp detection in colonoscopy scene utilizing vision foundation models
Gan, T., Cai, Y., & Ye, X. (2026) · Medical Image Analysis · doi:10.1016/j.media.2026.104268
Wan, R., Pu, H., Liang, Y., Xu, J., & Wang, D. (2026) · The Visual Computer · doi:10.1007/s00371-026-04733-2
- Authors
- Yu, F., Huang, K., Wang, M., Cheng, Y., Chu, W., & Cui, L.
- Affiliations
- Ant Group (China) (CN)
- Chinese Academy of Sciences (CN)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2022-06-28
- Volume
- 36
- Issue
- 3
- Pages
- 3143-3151
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W4283811336
- Cited by
- 95
- References
- 39
- Citations by year
- 2026: 11 · 2025: 37 · 2024: 32 · 2023: 13 · 2022: 2
- FWCI
- 7.62
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- Funders
- National Natural Science Foundation of China (61672498)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- vision transformer, neural network pruning, shallow classifiers, intermediate features, computational complexity reduction, edge deployment, DeiT, Swin Transformer, model compression
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
