Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space
Open accessAbstract
This paper explores the feasibility of finding an optimal sub-model from a vision transformer and introduces a pure vision transformer slimming (ViT-Slim) framework. It can search a sub-structure from the original model end-to-end across multiple dimensions, including the input tokens, MHSA and MLP modules with state-of-the-art performance. Our method is based on a learnable and unified ℓ1sparsity constraint with pre-defined factors to reflect the global importance in the continuous searching space of different dimensions. The searching process is highly efficient through a single-shot training scheme. For instance, on DeiT-S, ViT-Slim only takes ~43 GPU hours for the searching process, and the searched structure is flexible with diverse dimensionalities in different modules. Then, a budget threshold is employed according to the requirements of accuracy-FLOPs trade-off on running devices, and a retraining process is performed to obtain the final model. The extensive experiments show that our ViT-Slim can compress up to 40% of parameters and 40% FLOPs on various vision transformers while increasing the accuracy by ~0.6% on ImageNet. We also demonstrate the advantage of our searched models on several downstream datasets. Our code is available at https://github.com/Arnav0400/ViT-Slim.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ViT-Slim improves baseline DeiT-small accuracy and reduces the FLOP count.
“A few techniques like ViT-Slim [111] improve the baseline DeiT-small accuracy and reduce the FLOP count.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [111]
ViTSlim is a one-shot framework to search for an efficient architecture over three important modules: MHA, FFN and patching mechanism.
“ViTSlim [111] is another one-shot framework to search for an efficient architecture over three important modules - MHA, FFN and patching mechanism.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [111]
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · see citing works
Self-citationMobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755 · see citing works
Xception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380 · see citing works
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015 · see citing works
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285 · see citing works
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072 · see citing works
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · see citing works
ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
Ma, N., Zhang, X., Zheng, H.-T., & Sun, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01264-9_8 · Cited by 7,356 · cited 1× · see citing works
“The compression research extends from channel pruning [17,27,56], quantization and binarization [1, 9, 21, 29, 33, 63], knowledge distillation [18, 32, 36, 39–41], compact neural network de-sign [10,22,31,52,60] to architecture search [35,65,66].”
MethodLearning Transferable Architectures for Scalable Image Recognition
Zoph, B., Vasudevan, V., Shlens, J., & Le, Q. V. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00907 · Cited by 6,631 · see citing works
SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., & Keutzer, K. (2016) · arXiv · doi:10.48550/arxiv.1602.07360 · Cited by 6,201 · see citing works
mixup: Beyond Empirical Risk Minimization
Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2017) · arXiv · doi:10.48550/arxiv.1710.09412 · Cited by 5,831 · see citing works
CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features
Yun, S., Han, D., Chun, S., Oh, S. J., Yoo, Y., & Choe, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00612 · Cited by 5,400 · see citing works
Neural Architecture Search with Reinforcement LearningPreprint
Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 5,091 · see citing works
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J. M., & Luo, P. (2021) · arXiv · doi:10.48550/arxiv.2105.15203 · Cited by 5,090 · see citing works
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P. H. S., et al. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00681 · Cited by 4,008 · cited 1× · see citing works
“Transformer [49] has been a strong network model for various vision tasks, such as image classification [14, 28, 38,46,53], object detection [3,43,64], segmentation [50,54, 61], etc.”
MethodRandom Erasing Data Augmentation
Zhong, Z., Zheng, L., Kang, G., Li, S., & Yang, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.7000 · Cited by 3,617 · see citing works
DARTS: Differentiable architecture search
Liu, H., Simonyan, K., & Yang, Y. (2018) · arXiv · doi:10.48550/arxiv.1806.09055 · Cited by 3,248 · see citing works
Randaugment: Practical automated data augmentation with a reduced search space
Cubuk, E. D., Zoph, B., Shlens, J., & Le, Q. V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00359 · Cited by 3,223 · see citing works
FitNets: Hints for Thin Deep Nets
Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.6550 · Cited by 2,859 · see citing works
Learning Efficient Convolutional Networks through Network Slimming
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., & Zhang, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.298 · Cited by 2,761 · see citing works
Self-citationChannel Pruning for Accelerating Very Deep Neural Networks
He, Y., Zhang, X., & Sun, J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.155 · Cited by 2,647 · see citing works
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and ActivationsPreprint
Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., & Bengio, Y. (2016) · Journal of machine learning research · doi:10.48550/arxiv.1609.07061 · Cited by 2,196 · see citing works
Deep Networks with Stochastic Depth
Huang, G., Sun, Y., Liu, Z., Sedra, D., & Weinberger, K. Q. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_39 · Cited by 2,104 · see citing works
Self-citationDeformable DETR: Deformable Transformers for End-to-End Object Detection
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., & Dai, J. (2020) · arXiv · doi:10.48550/arxiv.2010.04159 · Cited by 1,837 · see citing works
DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., & Zou, Y. (2016) · arXiv · doi:10.48550/arxiv.1606.06160 · Cited by 1,690 · see citing works
ProxylessNAS: Direct Neural Architecture Search on Target Task and HardwarePreprint
Cai, H., Zhu, L., & Han, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.00332 · Cited by 1,652 · see citing works
The iNaturalist Species Classification and Detection Dataset
Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., & Belongie, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00914 · Cited by 1,451 · see citing works
AMC: AutoML for Model Compression and Acceleration on Mobile Devices
He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J., & Han, S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_48 · Cited by 1,400 · see citing works
Self-citationFBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
Wu, B., Keutzer, K., Dai, X., Zhang, P., Wang, Y., Sun, F., Wu, Y., Tian, Y., Vajda, P., & Jia, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01099 · Cited by 1,302 · see citing works
Transformer Interpretability Beyond Attention Visualization
Chefer, H., Gur, S., & Wolf, L. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00084 · Cited by 903 · see citing works
Single Path One-Shot Neural Architecture Search with Uniform Sampling
Guo, Z., Zhang, X., Mu, H., Heng, W., Liu, Z., Wei, Y., & Sun, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58517-4_32 · Cited by 882 · see citing works
Self-citationWhen does label smoothing help?
Müller, R., Kornblith, S., & Hinton, G. (2019) · arXiv · doi:10.48550/arxiv.1906.02629 · Cited by 859 · see citing works
End-to-End Video Instance Segmentation with Transformers
Wang, Y., Xu, Z., Wang, X., Shen, C., Cheng, B., Shen, H., & Xia, H. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00863 · Cited by 706 · see citing works
NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications
Yang, T.-J., Howard, A., Chen, B., Zhang, X., Go, A., Sandler, M., Sze, V., & Adam, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01249-6_18 · Cited by 572 · see citing works
Visual Transformers: Token-based Image Representation and Processing for Computer VisionPreprint
Wu, B., Xu, C., Dai, X., Wan, A., Zhang, P., Yan, Z., Tomizuka, M., Gonzalez, J., Keutzer, K., & Vajda, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03677 · Cited by 529 · see citing works
DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
Rao, Y., Zhao, W., Liu, B., Lu, J., Zhou, J., & Hsieh, C.-J. (2021) · arXiv · doi:10.48550/arxiv.2106.02034 · Cited by 448 · cited 5× · see citing works
“PS-ViT-S [45] and Dynamic-ViT-S [34] performs comparably to our method with respect to FLOPs, but they do not bring in any improvement in parameter efficiency.”
BackgroundMethodInfluentialShift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions
Wu, B., Wan, A., Yue, X., Jin, P., Zhao, S., Golmant, N., Gholaminejad, A., Gonzalez, J. E., & Keutzer, K. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00951 · Cited by 395 · see citing works
Rethinking Transformer-based Set Prediction for Object Detection
Sun, Z., Cao, S., Yang, Y., & Kitani, K. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00359 · Cited by 389 · see citing works
ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions
Liu, Z., Shen, Z., Savvides, M., & Cheng, K.-T. T. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58568-6_9 · Cited by 388 · see citing works
Self-citationFairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search
Chu, X., Zhang, B., & Xu, R. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01202 · Cited by 358 · see citing works
Regularized Evolution for Image Classifier Architecture Search
Real, E., Aggarwal, A., Huang, Y., & Le, Q. V. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33014780 · Cited by 328 · see citing works
Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
Ye, J., Lu, X., Lin, Z., & Wang, J. Z. (2018) · arXiv · doi:10.48550/arxiv.1802.00124 · Cited by 328 · see citing works
AutoFormer: Searching Transformers for Visual Recognition
Chen, M., Peng, H., Fu, J., & Ling, H. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01205 · Cited by 323 · see citing works
BigNAS: Scaling up Neural Architecture Search with Big Single-Stage Models
Yu, J., Jin, P., Liu, H., Bender, G., Kindermans, P.-J., Tan, M., Huang, T. S., Song, X., Pang, R., & Le, Q. V. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58571-6_41 · Cited by 252 · see citing works
Patch Slimming for Efficient Vision Transformers
Tang, Y., Han, K., Wang, Y., Xu, C., Guo, J., Xu, C., & Tao, D. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01185 · Cited by 201 · see citing works
Training data-efficient image transformers & distillation through attention
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jégou, H. (2020) · arXiv · doi:10.48550/arxiv.2012.12877 · Cited by 182 · see citing works
XNOR-Net++: Improved Binary Neural Networks
Bulat, A., & Tzimiropoulos, G. (2019) · arXiv · doi:10.48550/arxiv.1909.13863 · Cited by 165 · see citing works
Chasing Sparsity in Vision Transformers: An End-to-End Exploration
Chen, T., Cheng, Y., Gan, Z., Yuan, L., Zhang, L., & Wang, Z. (2021) · arXiv · doi:10.48550/arxiv.2106.04533 · Cited by 111 · see citing works
Operation-Aware Soft Channel Pruning using Differentiable MasksPreprint
Kang, M. S., & Han, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.03938 · Cited by 100 · see citing works
GLiT: Neural Architecture Search for Global and Local Image Transformer
Chen, B., Li, P., Li, C., Li, B., Bai, L., Chen, L., Sun, M., Yan, J., & Ouyang, W. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00008 · Cited by 100 · see citing works
Binarizing MobileNet via Evolution-Based Searching
Phan, H., Liu, Z., Huynh, D. N., Savvides, M., Cheng, K.-T. T., & Shen, Z. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01343 · Cited by 48 · see citing works
Self-citationMEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without TricksPreprint
Shen, Z., & Savvides, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.08453 · Cited by 45 · see citing works
Self-citationA study on the coexistence analysis of intelligent transport systems in 60GHz bandsPreprint
Choi, S. (2018) · EasyChair preprint · doi:10.29007/shbv · Cited by 43 · see citing works
A Fast Knowledge Distillation Framework for Visual Recognition
Shen, Z., & Xing, E. P. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20053-3_39 · Cited by 40 · see citing works
Self-citationPruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Prior
Lin, Z. J., Liu, J. Z., Yang, Z., Hua, N., & Roth, D. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.64 · Cited by 38 · see citing works
Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical StudyPreprint
Shen, Z., Liu, Z., Xu, D., Chen, Z., Cheng, K.-T. T., & Savvides, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.00676 · Cited by 38 · see citing works
Self-citationBN-NAS: Neural Architecture Search with Batch Normalization
Chen, B., Li, P., Li, B., Chen, L., Li, C., Sun, M., Yan, J., & Ouyang, W. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00037 · Cited by 32 · see citing works
Joint Multi-Dimension Pruning via Numerical Gradient Update
Liu, Z., Zhang, X., Shen, Z., Wei, Y., Cheng, K.-T. T., & Sun, J. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3112041 · Cited by 28 · see citing works
Self-citationShen, Z., Liu, Z., & Xing, E. P. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20053-3_42 · Cited by 19 · see citing works
Self-citationVision Transformer Architecture SearchPreprint
Su, X. Q., You, S., Xie, J., Zheng, M., Wang, F., Chen, Q., Zhang, C., Wang, X., & Xu, C. (2021) · arXiv (Cornell University) · Cited by 10 · see citing works
Flagged first, then by citations
2 of the 64 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Structured Pruning for Deep Convolutional Neural Networks: A Survey
He, Y., & Xiao, L. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3334614 · Cited by 398 · see citing works
Rethinking Vision Transformers for MobileNet Size and Speed
Li, Y., Ju, H., Wen, Y., Evangelidis, G., Salahi, K., Wang, Y., Tulyakov, S., & Ren, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01549 · Cited by 369 · see citing works
A Survey of Techniques for Optimizing Transformer Inference
Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., & Somani, A. K. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102990 · Cited by 154 · see citing works
TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance
Wu, K., Peng, H., Zhou, Z., Xiao, B., Liu, M., Yuan, L., Xuan, H., Valenzuela, M., Chen, X. S., Wang, X., et al. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02008 · Cited by 89 · see citing works
X-Pruner: eXplainable Pruning for Vision Transformers
Lu, Y., & Xiang, W. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02333 · Cited by 80 · see citing works
Which Tokens to Use? Investigating Token Reduction in Vision Transformers
Haurum, J. B., Escalera, S., Taylor, G. W., & Moeslund, T. B. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00085 · Cited by 47 · see citing works
DiffRate : Differentiable Compression Rate for Efficient Vision Transformers
Chen, M., Shao, W., Xu, P., Lin, M., Zhang, K., Chao, F., Ji, R., Qiao, Y., & Luo, P. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01574 · Cited by 43 · see citing works
Lin, M., Chen, M., Zhang, Y.-X., Shen, C.-H., Ji, R.-R., & Cao, L.-J. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-023-01861-3 · Cited by 35 · see citing works
Bennetot, A., Franchi, G., Ser, J. D., Chatila, R., & Díaz-Rodríguez, N. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109947 · Cited by 33 · see citing works
Boost Vision Transformer with GPU-Friendly Sparsity and Quantization
Yu, C., Chen, T., Gan, Z., & Fan, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02170 · Cited by 33 · see citing works
Contemporary Advances in Neural Network Quantization: A Survey
Li, M., Huang, Z., Chen, L., Ren, J., Jiang, M., Li, F., Fu, J., & Gao, C. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650109 · Cited by 33 · see citing works
Isomorphic Pruning for Vision Models
Fang, G., Ma, X., Mi, M. B., & Wang, X. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73404-5_14 · Cited by 24 · see citing works
Zhang, Y., Zhong, L., Ding, Y., Yu, H., & Zhai, Z. (2023) · Agriculture · doi:10.3390/agriculture13061264 · Cited by 24 · see citing works
Cao, J., Ye, P., Li, S., Yu, C., Tang, Y., Lu, J., & Chen, T. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01487 · Cited by 29 · see citing works
Lin, H., Bai, H., Liu, Z., Hou, L., Sun, M., Song, L., Wei, Y., & Surr, Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02584 · Cited by 22 · see citing works
Self-citationTraining-Free Transformer Architecture Search With Zero-Cost Proxy Guided Evolution
Zhou, Q., Sheng, K., Zheng, X., Li, K., Tian, Y., Chen, J., & Ji, R. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3378781 · Cited by 18 · see citing works
GhostViT: Expediting Vision Transformers via Cheap Operations
Cao, H., Qu, Z., Chen, G., Li, X.-Y., Thiele, L., & Knoll, A. C. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3326795 · Cited by 15 · see citing works
Global-Group Attention Network With Focal Attention Loss for Aerial Scene Classification
Zhao, Y., Chen, Y., Rong, Y., Xiong, S., & Lu, X. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3342171 · Cited by 14 · see citing works
DCT-ViT: High-Frequency Pruned Vision Transformer With Discrete Cosine Transform
Lee, J., & Kim, H. (2024) · IEEE Access · doi:10.1109/access.2024.3410231 · Cited by 14 · see citing works
Efficient Transformer-based 3D Object Detection with Dynamic Token Halting
Ye, M., Meyer, G. P., Chai, Y., & Liu, Q. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00775 · Cited by 11 · see citing works
MULTIFLOW: Shifting Towards Task-Agnostic Vision-Language Pruning
Farina, M., Mancini, M., Cunegatti, E., Cunegatti, E., Iacca, G., & Ricci, E. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01532 · Cited by 10 · see citing works
SceneFormer: Neural Architecture Search of Transformers for Remote Sensing Scene Classification
Tong, L., Liu, J., & Du, B. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3555207 · Cited by 10 · see citing works
Expediting Large-Scale Vision Transformer for Dense Prediction Without Fine-Tuning
Yuan, Y., Liang, W., Ding, H., Liang, Z., Zhang, C., & Hu, H. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3327511 · Cited by 14 · see citing works
Dense Vision Transformer Compression with Few Samples
Zhang, H., Zhou, Y., & Wang, G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01498 · Cited by 9 · see citing works
Agglomerative Token Clustering
Haurum, J. B., Escalera, S., Taylor, G. W., & Moeslund, T. B. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72998-0_12 · Cited by 9 · see citing works
Efficient feature selection for pre-trained vision transformers
Huang, L., Zeng, J., Yu, M., Ding, W., Bai, X., & Wang, K. (2025) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2025.104326 · Cited by 8 · see citing works
Building Vision Transformers with Hierarchy Aware Feature Aggregation
Chen, Y., Liu, H., Yin, H., & Fan, B. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00543 · Cited by 8 · see citing works
UAV-DETR: Few-Parameter DETR for Small Object Detection in High-Altitude UAV Images
Liao, N., Zhang, Y., Yu, Z., Huang, J., Zhu, M., & Peng, B. (2025) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2025.3645731 · Cited by 8 · see citing works
Spatiotemporal Attention-based Semantic Compression for Real-time Video Recognition
Li, N., Bennis, M., Iosifidis, A., & Zhang, Q. (2023) · IEEE Globecom Workshops (GC Wkshps) · doi:10.1109/gcwkshps58843.2023.10465187 · Cited by 8 · see citing works
BinaryFormer: A Hierarchical-Adaptive Binary Vision Transformer (ViT) for Efficient Computing
Wang, M., Xu, Z., Zheng, B., & Xie, W. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3396520 · Cited by 7 · see citing works
Efficient Visual Transformer by Learnable Token Merging
Wang, Y., & Yang, Y. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3588186 · Cited by 6 · see citing works
Cross-ViT: Cross-attention Vision Transformer for Image Duplicate Detection
Chandrasiri, M. D. N., & Talagala, P. D. (2023) · International Conference on Information Technology Research (ICITR) · doi:10.1109/icitr61062.2023.10382916 · Cited by 6 · see citing works
Three Guidelines You Should Know for Universally Slimmable Self-Supervised Learning
Cao, Y.-H., Sun, P., & Zhou, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01511 · Cited by 6 · see citing works
PYRA: Parallel Yielding Re-activation for Training-Inference Efficient Task Adaptation
Xiong, Y., Chen, H., Hao, T., Lin, Z., Han, J., Zhang, Y., Wang, G., Bao, Y., & Ding, G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72673-6_25 · Cited by 9 · see citing works
Slimmable Networks for Contrastive Self-supervised Learning
Zhao, S., Zhu, L., Wang, X., & Yang, Y. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02211-7 · Cited by 5 · see citing works
Token Compensator: Altering Inference Cost of Vision Transformer Without Re-tuning
Jie, S., Tang, Y., Guo, J., Deng, Z., Han, K., & Wang, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72640-8_5 · Cited by 5 · see citing works
PHD-NAS: Preserving helpful data to promote Neural Architecture Search
Lu, S., Hu, Y. H., Yang, L., Mei, J., Sun, Z., Tan, J., & Song, C. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127646 · Cited by 5 · see citing works
PRANCE: Joint Token-Optimization and Structural Channel-Pruning for Adaptive ViT Inference
Li, Y., Tang, C., Meng, Y., Fan, J., Chai, Z., Ma, X., Wang, Z., & Zhu, W. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3605239 · Cited by 4 · see citing works
Yang, X., & Wang, X. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72754-2_1 · Cited by 4 · see citing works
Zhou, S., Meng, S., Tian, H., Yu, J. W., & Wang, K. (2024) · doi:10.1145/3676536.3676667 · Cited by 4 · see citing works
XU, G., Huang, W., & Jia, W. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4893335 · Cited by 4 · see citing works
Straightforward Layer-Wise Pruning for More Efficient Visual Adaptation
Han, R., & Tang, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73220-1_14 · Cited by 3 · see citing works
Compressing Vision Transformer from the View of Model Property in Frequency Domain
Wang, Z., Xie, X., Luo, H., Huang, T., Dong, W., Xiong, K., Liu, Y., Li, X., Wang, F., & Shi, G. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02561-w · Cited by 3 · see citing works
MPNAS: Multimodal Sentiment Analysis Pruning via Neural Architecture Search
Zhang, B., Ren, A., Zhang, Z., Duan, M., Liu, D., Tan, Y., & Zhong, K. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887670 · Cited by 3 · see citing works
TSOSVNet: Teacher-student collaborative knowledge distillation for Online Signature Verification
Sekhar, C., V, Gautam, A., Pulabaigari, V., Sreeja, S. R., & Rama Krishna Sai, G. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00082 · Cited by 3 · see citing works
VMMP: Verifiable privacy-preserving multi-modal multi-task prediction
Bian, M., Ren, Y., He, G., Feng, G., & Zhang, X. (2024) · Information Sciences · doi:10.1016/j.ins.2024.120547 · Cited by 2 · see citing works
A Static-Dynamic Composition Framework for Efficient Action Recognition
Chen, X., Han, Y., Li, C., Chang, X., Sun, Y., & Yang, Y. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3525187 · Cited by 2 · see citing works
An Automatic Vision Transformer Pruning Method Based on Binary Particle Swarm Optimization
Tmamna, J., Ayed, E. B., Fourati, R., & Ayed, M. B. (2023) · IEEE Symposium on Computers and Communications (ISCC) · doi:10.1109/iscc58397.2023.10218144 · Cited by 2 · see citing works
RCV2023 Challenges: Benchmarking Model Training and Inference for Resource-Constrained Deep Learning
Tiwari, R., Chavan, A., Gupta, D. K., Mago, G., Gupta, A., Gupta, A., Sharan, S., Yang, Y., Zhao, S., Wang, S., et al. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00168 · Cited by 2 · see citing works
Self-citationTowards Reliable and Efficient Vegetation Segmentation for Australian Wheat Data Analysis
Yuan, B., Wang, Z., & Yu, X. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47843-7_9 · Cited by 1 · see citing works
AdaVid: Adaptive Video-Language Pretraining
Patel, C., Niebles, J. C., & Adeli, E. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00635 · Cited by 1 · see citing works
A Survey on Regularization-Based Structured Neural Network Pruning
Lu, Z. (2025) · International Symposium on Artificial Intelligence and Computational Social Sciences (AICSS) · doi:10.1145/3776759.3776809 · Cited by 1 · see citing works
DynaSlim: Dynamic Slimming for Vision Transformers
Shi, D., Gao, J., Liu, T., & Fu, Y. (2023) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme55011.2023.00251 · Cited by 1 · see citing works
Cross-Checking-Based Trademark Image Retrieval for Hot Company Detection
Wu, H., Zhang, Z., & Zhu, Z. (2023) · Journal of Organizational and End User Computing · doi:10.4018/joeuc.335455 · Cited by 1 · see citing works
On Designing Light-Weight Object Trackers Through Network Pruning: Use CNNS or Transformers?
Aggarwal, S., Gupta, T., Sahu, P. K., Chavan, A., Tiwari, R., Prasad, D. K., & Gupta, D. K. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094920 · Cited by 1 · see citing works
Self-citationAutomatic Hyper-Parameter Search for Vision Transformer Pruning
Feng, J., Zhao, S., Peng, L., Pan, S., Chen, H., Li, Z., Ke, G., Wang, G., & Long, Y. (2023) · International Conference on Pattern Recognition and Artificial Intelligence (PRAI) · doi:10.1109/prai59366.2023.10332058
Zhang, X., He, J. Y., Xu, T., Zhu, M., Wang, K., Jiang, B., & Liu, X. (2024) · International Conference on Artificial Intelligence, Big Data and Algorithms (CAIBDA) · doi:10.1145/3690407.3690550
AdaE: Knowledge Graph Embedding With Adaptive Embedding Sizes
Guan, Z., Zhang, F., Zhang, Z., Zhuang, F., Wang, F., An, Z., & Xu, Y. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3566270
Intra-head pruning for vision transformers via inter-layer dimension relationship modeling
Zhang, P., Tian, C., Zhao, L., & Duan, Z. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107656
Dual Cuts for Efficiency: Streamlining Nas with Train-Time Pruning of Supernet and Search SpacePreprint
NIU, D., Jiang, J., Pan, H., Xu, J., Niu, X., & Lv, Q. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5387391
Weight-Sharing NAS with Architecture-Agnostic Intermediate Representation
Yu, S., Mazaheri, A., & Jannesari, A. (2025) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1145/3731545.3731587
How to Train Your Metamorphic Deep Neural Network
Sommariva, T., Calderara, S., & Porrello, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-10192-1_25
Structured pruning via cross-layer metric and ℓ2,0-norm sparse reconstruction
Yang, H., Yi, S., Meng, F., Liu, W., & Liang, Y. (2026) · Displays · doi:10.1016/j.displa.2026.103362
XAI-Pruner: Explainability-Driven Pruning of CNN and Transformer
Niu, D., Pan, H., Jiang, J., Xu, J., & Niu, X. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462953
Sparse Matrix Accelerator Based on Dynamic Pruning
Chen, M.-Y., Fan, Y., & Lee, C.-M. (2026) · IEEE International Conference on Consumer Technology (ICCT-Pacific) · doi:10.1109/icct-pacific69083.2026.11518803
Liu, Y., Wang, T., Zhang, X., Wang, L., Zhao, Y., Ding, C., & Song, Q. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-00607-8
The Calibration Witness and Its Propagation AlgebraPreprint
Dudaboyina, N. S. M., Hassan, M. H. S., & Gunnu, G. R. (2026) · Research Square · doi:10.21203/rs.3.rs-9829547/v1
The Calibration Witness and Its Propagation AlgebraPreprint
Dudaboyina, N. S. M., Hassan, M. H. S., & Gunnu, G. R. (2026) · Research Square · doi:10.21203/rs.3.rs-9829547/v2
HypTM: Efficient Vision Transformers With Hyperbolic Token Merging
Chen, X. W., Zhang, X., Xie, W., Ma, J., Li, Y., & Fang, L. (2026) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2026.3709249
Padeiro, C. V., Chen, T.-W., Komamizu, T., & Ide, I. (2026) · IEEE Transactions on AgriFood Electronics · doi:10.1109/tafe.2026.3711909
Diversity-guided MLP pruning for efficient large vision transformers
Shen, C., Zhu, H., Fang, G., Wang, J., & Wang, X. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.114860
Zhang, M., Ye, H., Tong, Y., & Wang, Z. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11729540
- Authors
- Chavan, A., Shen, Z., Liu, Z., Liu, Z., Cheng, K.-T., & Xing, E.
- Affiliations
- Indian Institute of Technology Dhanbad (IN)
- Mohamed bin Zayed University of Artificial Intelligence (AE)
- University of California, Berkeley (US)
- Hong Kong University of Science and Technology (HK)
- Venue
- 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2022-06
- Pages
- 4921-4931
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4312772337
- Cited by
- 72
- References
- 64
- Citations by year
- 2026: 11 · 2025: 14 · 2024: 20 · 2023: 26 · 2022: 1
- FWCI
- 5.53
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- neural architecture search, model compression, DeiT
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
