The DOI matches its Crossref record; 150 of the 150 references it cites were found in a registry or an index.
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations
Open accessAbstract
Modern deep neural networks, particularly recent large language models, come with massive model sizes that require significant computational and storage resources. To enable the deployment of modern models on resource-constrained environments and to accelerate inference time, researchers have increasingly explored pruning techniques as a popular research direction in neural network compression. More than three thousand pruning papers have been published from 2020 to 2024. However, there is a dearth of up-to-date comprehensive review papers on pruning. To address this issue, in this survey, we provide a comprehensive review of existing research works on deep neural network pruning in a taxonomy of 1) universal/specific speedup, 2) when to prune, 3) how to prune, and 4) fusion of pruning and other compression techniques. We then provide a thorough comparative analysis of eight pairs of contrast settings for pruning (e.g., unstructured/structured, one-shot/iterative, data-free/data-driven, initialized/pre-trained weights, etc.) and explore several emerging topics, including pruning for large language models, vision transformers, diffusion models, and large multimodal models, post-training pruning, and different levels of supervision for pruning to shed light on the commonalities and differences of existing methods and lay the foundation for further method development. Finally, we provide some valuable recommendations on selecting pruning methods and prospect several promising research directions for neural network pruning. To facilitate future research on deep neural network pruning, we summarize broad pruning applications (e.g., adversarial robustness, natural language understanding, etc.) and build a curated collection of datasets, networks, and evaluations on different applications. We maintain a repository on https://github.com/hrcheng1066/awesome-pruning that serves as a comprehensive resource for neural network pruning papers and corresponding open-source codes. We will keep updating this repository to include the latest advancements in the field.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Boyd, S. (2010) · now publishers, Inc. eBooks · doi:10.1561/9781601984616 · Cited by 12,677
Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
Zhang, K., Zuo, W., Chen, Y., Meng, D., & Zhang, L. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2662206 · Cited by 8,983
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Model Selection and Estimation in Regression with Grouped Variables
Yuan, M., & Lin, Y. (2006) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00532.x · Cited by 7,261
Dimensionality Reduction by Learning an Invariant Mapping
Hadsell, R., Chopra, S., & LeCun, Y. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.100 · Cited by 5,336
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 3,832
DARTS: Differentiable architecture search
Liu, H., Simonyan, K., & Yang, Y. (2018) · arXiv · doi:10.48550/arxiv.1806.09055 · Cited by 3,248
RepVGG: Making VGG-style ConvNets Great Again
Ding, X., Zhang, X., Ma, N., Han, J., Ding, G., & Sun, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01352 · Cited by 2,907
Learning Efficient Convolutional Networks through Network Slimming
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., & Zhang, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.298 · Cited by 2,761
Channel Pruning for Accelerating Very Deep Neural Networks
He, Y., Zhang, X., & Sun, J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.155 · Cited by 2,647 · cited 6×
“For example, He et al. [134] propose an alternative two-step algorithm that introduces a scalar mask to each channel of a pre-trained CNN model, selects redundant channels based on LASSO regression [138], and reconstructs the outputs of un-pruned channels using linear least squares.”
InfluentialPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Improved Baselines with Momentum Contrastive Learning
Chen, X., Fan, H., Girshick, R., & He, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04297 · Cited by 1,956
ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression
Luo, J.-H., Wu, J., & Lin, W. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.541 · Cited by 1,832
Reed, R. (1993) · IEEE Transactions on Neural Networks · doi:10.1109/72.248452 · Cited by 1,708
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
Frankle, J., & Carbin, M. (2018) · arXiv · doi:10.48550/arxiv.1803.03635 · Cited by 1,529
Dynamic Convolution: Attention Over Convolution Kernels
Chen, Y., Dai, X., Liu, M., Chen, D., Yuan, L., & Liu, Z. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01104 · Cited by 1,477
Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
Denton, R., Zaremba, W., Bruna, J., LeCun, Y., & Fergus, R. (2014) · arXiv · doi:10.48550/arxiv.1404.0736 · Cited by 1,414
AMC: AutoML for Model Compression and Acceleration on Mobile Devices
He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J., & Han, S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_48 · Cited by 1,400
Learning methods for generic object recognition with invariance to pose and lighting
LeCun, Y., Huang, F. J., & Bottou, L. (2004) · Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. · doi:10.1109/cvpr.2004.1315150 · Cited by 1,324
Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration
He, Y., Liu, P., Wang, Z., Hu, Z., & Yang, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00447 · Cited by 1,217
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
Voita, E., Talbot, D., Moiseev, F., Sennrich, R., & Titov, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1580 · Cited by 1,146
Self-supervised Learning: Generative or Contrastive
Liu, X., Zhang, F., Hou, Z., Mian, L., Wang, Z., Zhang, J., & Tang, J. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3090866 · Cited by 1,087
Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks
He, Y., Kang, G., Dong, X., Fu, Y., & Yang, Y. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/309 · Cited by 1,020
Importance Estimation for Neural Network Pruning
Molchanov, P., Mallya, A., Tyree, S., Frosio, I., & Kautz, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01152 · Cited by 983 · cited 1×
“Pruning After Training ( PAT ) (as shown in Table IV) is the most popular type of pruning pipeline because it is commonly believed that pre-training the dense network is necessary to obtain an efficient subnetwork [110].”
Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision ModelsPreprint
Liu, Y., Zhang, K., Li, Y., Yan, Z., Gao, C., Chen, R., Yuan, Z., Huang, Y., Sun, H., Gao, J., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.17177 · Cited by 872
A Survey on Knowledge Distillation of Large Language ModelsPreprint
Xu, X., Li, M., Tao, C., Shen, T., Cheng, R., Li, J., Xu, C., Tao, D., & Zhou, T. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.13116 · Cited by 788
Model Pruning Enables Efficient Federated Learning on Edge Devices
Jiang, Y., Wang, S., Valls, V., Ko, B. J., Lee, W.-H., Leung, K. K., & Tassiulas, L. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3166101 · Cited by 589
MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning
Liu, Z., Mu, H., Zhang, X., Guo, Z., Yang, X., Cheng, K.-T., & Sun, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00339 · Cited by 563
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
Men, X., Xu, M., Zhang, Q., Yuan, Q., Wang, B., Lin, H., Lu, Y., Han, X., & Chen, W. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.1035 · Cited by 561
What is the State of Neural Network Pruning?
Blalock, D., Ortiz, J. J. G., Frankle, J., & Guttag, J. (2020) · arXiv · doi:10.48550/arxiv.2003.03033 · Cited by 560
Towards Optimal Structured CNN Pruning via Generative Adversarial Learning
Lin, S., Ji, R., Yan, C., Zhang, B., Cao, L., Ye, Q., Huang, F., & Doermann, D. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00290 · Cited by 555
Data-Driven Sparse Structure Selection for Deep Neural Networks
Huang, Z., & Wang, N. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01270-0_19 · Cited by 549
Mocanu, D. C., Mocanu, E., Stone, P., Nguyen, P. H., Gibescu, M., & Liotta, A. (2018) · Nature Communications · doi:10.1038/s41467-018-04316-3 · Cited by 524 · cited 9×
“In addition, DSR [95] reallo-catesweightsacrosslayersanddoesnotrestricttotheinner-layer weight redistribution in SET [91].”
InfluentialPractical Block-Wise Neural Network Architecture Generation
Zhong, Z., Yan, J., Wu, W., Shao, J., & Liu, C.-L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00257 · Cited by 522
A comprehensive survey on model compression and acceleration
Choudhary, T., Mishra, V., Goswami, A., & Sarangapani, J. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-020-09816-7 · Cited by 513
DepGraph: Towards Any Structural Pruning
Fang, G., Ma, X., Song, M., Bi Mi, M., & Wang, X. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01544 · Cited by 501
A Systematic DNN Weight Pruning Framework Using Alternating Direction Method of Multipliers
Zhang, T., Ye, S., Zhang, K., Tang, J., Wen, W., Fardad, M., & Wang, Y. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01237-3_12 · Cited by 473
Cao, Y., Li, S., Liu, Y., Yan, Z., Dai, Y., Yu, P. S., & Sun, L. (2023) · arXiv · doi:10.48550/arxiv.2303.04226 · Cited by 463
Structured Pruning for Deep Convolutional Neural Networks: A Survey
He, Y., & Xiao, L. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3334614 · Cited by 363
Variational Convolutional Neural Network Pruning
Zhao, C., Ni, B., Zhang, J., Zhao, Q., Zhang, W., & Tian, Q. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00289 · Cited by 361
MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks
Gordon, A., Eban, E., Nachum, O., Chen, B., Wu, H., Yang, T.-J., & Choi, E. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00171 · Cited by 341
AI-Generated Content (AIGC): A SurveyPreprint
Wu, J., Gan, W., Chen, Z., Wan, S., & Hong, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.06632 · Cited by 330
MobileVLM V2: Faster and Stronger Baseline for Vision Language ModelPreprint
Chu, X., Qiao, L., Zhang, X., Xu, S., Wei, F., Yang, Y., Sun, X., Hu, Y.-M., Lin, X., Zhang, B., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.03766 · Cited by 300
More is Less: A More Complicated Network with Less Inference Complexity
Dong, X., Huang, J., Yang, Y., & Yan, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.205 · Cited by 291
A Survey on Model Compression for Large Language Models
Zhu, X., Li, J., Liu, Y., Ma, C., & Wang, W. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00704 · Cited by 273
VILA: On Pre-training for Visual Language Models
Lin, J., Yin, H., Ping, W., Molchanov, P., Shoeybi, M., & Han, S. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02520 · Cited by 243
NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm
Dai, X., Yin, H., & Jha, N. K. (2019) · IEEE Transactions on Computers · doi:10.1109/tc.2019.2914438 · Cited by 233
CLIP-Q: Deep Network Compression Learning by In-parallel Pruning-Quantization
Tung, F., & Mori, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00821 · Cited by 220
Movement pruning: Adaptive sparsity by fine-tuning
Sanh, V., Wolf, T., & Rush, A. M. (2020) · arXiv · doi:10.48550/arxiv.2005.07683 · Cited by 218
What’s Hidden in a Randomly Weighted Neural Network?
Ramanujan, V., Wortsman, M., Kembhavi, A., Farhadi, A., & Rastegari, M. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01191 · Cited by 217
When BERT Plays the Lottery, All Tickets Are Winning
Prasanna, S., Rogers, A., & Rumshisky, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.259 · Cited by 216
Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compression
Li, Y., Gu, S., Mayer, C., Van Gool, L., & Timofte, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00804 · Cited by 208
DMCP: Differentiable Markov Channel Pruning for Neural Networks
Guo, S., Wang, Y., Li, Q., & Yan, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00161 · Cited by 198
A Survey on Efficient Convolutional Neural Networks and Hardware Acceleration
Ghimire, D., Kil, D., & Kim, S.-H. (2022) · Electronics · doi:10.3390/electronics11060945 · Cited by 198
Model Compression and Efficient Inference for Large Language Models: A SurveyPreprint
Wang, W., Chen, W., Luo, Y., Yu, L. O. N. G., Lin, Z., Zhang, L., Lin, B., Cai, D., & He, X. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.09748 · Cited by 193
Compression of Neural Machine Translation Models via Pruning
See, A., Luong, M.-T., & Manning, C. D. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1029 · Cited by 184
Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining MethodsPreprint
Kim, B.-K., Kim, G., Kim, T., Castells, T., Choi, S., Shin, J., & Song, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.02834 · Cited by 184
Toward Compact ConvNets via Structure-Sparsity Regularized Filter Pruning
Lin, S., Ji, R., Li, Y., Deng, C., & Li, X. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2906563 · Cited by 180
LaCo: Large Language Model Pruning via Layer Collapse
Yang, Y., Cao, Z., & Zhao, H. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.372 · Cited by 177
Sparse Networks from Scratch: Faster Training without Losing Performance
Dettmers, T., & Zettlemoyer, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.04840 · Cited by 173
Network Pruning via Performance Maximization
Gao, S., Huang, F., Cai, W., & Huang, H. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00915 · Cited by 167
Are All Layers Created Equal?Preprint
Zhang, C., Samy Bengio, A., & Singer, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.01996 · Cited by 166
Towards Efficient Model Compression via Learned Global Ranking
Chin, T.-W., Ding, R., Zhang, C., & Marculescu, D. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00159 · Cited by 165
DHP: Differentiable Meta Pruning via HyperNetworks
Li, Y., Gu, S., Zhang, K., Gool, L. V., & Timofte, R. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58598-3_36 · Cited by 165
LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning
Zhang, M., Chen, H., Shen, C., Yang, Z., Ou, L., Yu, X., & Zhuang, B. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.178 · Cited by 165
Self-citationHolistic CNN Compression via Low-Rank Decomposition with Knowledge Transfer
Lin, S., Ji, R., Chen, C., Tao, D., & Luo, J. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2873305 · Cited by 164
Global Vision Transformer Pruning with Hessian-Aware Saliency
Yang, H., Yin, H., Shen, M., Molchanov, P. A., Li, H. H., & Kautz, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01779 · Cited by 164
Weight Agnostic Neural Networks
Gaier, A., & Ha, D. (2019) · Unpublished · doi:10.13140/rg.2.2.16025.88169 · Cited by 161
Exploiting Kernel Sparsity and Entropy for Interpretable CNN Compression
Li, Y., Lin, S., Zhang, B., Liu, J., Doermann, D., Wu, Y., Huang, F., & Ji, R. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00291 · Cited by 160
BERT Busters: Outlier Dimensions that Disrupt Transformers
Kovaleva, O., Kulshreshtha, S., Rogers, A., & Rumshisky, A. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.300 · Cited by 152
X-Pruner: eXplainable Pruning for Vision Transformers
Lu, Y., & Xiang, W. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02333 · Cited by 151
A Survey on Deep Neural Network Compression: Challenges, Overview, and SolutionsPreprint
Mishra, R., Gupta, H. P., & Dutta, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.03954 · Cited by 150
Chen, T., Frankle, J., Chang, S., Liu, S., Zhang, Y., Carbin, M., & Wang, Z. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01604 · Cited by 148
Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity
Yin, L., Wu, Y., Zhang, Z., Hsieh, C.-Y., Wang, Y., Jia, Y., Li, G., Jaiswal, A., Pechenizkiy, M., Liang, Y., et al. (2023) · TU/e Research Portal · doi:10.48550/arxiv.2310.05175 · Cited by 145
Structured Pruning Learns Compact and Accurate Models
Xia, M., Zhong, Z., & Chen, D. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.107 · Cited by 136
DSA: More Efficient Budgeted Pruning via Differentiable Sparsity Allocation
Ning, X., Zhao, T., Li, W., Lei, P., Wang, Y., & Yang, H. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58580-8_35 · Cited by 133
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
Frantar, E., & Alistarh, D. (2023) · arXiv · doi:10.48550/arxiv.2301.00774 · Cited by 131 · cited 10×
“Unstructured pruning methods ([13], [17]) remove weights anywhere and can achieve high prune ratios with little impact on accuracy.”
InfluentialTowards Compact CNNs via Collaborative Compression
Li, Y., Lin, S., Liu, J., Ye, Q., Wang, M., Chao, F., Yang, F., Ma, J., Tian, Q. C., & Ji, R. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00637 · Cited by 129
Dynamic Model Pruning with Feedback
Lin, T., Stich, S. U., Barba, L., Dmitriev, D., & Jaggi, M. (2020) · arXiv · doi:10.48550/arxiv.2006.07253 · Cited by 126
Manifold Regularized Dynamic Network Pruning
Tang, Y., Wang, Y., Xu, Y., Deng, Y., Xu, C., Tao, D., & Xu, C. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00498 · Cited by 122
Auto-Balanced Filter Pruning for Efficient Convolutional Neural Networks
Ding, X., Ding, G., Han, J., & Tang, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12262 · Cited by 117
Robust statistics on Riemannian manifolds via the geometric median
Fletcher, P. T., Venkatasubramanian, S., & Joshi, S. C. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587747 · Cited by 114
EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets
Chen, X., Cheng, Y., Wang, S., Gan, Z., Wang, Z., & Liu, J. (2021) · doi:10.18653/v1/2021.acl-long.171 · Cited by 109
Compressing Convolutional Neural Networks via Factorized Convolutional Filters
Li, T., Wu, B., Yang, Y., Fan, Y., Zhang, Y., & Liu, W. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00410 · Cited by 108
Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win
Evci, U., Ioannou, Y., Keskin, C., & Dauphin, Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i6.20611 · Cited by 101
Fluctuation-Based Adaptive Structured Pruning for Large Language Models
An, Y., Xu, Z., Yu, C., Tang, M., & Wang, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i10.28960 · Cited by 97
Width & Depth Pruning for Vision Transformers
Yu, F., Huang, K., Wang, M., Cheng, Y., Chu, W., & Cui, L. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20222 · Cited by 95
Recent Advances on Neural Network Pruning at Initialization
Wang, H., Qin, C., Bai, Y., Zhang, Y., & Fu, Y. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/786 · Cited by 94
A Simple and Effective Pruning Approach for Large Language Models
Sun, M., Liu, Z., Bair, A., & Kolter, J. Z. (2023) · arXiv · doi:10.48550/arxiv.2306.11695 · Cited by 92 · cited 9×
“It is usually divided into three types: un-structured ([48], [49], [50]), semi-structured (also called pattern-based) ([17], [51], [52]) and structured ([20], [39], [53]).”
InfluentialGAN Slimming: All-in-One GAN Compression by a Unified Optimization Framework
Wang, H., Gui, S., Yang, H., Liu, J., & Wang, Z. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58548-8_4 · Cited by 90
LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression
Mao, Y., Wang, Y., Wu, C., Zhang, C., Wang, Y., Zhang, Q., Yang, Y., Tong, Y., & Bai, J. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.287 · Cited by 89
PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech RecognitionPreprint
Lai, C.-I., Zhang, Y., Liu, A. H., Chang, S., Liao, Y.-L., Chuang, Y.-S., Qian, K., Khurana, S., Cox, D., & Glass, J. R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05933 · Cited by 81
Streamlining Redundant Layers to Compress Large Language ModelsPreprint
Chen, X. D., Hu, Y., Zhang, J., Wang, Y., Li, C., & Chen, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.19135 · Cited by 81
Dynamic Sparse Training for Deep Reinforcement Learning
Sokar, G., Mocanu, E., Mocanu, D. C., Pechenizkiy, M., & Stone, P. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/477 · Cited by 78
Fire Together Wire Together: A Dynamic Pruning Approach with Self-Supervised Mask Prediction
Elkerdawy, S., Elhoushi, M., Zhang, H., & Ray, N. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01213 · Cited by 74
AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference With Transformers
Tuli, S., & Jha, N. K. (2023) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2023.3273992 · Cited by 74
Liu, Y., Shu, Z., Li, Y., Lin, Z., Perazzi, F., & Kung, S. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01198 · Cited by 73
Dynamic Dual Gating Neural Networks
Li, F., Li, G., He, X., & Cheng, J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00528 · Cited by 72
Pruning Algorithms to Accelerate Convolutional Neural Networks for Edge Applications: A SurveyPreprint
Liu, J., Tripathi, S., Kurup, U., & Shah, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.04275 · Cited by 66
Playing Lottery Tickets with Vision and Language
Gan, Z., Chen, Y.-C., Li, L., Chen, T., Cheng, Y., Wang, S., Liu, J., Wang, L., & Liu, Z. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i1.19945 · Cited by 66
Prune Your Model Before Distill It
Park, J., & No, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20083-0_8 · Cited by 66
The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models
Kurtic, E., Campos, D., Nguyen, T., Frantar, E., Kurtz, M., Fineran, B., Goin, M., & Alistarh, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.279 · Cited by 65
A Fast Post-Training Pruning Framework for Transformers
Kwon, W., Kim, S., Mahoney, M. W., Hassoun, J., Keutzer, K., & Gholami, A. (2022) · arXiv · doi:10.48550/arxiv.2204.09656 · Cited by 64
Everybody Prune Now: Structured Pruning of LLMs with only Forward PassesPreprint
Kolawole, S., Dery, L. M., Kagy, J.-F., Smith, V., Neubig, G., & Talwalkar, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.05406 · Cited by 64
Yang, H., Zhu, Y., & Liu, J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01146 · Cited by 56
Runtime Network Routing for Efficient Image Classification
Rao, Y., Lu, J., Lin, J., & Zhou, J. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2878258 · Cited by 55
Li, Z., Yuan, G., Niu, W., Zhao, P., Li, Y., Cai, Y., Shen, X., Zhan, Z., Kong, Z., Jin, Q., et al. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01403 · Cited by 53
What Matters In The Structured Pruning of Generative Language Models?Preprint
Santacroce, M., Wen, Z., Shen, Y., & Li, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.03773 · Cited by 53
Knowledge from the Original Network: Restore a Better Pruned Network with Knowledge Distillation
Chen, L., Chen, Y., Xi, J., & Le, X. (2022) · Complex & Intelligent Systems · doi:10.1007/s40747-020-00248-y · Cited by 48
Learning Bayesian Sparse Networks with Full Experience Replay for Continual Learning
Yan, Q., Gong, D., Liu, Y., van den Hengel, A., & Shi, Q. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00021 · Cited by 47
An Image Enhancing Pattern-Based Sparsity for Real-Time Inference on Mobile Devices
Ma, X., Niu, W., Zhang, T., Liu, S., Lin, S., Li, H., Wen, W., Chen, X., Tang, J., Ma, K., et al. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58601-0_37 · Cited by 44
A Survey of Resource-efficient LLM and Multimodal Foundation Models
Xu, M., Yin, W., Cai, D., Yi, R., Xu, D., Wang, Q., Wu, B., Zhao, Y., Yang, C., Wang, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.08092 · Cited by 44
Liu, S., Chen, T., Chen, X., Shen, L., Mocanu, D. C., Wang, Z., & Pechenizkiy, M. (2022) · arXiv · doi:10.48550/arxiv.2202.02643 · Cited by 43
Diffenderfer, J., & Kailkhura, B. (2021) · arXiv · doi:10.48550/arxiv.2103.09377 · Cited by 39 · cited 4×
“Inspired by LTH, there are various follow-up works to identify wider tickets across multiple kinds of neural networks (such as CNNs [113], Generative Adversarial Networks ( GANs ) [114], Variational AutoEncoders ( VAEs ) [115], GraphNeural Networks ( GNNs ) [116], Transformer basedmod-els [117])…”
InfluentialA Hardware-Friendly High-Precision CNN Pruning Method and Its FPGA Implementation
Sui, X., Lv, Q., Zhi, L., Zhu, B., Yang, Y., Zhang, Y., & Tan, Z. (2023) · Sensors · doi:10.3390/s23020824 · Cited by 37
Towards Compact and Robust Deep Neural NetworksPreprint
Sehwag, V., Wang, S., Mittal, P., & Jana, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.06110 · Cited by 36
Pruning Convolutional Neural Networks with Self-SupervisionPreprint
Caron, M., Morcos, A. S., Bojanowski, P., Mairal, J., & Joulin, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.03554 · Cited by 35
Wang, H., Qin, C., Bai, Y., & Fu, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.05219 · Cited by 35
Joint-DetNAS: Upgrade Your Detector with NAS, Pruning and Dynamic Distillation
Yao, L., Pi, R., Xu, H., Zhang, W., Li, Z., & Zhang, T. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01004 · Cited by 34
Dreaming to Prune Image Deraining Networks
Zou, W., Wang, Y., Fu, X., & Cao, Y. H. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00593 · Cited by 30
A Comprehensive Survey of Compression Algorithms for Language Models
Park, S., Choi, J., Lee, S., & Kang, U. (2026) · ACM Computing Surveys · doi:10.1145/3819816 · Cited by 30
Single Shot Structured Pruning Before TrainingPreprint
van Amersfoort, J., Alizadeh, M., Farquhar, S., Lane, N. D., & Gal, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.00389 · Cited by 29
A Survey of Lottery Ticket HypothesisPreprint
Liu, B., Zhang, Z., He, P., Wang, Z., Xiao, Y., Ye, R., Zhou, Y., Ku, W., & Hui, B. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.04861 · Cited by 28
Network Pruning That Matters: A Case Study on Retraining Variants
Le, D. H., & Hua, B.-S. (2021) · arXiv · doi:10.48550/arxiv.2105.03193 · Cited by 28
Winning Lottery Tickets in Deep Generative Models
Kalibhat, N., Balaji, Y., & Feizi, S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i9.16980 · Cited by 26
How Well Do Sparse ImageNet Models Transfer?
Iofinova, E., Peste, A., Kurtz, M., & Alistarh, D. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01195 · Cited by 22
A Unified Paths Perspective for Pruning at InitializationPreprint
Gebhart, T., Saxena, U., & Schrater, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.10552 · Cited by 19
Sparks of Large Audio Models: A Survey and Outlook
Latif, S., Shoukat, M., Shamshad, F., Usama, M., Ren, Y., Cuayáhuitl, H., Wang, W., Zhang, X., Togneri, R., Cambria, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12792 · Cited by 17 · cited 1×
“O VER the past several years, Deep Neural Networks ( DNNs ) have achieved conspicuous progress in various domains and applications, such as Computer Vision ( CV ) [1], [2], Natural Language Processing ( NLP ) [3], [4], Audio Signal Processing ( ASP ) [5], [6], and cross-modal applications [7], [8].”
No Free Prune: Information-Theoretic Barriers to Pruning at InitializationPreprint
Kumar, T., Luo, K., & Sellke, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.01089 · Cited by 14
Large Multimodal Model Compression via Efficient Pruning and Distillation at AntGroupPreprint
Wang, M., Zhao, Y., Liu, J., Chen, J., Zhuang, C., Gu, J., Guo, R., & Zhao, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.05795 · Cited by 13
Graph pruning for model compression
Zhang, M., Yu, X., Rong, J., & Ou, L. (2022) · Applied Intelligence · doi:10.1007/s10489-021-02802-8 · Cited by 11
Self-citationGong, R., Yong, Y., Wang, Z., Guo, J., Wei, X., Ma, Y., & Liu, X. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i11.29108 · Cited by 9
Wang, Y., Zhang, X., Xie, L., Zhou, J., Su, H., Zhang, B., & Hu, X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6910 · Cited by 8
Contrastive Dual Gating: Learning Sparse Features With Contrastive Learning
Meng, J., Yang, L., Shin, J., Fan, D., & Seo, J.-S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01194 · Cited by 6
Why Lift so Heavy? Slimming Large Language Models by Cutting Off the Layers
Yuan, S., Nie, E., Ma, B., & Färber, M. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228532 · Cited by 5
How much pre-training is enough to discover a good subnetwork?Preprint
Wolfe, C. R., Liao, F., Wang, Q., Kim, J. L., & Kyrillidis, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.00259 · Cited by 4
Differentiable Network Pruning via Polarization of Probabilistic Channelwise Soft Masks
Ma, M., Wang, J., & Yu, Z. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/7775419 · Cited by 2
Pan, S., Qin, Y., Li, T., Li, X., & Hou, L. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00298 · Cited by 2
Why is Pruning at Initialization Immune to Reinitializing and Shuffling?Preprint
Singh, S., & Liu, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.01808 · Cited by 1
OPT: Open Pre-trained Transformer Language Models
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.01068
Cited by
From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks
Geng, X., Wang, Z., Chen, C., Xu, Q., Xu, K., Jin, C., Gupta, M., Yang, X., Chen, Z., Sabry Aly, M. M., et al. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3394494 · Cited by 26
Raza, S. M., Abidi, S. M. H., Masuduzzaman, M., & Shin, S. Y. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131357 · Cited by 13
Metadata
- Authors
- Cheng, H., Zhang, M., & Shi, J. Q.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2024-12
- Volume
- 46
- Issue
- 12
- Pages
- 10558-10578
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W4401726555
- Cited by
- 497
- References
- 150
- FWCI
- 69.92
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.