Non-Structured DNN Weight Pruning—Is It Beneficial in Any Platform?
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Large deep neural network (DNN) models pose the key challenge to energy efficiency due to the significantly higher energy consumption of off-chip DRAM accesses than arithmetic or SRAM operations. It motivates the intensive research on model compression with two main approaches. Weight pruning leverages the redundancy in the number of weights and can be performed in a non-structured, which has higher flexibility and pruning rate but incurs index accesses due to irregular weights, or structured manner, which preserves the full matrix structure with a lower pruning rate. Weight quantization leverages the redundancy in the number of bits in weights. Compared to pruning, quantization is much more hardware-friendly and has become a "must-do" step for FPGA and ASIC implementations. Thus, any evaluation of the effectiveness of pruning should be on top of quantization. The key open question is, with quantization, what kind of pruning (non-structured versus structured) is most beneficial? This question is fundamental because the answer will determine the design aspects that we should really focus on to avoid the diminishing return of certain optimizations. This article provides a definitive answer to the question for the first time. First, we build ADMM-NN-S by extending and enhancing ADMM-NN, a recently proposed joint weight pruning and quantization framework, with the algorithmic supports for structured pruning, dynamic ADMM regulation, and masked mapping and retraining. Second, we develop a methodology for fair and fundamental comparison of non-structured and structured pruning in terms of both storage and computation efficiency. Our results show that ADMM-NN-S consistently outperforms the prior art: 1) it achieves 348× , 36× , and 8× overall weight pruning on LeNet-5, AlexNet, and ResNet-50, respectively, with (almost) zero accuracy loss and 2) we demonstrate the first fully binarized (for all layers) DNNs can be lossless in accuracy in many cases. These results provide a strong baseline and credibility of our study. Based on the proposed comparison framework, with the same accuracy and quantization, the results show that non-structured pruning is not competitive in terms of both storage and computation efficiency. Thus, we conclude that structured pruning has a greater potential compared to non-structured pruning. We encourage the community to focus on studying the DNN inference acceleration with structured sparsity.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755 · cited 1×
“MobileNet-V2 [60], and LeNet-5 DNN models.”
Han, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_32 · Cited by 4,408
Pruning Filters for Efficient ConvNets
Li, H., Kadav, A., Durdanovic, I., Samet, H., & Graf, H. P. (2016) · arXiv · doi:10.48550/arxiv.1608.08710 · Cited by 2,928
Channel Pruning for Accelerating Very Deep Neural Networks
He, Y., Zhang, X., & Sun, J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.155 · Cited by 2,647
BinaryConnect: Training Deep Neural Networks with binary weights during propagations
Courbariaux, M., Bengio, Y., & David, J.-P. (2015) · arXiv · doi:10.48550/arxiv.1511.00363 · Cited by 2,180
Optimizing FPGA-based Accelerator Design for Deep Convolutional Neural Networks
Zhang, C., Li, P., Sun, G., Guan, Y., Xiao, B., & Cong, J. (2015) · Proceedings of the 2015 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays · doi:10.1145/2684746.2689060 · Cited by 2,158
Hubara, I., Soudry, D., & Yaniv, R. E. (2016) · arXiv · doi:10.48550/arxiv.1602.02505 · Cited by 1,667
Rethinking the Value of Network Pruning
Liu, Z., Sun, M., Zhou, T., Huang, G., & Darrell, T. (2018) · arXiv · doi:10.48550/arxiv.1810.05270 · Cited by 1,625
Research works printed but not linked: 55
- 10.1109/MICRO.2018.00023 · DOI · Paper page
- 10.1109/ISCA.2018.00051 · DOI
- 10.1109/ISCA.2018.00062 · DOI · Paper page
- 10.1145/3093315.3037746 · DOI
- 10.1145/3173162.3173212 · DOI
- 10.1145/3289602.3293947 · doi:10.1145/3289602.3293947
- 10.1145/3289602.3293951 · doi:10.1145/3289602.3293951
- 10.1145/3289602.3293964 · doi:10.1145/3289602.3293964
- 10.1145/3289602.3293967 · doi:10.1145/3289602.3293967
- 10.1145/3289602.3293970 · doi:10.1145/3289602.3293970
- 10.1145/3289602.3293976 · DOI
- Chetlur, 2014, CuDNN: Efficient primitives for deep learning
- 10.1145/3410463.3414648 · DOI
- 10.1145/2966986.2967011 · DOI · Paper page
- 2018, Google’s Tensor Processing Unit
- 2018, Intel’s Deep Learning Chips
- Han, Learning both weights and connections for efficient neural network, 1135
- Guo, Dynamic network surgery for efficient DNNs, 1379
- 10.1609/aaai.v34i04.5924 · DOI · Paper page
- Wen, Learning structured sparsity in deep neural networks, 2074
- 10.1145/3079856.3080215 · DOI · Paper page
- 10.1609/aaai.v32i1.11713 · DOI · Paper page
- 10.1109/CVPR.2017.761 · DOI · Paper page
- Zhou, Incremental network quantization: Towards lossless CNNs with low-precision weights, 1
- 10.1109/CVPR.2016.521 · DOI · Paper page
- 10.1007/978-1-4842-6666-3_1 · doi:10.1007/978-1-4842-6666-3_1
- Paszke, 2017, Pytorch
- 10.1145/3297858.3304076 · DOI · Paper page
- Ouyang, Stochastic alternating direction method of multipliers, 80
- Suzuki, Dual averaging and proximal gradient descent for online alternating direction multiplier method, 392
- 10.1561/9781601984616 · DOI · Paper page
- 10.1007/978-3-030-01237-3_12 · DOI · Paper page
- 10.1109/VLSIC.2018.8502404 · DOI · Paper page
- 10.1109/MICRO.2016.7783723 · DOI · Paper page
- 10.1145/3140659.3080254 · DOI · Paper page
- He, 2018, Simultaneously optimizing weight and quantizer of ternary neural network using truncated Gaussian approximation
- 10.1145/3349567.3351718 · doi:10.1145/3349567.3351718
- 10.1609/aaai.v34i04.5954 · DOI · Paper page
- 10.1007/978-3-030-58601-0_37 · DOI
- 2018, FBGEMM
- 10.1137/140990309 · DOI · Paper page
- Liu, Zeroth-order online alternating direction method of multipliers: Convergence analysis and applications, 288
- 10.1007/978-3-319-99223-5 · DOI
- 10.1109/TNNLS.2019.2910073 · DOI · Paper page
- 10.1109/JSTSP.2020.2969554 · DOI · Paper page
- 10.5555/2999134.2999257 · doi:10.5555/2999134.2999257
- Simonyan, 2014, Very deep convolutional networks for large-scale image recognition
- 10.1109/ICME.2018.8486500 · DOI
- 2019, Google Supercharges Machine Learning Tasks With TPU Custom Chip
- Molchanov, 2016, Pruning convolutional neural networks for resource efficient inference
- 10.1109/CVPR.2018.00958 · DOI · Paper page
- Zhuang, Discrimination-aware channel pruning for deep neural networks, 875
- 10.1109/ICCV.2017.541 · DOI · Paper page
- 10.1109/CVPR42600.2020.01464 · DOI · Paper page
- Min, 2018, 2PFPCE: Two-phase filter pruning based on conditional entropy
Cited by
Structured Pruning for Deep Convolutional Neural Networks: A Survey
He, Y., & Xiao, L. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3334614 · Cited by 363
A comprehensive review of model compression techniques in machine learning
Dantas, P. V., Sabino da Silva, W., Cordeiro, L. C., & Carvalho, C. B. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05747-w · Cited by 261
SPViT: Enabling Faster Vision Transformers via Latency-Aware Soft Token Pruning
Kong, Z., Dong, P., Ma, X., Meng, X., Niu, W., Sun, M., Shen, X., Yuan, G., Ren, B., Tang, H., et al. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20083-0_37 · Cited by 169
“Pruning has been proved as the one of the most effective methods to reduce network dimensions in convolution-based neural networks [70,101,57,47,107,62,72,52,108,11,36,55,54,4].”
MethodSelf-citationXu, W., Fang, W., Ding, Y., Zou, M., & Xiong, N. (2021) · IEEE Access · doi:10.1109/access.2021.3063291 · Cited by 112
“Our choice is based on the following three reasons: (1) Other than weight pruning [7], channel pruning produces hardware-friendly models without introducing irregular sparsity [11].”
BackgroundMethodNaveen, S., & Kounte, M. R. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-20523-1 · Cited by 33
Metadata
- Authors
- Ma, X., Lin, S., Ye, S., He, Z., Zhang, L., Yuan, G., Tan, S. H., Li, Z., Fan, D., Qian, X., Lin, X., Ma, K., & Wang, Y.
- Venue
- IEEE Transactions on Neural Networks and Learning Systems
- Published
- 2022-09
- Volume
- 33
- Issue
- 9
- Pages
- 4930-4944
- ISSN
- 2162-237X (ISSN-L)
- 2162-2388
- OpenAlex
- W3139203094
- Cited by
- 115
- References
- 21
- FWCI
- 7.50
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- Funders
- National Science Foundation (CCF-1919117, CCF-1937500, CCF-1901378)
- Army Research Office (ARO) Young Investigator Program (76598CSYIP)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Non-Structured DNN Weight Pruning—Is It Beneficial in Any Platform? https://app.cochat.ai/papers/10.1109/tnnls.2021.3063265
- The paper
- Ma, X., Lin, S., Ye, S., He, Z., Zhang, L., Yuan, G., Tan, S. H., Li, Z., Fan, D., Qian, X., Lin, X., Ma, K., & Wang, Y. (2022). Non-Structured DNN Weight Pruning—Is It Beneficial in Any Platform? IEEE Transactions on Neural Networks and Learning Systems. https://doi.org/10.1109/tnnls.2021.3063265
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.