CoChat

Non-Structured DNN Weight Pruning—Is It Beneficial in Any Platform?

Open access
Ma, Xiaolong, Lin, Sheng, Ye, Shaokai, He, Zhezhi, Zhang, Linfeng, Yuan, Geng, Tan, Sia Huat, Li, Zhengang, Fan, Deliang, Qian, Xuehai, Lin, Xue, Ma, Kaisheng, Wang, Yanzhi · 2022 · IEEE Transactions on Neural Networks and Learning Systems
Cited by 115

The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.

Abstract

Large deep neural network (DNN) models pose the key challenge to energy efficiency due to the significantly higher energy consumption of off-chip DRAM accesses than arithmetic or SRAM operations. It motivates the intensive research on model compression with two main approaches. Weight pruning leverages the redundancy in the number of weights and can be performed in a non-structured, which has higher flexibility and pruning rate but incurs index accesses due to irregular weights, or structured manner, which preserves the full matrix structure with a lower pruning rate. Weight quantization leverages the redundancy in the number of bits in weights. Compared to pruning, quantization is much more hardware-friendly and has become a "must-do" step for FPGA and ASIC implementations. Thus, any evaluation of the effectiveness of pruning should be on top of quantization. The key open question is, with quantization, what kind of pruning (non-structured versus structured) is most beneficial? This question is fundamental because the answer will determine the design aspects that we should really focus on to avoid the diminishing return of certain optimizations. This article provides a definitive answer to the question for the first time. First, we build ADMM-NN-S by extending and enhancing ADMM-NN, a recently proposed joint weight pruning and quantization framework, with the algorithmic supports for structured pruning, dynamic ADMM regulation, and masked mapping and retraining. Second, we develop a methodology for fair and fundamental comparison of non-structured and structured pruning in terms of both storage and computation efficiency. Our results show that ADMM-NN-S consistently outperforms the prior art: 1) it achieves 348× , 36× , and 8× overall weight pruning on LeNet-5, AlexNet, and ResNet-50, respectively, with (almost) zero accuracy loss and 2) we demonstrate the first fully binarized (for all layers) DNNs can be lossless in accuracy in many cases. These results provide a strong baseline and credibility of our study. Based on the proposed comparison framework, with the same accuracy and quantization, the results show that non-structured pruning is not competitive in terms of both storage and computation efficiency. Thus, we conclude that structured pruning has a greater potential compared to non-structured pruning. We encourage the community to focus on studying the DNN inference acceleration with structured sparsity.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tnnls.2021.3063265

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 21 references

Research works printed but not linked: 55

  1. 10.1109/MICRO.2018.00023 · DOI · Paper page
  2. 10.1109/ISCA.2018.00051 · DOI
  3. 10.1109/ISCA.2018.00062 · DOI · Paper page
  4. 10.1145/3093315.3037746 · DOI
  5. 10.1145/3173162.3173212 · DOI
  6. 10.1145/3289602.3293947 · doi:10.1145/3289602.3293947
  7. 10.1145/3289602.3293951 · doi:10.1145/3289602.3293951
  8. 10.1145/3289602.3293964 · doi:10.1145/3289602.3293964
  9. 10.1145/3289602.3293967 · doi:10.1145/3289602.3293967
  10. 10.1145/3289602.3293970 · doi:10.1145/3289602.3293970
  11. 10.1145/3289602.3293976 · DOI
  12. Chetlur, 2014, CuDNN: Efficient primitives for deep learning
  13. 10.1145/3410463.3414648 · DOI
  14. 10.1145/2966986.2967011 · DOI · Paper page
  15. 2018, Google’s Tensor Processing Unit
  16. 2018, Intel’s Deep Learning Chips
  17. Han, Learning both weights and connections for efficient neural network, 1135
  18. Guo, Dynamic network surgery for efficient DNNs, 1379
  19. 10.1609/aaai.v34i04.5924 · DOI · Paper page
  20. Wen, Learning structured sparsity in deep neural networks, 2074
  21. 10.1145/3079856.3080215 · DOI · Paper page
  22. 10.1609/aaai.v32i1.11713 · DOI · Paper page
  23. 10.1109/CVPR.2017.761 · DOI · Paper page
  24. Zhou, Incremental network quantization: Towards lossless CNNs with low-precision weights, 1
  25. 10.1109/CVPR.2016.521 · DOI · Paper page
  26. 10.1007/978-1-4842-6666-3_1 · doi:10.1007/978-1-4842-6666-3_1
  27. Paszke, 2017, Pytorch
  28. 10.1145/3297858.3304076 · DOI · Paper page
  29. Ouyang, Stochastic alternating direction method of multipliers, 80
  30. Suzuki, Dual averaging and proximal gradient descent for online alternating direction multiplier method, 392
  31. 10.1561/9781601984616 · DOI · Paper page
  32. 10.1007/978-3-030-01237-3_12 · DOI · Paper page
  33. 10.1109/VLSIC.2018.8502404 · DOI · Paper page
  34. 10.1109/MICRO.2016.7783723 · DOI · Paper page
  35. 10.1145/3140659.3080254 · DOI · Paper page
  36. He, 2018, Simultaneously optimizing weight and quantizer of ternary neural network using truncated Gaussian approximation
  37. 10.1145/3349567.3351718 · doi:10.1145/3349567.3351718
  38. 10.1609/aaai.v34i04.5954 · DOI · Paper page
  39. 10.1007/978-3-030-58601-0_37 · DOI
  40. 2018, FBGEMM
  41. 10.1137/140990309 · DOI · Paper page
  42. Liu, Zeroth-order online alternating direction method of multipliers: Convergence analysis and applications, 288
  43. 10.1007/978-3-319-99223-5 · DOI
  44. 10.1109/TNNLS.2019.2910073 · DOI · Paper page
  45. 10.1109/JSTSP.2020.2969554 · DOI · Paper page
  46. 10.5555/2999134.2999257 · doi:10.5555/2999134.2999257
  47. Simonyan, 2014, Very deep convolutional networks for large-scale image recognition
  48. 10.1109/ICME.2018.8486500 · DOI
  49. 2019, Google Supercharges Machine Learning Tasks With TPU Custom Chip
  50. Molchanov, 2016, Pruning convolutional neural networks for resource efficient inference
  51. 10.1109/CVPR.2018.00958 · DOI · Paper page
  52. Zhuang, Discrimination-aware channel pruning for deep neural networks, 875
  53. 10.1109/ICCV.2017.541 · DOI · Paper page
  54. 10.1109/CVPR42600.2020.01464 · DOI · Paper page
  55. Min, 2018, 2PFPCE: Two-phase filter pruning based on conditional entropy

Cited by

See all 115

Metadata

Authors
Ma, X., Lin, S., Ye, S., He, Z., Zhang, L., Yuan, G., Tan, S. H., Li, Z., Fan, D., Qian, X., Lin, X., Ma, K., & Wang, Y.
Venue
IEEE Transactions on Neural Networks and Learning Systems
Published
2022-09
Volume
33
Issue
9
Pages
4930-4944
ISSN
  • 2162-237X (ISSN-L)
  • 2162-2388
OpenAlex
W3139203094
Cited by
115
References
21
FWCI
7.50
Type
journal article
Language
en
Open access
green
Funders
  • National Science Foundation (CCF-1919117, CCF-1937500, CCF-1901378)
  • Army Research Office (ARO) Young Investigator Program (76598CSYIP)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Non-Structured DNN Weight Pruning—Is It Beneficial in Any Platform? https://app.cochat.ai/papers/10.1109/tnnls.2021.3063265
The paper
Ma, X., Lin, S., Ye, S., He, Z., Zhang, L., Yuan, G., Tan, S. H., Li, Z., Fan, D., Qian, X., Lin, X., Ma, K., & Wang, Y. (2022). Non-Structured DNN Weight Pruning—Is It Beneficial in Any Platform? IEEE Transactions on Neural Networks and Learning Systems. https://doi.org/10.1109/tnnls.2021.3063265

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.