Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Cited by 363

The DOI matches its Crossref record; 193 of the 193 references it cites were found in a registry or an index.

Structured Pruning for Deep Convolutional Neural Networks: A Survey

Open access
He, Yang, Xiao, Lingao · 2024 · IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract

The remarkable performance of deep Convolutional neural networks (CNNs) is generally attributed to their deeper and wider architectures, which can come with significant computational costs. Pruning neural networks has thus gained interest since it effectively lowers storage and computational costs. In contrast to weight pruning, which results in unstructured models, structured pruning provides the benefit of realistic acceleration by producing models that are friendly to hardware implementation. The special requirements of structured pruning have led to the discovery of numerous new challenges and the development of innovative solutions. This article surveys the recent progress towards structured pruning of deep CNNs. We summarize and compare the state-of-the-art structured pruning techniques with respect to filter ranking methods, regularization methods, dynamic execution, neural architecture search, the lottery ticket hypothesis, and the applications of pruning. While discussing structured pruning algorithms, we briefly introduce the unstructured pruning counterpart to emphasize their differences. Furthermore, we provide insights into potential research opportunities in the field of structured pruning. A curated list of neural network pruning papers can be found at: https://github.com/he-y/Awesome-Pruning. A dedicated website offering a more interactive comparison of structured pruning methods can be found at: https://huggingface.co/spaces/he-yang/Structured-Pruning-Survey.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tpami.2023.3334614

Cites

193 of 193 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 89

  1. Han, Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding
  2. Simonyan, Very deep convolutional networks for large-scale image recognition
  3. Han, Learning both weights and connections for efficient neural network, 1135
  4. Li, Pruning filters for efficient convnets
  5. 10.48550/ARXIV.1706.03762 · DOI · Paper page
  6. Hoefler, 2021, Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks, J. Mach. Learn. Res., 22, 1
  7. Yvinec, RED: Looking for redundancies for data-freestructured compression of deep neural networks, 20863
  8. Sui, Chip: Channel independence-based pruning for compact neural networks, 24604
  9. Tan, DropNet: Reducing neural network complexity via iterative pruning, 9356
  10. Ye, Good subnetworks provably exist: Pruning via greedy forward selection, 10820
  11. Zhuang, Discrimination-aware channel pruning for deep neural networks, 883
  12. Liebenwein, Provable filter pruning for efficient neural networks
  13. You, Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks
  14. Zhuang, Neuron-level structured pruning using polarization regularizer, 9865
  15. Ye, Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers
  16. Kang, Operation-aware soft channel pruning using differentiable masks, 5122
  17. Tang, SCOP: Scientific control for reliable neural network pruning, 10936
  18. 10.5555/3157096.3157329 · doi:10.5555/3157096.3157329
  19. Chen, Only train once: A one-shot neural network training and pruning framework, 19637
  20. Wang, Neural pruning via growing regularization
  21. Molchanov, Pruning convolutional neural networks for resource efficient inference
  22. Peng, Collaborative channel pruning for deep networks, 5113
  23. Wang, Eigendamage: Structured pruning in the kronecker-factored eigenbasis, 6566
  24. Liu, Group Fisher pruning for practical network compression, 7021
  25. Nonnenmacher, SOSP: Efficiently capturing global correlations by second-order structured pruning
  26. Dai, Compressing neural networks using the variational information bottleneck, 1135
  27. Louizos, Bayesian compression for deep learning, 3290
  28. Neklyudov, Structured Bayesian pruning via log-normal multiplicative noise, 6778
  29. Huang, Training structured neural networks through manifold identification and variance reduction
  30. Kusupati, Soft threshold weight reparameterization for learnable sparsity, 5544
  31. Liu, Dynamic sparse graph for efficient deep learning
  32. Lin, Runtime neural pruning
  33. Gao, Dynamic channel pruning: Feature boosting and suppression
  34. Chen, Storage efficient and dynamic flexible runtime channel pruning via deep reinforcement learning, 14747
  35. Yu, Topology-aware network pruning using multi-stage graph embedding and reinforcement learning, 25656
  36. Dong, Network pruning via transformable architecture search, 760
  37. Liu, Rethinking the value of network pruning
  38. You, Drawing early-bird tickets: Towards more efficient training of deep networks
  39. Alizadeh, Prospect pruning: Finding trainable weights at initialization using meta-gradients
  40. Rachwan, Winning the lottery ahead of time: Efficient early network pruning
  41. Fischer, Plant’n’seek: Can you find the winning ticket?
  42. Cunha, Proving the lottery ticket hypothesis for convolutional neural networks
  43. Wang, Fast lossless neural compression with integer-only discrete flows, 22562
  44. Kim, Neuron merging: Compensating for pruned neurons, 585
  45. Meng, Pruning filter in filter, 17629
  46. Zhong, Revisit kernel pruning with lottery regulated grouped convolutions
  47. Tanaka, Pruning neural networks without any data by iteratively conserving synaptic flow, 6377
  48. Lee, A signal propagation perspective for pruning neural networks at initialization
  49. Bartoldson, The generalization-stability tradeoff in neural network pruning, 20852
  50. Orseau, Logarithmic pruning is all you need, 2925
  51. Yang, PROXSGD: Training structured neural networks under regularization and constraints
  52. Ding, Global sparse momentum SGD for pruning very deep neural networks, 6382
  53. Barsbey, Heavy tails in SGD and compressibility of overparametrized neural networks, 29364
  54. Miao, Learning pruning-friendly networks via Frank-Wolfe: One-shot, any-sparsity, and no retraining
  55. Zhang, One-shot pruning of recurrent neural networks by Jacobian spectrum evaluation
  56. Peste, AC/DC: Alternating compressed/decompressed training of deep neural networks, 8557
  57. Lee, Layer-adaptive sparsity for the magnitude-based pruning
  58. Liebenwein, Compressing neural networks: Towards determining the optimal layer-wise decomposition, 8557
  59. Liu, Frequency-domain dynamic pruning for convolutional neural networks, 1051
  60. Golkar, Continual learning via neural pruning
  61. Li, Revisiting discriminator in GAN compression: A generator-discriminator cooperative compression scheme, 28560
  62. Reed, 2022, A generalist agent, Trans. Mach. Learn. Res.
  63. Serra, Scaling up exact neural network compression by RELU stability, 27081
  64. 10.1007/978-3-031-20083-0_42 · doi:10.1007/978-3-031-20083-0_42
  65. Chen, Linearity grafting: Relaxed neuron pruning helps certifiable robustness, 3760
  66. Madaan, Adversarial neural pruning with latent vulnerability suppression, 6575
  67. Baykal, Data-dependent coresets for compressing neural networks with applications to generalization bounds
  68. 10.1155/2019/1978154 · doi:10.1155/2019/1978154
  69. Nitanda, Stochastic proximal gradient descent with acceleration techniques, 1574
  70. Kingma, Variational dropout and the local reparameterization trick, 2575
  71. 10.1137/1015034 · doi:10.1137/1015034
  72. LeCun, Optimal brain damage, 598
  73. Hassibi, Second order derivatives for network pruning: Optimal brain surgeon, 164
  74. Martens, Optimizing neural networks with kronecker-factored approximate curvature, 2408
  75. George, Fast approximate natural gradient descent in a kronecker factored eigenbasis, 9573
  76. Xiao, Dual averaging method for regularized stochastic learning and online optimization, 2116
  77. Srinivas, Gaussian process optimization in the bandit setting: No regret and experimental design, 1015
  78. Guo, Dynamic network surgery for efficient DNNs, 1387
  79. Hua, Channel gating neural networks, 1886
  80. Kipf, Semi-supervised classification with graph convolutional networks
  81. Chen, Stochastic gradient Hamiltonian Monte Carlo, 1683
  82. Frankle, Linear mode connectivity and the lottery ticket hypothesis, 3259
  83. Lee, Snip: Single-shot network pruning based on connection sensitivity
  84. Wang, Picking winning tickets before training by preserving gradient flow
  85. Jacot, Neural tangent kernel: Convergence and generalization in neural networks, 8580
  86. Lubana, A gradient flow framework for analyzing network pruning
  87. Renda, Comparing rewinding and fine-tuning in neural network pruning
  88. Yang, Dataset pruning: Reducing training data by examining generalization influence
  89. Guo, Sparse DNNs with improved adversarial robustness, 240

Cited by

See all 363

Metadata

Authors
He, Y., & Xiao, L.
Venue
IEEE Transactions on Pattern Analysis and Machine Intelligence
Published
2024-05
Volume
46
Issue
5
Pages
2900-2919
ISSN
  • 0162-8828 (ISSN-L)
  • 2160-9292
  • 1939-3539
OpenAlex
W4389104738
Cited by
363
References
193
FWCI
33.00
Type
journal article
Language
en
Open access
green
Funders
  • A*STAR Centre for Frontier AI Research
  • CFAR Internship Award and Research Excellence
  • A*STAR Science and Engineering Research Council
  • Maritime AI Research Programme (SMI-2022-MTP-06)
  • Singapore Maritime Institute

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.