CoChat
Sign Up
PreprintCited by 339Advanced Neural Network ApplicationsMachine Learning and Data ClassificationNeural Networks and Applications

Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon

Open access
Dong, Xin, Chen, Shangyu, Pan, Sinno Jialin·2017·arXiv
DOI 10.48550/arxiv.1705.07565arXiv 1705.07565
Open this paper in CoChat — free accountOpen PDF

Abstract

How to develop slim and accurate deep neural networks has become crucial for real- world applications, especially for those employed in embedded systems. Though previous work along this research line has shown some promising results, most existing methods either fail to significantly compress a well-trained deep network or require a heavy retraining process for the pruned deep network to re-boost its prediction performance. In this paper, we propose a new layer-wise pruning method for deep neural networks. In our proposed method, parameters of each individual layer are pruned independently based on second order derivatives of a layer-wise error function with respect to the corresponding parameters. We prove that the final prediction performance drop after pruning is bounded by a linear combination of the reconstructed errors caused at each layer. Therefore, there is a guarantee that one only needs to perform a light retraining process on the pruned network to resume its original prediction performance. We conduct extensive experiments on benchmark datasets to demonstrate the effectiveness of our pruning method compared with several state-of-the-art baseline methods.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1705.07565

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The trace of the Hessian can measure a layer's sensitivity to quantization.

    “It was theoretically shown that the trace of the second-order operator (i.e., the Hessian) can be used to measure the sensitivity of a layer to quantization [50], similar to results for pruning in the seminal work of Optimal Brain Damage [139].”

    From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [50]

    Section 3.2 (p. 4) uses the inverse Hessian to measure parameter sensitivity for pruning, and Section 3.4.1 (p. 5) describes computing it layer by layer. These passages support a connection between Hessians and pruning sensitivity, but they do not show that the trace of the Hessian measures a layer’s sensitivity to quantization. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • HAWQv2 extends this to mixed-precision activation quantization and is over 100 times faster than RL-based mixed-precision methods.

    “In HAWQv2, this method was extended to mixedprecision activation quantization [50], and was shown to be more than 100x faster than RL based mixed-precision methods [246].”

    From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [50]

    The cited paper’s Results (p. 8) discuss the speed and computational cost of L-OBS pruning, not HAWQv2. Searches for HAWQv2, mixed-precision activation quantization, and a 100× speed comparison with RL-based methods found no passage addressing the claim. (LLM, conf=0.77)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 21

  1. [1][1] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature , 521(7553):436-444, 2015. -
  2. [2][2] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998. -
  3. [3][3] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014. - · arXiv
  4. [4][4] Luisa de Vivo, Michele Bellesi, William Marshall, Eric A Bushong, Mark H Ellisman, Giulio Tononi, and Chiara Cirelli. Ultrastructural evidence for synaptic scaling across the wake/sleep cycle. Science , 355(6324):507-510, 2017. -
  5. [5][5] Misha Denil, Babak Shakibi, Laurent Dinh, Nando de Freitas, et al. Predicting parameters in deep learning. In Advances in Neural Information Processing Systems , pages 2148-2156, 2013. -
  6. [6][6] Nguyen N. Aghasi, A. and J. Romberg. Net-trim: A layer-wise convex pruning of deep neural networks. Journal of Machine Learning Research , 2016. -
  7. [7][7] Russell Reed. Pruning algorithms-a survey. IEEE transactions on Neural Networks , 4(5):740747, 1993. -
  8. [8][8] Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115 , 2014. - · arXiv
  9. [9][9] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems , pages 1135-1143, 2015. -
  10. [10][10] Yi Sun, Xiaogang Wang, and Xiaoou Tang. Sparsifying neural network connections for face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4856-4864, 2016. -
  11. [11][11] Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. In Advances In Neural Information Processing Systems , pages 1379-1387, 2016. -
  12. [12][12] Yann LeCun, John S Denker, Sara A Solla, Richard E Howard, and Lawrence D Jackel. Optimal brain damage. In NIPs , volume 2, pages 598-605, 1989. -
  13. [13][13] Babak Hassibi, David G Stork, et al. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems , pages 164-164, 1993. -
  14. [15][15] Nikolas Wolfe, Aditya Sharma, Lukas Drude, and Bhiksha Raj. The incredible shrinking neural network: New perspectives on learning representations through the lens of pruning. arXiv preprint arXiv:1701.04465 , 2017. - · arXiv
  15. [16][16] Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03250 , 2016. - · arXiv
  16. [17][17] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 , 2016. - · arXiv
  17. [18][18] Xiaojie Jin, Xiaotong Yuan, Jiashi Feng, and Shuicheng Yan. Training skinny deep neural networks with iterative hard thresholding methods. arXiv preprint arXiv:1607.05423 , 2016. - · arXiv
  18. [19][19] Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang, et al. Convolutional neural networks with low-rank regularization. arXiv preprint arXiv:1511.06067 , 2015. - · arXiv
  19. [20][20] Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky. Sparse convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 806-814, 2015. -
  20. [23][23] Ian J Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron C Courville, and Yoshua Bengio. Maxout networks. ICML (3) , 28:1319-1327, 2013. -
  21. [25][25] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , pages 1097-1105, 2012. -

Cited by

Sorted by citations

Authors
Dong, X., Chen, S., & Pan, S. J.
Affiliations
  • University of Electronic Science and Technology of China (CN)
  • Sun Yat-sen University (CN)
  • Nanyang Technological University (SG)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2017
DOI
10.48550/arxiv.1705.07565
arXiv
1705.07565
OpenAlex
W2618305643
Cited by
339
References
2
Citations by year
2026: 1 · 2025: 20 · 2024: 32 · 2023: 38 · 2022: 46 · 2021: 70 · 2020: 56 · 2019: 52 · 2018: 23 · 2017: 1
FWCI
10.39
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • DR-NTU (Nanyang Technological University) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
Keywords
neural network pruning, layer-wise pruning, optimal brain surgeon, second derivative, model compression, embedded systems

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review