CoChat
Sign Up
PreprintCited by 126Advanced Neural Network ApplicationsMachine Learning and Data ClassificationDomain Adaptation and Few-Shot Learning

Dynamic Model Pruning with Feedback

Open access
Lin, Tao, Stich, Sebastian U., Barba, Luis, Dmitriev, Daniil, Jaggi, Martin·2020·arXiv
DOI 10.48550/arxiv.2006.07253arXiv 2006.07253
Open this paper in CoChat — free accountOpen PDF

Abstract

Deep neural networks often have millions of parameters. This can hinder their deployment to low-end devices, not only due to high memory requirements but also because of increased latency at inference. We propose a novel model compression method that generates a sparse trained model without additional overhead: by allowing (i) dynamic allocation of the sparsity pattern and (ii) incorporating feedback signal to reactivate prematurely pruned weights we obtain a performant sparse model in one single training pass (retraining is not needed, but can further improve the performance). We evaluate our method on CIFAR-10 and ImageNet, and show that the obtained sparse models can reach the state-of-the-art performance of dense models. Moreover, their performance surpasses that of models generated by all previously proposed pruning schemes.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2006.07253

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Iterative pruning generally outperforms one-shot pruning.

    “Iterative pruning [1] is widely used in machine learning because it generally outperforms the one-shot pruning method [2].”

    From PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation (2021) · cites this paper as [1]

    The Method (p. 4) and Discussion (p. 8) say that the paper’s dynamic pruning method, DPF, can find better sparse models than one-shot pruning. They do not establish that iterative pruning generally outperforms one-shot pruning; Related Work (p. 2) instead describes one-shot pruning with fine-tuning as a standard method. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 41

  1. Jose M Alvarez and Mathieu Salzmann. Learning the number of neurons in deep networks. In Advances in Neural Information Processing Systems , pp. 2270-2278, 2016.
  2. Jose M Alvarez and Mathieu Salzmann. Compression-aware training of deep networks. In Advances in Neural Information Processing Systems , pp. 856-867, 2017.
  3. Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. In ICLR - International Conference on Learning Representations , 2018. URL https://openreview.net/forum?id=BJ\_wN01C-.
  4. Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 , 2013. · arXiv
  5. Miguel A Carreira-Perpinán and Yerlan Idelbayev. 'learning-compression' algorithms for neural net pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 8532-8541, 2018.
  6. Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In NeurIPS - Advances in Neural Information Processing Systems , pp. 3123-3131, 2015.
  7. Xiaoliang Dai, Hongxu Yin, and Niraj Jha. Nest: A neural network synthesis tool based on a grow-and-prune paradigm. IEEE Transactions on Computers , 2019.
  8. Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.04840 , 2019. · arXiv
  9. Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR - International Conference on Learning Representations , 2019. URL https://openreview.net/forum?id=rJl-b3RcF7 .
  10. Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis. arXiv preprint arXiv:1903.01611 , 2019. · arXiv
  11. Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. In NeurIPS - Advances in Neural Information Processing Systems , pp. 3581-3590, 2017.
  12. Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet in 1 hour. arXiv preprint arXiv:1706.02677 , 2017. · arXiv
  13. Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient DNNs. In NeurIPS - Advances in Neural Information Processing Systems , pp. 1379-1387, 2016.
  14. Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NeurIPS - Advances in Neural Information Processing Systems , pp. 1135-1143, 2015.
  15. Song Han, Jeff Pool, Sharan Narang, Huizi Mao, Shijian Tang, Erich Elsen, Bryan Catanzaro, John Tran, and William J Dally. DSD: regularizing deep neural networks with dense-sparse-dense training flow. In ICLR - International Conference on Learning Representations , 2017.
  16. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision , pp. 630-645. Springer, 2016b.
  17. Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. In International Joint Conference on Artificial Intelligence (IJCAI) , pp. 2234-2240, 2018.
  18. Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2019.
  19. Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision , pp. 646-661. Springer, 2016.
  20. Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 4700-4708, 2017.
  21. Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian Stich, and Martin Jaggi. Error feedback fixes SignSGD and other gradient compression schemes. In International Conference on Machine Learning , pp. 3252-3261, 2019.
  22. Simon Lacoste-Julien, Mark W. Schmidt, and Francis R. Bach. A simpler approach to obtaining an O (1 /t ) convergence rate for the projected stochastic subgradient method. arXiv preprint arXiv:1212.2002 , 2012. · arXiv
  23. Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In NeurIPS - Advances in Neural Information Processing Systems , pp. 598-605, 1990.
  24. Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. SNIP: Single-shot network pruning based on connection sensitivity. In ICLR - International Conference on Learning Representations , 2019.
  25. Hao Li, Soham De, Zheng Xu, Christoph Studer, Hanan Samet, and Tom Goldstein. Training quantized nets: A deeper understanding. In NeurIPS - Advances in Neural Information Processing Systems , pp. 5811-5821, 2017.
  26. Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In ICLR - International Conference on Learning Representations , 2019.
  27. Sangkug Lym, Esha Choukse, Siavash Zangeneh, Wei Wen, Mattan Erez, and Sujay Shanghavi. Prunetrain: Gradual structured pruning from scratch for faster neural network training. In International Conference for High Performance Computing, Networking, Storage and Analysis (SC) , 2019.
  28. Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications , 9(1):2383, 2018.
  29. Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In ICML - International Conference on Machine Learning , pp. 2498-2507. JMLR. org, 2017.
  30. Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In ICML - International Conference on Machine Learning , 2019.
  31. Michael C Mozer and Paul Smolensky. Skeletonization: A technique for trimming the fat from a network via relevance assessment. In NeurIPS - Advances in Neural Information Processing Systems , pp. 107-115, 1989.
  32. Sharan Narang, Erich Elsen, Gregory Diamos, and Shubho Sengupta. Exploring sparsity in recurrent neural networks. In ICLR - International Conference on Learning Representations , 2017.
  33. Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, and Dmitry P Vetrov. Structured bayesian pruning via log-normal multiplicative noise. In NeurIPS - Advances in Neural Information Processing Systems , pp. 6775-6784, 2017.
  34. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision , 115(3):211-252, 2015.
  35. Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR - International Conference on Learning Representations , 2015.
  36. Suraj Srinivas, Akshayvarun Subramanya, and R Venkatesh Babu. Training sparse neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , pp. 138-145, 2017.
  37. Sebastian U. Stich and Sai P. Karimireddy. The error-feedback framework: Better rates for SGD with delayed gradients and compressed communication. arXiv preprint arXiv:1909.05350 , 2019. URL https://arxiv.org/abs/1909.05350 . · arXiv
  38. Jianbo Ye, Xin Lu, Zhe Lin, and James Z. Wang. Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers. In ICLR - International Conference on Learning Representations , 2018. URL https://openreview.net/forum?id=HJ94fqApW .
  39. Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. Understanding straight-through estimator in training activation quantized neural nets. In ICLR - International Conference on Learning Representations , 2019.
  40. Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. In NeurIPS - Advances in Neural Information Processing Systems , 2019.
  41. Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878 , 2017. · arXiv

Cited by

Sorted by citations

Authors
Lin, T., Stich, S. U., Barba, L., Dmitriev, D., & Jaggi, M.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2006.07253
arXiv
2006.07253
OpenAlex
W2995463996
Cited by
126
References
5
Citations by year
2025: 7 · 2024: 22 · 2023: 25 · 2022: 22 · 2021: 36 · 2020: 12 · 2019: 2
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (175796)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
sparse neural networks, model compression, inference latency reduction, memory-efficient deep learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review