CoChat
Sign Up
PreprintCited by 857Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningMachine Learning and Data Classification

A Survey of Model Compression and Acceleration for Deep Neural Networks

Open access
Cheng, Yu, Wang, Duo, Zhou, Pan, Zhang, Tao·2017·arXiv
DOI 10.48550/arxiv.1710.09282arXiv 1710.09282
Open this paper in CoChat — free accountOpen PDF

Abstract

Deep neural networks (DNNs) have recently achieved great success in many visual recognition tasks. However, existing deep neural network models are computationally expensive and memory intensive, hindering their deployment in devices with low memory resources or in applications with strict latency requirements. Therefore, a natural thought is to perform model compression and acceleration in deep networks without significantly decreasing the model performance. During the past five years, tremendous progress has been made in this area. In this paper, we review the recent techniques for compacting and accelerating DNN models. In general, these techniques are divided into four categories: parameter pruning and quantization, low-rank factorization, transferred/compact convolutional filters, and knowledge distillation. Methods of parameter pruning and quantization are described first, after that the other techniques are introduced. For each category, we also provide insightful analysis about the performance, related applications, advantages, and drawbacks. Then we go through some very recent successful methods, for example, dynamic capacity networks and stochastic depths networks. After that, we survey the evaluation matrices, the main datasets used for evaluating the model performance, and recent benchmark efforts. Finally, we conclude this paper, discuss remaining the challenges and possible directions for future work.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1710.09282

Cites

80 of 80 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Most reviews in related fields review object detection models and model compression technology separately, and there is a lack of comprehensive reviews covering both.

    “In addition to that, most of the reviews in related fields review the object detection model and model compression technology separately, and there is a lack of comprehensive reviews for both.”

    From A survey of model compression strategies for object detection (2023) · cites this paper as [10]

    The title and opening passage (p. 1) identify this paper as a review of model compression and acceleration, so it provides an example of a review focused on compression. They do not establish that most reviews treat object detection and compression separately, or that comprehensive reviews covering both are lacking. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Model compression and hardware acceleration using GPUs or TPUs further enhance efficiency.

    “Additional optimization techniques, such as model compression [165] and hardware acceleration using graphics processing units (GPUs) or tensor processing units (TPUs) [166], further enhance efficiency.”

    From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [165]

  • [205] reduces downlink by 14×, uplink by 28×, and local computation by 1.7×.

    “Downlink reduction: 14×; Uplink reduction: 28×; Local computation reduction: 1.7×”

    From Convergence of Edge Computing and Deep Learning: A Comprehensive Survey (2020) · cites this paper as [205]

    The Introduction (p. 1) discusses reducing storage and computational costs, and Quantization and Binarization (p. 2) describes model compression. Neither passage reports downlink, uplink, and local-computation reductions of 14×, 28×, and 1.7×. Searches for those terms and figures found no passage addressing the claimed result. (LLM, conf=0.73)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 22

  1. [1][1] A. Krizhevsky, I. Sutskever, and G. Hinton, 'Imagenet classification with deep convolutional neural networks,' in NIPS , 2012. -
  2. [10][10] S. Han, H. Mao, and W. J. Dally, 'Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,' International Conference on Learning Representations (ICLR) , 2016. -
  3. [14][14] M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi, 'Xnor-net: Imagenet classification using binary convolutional neural networks,' in ECCV , 2016. -
  4. [18][18] Z. Cai, X. He, J. Sun, and N. Vasconcelos, 'Deep learning with low precision by half-wave gaussian quantization,' in CVPR . IEEE Computer Society, 2017, pp. 5406-5414. -
  5. [20][20] Y. L. Cun, J. S. Denker, and S. A. Solla, 'Advances in neural information processing systems 2,' D. S. Touretzky, Ed., 1990, ch. Optimal Brain Damage, pp. 598-605. -
  6. [23][23] S. Han, J. Pool, J. Tran, and W. J. Dally, 'Learning both weights and connections for efficient neural networks,' in Proceedings of the 28th International Conference on Neural Information Processing Systems , ser. NIPS'15, 2015. -
  7. [31][31] Y. Cheng, F. X. Yu, R. Feris, S. Kumar, A. Choudhary, and S.-F. Chang, 'An exploration of parameter redundancy in deep networks with circulant projections,' in International Conference on Computer Vision (ICCV) , 2015. -
  8. [36][36] M. Moczulski, M. Denil, J. Appleyard, and N. de Freitas, 'Acdc: A structured efficient linear layer,' in International Conference on Learning Representations (ICLR) , 2016. -
  9. [37][37] R. Rigamonti, A. Sironi, V. Lepetit, and P. Fua, 'Learning separable filters,' in 2013 IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, June 23-28, 2013 , 2013, pp. 27542761. -
  10. [45][45] S. Zhai, Y. Cheng, and Z. M. Zhang, 'Doubly convolutional neural networks,' in Advances In Neural Information Processing Systems , 2016, pp. 1082-1090. -
  11. [48][48] S. Dieleman, J. De Fauw, and K. Kavukcuoglu, 'Exploiting cyclic symmetry in convolutional neural networks,' in Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48 , ser. ICML'16, 2016. -
  12. [62][62] A. Almahairi, N. Ballas, T. Cooijmans, Y. Zheng, H. Larochelle, and A. C. Courville, 'Dynamic capacity networks,' in Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016 , 2016, pp. 2549-2558. -
  13. [76][76] M. Lin, Q. Chen, and S. Yan, 'Network in network,' in ICLR , 2014. -
  14. [82][82] K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han, 'Haq: Hardware-aware automated quantization with mixed precision,' in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2019. -
  15. [84][84] J. M. Alvarez and M. Salzmann, 'Learning the number of neurons in deep networks,' pp. 2270-2278, 2016. -
  16. [86][86] Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang, 'Learning efficient convolutional networks through network slimming,' in ICCV , 2017. -
  17. [90][90] Y. Tian, D. Krishnan, and P. Isola, 'Contrastive representation distillation,' in International Conference on Learning Representations , 2020. [Online]. Available: https://openreview.net/forum?id=SkgpBJrtvS -
  18. [91][91] G. Chen, W. Choi, X. Yu, T. Han, and M. Chandraker, 'Learning efficient object detection models with knowledge distillation,' in Advances in Neural Information Processing Systems 30 , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., 2017, pp. 742-751. -
  19. [93][93] S. Sun, Y. Cheng, Z. Gan, and J. Liu, 'Patient knowledge distillation for BERT model compression,' in Empirical Methods in Natural Language Processing (EMNLP) , 2019. -
  20. [94][94] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, 'Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter,' CoRR , vol. abs/1910.01108, 2019. - · arXiv
  21. [98][98] Y. Mroueh, C.-L. Li, T. Sercu, A. Raj, and Y. Cheng, 'Sobolev GAN,' in ICLR , 2018. -
  22. [99][99] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, 'Generative adversarial nets,' in NeurIPS , 2014.

Cited by

Sorted by citations

Related

Authors
Cheng, Y., Wang, D., Zhou, P., & Zhang, T.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2017
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1710.09282
arXiv
1710.09282
OpenAlex
W2766839578
Cited by
857
References
80
Citations by year
2026: 8 · 2025: 44 · 2024: 71 · 2023: 105 · 2022: 91 · 2021: 216 · 2020: 180 · 2019: 112 · 2018: 30
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (publishedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
model compression, deep neural network, inference acceleration, parameter pruning, quantization, low-rank factorization, knowledge distillation, stochastic depth

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review