CoChat
Sign Up
Cited by 2,761Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningMachine Learning and Data Classification

Learning Efficient Convolutional Networks through Network Slimming

Open access
Liu, Zhuang, Li, Jianguo, Shen, Zhiqiang, Huang, Gao, Yan, Shoumeng, Zhang, Changshui·2017·2017 IEEE International Conference on Computer Vision (ICCV)
DOI 10.1109/iccv.2017.298
Open this paper in CoChat — free accountOpen PDF

Abstract

The deployment of deep convolutional neural networks (CNNs) in many real world applications is largely hindered by their high computational cost. In this paper, we propose a novel learning scheme for CNNs to simultaneously 1) reduce the model size; 2) decrease the run-time memory footprint; and 3) lower the number of computing operations, without compromising accuracy. This is achieved by enforcing channel-level sparsity in the network in a simple but effective way. Different from many existing approaches, the proposed method directly applies to modern CNN architectures, introduces minimum overhead to the training process, and requires no special software/hardware accelerators for the resulting models. We call our approach network slimming, which takes wide and large networks as input models, but during training insignificant channels are automatically identified and pruned afterwards, yielding thin and compact models with comparable accuracy. We empirically demonstrate the effectiveness of our approach with several state-of-the-art CNN models, including VGGNet, ResNet and DenseNet, on various image classification datasets. For VGGNet, a multi-pass version of network slimming gives a 20× reduction in model size and a 5× reduction in computing operations.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/iccv.2017.298

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Operations with the same FLOPs can have different running time depending on the platform.

    “Second, operations with the same FLOPs could have different running time, depending on the platform.”

    From ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design (2018) · cites this paper as [19]

    The Introduction (page unavailable) indicates that hardware affects inference speed, noting that a large CNN may take minutes to process an image on a mobile device and that some techniques need special hardware for speedup. The Additional information section reports FLOP and wall-clock savings on one GPU, but does not compare equal-FLOP operations across platforms; that part of the claim is unsupported. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • FLOPs is an indirect metric and is usually not equivalent to the direct metric such as speed or latency.

    “However, FLOPs is an indirect metric. It is an approximation of, but usually not equivalent to the direct metric that we really care about, such as speed or latency.”

    From ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design (2018) · cites this paper as [19]

    The Additional information section reports that wall-clock time savings for one pruned VGGNet only “roughly” matched its FLOP savings; Tables 1(a) and 6 give different FLOP and timing reductions. This supports treating FLOPs as an approximation of speed for that model, but the paper does not establish the broader claim that the two are usually not equivalent. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 29

  1. B. Baker, O. Gupta, N. Naik, and R. Raskar. Designing neu- ral network architectures using reinforcement learning. In ICLR, 2017.
  2. S. Changpinyo, M. Sandler, and A. Zhmoginov. The power of sparsity in convolutional neural networks. arXiv preprint arXiv:1702.06257, 2017. · arXiv
  3. R. Collobert, K. Kavukcuoglu, and C. Farabet. Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop, number EPFL-CONF-192376, 2011.
  4. M. Courbariaux and Y. Bengio. Binarynet: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830, 2016. · arXiv
  5. E. L. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fer- gus. Exploiting linear structure within convolutional net- works for efficient evaluation. In NIPS, 2014.
  6. R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich fea- ture hierarchies for accurate object detection and semantic segmentation. In CVPR, pages 580-587, 2014.
  7. I. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio. Maxout networks. In ICML, 2013.
  8. S. Han, H. Mao, and W. J. Dally. Deep compression: Com- pressing deep neural network with pruning, trained quanti- zation and huffman coding. In ICLR, 2016.
  9. S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural network. In NIPS, pages 1135-1143, 2015.
  10. K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In ECCV, pages 630-645. Springer, 2016.
  11. G. Huang, D. Chen, T. Li, F. Wu, L. van der Maaten, and K. Q. Weinberger. Multi-scale dense convolutional networks for efficient prediction. arXiv preprint arXiv:1703.09844, 2017. · arXiv
  12. G. Huang, Z. Liu, K. Q. Weinberger, and L. van der Maaten. Densely connected convolutional networks. In CVPR, 2017.
  13. G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger. Deep networks with stochastic depth. In ECCV, 2016.
  14. J. Jin, Z. Yan, K. Fu, N. Jiang, and C. Zhang. Neural network architecture optimization through submodularity and super- modularity. arXiv preprint arXiv:1609.00074, 2016. · arXiv
  15. H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710, 2016. · arXiv
  16. M. Lin, Q. Chen, and S. Yan. Network in network. In ICLR, 2014.
  17. B. Liu, M. Wang, H. Foroosh, M. Tappen, and M. Pensky. Sparse convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 806-814, 2015.
  18. J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR, pages 3431- 3440, 2015.
  19. Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised fea- ture learning, 2011. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
  20. M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. Xnor- net: Imagenet classification using binary convolutional neu- ral networks. In ECCV, 2016.
  21. S. Scardapane, D. Comminiello, A. Hussain, and A. Uncini. Group sparse regularization for deep neural networks. arXiv preprint arXiv:1607.00485, 2016. · arXiv
  22. M. Schmidt, G. Fung, and R. Rosales. Fast optimization methods for l1 regularization: A comparative study and two new approaches. In ECML, pages 286-297, 2007.
  23. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
  24. S. Srinivas, A. Subramanya, and R. V. Babu. Training sparse neural networks. CoRR, abs/1611.06694, 2016. · arXiv
  25. I. Sutskever, J. Martens, G. Dahl, and G. Hinton. On the importance of initialization and momentum in deep learning. In ICML, 2013.
  26. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, et al. Going deeper with convolu- tions. In CVPR, pages 1-9, 2015.
  27. W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li. Learning structured sparsity in deep neural networks. In NIPS, 2016.
  28. H. Zhou, J. M. Alvarez, and F. Porikli. Less is more: Towards compact cnns. In ECCV, 2016.
  29. B. Zoph and Q. V. Le. Neural architecture search with rein- forcement learning. In ICLR, 2017.

Cited by

Sorted by citations

Authors
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., & Zhang, C.
Affiliations
  • Tsinghua University (CN)
  • Intel (United States) (US)
  • Fudan University (CN)
  • Cornell University (US)
ORCID
Venue
2017 IEEE International Conference on Computer Vision (ICCV)
Publisher
IEEE
Published
2017-10
Pages
2755-2763
DOI
10.1109/iccv.2017.298
OpenAlex
W2962851801
Cited by
2,761
References
5
Citations by year
2026: 176 · 2025: 306 · 2024: 386 · 2023: 435 · 2022: 430 · 2021: 445 · 2020: 332 · 2019: 200 · 2018: 49 · 2017: 1
FWCI
46.53
Type
conference paper
Language
en
Open access
Locations
  • IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Deutsche Forschungsgemeinschaft (DFG TRR 169, NSFC 61621136008/DFG TRR-169, TRR169, 61621136008, NSFC 61621136008, 61621136008/DFG TRR-169)
  • National Natural Science Foundation of China (61621136008, NSFC 61621136008, TRR-169, 61621136008/DFG TRR-169)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
channel pruning, channel sparsity, convolutional neural networks, model compression, memory footprint reduction, computational complexity reduction, training overhead, image classification, VGG, ResNet, DenseNet

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review