CoChat
Sign Up
PreprintCited by 270Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningMachine Learning and Data Classification

Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy

Open access
Mishra, Asit, Marr, Debbie·2017·arXiv
DOI 10.48550/arxiv.1711.05852arXiv 1711.05852
Open this paper in CoChat — free accountOpen PDF

Abstract

Deep learning networks have achieved state-of-the-art accuracies on computer vision workloads like image classification and object detection. The performant systems, however, typically involve big models with numerous parameters. Once trained, a challenging aspect for such top performing models is deployment on resource constrained inference systems - the models (often deep networks or wide networks or both) are compute and memory intensive. Low-precision numerics and model compression using knowledge distillation are popular techniques to lower both the compute requirements and memory footprint of these deployed models. In this paper, we study the combination of these two techniques and show that the performance of low-precision networks can be significantly improved by using knowledge distillation techniques. Our approach, Apprentice, achieves state-of-the-art accuracies using ternary precision and 4-bit precision for variants of ResNet architecture on ImageNet dataset. We present three schemes using which one can apply knowledge distillation techniques to various stages of the train-and-deploy pipeline.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1711.05852

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Binarization of wider networks can achieve a good trade-off between accuracy and model size.

    “Inspired by the fact that binarizing the activations reduces their representational capability for the succeeding convolution block, [178] and [34] show that binarization of wider networks (i.e., networks with larger number of filters) can achieve a good trade-off between the accuracy and the model size.”

    From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [178]

    Section 5.5 (p. 10) says ternary-precision models can combine reduced model size with strong accuracy. The Appendix (p. 14) finds that increasing ResNet depth diminishes the accuracy impact of ternarization, but it does not establish the claimed trade-off for binarized networks widened by adding filters. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 20

  1. [2]2. Lei Jimmy Ba and Rich Caurana. Do deep nets really need to be deep? CoRR , abs/1312.6184, 2013. URL http://arxiv.org/abs/1312.6184 . · arXiv
  2. [3]3. Cristian Buciluˇ a, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '06, pp. 535-541, New York, NY, USA, 2006. ACM. ISBN 1-59593-339-5. doi: 10. 1145/1150402.1150464. URL http://doi.acm.org/10.1145/1150402.1150464 . · DOI · Paper page
  3. [4]4. Alfredo Canziani, Adam Paszke, and Eugenio Culurciello. An analysis of deep neural network models for practical applications. CoRR , abs/1605.07678, 2016. URL http://arxiv.org/ abs/1605.07678 . · arXiv
  4. [5]5. Tianqi Chen, Ian J. Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. CoRR , abs/1511.05641, 2015a. URL http://arxiv.org/abs/1511.05641 . · arXiv
  5. [7]7. Matthieu Courbariaux and Yoshua Bengio. Binarynet: Training deep neural networks with weights and activations constrained to +1 or -1. CoRR , abs/1602.02830, 2016. URL http://arxiv. org/abs/1602.02830 . · arXiv
  6. [8]8. Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. CoRR , abs/1511.00363, 2015. URL http: //arxiv.org/abs/1511.00363 . · arXiv
  7. [9]9. Misha Denil, Babak Shakibi, Laurent Dinh, Marc'Aurelio Ranzato, and Nando de Freitas. Predicting parameters in deep learning. CoRR , abs/1306.0543, 2013. URL http://arxiv.org/abs/ 1306.0543 . · arXiv
  8. [10]10. Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. CoRR , abs/1502.02551, 2015. URL http://arxiv.org/abs/ 1502.02551 . · arXiv
  9. [11]11. Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. CoRR , abs/1510.00149, 2015a. URL http://arxiv.org/abs/1510.00149 . · arXiv
  10. [12]12. Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada , pp. 1135-1143, 2015b.
  11. [13]13. Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A. Horowitz, and William J. Dally. EIE: efficient inference engine on compressed deep neural network. In 43rd ACM/IEEE Annual International Symposium on Computer Architecture, ISCA 2016, Seoul, South Korea, June 18-22, 2016 , pp. 243-254, 2016. doi: 10.1109/ISCA.2016.30. URL https://doi.org/10. 1109/ISCA.2016.30 . · DOI · Paper page
  12. [2]2. Vincent Vanhoucke, Andrew Senior, and Mark Z. Mao. Improving the speed of neural networks on cpus. In Deep Learning and Unsupervised Feature Learning Workshop, NIPS 2011 , 2011.
  13. [3]3. Ganesh Venkatesh, Eriko Nurvitadhi, and Debbie Marr. Accelerating deep convolutional networks using low-precision and sparsity. CoRR , abs/1610.00324, 2016. URL http://arxiv.org/ abs/1610.00324 . · arXiv
  14. [4]4. Kilian Q. Weinberger, Anirban Dasgupta, Josh Attenberg, John Langford, and Alexander J. Smola. Feature hashing for large scale multitask learning. CoRR , abs/0902.2206, 2009. URL http: //arxiv.org/abs/0902.2206 . · arXiv
  15. [5]5. Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. CoRR , abs/1608.03665, 2016. URL http://arxiv.org/abs/ 1608.03665 . · arXiv
  16. [6]6. Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , July 2017.
  17. [7]7. Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. CoRR , abs/1612.03928, 2016. URL http://arxiv.org/abs/1612.03928 . · arXiv
  18. [8]8. Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantization: Towards lossless cnns with low-precision weights. CoRR , abs/1702.03044, 2017. URL http://arxiv.org/abs/1702.03044 . · arXiv
  19. [9]9. Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. CoRR , abs/1606.06160, 2016. URL http://arxiv.org/abs/1606.06160 . · arXiv
  20. [10]10. Chenzhuo Zhu, Song Han, Huizi Mao, and William J. Dally. Trained ternary quantization. CoRR , abs/1612.01064, 2016. URL http://arxiv.org/abs/1612.01064 . · arXiv

Cited by

Sorted by citations

Authors
Mishra, A., & Marr, D.
Affiliations
Intel (United States) (US)
Venue
arXiv
Publisher
arXiv
Published
2017
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1711.05852
arXiv
1711.05852
OpenAlex
W2769312802
Cited by
270
References
5
Citations by year
2025: 6 · 2024: 15 · 2023: 26 · 2022: 29 · 2021: 68 · 2020: 60 · 2019: 44 · 2018: 21 · 2017: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
knowledge distillation, low-precision arithmetic, 4-bit quantization, model compression, ResNet, ImageNet, inference efficiency, resource-constrained deployment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review