CoChat

Explicit Loss-Error-Aware Quantization for Low-Bit Deep Neural Networks

Zhou, Aojun, Yao, Anbang, Wang, Kuan, Chen, Yurong · 2018 · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
Cited by 84

The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.

Abstract

Benefiting from tens of millions of hierarchically stacked learnable parameters, Deep Neural Networks (DNNs) have demonstrated overwhelming accuracy on a variety of artificial intelligence tasks. However reversely, the large size of DNN models lays a heavy burden on storage, computation and power consumption, which prohibits their deployments on the embedded and mobile systems. In this paper, we propose Explicit Loss-error-aware Quantization (ELQ), a new method that can train DNN models with very low-bit parameter values such as ternary and binary ones to approximate 32-bit floating-point counterparts without noticeable loss of predication accuracy. Unlike existing methods that usually pose the problem as a straightforward approximation of the layer-wise weights or outputs of the original full-precision model (specifically, minimizing the error of the layer-wise weights or inner products of the weights and the inputs between the original and respective quantized models), our ELQ elaborately bridges the loss perturbation from the weight quantization and an incremental quantization strategy to address DNN quantization. Through explicitly regularizing the loss perturbation and the weight approximation error in an incremental way, we show that such a new optimization method is theoretically reasonable and practically effective. As validated with two mainstream convolutional neural network families (i.e., fully convolutional and non-fully convolutional), our ELQ shows better results than state-of-the-art quantization methods on the large scale ImageNet classification dataset. Code will be made publicly available.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr.2018.00982

Cites

16 of 16 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 16 references

Research works printed but not linked: 30

  1. yann, 1990, Optimal brain damage, NIPS
  2. 10.1109/CVPR.2013.75 · DOI · Paper page
  3. soudry, 2014, Expectation backpropagation: Parameter-free training of multilayer neural networks with continuous or discrete weights, NIPS
  4. simonyan, 2015, Very deep convolutional networks for large-scale image recognition, ICLRE
  5. 10.1007/s11263-015-0816-y · DOI · Paper page
  6. 10.1109/CVPR.2017.634 · DOI · Paper page
  7. vanhoucke, 2011, Improving the speed of neural networks on cpus, NIPS Workshop
  8. 10.1109/CVPR.2015.7298594 · DOI · Paper page
  9. 10.1109/CVPR.2017.754 · DOI · Paper page
  10. 10.1109/MSP.2012.2205597 · DOI · Paper page
  11. hinton, 2014, Distilling the knowledge in a neural network, NIPS
  12. hou, 2017, Loss-aware binarization of deep networks, ICLRE
  13. howard, 2017, Mobilenets Efficient convolutional neural networks for mobile vision applications
  14. hu, 2016, A data-driven neuron pruning approach towards efficient deep architectures
  15. 10.1109/CVPR.2017.243 · DOI · Paper page
  16. hubara, 2016, Quantized neural networks Training neural networks with low precision weights and activations
  17. courbariaux, 2016, Binarized neural networks: Training neural networks with weights and activations constrained to +1 or ?1, NIPS
  18. polino, 2018, Model compression via distillation and quantization, ICLRE
  19. courbariaux, 2015, Binaryconnect: Training deep neural networks with binary weights during propagations, NIPS
  20. guo, 2016, Dynamic network surgery for efficient dnns, NIPS
  21. 10.1145/3007787.3001163 · DOI · Paper page
  22. collobert, 2011, Natural language processing (almost) from scratch, Journal of Machine Learning Research
  23. hassibi, 1993, Second order derivatives for network pruning: Optimal brain surgeon, NIPS
  24. 10.1109/CVPR.2017.195 · DOI · Paper page
  25. 10.1145/2647868.2654889 · DOI · Paper page
  26. zhu, 2017, Trained ternary quantization, ICLRE
  27. krizhevsky, 2009, Learning multiple layers of features from tiny images
  28. li, 2017, Training quantized nets: A deeper understanding, NIPS
  29. mishra, 2018, Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy, ICLRE
  30. zhou, 2017, Incremental network quantization: Towards lossless cnns with low-precision weights, ICLRE

Cited by

Metadata

Authors
Zhou, A., Yao, A., Wang, K., & Chen, Y.
Venue
2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
Publisher
IEEE
Published
2018-06
Pages
9426-9435
ISSN
1063-6919 (ISSN-L)
OpenAlex
W2799238091
Cited by
84
References
16
FWCI
4.38
Type
conference paper
Language
en
Open access
closed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.