Explicit Loss-Error-Aware Quantization for Low-Bit Deep Neural Networks
The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
Benefiting from tens of millions of hierarchically stacked learnable parameters, Deep Neural Networks (DNNs) have demonstrated overwhelming accuracy on a variety of artificial intelligence tasks. However reversely, the large size of DNN models lays a heavy burden on storage, computation and power consumption, which prohibits their deployments on the embedded and mobile systems. In this paper, we propose Explicit Loss-error-aware Quantization (ELQ), a new method that can train DNN models with very low-bit parameter values such as ternary and binary ones to approximate 32-bit floating-point counterparts without noticeable loss of predication accuracy. Unlike existing methods that usually pose the problem as a straightforward approximation of the layer-wise weights or outputs of the original full-precision model (specifically, minimizing the error of the layer-wise weights or inner products of the weights and the inputs between the original and respective quantized models), our ELQ elaborately bridges the loss perturbation from the weight quantization and an incremental quantization strategy to address DNN quantization. Through explicitly regularizing the loss perturbation and the weight approximation error in an incremental way, we show that such a new optimization method is theoretically reasonable and practically effective. As validated with two mainstream convolutional neural network families (i.e., fully convolutional and non-fully convolutional), our ELQ shows better results than state-of-the-art quantization methods on the large scale ImageNet classification dataset. Code will be made publicly available.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2017) · arXiv · doi:10.48550/arxiv.1707.01083 · Cited by 9,982
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00716 · Cited by 9,982
Zagoruyko, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.87 · Cited by 7,244
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_32 · Cited by 4,408
Efficient Processing of Deep Neural Networks: A Tutorial and Survey
Sze, V., Chen, Y.-H., Yang, T.-J., & Emer, J. S. (2017) · Proceedings of the IEEE · doi:10.1109/jproc.2017.2761740 · Cited by 4,215 · cited 1×
“In the literature, there exists numerous methods for DNN compression and acceleration, and the reader is referred to [34] for a recent survey.”
MethodSelf-citationPruning Filters for Efficient ConvNets
Li, H., Kadav, A., Durdanovic, I., Samet, H., & Graf, H. P. (2016) · arXiv · doi:10.48550/arxiv.1608.08710 · Cited by 2,928
FitNets: Hints for Thin Deep Nets
Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.6550 · Cited by 2,859
Channel Pruning for Accelerating Very Deep Neural Networks
He, Y., Zhang, X., & Sun, J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.155 · Cited by 2,647 · cited 1×
“In the latest works [17] [11] [25], the authors propose to either trim convolutional filters or prune feature map channels.”
BackgroundDoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., & Zou, Y. (2016) · arXiv · doi:10.48550/arxiv.1606.06160 · Cited by 1,690
Learning both Weights and Connections for Efficient Neural Networks
Han, S., Pool, J., Tran, J., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1506.02626 · Cited by 1,481 · cited 2×
“[5] further present a more efficient method that can enable the recovery of wrongly pruned connections appeared in [8], yielding better compression performance.”
BackgroundMethodDeep Learning with Limited Numerical Precision
Gupta, S., Agrawal, A., Gopalakrishnan, K., & Narayanan, P. (2015) · arXiv · doi:10.48550/arxiv.1502.02551 · Cited by 1,364 · cited 1×
“In [36] [6], the authors use 8-bit fixed-point or 16-bit fixed-point representation to substitute 32-bit floating-point representation.”
MethodLi, F., Liu, B., Wang, X., Zhang, B., & Yan, J. (2016) · arXiv · doi:10.48550/arxiv.1605.04711 · Cited by 732
Loss-aware Weight Quantization of Deep Networks
Hou, L., & Kwok, J. T. (2018) · arXiv · doi:10.48550/arxiv.1802.08635 · Cited by 99
Research works printed but not linked: 30
- yann, 1990, Optimal brain damage, NIPS
- 10.1109/CVPR.2013.75 · DOI · Paper page
- soudry, 2014, Expectation backpropagation: Parameter-free training of multilayer neural networks with continuous or discrete weights, NIPS
- simonyan, 2015, Very deep convolutional networks for large-scale image recognition, ICLRE
- 10.1007/s11263-015-0816-y · DOI · Paper page
- 10.1109/CVPR.2017.634 · DOI · Paper page
- vanhoucke, 2011, Improving the speed of neural networks on cpus, NIPS Workshop
- 10.1109/CVPR.2015.7298594 · DOI · Paper page
- 10.1109/CVPR.2017.754 · DOI · Paper page
- 10.1109/MSP.2012.2205597 · DOI · Paper page
- hinton, 2014, Distilling the knowledge in a neural network, NIPS
- hou, 2017, Loss-aware binarization of deep networks, ICLRE
- howard, 2017, Mobilenets Efficient convolutional neural networks for mobile vision applications
- hu, 2016, A data-driven neuron pruning approach towards efficient deep architectures
- 10.1109/CVPR.2017.243 · DOI · Paper page
- hubara, 2016, Quantized neural networks Training neural networks with low precision weights and activations
- courbariaux, 2016, Binarized neural networks: Training neural networks with weights and activations constrained to +1 or ?1, NIPS
- polino, 2018, Model compression via distillation and quantization, ICLRE
- courbariaux, 2015, Binaryconnect: Training deep neural networks with binary weights during propagations, NIPS
- guo, 2016, Dynamic network surgery for efficient dnns, NIPS
- 10.1145/3007787.3001163 · DOI · Paper page
- collobert, 2011, Natural language processing (almost) from scratch, Journal of Machine Learning Research
- hassibi, 1993, Second order derivatives for network pruning: Optimal brain surgeon, NIPS
- 10.1109/CVPR.2017.195 · DOI · Paper page
- 10.1145/2647868.2654889 · DOI · Paper page
- zhu, 2017, Trained ternary quantization, ICLRE
- krizhevsky, 2009, Learning multiple layers of features from tiny images
- li, 2017, Training quantized nets: A deeper understanding, NIPS
- mishra, 2018, Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy, ICLRE
- zhou, 2017, Incremental network quantization: Towards lossless cnns with low-precision weights, ICLRE
Cited by
A Survey of Quantization Methods for Efficient Neural Network Inference
Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., & Keutzer, K. (2022) · Low-Power Computer Vision · doi:10.1201/9781003162810-13 · Cited by 1,219
“Some work in the literature has also explored non-uniform quantization [25, 38, 62, 74, 78, 98, 117, 123, 150, 156, 175, 184, 185, 230, 240, 248, 256, 258, 268, 275], where quantization steps as well as quantization levels are allowed to be non-uniformly spaced.”
Similarity-Preserving Knowledge Distillation
Tung, F., & Mori, G. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00145 · Cited by 1,079
HAQ: Hardware-Aware Automated Quantization With Mixed Precision
Wang, K., Liu, Z., Lin, Y., Lin, J., & Han, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00881 · Cited by 978
Self-citationMetaPruning: Meta Learning for Automatic Neural Network Channel Pruning
Liu, Z., Mu, H., Zhang, X., Guo, Z., Yang, X., Cheng, K.-T., & Sun, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00339 · Cited by 563
Fully Quantized Network for Object Detection
Li, R., Wang, Y., Liang, F., Qin, H., Yan, J., & Fan, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00292 · Cited by 214
“Most network quantization research has focused on object classification, including BNN [13], QNN [14], XNOR-Net [24], DoReFa-Net [35], INQ [33], ELQ [34], LQ-Nets [31] etc.”
Background
Metadata
- Authors
- Zhou, A., Yao, A., Wang, K., & Chen, Y.
- Venue
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2018-06
- Pages
- 9426-9435
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2799238091
- Cited by
- 84
- References
- 16
- FWCI
- 4.38
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.