CoChat
Sign Up
PreprintCited by 76Advanced Neural Network ApplicationsStochastic Gradient Optimization TechniquesMachine Learning and Data Classification

ProxQuant: Quantized Neural Networks via Proximal Operators

Open access
Bai, Yu, Wang, Yu-Xiang, Liberty, Edo·2018·arXiv
DOI 10.48550/arxiv.1810.00861arXiv 1810.00861
Open this paper in CoChat — free accountOpen PDF

Abstract

To make deep neural networks feasible in resource-constrained environments (such as mobile devices), it is beneficial to quantize models by using low-precision weights. One common technique for quantizing neural networks is the straight-through gradient method, which enables back-propagation through the quantization mapping. Despite its empirical success, little is understood about why the straight-through gradient method works. Building upon a novel observation that the straight-through gradient method is in fact identical to the well-known Nesterov's dual-averaging algorithm on a quantization constrained optimization problem, we propose a more principled alternative approach, called ProxQuant, that formulates quantized network training as a regularized learning problem instead and optimizes it via the prox-gradient method. ProxQuant does back-propagation on the underlying full-precision vector and applies an efficient prox-operator in between stochastic gradient steps to encourage quantizedness. For quantizing ResNets and LSTMs, ProxQuant outperforms state-of-the-art results on binary quantization and is on par with state-of-the-art on multi-bit quantization. For binary quantization, our analysis shows both theoretically and experimentally that ProxQuant is more stable than the straight-through gradient method (i.e. BinaryConnect), challenging the indispensability of the straight-through gradient method and providing a powerful alternative.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1810.00861

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • STE often performs well despite its coarse approximation, except in ultra low-precision cases such as binary quantization.

    “Despite the coarse approximation of STE, it often works well in practice, except for ultra low-precision quantization such as binary quantization [8].”

    From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [8]

    Section 2.1 (p. 4) says the straight-through method gives good empirical performance in quantized networks, supporting the claim that it often works well. The paper also finds limitations for BinaryConnect in binary quantization (Section 4.1, p. 8; Appendix C, p. 15), but it does not treat binary quantization as a general exception to STE’s practical success; the Introduction (p. 2) calls it empirically effective even in the binary case. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Non-STE methods use regularization to force weights to quantized values, avoiding the non-differentiable quantization operator.

    “Another different class of alternative methods tries to use regularization operators to enforce the weight to be quantized.This removes the need to use the non-differentiable quantization operator in Eq. 2. These are often referred to as Non-STE methods [4, 8, 39, 99, 144, 184, 283].”

    From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [8]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 24

  1. [1][1] A. G. Anderson and C. P. Berg. The high-dimensional geometry of binary neural networks. arXiv preprint arXiv:1705.07199 , 2017. - · arXiv
  2. [2][2] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein gan. arXiv preprint arXiv:1701.07875 , 2017. - · arXiv
  3. [3][3] M. A. Carreira-Perpin´ an. Model compression as constrained optimization, with application to neural nets. part i: General framework. arXiv preprint arXiv:1707.01209 , 2017. - · arXiv
  4. [4][4] M. A. Carreira-Perpin´ an and Y. Idelbayev. Model compression as constrained optimization, with application to neural nets. part ii: Quantization. arXiv preprint arXiv:1707.04319 , 2017. - · arXiv
  5. [5][5] M. Courbariaux, Y. Bengio, and J.-P. David. BinaryConnect: Training deep neural networks with binary weights during propagations. In Advances in neural information processing systems , pages 3123-3131, 2015. -
  6. [6][6] Y. Ding, J. Liu, and Y. Shi. On the universal approximability of quantized relu neural networks. arXiv preprint arXiv:1802.03646 , 2018. - · arXiv
  7. [7][7] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio. Deep learning , volume 1. MIT press Cambridge, 2016. -
  8. [8][8] S. Han, H. Mao, and W. J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 , 2015. - · arXiv
  9. [9][9] S. Han, X. Liu, H. Mao, J. Pu, A. Pedram, M. A. Horowitz, and W. J. Dally. EIE: Efficient inference engine on compressed deep neural network. In Computer Architecture (ISCA), 2016 ACM/IEEE 43rd Annual International Symposium on , pages 243-254. IEEE, 2016. -
  10. [11][11] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation , 9(8):17351780, 1997. -
  11. [12][12] L. Hou and J. T. Kwok. Loss-aware weight quantization of deep networks. In International Conference on Learning Representations , 2018. URL https://openreview.net/forum?id= BkrSv0lA-. -
  12. [13][13] L. Hou, Q. Yao, and J. T. Kwok. Loss-aware binarization of deep networks. In International Conference on Learning Representations , 2017. URL https://openreview.net/forum?id= S1oWlN9ll . -
  13. [14][14] I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. Journal of Machine Learning Research , 18:187-1, 2017. -
  14. [15][15] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. - · arXiv
  15. [17][17] H. Li, S. De, Z. Xu, C. Studer, H. Samet, and T. Goldstein. Training quantized nets: A deeper understanding. In Advances in Neural Information Processing Systems , pages 5811-5821, 2017. -
  16. [18][18] M. P. Marcus, M. A. Marcinkiewicz, and B. Santorini. Building a large annotated corpus of english: The penn treebank. Computational linguistics , 19(2):313-330, 1993. -
  17. [20][20] M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision , pages 525-542. Springer, 2016. -
  18. [22][22] R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological) , pages 267-288, 1996. -
  19. [23][23] L. Xiao. Dual averaging methods for regularized stochastic learning and online optimization. Journal of Machine Learning Research , 11(Oct):2543-2596, 2010. -
  20. [24][24] C. Xu, J. Yao, Z. Lin, W. Ou, Y. Cao, Z. Wang, and H. Zha. Alternating multi-bit quantization for recurrent neural networks. In International Conference on Learning Representations , 2018. URL https://openreview.net/forum?id=S19dR9x0b . -
  21. [25][25] P. Yin, S. Zhang, Y. Qi, and J. Xin. Quantization and training of low bit-width convolutional neural networks for object detection. arXiv preprint arXiv:1612.06052 , 2016. - · arXiv
  22. [26][26] P. Yin, S. Zhang, J. Lyu, S. Osher, Y. Qi, and J. Xin. Binaryrelax: A relaxation approach for training deep neural networks with quantized weights. arXiv preprint arXiv:1801.06313 , 2018. - · arXiv
  23. [27][27] S. Zhou, Y. Wu, Z. Ni, X. Zhou, H. Wen, and Y. Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160 , 2016. - · arXiv
  24. [28][28] C. Zhu, S. Han, H. Mao, and W. J. Dally. Trained ternary quantization. arXiv preprint arXiv:1612.01064 , 2016. · arXiv

Cited by

Sorted by citations

Authors
Bai, Y., Wang, Y.-X., & Liberty, E.
Affiliations
  • Stanford University (US)
  • University of California, Santa Barbara (US)
  • Amazon (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1810.00861
arXiv
1810.00861
OpenAlex
W2892684141
Cited by
76
References
3
Citations by year
2025: 4 · 2024: 6 · 2023: 9 · 2022: 9 · 2021: 23 · 2020: 14 · 2019: 10 · 2018: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
neural network quantization, proximal operator, binary quantization, multi-bit quantization, resource-constrained devices, ResNet, LSTM

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review