CoChat
Sign Up
PreprintCited by 252Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningNumerical Methods and Algorithms

Up or Down? Adaptive Rounding for Post-Training Quantization

Open access
Nagel, Markus, Amjad, Rana Ali, van Baalen, Mart, Louizos, Christos, Blankevoort, Tijmen·2020·arXiv
DOI 10.48550/arxiv.2004.10568arXiv 2004.10568
Open this paper in CoChat — free accountOpen PDF

Abstract

When quantizing neural networks, assigning each floating-point weight to its nearest fixed-point value is the predominant approach. We find that, perhaps surprisingly, this is not the best we can do. In this paper, we propose AdaRound, a better weight-rounding mechanism for post-training quantization that adapts to the data and the task loss. AdaRound is fast, does not require fine-tuning of the network, and only uses a small amount of unlabelled data. We start by theoretically analyzing the rounding problem for a pre-trained neural network. By approximating the task loss with a Taylor series expansion, the rounding task is posed as a quadratic unconstrained binary optimization problem. We simplify this to a layer-wise local loss and propose to optimize this loss with a soft relaxation. AdaRound not only outperforms rounding-to-nearest by a significant margin but also establishes a new state-of-the-art for post-training quantization on several networks and tasks. Without fine-tuning, we can quantize the weights of Resnet18 and Resnet50 to 4 bits while staying within an accuracy loss of 1%.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2004.10568

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The Level 1 method in [182] can be sub-optimal for networks with non-linear activations because it relies on scale-equivariance of piece-wise linear activations.

    “However, as this method is based on the scaleequivariance property of (piece-wise) linear activation functions, it can be sub-optimal for NNs with non-linear activations, such as BERT [46] with GELU [94] activation or MobileNetV3 [100] with swish activation [203].”

    From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [182]

    Section 3.3 (p. 5) describes AdaRound as optimizing a reconstruction objective that explicitly accounts for the activation function, rather than relying on scale-equivariance. The ablation study (Section 5.1, p. 7) reports an accuracy gain from accounting for the activation. The paper does not test the claimed limitation on BERT or MobileNetV3. (LLM, conf=0.76)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Equalizing weight ranges across layers or channels can reduce quantization errors.

    “and [174,182] show that equalizing the weight ranges (and implicitly activation ranges) between different layers or channels can reduce quantization errors.”

    From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [182]

    The Literature comparison (p. 7) mentions cross-layer equalization (CLE) only as preprocessing and attributes it to Nagel et al. (2019). The Background and related work (p. 5) discusses other ways to improve quantization, but our copy does not say that equalizing weight or activation ranges across layers or channels reduces quantization errors. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 31

  1. Banner, R., Nahshan, Y., and Soudry, D. Post training 4-bit quantization of convolutional networks for rapiddeployment. Neural Information Processing Systems (NeuRIPS) , 2019.
  2. Bengio, Y., L´ eonard, N., and Courville, A. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 , 2013. · arXiv
  3. Botev, A., Ritter, H., and Barber, D. Practical gauss-newton optimisation for deep learning. International Conference on Machine Learning (ICML) , 2017.
  4. Cai, Y., Yao, Z., Dong, Z., Gholami, A., Mahoney, M. W., and Keutzer, K. Zeroq: A novel zero shot quantization framework. arXiv preprint arXiv:2001.00281 , 2020. · arXiv
  5. Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H. Encoder-decoder with atrous separable convolution for semantic image segmentation. The European Conference on Computer Vision (ECCV) , 2018.
  6. Choi, J., Wang, Z., Venkataramani, S., Chuang, P. I., Srinivasan, V., and Gopalakrishnan, K. PACT: parameterized clipping activation for quantized neural networks. arXiv preprint arxiv:805.06085 , 2018.
  7. Esser, S. K., McKinstry, J. L., Bablani, D., Appuswamy, R., and Modha, D. S. Learned step size quantization. International Conference on Learning Representations (ICLR) , 2020.
  8. Finkelstein, A., Almog, U., and Grobman, M. Fighting quantization bias with bias. arXiv preprint arxiv:1906.03193 , 2019. · arXiv
  9. Guo, Y. A survey on methods and theories of quantized neural networks. arXiv preprint: arxiv:1808.04752 , 2018. · arXiv
  10. Gupta, S., Agrawal, A., Gopalakrishnan, K., and Narayanan, P. Deep learning with limited numerical precision. International Conference on Machine Learning, ICML , 2015.
  11. Hammerstrom, D. A vlsi architecture for high-performance, low-cost, on-chip learning. International Joint Conference on Neural Networks (IJCNN) , 1990.
  12. Hoehfeld, M. and Fahlman, S. E. Learning with limited numerical precision using the cascade-correlation algorithm. IEEE Transactions on Neural Networks , 3(4):602-611, 1992.
  13. Holi, J. L. and Hwang, J. N. Finite precision error analysis of neural network hardware implementations. IEEE Trans. Comput. , 42(3):281290, 1993.
  14. Hopfield, J. J. and Tank, D. W. 'neural' computation of decisions in optimization problems. Biological Cybernetics , 52(3):141-152, 1985.
  15. Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. Quantization and training of neural networks for efficient integerarithmetic-only inference. Conference on Computer Vision and Pattern Recognition (CVPR) , 2018.
  16. Jain, S. R., Gural, A., Wu, M., and Dick, C. Trained uniform quantization for accurate and efficient neural network inference on fixed-point hardware. arxiv preprint arxiv:1903.08066 , 2019. · arXiv
  17. Kim, J., Bhalgat, Y., Lee, J., Patel, C., and Kwak, N. QKD: quantization-aware knowledge distillation. arxiv preprint arxiv:1911.12491 , 2019. · arXiv
  18. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. International Conference for Learning Representations (ICLR) , 2015.
  19. Kochenberger, G., Hao, J.-K., Glover, F., Lewis, M., L¨ u, Z., Wang, H., and Wang, Y. The unconstrained binary quadratic programming problem: a survey. Journal of Combinatorial Optimization , 28(1):58-81, Jul 2014.
  20. Krishnamoorthi, R. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.08342 , 2018. · arXiv
  21. Louizos, C., Reisser, M., Blankevoort, T., Gavves, E., and Welling, M. Relaxed quantization for discretized neural networks. In International Conference on Learning Representations (ICLR) , 2019.
  22. Mishra, A. K. and Marr, D. Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy. arXiv preprint arxiv:1711.05852 , 2017. · arXiv
  23. Nagel, M., van Baalen, M., Blankevoort, T., and Welling, M. Data-free quantization through weight equalization and bias correction. International Conference on Computer Vision (ICCV) , 2019.
  24. L., Bai, J., and Chintala, S. Pytorch: An imperative style, high-performance deep learning library. In Neural Information Processing Systems (NeuRIPS) . 2019.
  25. Rubinstein, R. The cross-entropy method for combinatorial and continuous optimization. Methodology And Computing In Applied Probability , 1(2):127-190, Sep 1999.
  26. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) , 115(3):211-252, 2015.
  27. Stock, P., Joulin, A., Gribonval, R., Graham, B., and Jgou, H. And the bit goes down: Revisiting the quantization of neural networks. In International Conference on Learning Representations , 2020.
  28. Uhlich, S., Mauch, L., Yoshiyama, K., Cardinaux, F., Garc´ ıa, J. A., Tiedemann, S., Kemp, T., and Nakamura, A. Mixed precision dnns: All you need is a good parametrization. International Conference on Learning Representations (ICLR) , 2020.
  29. Wang, P., Hu, Q., Zhang, Y., Zhang, C., Liu, Y., and Cheng, J. Two-step quantization for low-bit neural networks. Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 4376-4384, 2018.
  30. Zhang, X., Zou, J., He, K., and Sun, J. Accelerating very deep convolutional networks for classification and detection. IEEE Trans. Pattern Anal. Mach. Intell. , 38(10): 1943-1955, 2016.
  31. Zhao, R., Hu, Y., Dotzel, J., Sa, C. D., and Zhang, Z. Improving neural network quantization without retraining using outlier channel splitting. International Conference on Machine Learning, ICML , 2019.

Cited by

Sorted by citations

Authors
Nagel, M., Amjad, R. A., van Baalen, M., Louizos, C., & Blankevoort, T.
Affiliations
Qualcomm (United States) (US)
ORCID
Amjad, Rana Ali
Venue
arXiv
Publisher
arXiv
Published
2020
Volume
1
Pages
7197-7206
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2004.10568
arXiv
2004.10568
OpenAlex
W3020212829
Cited by
252
References
7
Citations by year
2026: 1 · 2025: 51 · 2024: 66 · 2023: 72 · 2022: 38 · 2021: 20 · 2020: 4
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
  • International Conference on Machine Learning
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Numerical Methods and Algorithms
Keywords
post-training quantization, neural network quantization, QUBO, Taylor series expansion, 4-bit quantization, ResNet

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review