CoChat
Sign Up
PreprintCited by 323Advanced Neural Network ApplicationsLow-power high-performance VLSI designDomain Adaptation and Few-Shot Learning

A White Paper on Neural Network Quantization

Open access
Nagel, Markus, Fournarakis, Marios, Amjad, Rana Ali, Bondarenko, Yelysei, van Baalen, Mart, Blankevoort, Tijmen·2021·arXiv
DOI 10.48550/arxiv.2106.08295arXiv 2106.08295
Open this paper in CoChat — free accountOpen PDF

Abstract

While neural networks have advanced the frontiers in many applications, they often come at a high computational cost. Reducing the power and latency of neural network inference is key if we want to integrate modern networks into edge devices with strict power and compute requirements. Neural network quantization is one of the most effective ways of achieving these savings but the additional noise it induces can lead to accuracy degradation. In this white paper, we introduce state-of-the-art algorithms for mitigating the impact of quantization noise on the network's performance while maintaining low-bit weights and activations. We start with a hardware motivated introduction to quantization and then consider two main classes of algorithms: Post-Training Quantization (PTQ) and Quantization-Aware-Training (QAT). PTQ requires no re-training or labelled data and is thus a lightweight push-button approach to quantization. In most cases, PTQ is sufficient for achieving 8-bit quantization with close to floating-point accuracy. QAT requires fine-tuning and access to labeled training data but enables lower bit quantization with competitive results. For both solutions, we provide tested pipelines based on existing literature and extensive experimentation that lead to state-of-the-art performance for common deep learning models and tasks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2106.08295

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Non-STE methods use regularization to force weights to quantized values, avoiding the non-differentiable quantization operator.

    “Another different class of alternative methods tries to use regularization operators to enforce the weight to be quantized.This removes the need to use the non-differentiable quantization operator in Eq. 2. These are often referred to as Non-STE methods [4, 8, 39, 99, 144, 184, 283].”

    From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [184]

    Section 3.4 (p. 15) describes AdaRound using a regularizer to make continuous rounding variables converge to valid discrete choices. Section 4.1 (p. 19) explains that rounding is non-differentiable and presents STE as a workaround. The copy does not describe a general class called “Non-STE methods” or state that regularization, as a class, removes the need for the quantization operator. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Weight-quantization into as few as 2-4 bits can often be tolerated without significant accuracy reduction.

    “Weight-quantization into as few as 2–4 bits can often be tolerated without significant accuracy reduction8,19,20.”

    From Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators (2023) · cites this paper as [20]

    The Experiments (p. 24) show that 4-bit weight quantization can preserve accuracy closely for several models, including BERT-base, particularly with 8-bit activations or per-channel quantization. The cited passages do not establish the same result for weights as low as 2 bits, and they report larger accuracy drops for some models at 4 bits. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Starting from a pre-trained FP32 model is common and recommended practice in QAT, improving accuracy and reducing required iterations.

    “It is common and recommended practice in the literature to start from a pre-trained FP32 model [12,15,17]. This approach not only improves accuracy but also reduces the number of required iterations.”

    From HADQ-Net: A Power-Efficient and Hardware-Adaptive Deep Convolutional Neural Network Translator Based on Quantization-Aware Training for Hardware Accelerators (2025) · cites this paper as [12]

    Section 4.3 (p. 22) says starting QAT from a pre-trained FP32 model is common practice and beneficial. Section 4.4 (p. 23) says better initialization of quantization parameters can speed training and might improve final accuracy, but it does not establish that starting from a pre-trained FP32 model, rather than another starting point, improves accuracy or reduces the required iterations. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 25

  1. Banner, R., Nahshan, Y., and Soudry, D. Post training 4-bit quantization of convolutional networks for rapid-deployment. Neural Information Processing Systems (NeuRIPS) , 2019. 9
  2. Bengio, Y., Léonard, N., and Courville, A. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 , 2013. 19 · arXiv
  3. Chin, T.-W., Chuang, P. I.-J., Chandra, V., and Marculescu, D. One weight bitwidth to rule them all. In Bartoli, A. and Fusiello, A. (eds.), Computer Vision - ECCV 2020 Workshops , pp. 85-103, Cham, 2020. Springer International Publishing. ISBN 978-3-030-68238-5. 24
  4. Esser, S. K., McKinstry, J. L., Bablani, D., Appuswamy, R., and Modha, D. S. Learned step size quantization. International Conference on Learning Representations (ICLR) , 2020. 20, 22
  5. Finkelstein, A., Almog, U., and Grobman, M. Fighting quantization bias with bias. arXiv preprint arxiv:1906.03193 , 2019. 13 · arXiv
  6. Gupta, S., Agrawal, A., Gopalakrishnan, K., and Narayanan, P. Deep learning with limited numerical precision. International Conference on Machine Learning, ICML , 2015. 14
  7. Horowitz, M. 1.1 computing's energy problem (and what we can do about it). In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC) , pp. 10-14, 2014. doi: 10.1109/ISSCC.2014.6757323. 2, 3 · DOI
  8. Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 , 2017. 10 · arXiv
  9. Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. Quantization and training of neural networks for efficient integer-arithmetic-only inference. Conference on Computer Vision and Pattern Recognition (CVPR) , 2018. 3, 6, 21, 22
  10. Jain, S. R., Gural, A., Wu, M., and Dick, C. Trained uniform quantization for accurate and efficient neural network inference on fixed-point hardware. CoRR , abs/1903.08066, 2019. URL http: //arxiv.org/abs/1903.08066 . 20, 22 · arXiv
  11. Krishnamoorthi, R. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.08342 , 2018. 6, 10, 12, 20, 21, 22 · arXiv
  12. Meller, E., Finkelstein, A., Almog, U., and Grobman, M. Same, same but different: Recovering neural network quantization error through weight factorization. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , pp. 4486-4495, 2019. URL http://proceedings.mlr.press/v97/meller19a.html . 11, 12, 13
  13. Moons, B., Noorzad, P., Skliar, A., Mariani, G., Mehta, D., Lott, C., and Blankevoort, T. Distilling optimal neural networks: Rapid search in diverse spaces. arXiv preprint arXiv:2012.08859 , 2020. 15 · arXiv
  14. Nagel, M., van Baalen, M., Blankevoort, T., and Welling, M. Data-free quantization through weight equalization and bias correction. International Conference on Computer Vision (ICCV) , 2019. 10, 11, 12, 13
  15. Nagel, M., Amjad, R. A., Van Baalen, M., Louizos, C., and Blankevoort, T. Up or down? Adaptive rounding for post-training quantization. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pp. 7197-7206. PMLR, 13-18 Jul 2020. URL http://proceedings.mlr.press/ v119/nagel20a.html . 13, 14, 15
  16. Nascimento, M. G. d., Fawcett, R., and Prisacariu, V. A. Dsconv: Efficient convolution operator. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2019. 5
  17. Ramachandran, P., Zoph, B., and Le, Q. V. Searching for activation functions. CoRR , abs/1710.05941, 2017. URL http://arxiv.org/abs/1710.05941 . 7 · arXiv
  18. Rouhani, B., Lo, D., Zhao, R., Liu, M., Fowers, J., Ovtcharov, K., Vinogradsky, A., Massengill, S., Yang, L., Bittner, R., Forin, A., Zhu, H., Na, T., Patel, P., Che, S., Koppaka, L. C., Song, X., Som, S., Das, K., Tiwary, S., Reinhardt, S., Lanka, S., Chung, E., and Burger, D. Pushing the limits of narrow precision inferencing at cloud scale with microsoft floating point. In Neural Information Processing Systems (NeurIPS 2020) . ACM, November 2020. 5
  19. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 4510-4520, 2018. 11
  20. Sheng, T., Feng, C., Zhuo, S., Zhang, X., Shen, L., and Aleksic, M. A quantization-friendly separable convolution for mobilenets. 2018 1st Workshop on Energy Efficient Machine Learning and Cognitive Computing for Embedded Applications (EMC2) , Mar 2018a. doi: 10.1109/emc2.2018. 00011. URL http://dx.doi.org/10.1109/EMC2.2018.00011 . 24 · DOI
  21. Sheng, T., Feng, C., Zhuo, S., Zhang, X., Shen, L., and Aleksic, M. A quantization-friendly separable convolution for mobilenets. In 1st Workshop on Energy Efficient Machine Learning and Cognitive Computing for Embedded Applications (EMC2) , 2018b. URL https://ieeexplore.ieee.org/ abstract/document/8524017 . 10
  22. Stock, P., Joulin, A., Gribonval, R., Graham, B., and Jégou, H. And the bit goes down: Revisiting the quantization of neural networks. CoRR , abs/1907.05686, 2019. URL http://arxiv.org/abs/ 1907.05686 . 5 · arXiv
  23. Uhlich, S., Mauch, L., Cardinaux, F., Yoshiyama, K., Garcia, J. A., Tiedemann, S., Kemp, T., and Nakamura, A. Mixed precision dnns: All you need is a good parametrization. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id= Hyx0slrFvH . 8
  24. van Baalen, M., Louizos, C., Nagel, M., Amjad, R. A., Wang, Y., Blankevoort, T., and Welling, M. Bayesian bits: Unifying quantization and pruning. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H. (eds.), Advances in Neural Information Processing Systems , volume 33, pp. 5741-5752. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/ paper/2020/file/3f13cf4ddf6fc50c0d39a1d5aeb57dd8-Paper.pdf . 8
  25. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/ W18-5446. URL https://www.aclweb.org/anthology/W18-5446 . 16 · DOI

Cited by

Sorted by citations

Authors
Nagel, M., Fournarakis, M., Amjad, R. A., Bondarenko, Y., van Baalen, M., & Blankevoort, T.
Affiliations
Qualcomm (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2106.08295
arXiv
2106.08295
OpenAlex
W3170437634
Cited by
323
References
4
Citations by year
2026: 8 · 2025: 74 · 2024: 117 · 2023: 89 · 2022: 29 · 2021: 6
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Engineering › Electrical and Electronic Engineering › Low-power high-performance VLSI design
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
neural network quantization, post-training quantization, quantization-aware training, quantization noise, edge inference, 8-bit quantization, accuracy degradation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review