CoChat
Sign Up
Cited by 500Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningMachine Learning and Data Classification

Data-Free Quantization Through Weight Equalization and Bias Correction

Open access
Nagel, Markus, Baalen, Mart Van, Blankevoort, Tijmen, Welling, Max·2019·2019 IEEE/CVF International Conference on Computer Vision (ICCV)
DOI 10.1109/iccv.2019.00141
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce a data-free quantization method for deep neural networks that does not require fine-tuning or hyperparameter selection. It achieves near-original model performance on common computer vision architectures and tasks. 8-bit fixed-point quantization is essential for efficient inference on modern deep learning hardware. However, quantizing models to run in 8-bit is a non-trivial task, frequently leading to either significant performance reduction or engineering time spent on training a network to be amenable to quantization. Our approach relies on equalizing the weight ranges in the network by making use of a scale-equivariance property of activation functions. In addition the method corrects biases in the error that are introduced during quantization. This improves quantization accuracy performance, and can be applied to many common computer vision architectures with a straight forward API call. For common architectures, such as the MobileNet family, we achieve state-of-the-art quantized model performance. We further show that the method also extends to other computer vision architectures and tasks such as semantic segmentation and object detection.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/iccv.2019.00141

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Bias correction can improve the performance of PTQ4ViT at 4-bit quantization.

    “Although bias correction [17] can improve the performance of PTQ4ViT, the result at 4-bit quantization is lower than the mixed-precision of Liu et al.”

    From PTQ4ViT: Post-Training Quantization for Vision Transformers with Twin Uniform Quantization (2022) · cites this paper as [17]

    Section 5.1.2 reports that bias correction improves quantized MobileNetV2 performance, supporting the general benefit attributed to the method. The paper does not establish that it improves PTQ4ViT specifically at 4-bit quantization; its reported experiments concern different models and quantization settings. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Most existing post-training quantization methods are designed for convolutional neural networks or recurrent neural networks.

    “Most of the existing post-training quantization methods are designed for convolutional neural networks [3, 21, 30] or recurrent neural networks [35].”

    From Post-Training Quantization for Vision Transformer (2021) · cites this paper as [21]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 29

  1. Jan Achterhold, Jan Mathias Koehler, Anke Schmeink, and Tim Genewein. Variational network quantization. In Inter- national Conference on Learning Representations (ICLR), 2018.
  2. Raziel Alvarez, Rohit Prabhavalkar, and Anton Bakhtin. On the efficient representation and execution of deep acoustic models. In The Annual Conference of the International Speech Communication Association (Interspeech), 2016.
  3. Liang-Chieh Chen, Yukun Zhu, George Papandreou, Flo- rian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmen- tation. In The European Conference on Computer Vision (ECCV), September 2018.
  4. Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. PACT: parameterized clipping activa- tion for quantized neural networks. arXiv preprint arxiv:805.06085, 2018.
  5. Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Proceedings of the 28th International Conference on Neural Information Pro- cessing Systems -Volume 2, NIPS'15, pages 3123-3131, Cambridge, MA, USA, 2015. MIT Press.
  6. Alexander Finkelstein, Uri Almog, and Mark Grobman. Fighting quantization bias with bias. arXiv preprint arxiv:1906.03193, 2019. · arXiv
  7. Denis A. Gudovskiy and Luca Rigazio. Shiftcnn: Gener- alized low-precision architecture for inference of convolu- tional neural networks. arXiv preprint arxiv:1706.02393, 2017. · arXiv
  8. Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In Proceedings of the 32nd International Confer- ence on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, pages 1737-1746, 2015.
  9. Philipp Gysel, Jon J. Pimentel, Mohammad Motamedi, and Soheil Ghiasi. Ristretto: A framework for empirical study of resource-efficient inference in convolutional neu- ral networks. IEEE Trans. Neural Netw. Learning Syst., 29(11):5784-5789, 2018.
  10. Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco An- dreetto, and Hartwig Adam. Mobilenets: Efficient convolu- tional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017. · arXiv
  11. Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and ac- tivations. Journal of Machine Learning Research, 18:187:1- 187:30, 2017.
  12. Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  13. Urs Köster, Tristan Webb, Xin Wang, Marcel Nassar, Ar- jun K. Bansal, William Constable, Oguz Elibol, Stewart Hall, Luke Hornof, Amir Khosrowshahi, Carey Kloss, Ruby J. Pai, and Naveen Rao. Flexpoint: An adaptive numerical format for efficient training of deep neural networks. In Advances in Neural Information Processing Systems 30: Annual Confer- ence on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 1740-1750, 2017.
  14. Raghuraman Krishnamoorthi. Quantizing deep convolu- tional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.08342, Jun 2018. · arXiv
  15. Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E. Reed, Cheng-Yang Fu, and Alexander C. Berg. SSD: single shot multibox detector. In Computer Vi- sion -ECCV 2016 -14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I, pages 21-37, 2016.
  16. Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Ef- stratios Gavves, and Max Welling. Relaxed quantization for discretized neural networks. In International Conference on Learning Representations (ICLR), 2019.
  17. Eldad Meller, Alexander Finkelstein, Uri Almog, and Mark Grobman. Same, same but different: Recovering neural net- work quantization error through weight factorization. In Pro- ceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, Cali- fornia, USA, pages 4486-4495, 2019.
  18. Asit K. Mishra, Jeffrey J. Cook, Eriko Nurvitadhi, and Deb- bie Marr. WRPN: training and inference using wide reduced- precision networks. arXiv preprint arxiv 1704.03079, 2017. · arXiv
  19. Daisuke Miyashita, Edward H. Lee, and Boris Murmann. Convolutional neural networks using logarithmic data rep- resentation. arXiv preprint arxiv:1603.01025, 2016. · arXiv
  20. Vinod Nair and Geoffrey E. Hinton. Rectified linear units im- prove restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML- 10), June 21-24, 2010, Haifa, Israel, pages 807-814, 2010.
  21. Jorn W. T. Peters and Max Welling. Probabilistic binary neu- ral networks. arXiv preprint arxiv:1809.03368, 2018. · arXiv
  22. Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using bi- nary convolutional neural networks. In Computer Vision - ECCV 2016 -14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, pages 525-542, 2016.
  23. Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zh- moginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  24. Tao Sheng, Chen Feng, Shaojie Zhuo, Xiaopeng Zhang, Liang Shen, and Mickey Aleksic. A quantization-friendly separable convolution for mobilenets. In 1st Workshop on Energy Efficient Machine Learning and Cognitive Comput- ing for Embedded Applications (EMC2), 2018.
  25. Pierre Stock, Benjamin Graham, Rmi Gribonval, and Herv Jgou. Equi-normalization of neural networks. In Interna- tional Conference on Learning Representations, 2019.
  26. Karen Ullrich, Edward Meeds, and Max Welling. Soft weight-sharing for neural network compression. In Inter- national Conference on Learning Representations (ICLR), 2017.
  27. Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating very deep convolutional networks for classifi- cation and detection. IEEE Trans. Pattern Anal. Mach. In- tell., 38(10):1943-1955, 2016.
  28. Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantization: Towards lossless cnns with low-precision weights. arXiv preprint arxiv:1702.03044, abs/1702.03044, 2017. · arXiv
  29. Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. Dorefa-net: Training low bitwidth convo- lutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160, 2016. · arXiv

Cited by

Sorted by citations

Authors
Nagel, M., Baalen, M. V., Blankevoort, T., & Welling, M.
Affiliations
Qualcomm (United States) (US)
ORCID
Welling, Max
Venue
2019 IEEE/CVF International Conference on Computer Vision (ICCV)
Publisher
IEEE
Published
2019-10
Pages
1325-1334
DOI
10.1109/iccv.2019.00141
OpenAlex
W2950297794
Cited by
500
References
6
Citations by year
2026: 40 · 2025: 84 · 2024: 74 · 2023: 93 · 2022: 70 · 2021: 80 · 2020: 52 · 2019: 7
FWCI
17.03
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • TIB Data Manager
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
data-free quantization, bias correction, scale equivariance, activation functions, quantization error, MobileNet, computer vision architectures, semantic segmentation, object detection, inference efficiency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review