CoChat
Sign Up
PreprintCited by 110Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningLow-power high-performance VLSI design

HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks

Open access
Dong, Zhen, Yao, Zhewei, Cai, Yaohui, Arfeen, Daiyaan, Gholami, Amir, Mahoney, Michael W., Keutzer, Kurt·2019·arXiv
DOI 10.48550/arxiv.1911.03852arXiv 1911.03852
Open this paper in CoChat — free accountOpen PDF

Abstract

Quantization is an effective method for reducing memory footprint and inference time of Neural Networks, e.g., for efficient inference in the cloud, especially at the edge. However, ultra low precision quantization could lead to significant degradation in model generalization. A promising method to address this is to perform mixed-precision quantization, where more sensitive layers are kept at higher precision. However, the search space for a mixed-precision quantization is exponential in the number of layers. Recent work has proposed HAWQ, a novel Hessian based framework, with the aim of reducing this exponential search space by using second-order information. While promising, this prior work has three major limitations: (i) HAWQV1 only uses the top Hessian eigenvalue as a measure of sensitivity and do not consider the rest of the Hessian spectrum; (ii) HAWQV1 approach only provides relative sensitivity of different layers and therefore requires a manual selection of the mixed-precision setting; and (iii) HAWQV1 does not consider mixed-precision activation quantization. Here, we present HAWQV2 which addresses these shortcomings. For (i), we perform a theoretical analysis showing that a better sensitivity metric is to compute the average of all of the Hessian eigenvalues. For (ii), we develop a Pareto frontier based method for selecting the exact bit precision of different layers without any manual selection. For (iii), we extend the Hessian analysis to mixed-precision activation quantization. We have found this to be very beneficial for object detection. We show that HAWQV2 achieves new state-of-the-art results for a wide range of tasks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1911.03852

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 25

  1. [2][2] Haim Avron and Sivan Toledo. Randomized algorithms for estimating the trace of an implicit symmetric positive semi-definite matrix. Journal of the ACM (JACM) , 58(2):8, 2011. -
  2. [3][3] Zhaojun Bai, Gark Fahey, and Gene Golub. Some large-scale matrix computation problems. Journal of Computational and Applied Mathematics , 74(1-2):71-89, 1996. -
  3. [4][4] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 , 2013. - · arXiv
  4. [5][5] Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. PACT: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv:1805.06085 , 2018. - · arXiv
  5. [6][6] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. BinaryConnect: Training deep neural networks with binary weights during propagations. In Advances in neural information processing systems , pages 3123-3131, 2015. -
  6. [8][8] Amir Gholami, Kiseok Kwon, Bichen Wu, Zizheng Tai, Xiangyu Yue, Peter Jin, Sicheng Zhao, and Kurt Keutzer. SqueezeNext: Hardware-aware neural network design. Workshop paper in CVPR , 2018. -
  7. [9][9] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. International Conference on Learning Representations , 2016. -
  8. [11][11] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integerarithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2704-2713, 2018. -
  9. [12][12] Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.08342 , 2018. - · arXiv
  10. [13][13] Rundong Li, Yan Wang, Feng Liang, Hongwei Qin, Junjie Yan, and Rui Fan. Fully quantized network for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2810-2819, 2019. -
  11. [14][14] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object - detection. In Proceedings of the IEEE international conference on computer vision , pages 2980-2988, 2017. -
  12. [15][15] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision , pages 740-755. Springer, 2014. -
  13. [18][18] Eunhyeok Park, Sungjoo Yoo, and Peter Vajda. Value-aware quantization for training and inference of neural networks. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 580-595, 2018. -
  14. [19][19] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision , pages 525-542. Springer, 2016. -
  15. [20][20] Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Q-BERT: Hessian based ultra low precision quantization of bert. arXiv preprint arXiv:1909.05840 , 2019. - · arXiv
  16. [21][21] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2818-2826, 2016. -
  17. [22][22] Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. HAQ: Hardware-aware automated quantization. In Proceedings of the IEEE conference on computer vision and pattern recognition , 2019. -
  18. [23][23] Bichen Wu, Yanghan Wang, Peizhao Zhang, Yuandong Tian, Peter Vajda, and Kurt Keutzer. Mixed precision quantization of convnets via differentiable neural architecture search. arXiv preprint arXiv:1812.00090 , 2018. - · arXiv
  19. [24][24] Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng. Quantized convolutional neural networks for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4820-4828, 2016. -
  20. [25][25] Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W. Mahoney. Large batch size training of neural networks with adversarial training and second-order information. arXiv preprint arXiv:1810.01021 , 2018. - · arXiv
  21. [26][26] Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, and Michael W. Mahoney. Hessian-based analysis of large batch training and robustness to adversaries. Advances in Neural Information Processing Systems , 2018. -
  22. [27][27] Zhewei Yao, Amir Gholami, Peng Xu, Kurt Keutzer, and Michael W. Mahoney. Trust region based adversarial attack on neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 11350-11359, 2019. -
  23. [28][28] Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. LQ-Nets: Learned quantization for highly accurate and compact deep neural networks. In The European Conference on Computer Vision (ECCV) , September 2018. -
  24. [29][29] Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantization: Towards lossless CNNs with low-precision weights. International Conference on Learning Representations , 2017. -
  25. [30][30] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. DoReFa-Net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160 , 2016. Appendix A. Quantization Details. During the forward pass, each element in a weight or activation tensor X will be quantized as follows: <!-- formula-not-decoded --> where ⌊·⌉ is the round operator, ∆ is the distance between adjacent quantized points, X I is a set of integer indices, [ q 0 , q 2 k -1 ] stands for the quantization range of the floating point tensor, and the function Clamp sets all elements smaller than q 0 equal to q 0 , and all elements larger than q 2 k -1 to q 2 k -1 . It should be noted that [ q 0 , q 2 k -1 ] can be a subinterval of [ min,max ] , in order to get rid of outliers and better represent the majority of the given tensor. During inference, the expensive floating point arithmetic can be replaced by efficient integer arithmetic for the matrix multiplication with X I , and then followed by a gathered dequantization operation, which will significantly accelerate the computation process. Since we use the quantization-aware fine-tuning scheme, in the backward pass, the Straight-Through Estimator (STE) [4] is used for computing the gradient for X . Appendix B. Search Space. Suppose the number of different mixed-precision settings is B , and the number of different progressive quantization-aware fine-tuning orders is C . The whole search space can be written as B × C . We have: <!-- formula-not-decoded --> <!-- formula-not-decoded --> where m is the number of quantization precision options, L is the number of layers in a given model, S ( L, i ) stands for Stiring numbers of the second kind, which have a growth speed between O ( L !) and O ( L L ) . In the case of layer-wise fine-tuning, where only one layer can be fine-tuned at a time, C degrades to L ! . Given two layers B i and B j with average Hessian trace Tr ( B i ) /n i &gt; Tr ( B j ) /n j , if we set quantization precision q i ≥ q j , then based on that, we are able to order all L layers in the model according to their average Hessian trace. Considering the situation that j precision options are used out of total number m , the mixed-precision problem can be reduced to an integer partition problem, namely, to partition the ordered layers into j different groups, which results in ( j -1 L -1 ) possible solutions. Since there are ( j m ) different combinations of the j precision options, the total size of search space is ∑ m j =1 ( ( j m ) · ( j -1 L -1 ) ) . · arXiv

Cited by

Sorted by citations

Authors
Dong, Z., Yao, Z., Cai, Y., Arfeen, D., Gholami, A., Mahoney, M. W., & Keutzer, K.
Affiliations
  • University of California, Berkeley (US)
  • Peking University (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
Volume
33
Pages
18518-18529
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1911.03852
arXiv
1911.03852
OpenAlex
W2989530497
Cited by
110
References
2
Citations by year
2025: 5 · 2024: 26 · 2023: 28 · 2022: 10 · 2021: 29 · 2020: 12
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • eScholarship (California Digital Library) (submittedVersion, cc-by)
  • arXiv (Cornell University)
  • neural information processing systems
Funders
  • National Science Foundation
  • Intel Corporation
  • Nvidia
  • Defense Advanced Research Projects Agency
  • Office of Naval Research
  • Army Research Office
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Engineering › Electrical and Electronic Engineering › Low-power high-performance VLSI design
Keywords
neural network quantization, mixed-precision quantization, Hessian eigenvalues, Pareto front, activation quantization, edge inference, object detection, model compression

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review