CoChat
Sign Up
PreprintCited by 231Advanced Neural Network ApplicationsParallel Computing and Optimization TechniquesLow-power high-performance VLSI design

Improving Neural Network Quantization without Retraining using Outlier Channel Splitting

Open access
Zhao, Ritchie, Hu, Yuwei, Dotzel, Jordan, De Sa, Christopher, Zhang, Zhiru·2019·arXiv
DOI 10.48550/arxiv.1901.09504arXiv 1901.09504
Open this paper in CoChat — free accountOpen PDF

Abstract

Quantization can improve the execution latency and energy efficiency of neural networks on both commodity GPUs and specialized accelerators. The majority of existing literature focuses on training quantized DNNs, while this work examines the less-studied topic of quantizing a floating-point model without (re)training. DNN weights and activations follow a bell-shaped distribution post-training, while practical hardware uses a linear quantization grid. This leads to challenges in dealing with outliers in the distribution. Prior work has addressed this by clipping the outliers or using specialized hardware. In this work, we propose outlier channel splitting (OCS), which duplicates channels containing outliers, then halves the channel values. The network remains functionally identical, but affected outliers are moved toward the center of the distribution. OCS requires no additional training and works on commodity hardware. Experimental evaluation on ImageNet classification and language modeling shows that OCS can outperform state-of-the-art clipping techniques with only minor overhead.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1901.09504

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Most existing post-training quantization methods are designed for convolutional neural networks or recurrent neural networks.

    “Most of the existing post-training quantization methods are designed for convolutional neural networks [3, 21, 30] or recurrent neural networks [35].”

    From Post-Training Quantization for Vision Transformer (2021) · cites this paper as [35]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 29

  1. Banner, R., Nahshan, Y., Hoffer, E., and Soudry, D. ACIQ: Analytical Clipping for Integer Quantization of Neural Networks. arXiv e-print , arXiv:1810.05723, Oct 2018. · arXiv
  2. Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, C., and Zhang, Z. MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems. arXiv preprint , arXiv:1512.01274, Dec 2015. · arXiv
  3. Chen, T., Goodfellow, I., and Shlens, J. Net2net: Accelerating Learning via Knowledge Transfer. Int'l Conf. on Learning Representations (ICLR) , May 2016.
  4. Choi, J., Chuang, P. I.-J., Wang, Z., Venkataramani, S., Srinivasan, V., and Gopalakrishnan, K. Bridging the Accuracy Gap for 2-bit Quantized Neural Networks (QNN). arXiv e-print , arXiv:1807.06964, Jul 2018a. · arXiv
  5. Choi, J., Wang, Z., Venkataramani, S., Chuang, P. I.-J., Srinivasan, V., and Gopalakrishnan, K. PACT: Parameterized Clipping Activation for Quantized Neural Networks. arXiv e-print , arXiv:1805.0608, May 2018b.
  6. Chung, E., Fowers, J., Ovtcharov, K., Papamichael, M., Caulfield, A., Massengill, T., Liu, M., Lo, D., Alkalay, S., Haselman, M., Abeydeera, M., Adams, L., Angepat, H., Boehn, C., Chiou, D., Firestein, O., Forin, A., Gatlin, K. S., Ghandi, M., Heil, S., Holohan, K., Husseini, A. E., Juhasz, T., Kagi, K., Kovvuri, R. K., Lanka, S., van Megen, F., Mukhortov, D., Patel, P., Perez, B., Rapsang, A. G., Reinhardt, S. K., Rouhani, B. D., Sapek, A., Seera, R., Shekar, S., Sridharan, B., Weisz, G., Woods, L., Xiao, P. Y., Zhang, D., Zhao, R., , and Burger, D. Serving DNNs in Real Time at Datacenter Scale with Project Brainwave . IEEE Micro , 38(2):8-20, 2018.
  7. Courbariaux, M., Bengio, Y., and David, J.-P. BinaryConnect: Training Deep Neural Networks with binary weights during propagations. Advances in Neural Information Processing Systems (NIPS) , pp. 3123-3131, 2015.
  8. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and FeiFei, L. ImageNet: A Large-Scale Hierarchical Image Database. Conf. on Computer Vision and Pattern Recognition (CVPR) , pp. 248-255, 2009.
  9. Han, S., Mao, H., and Dally, W. J. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. Int'l Conf. on Learning Representations (ICLR) , Feb 2016.
  10. Huang, G., Liu, Z., Weinberger, K. Q., and van der Maaten, L. Densely connected convolutional networks. Conf. on Computer Vision and Pattern Recognition (CVPR) , 1(2): 3, 2017.
  11. Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., and Bengio, Y. Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations. Journal of Machine Learning Research (JMLR) , 18(187): 1-30, 2017.
  12. Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. Quantization and Training of Neural Networks for Efficient IntegerArithmetic-Only Inference. Conf. on Computer Vision and Pattern Recognition (CVPR) , pp. 2704-2713, Jun 2018.
  13. Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. In-Datacenter Performance Analysis of a Tensor Processing Unit. Int'l Symp. on Computer Architecture (ISCA) , pp. 1-12, 2017.
  14. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll´ ar, P., and Zitnick, C. L. Microsoft COCO: Common Objects in Context. European Conference on Computer Vision (ECCV) , pp. 740-755, 2014.
  15. Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer Sentinel Mixture Models. arXiv preprint , arXiv:1609.07843, Sep 2016. · arXiv
  16. Migacz, S. 8-bit Inference with TensorRT. NVIDIA GPU Technology Conference , May 2017.
  17. Park, E., Kim, D., and Yoo, S. Energy-Efficient Neural Network Accelerator Based on Outlier-Aware Low-Precision Computation. Int'l Symp. on Computer Architecture (ISCA) , Jun 2018a.
  18. Park, E., Yoo, S., and Vajda, P. Value-aware Quantization for Training and Inference of Neural Networks. arXiv e-print , arXiv:1804.07802, Apr 2018b. · arXiv
  19. Park, H. and Choi, K. Cell Division: Weight Bit-Width Reduction Technique for Convolutional Neural Network Hardware Accelerators. Asia and South Pacific Design Automation Conf. (ASP-DAC) , pp. 286-291, Jan 2019.
  20. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. Automatic Differentiation in PyTorch. Advances in Neural Information Processing Systems Workshops (NIPS-W) , 2017.
  21. Savchev, S. and Andreescu, T. Mathematical Miniatures , chapter 12. Hermite's Identity, pp. 41-44. Mathematical Association of America, 2003.
  22. Settle, S. O., Bollavaram, M., D'Alberto, P., Delaye, E., Fernandez, O., Fraser, N., Ng, A., Sirasao, A., and Wu, M. Quantizing Convolutional Neural Networks for Low-Power High-Throughput Inference Engines. arXiv preprint , arXiv:1805.07941, May 2018. · arXiv
  23. Shin, S., Hwang, K., and Sung, W. Fixed-Point Performance Analysis of Recurrent Neural Networks. Int'l Conf. on Acoustics, Speech and Signal Processing (ICASSP) , pp. 976-980, 2016.
  24. Simonyan, K. and Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv eprint , arXiv:1409.15568, Apr 2015.
  25. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going Deeper with Convolutions. Conf. on Computer Vision and Pattern Recognition (CVPR) , 2015.
  26. Wu, S., Li, G., Chen, F., and Shi, L. Training and Inference with Integers in Deep Neural Networks. Int'l Conf. on Learning Representations (ICLR) , May 2018.
  27. Zaremba, W., Sutskever, I., and Vinyals, O. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329 , 2014. · arXiv
  28. Zhou, A., Yao, A., Guo, Y., Xu, L., and Chen, Y. Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights. arXiv preprint , arXiv:1702.03044, 2017. · arXiv
  29. Zhuang, B., Shen, C., Tan, M., Liu, L., and Reid, I. Towards Effective Low-Bitwidth Convolutional Neural Networks. Conf. on Computer Vision and Pattern Recognition (CVPR) , pp. 7920-7928, Jun 2018.

Cited by

Sorted by citations

Authors
Zhao, R., Hu, Y., Dotzel, J., De Sa, C., & Zhang, Z.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
Pages
7543-7552
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1901.09504
arXiv
1901.09504
OpenAlex
W2915478146
Cited by
231
References
4
Citations by year
2025: 14 · 2024: 20 · 2023: 38 · 2022: 41 · 2021: 64 · 2020: 42 · 2019: 12
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • International Conference on Machine Learning
Funders
  • Semiconductor Research Corporation
  • Nvidia
  • Defense Advanced Research Projects Agency
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
  • Physical Sciences › Engineering › Electrical and Electronic Engineering › Low-power high-performance VLSI design
Keywords
neural network quantization, post-training quantization, outlier handling, activation distribution, floating-point to integer conversion, energy efficiency, execution latency, ImageNet classification, language modeling

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review