CoChat
Sign Up
PreprintCited by 60Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningNeural Networks and Applications

Same, Same But Different - Recovering Neural Network Quantization Error Through Weight Factorization

Open access
Meller, Eldad, Finkelstein, Alexander, Almog, Uri, Grobman, Mark·2019·arXiv
DOI 10.48550/arxiv.1902.01917arXiv 1902.01917
Open this paper in CoChat — free accountOpen PDF

Abstract

Quantization of neural networks has become common practice, driven by the need for efficient implementations of deep neural networks on embedded devices. In this paper, we exploit an oft-overlooked degree of freedom in most networks - for a given layer, individual output channels can be scaled by any factor provided that the corresponding weights of the next layer are inversely scaled. Therefore, a given network has many factorizations which change the weights of the network without changing its function. We present a conceptually simple and easy to implement method that uses this property and show that proper factorizations significantly decrease the degradation caused by quantization. We show improvement on a wide variety of networks and achieve state-of-the-art degradation results for MobileNets. While our focus is on quantization, this type of factorization is applicable to other domains such as network-pruning, neural nets regularization and network interpretability.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1902.01917

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Half-precision and mixed-precision training advances have been the primary drivers enabling an order-of-magnitude increase in throughput for AI accelerators.

    “In particular, the breakthroughs of half-precision and mixed-precision training [41,72,79,175] have been the main drivers that have enabled an order of magnitude higher throughput in AI accelerators.”

    From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [175]

    The Introduction (p. 1) discusses low-precision integer quantization for efficient inference, and the Discussion (p. 8) focuses on improving inference quantization without retraining. Searches for half-precision or mixed-precision training and order-of-magnitude accelerator throughput found no passage addressing that claim. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 29

  1. Banner, R., Hubara, I., Hoffer, E., and Soudry, D. Scalable Methods for 8-bit Training of Neural Networks. may 2018a. URL http://arxiv.org/abs/1805. 11046 .
  2. Banner, R., Nahshan, Y., Hoffer, E., and Soudry, D. ACIQ: Analytical Clipping for Integer Quantization of neural networks. oct 2018b. URL http://arxiv.org/ abs/1810.05723 . · arXiv
  3. Choi, J., Wang, Z., Venkataramani, S., Chuang, P. I.-J., Srinivasan, V., and Gopalakrishnan, K. PACT: Parameterized Clipping Activation for Quantized Neural Networks. may 2018. URL http://arxiv.org/abs/1805. 06085 .
  4. Choi, Y., El-Khamy, M., and Lee, J. Towards the limit of network quantization. CoRR , abs/1612.01543, 2016. URL http://arxiv.org/abs/1612.01543 . · arXiv
  5. Courbariaux, M., Bengio, Y., and David, J.-P. Training deep neural networks with low precision multiplications. dec 2014. URL http://arxiv.org/abs/1412. 7024 .
  6. Gupta, S., Agrawal, A., Gopalakrishnan, K., and Narayanan, P. Deep Learning with Limited Numerical Precision. feb 2015. URL https://arxiv.org/abs/1502. 02551 .
  7. Han, S., Mao, H., and Dally, W. J. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. oct 2015. URL http://arxiv.org/abs/1510.00149 . · arXiv
  8. He, K., Zhang, X., Ren, S., and Sun, J. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. feb 2015a. URL http: //arxiv.org/abs/1502.01852 .
  9. He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. CoRR , abs/1512.03385, 2015b. URL http://arxiv.org/abs/1512.03385 . · arXiv
  10. Hinton, G., Vinyals, O., and Dean, J. Distilling the Knowledge in a Neural Network. mar 2015. URL https: //arxiv.org/abs/1503.02531 .
  11. Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. apr 2017. URL http: //arxiv.org/abs/1704.04861 .
  12. Huang, G., Liu, Z., van der Maaten, L., and Weinberger, K. Q. Densely Connected Convolutional Networks. aug 2016. URL http://arxiv.org/abs/1608. 06993 .
  13. Ignatov, A., Timofte, R., Chou, W., Wang, K., Wu, M., Hartley, T., and Van Gool, L. AI Benchmark: Running Deep Neural Networks on Android Smartphones. oct 2018. URL http://arxiv.org/abs/1810.01109 . · arXiv
  14. Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR , abs/1502.03167, 2015. URL http://arxiv. org/abs/1502.03167 . · arXiv
  15. Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. Quantization and Training of Neural Networks for Efficient IntegerArithmetic-Only Inference. 2017. doi: 10.1109/CVPR. 2018.00286. · DOI
  16. Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., Boyle, R., Cantin, P.-l., Chao, C., Clark, C., Coriell, J., Daley, M., Dau, M., Dean, J., Gelb, B., Ghaemmaghami, T. V., Gottipati, R., Gulland, W., Hagmann, R., Ho, C. R., Hogberg, D., Hu, J., Hundt, R., Hurt, D., Ibarz, J., Jaffey, A., Jaworski, A., Kaplan, A., Khaitan, H., Koch, A., Kumar, N., Lacy, S., Laudon, J., Law, J., Le, D., Leary, C., Liu, Z., Lucke, K., Lundin, A., MacKean, G., Maggiore, A., Mahony, M., Miller, K., Nagarajan, R., Narayanaswami, R., Ni, R., Nix, K., Norrie, T., Omernick, M., Penukonda, N., Phelps, A., Ross, J., Ross, M., Salek, A., Samadiani, E., Severn, C., Sizikov, G., Snelham, M., Souter, J., Steinberg, D., Swing, A., Tan, M., Thorson, G., Tian, B., Toma, H., Tuttle, E., Vasudevan, V., Walter, R., Wang, W., Wilcox, E., and Yoon, D. H. In-Datacenter Performance Analysis of a Tensor Processing Unit. apr 2017. URL http://arxiv.org/abs/1704.04760 . · arXiv
  17. Lee, J. H., Ha, S., Choi, S., Lee, W., and Lee, S. Quantization for rapid deployment of deep neural networks. CoRR , abs/1810.05488, 2018. URL http://arxiv. org/abs/1810.05488 . · arXiv
  18. Lin, D. D., Talathi, S. S., and Annapureddy, V. S. Fixed Point Quantization of Deep Convolutional Networks. nov 2015. URL https://arxiv.org/abs/1511. 06393 .
  19. Marco, D. and Neuhoff, D. L. The validity of the additive noise model for uniform scalar quantizers. IEEE Transactions on Information Theory , 51(5):1739-1755, May 2005. ISSN 0018-9448. doi: 10.1109/TIT.2005.846397. · DOI
  20. McKinstry, J. L., Esser, S. K., Appuswamy, R., Bablani, D., Arthur, J. V ., Yildiz, I. B., and Modha, D. S. Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference. sep 2018. URL http://arxiv.org/abs/1809.04191 . · arXiv
  21. Nair, V. and Hinton, G. E. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning , ICML'10, pp. 807814, USA, 2010. Omnipress. ISBN 978-1-60558-9077. URL http://dl.acm.org/citation.cfm? id=3104322.3104425 .
  22. Park, J., Naumov, M., Basu, P., Deng, S., Kalaiah, A., Khudia, D., Law, J., Malani, P., Malevich, A., Nadathur, S., Pino, J., Schatz, M., Sidorov, A., Sivakumar, V., Tulloch, A., Wang, X., Wu, Y., Yuen, H., Diril, U., Dzhulgakov, D., Hazelwood, K., Jia, B., Jia, Y., Qiao, L., Rao, V., Rotem, N., Yoo, S., and Smelyanskiy, M. Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications. nov 2018. URL http://arxiv.org/abs/1811. 09886 .
  23. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) , 115(3):211-252, 2015. doi: 10.1007/s11263-015-0816-y. · DOI · Paper page
  24. Sakr, C., Kim, Y., and Shanbhag, N. Analytical guarantees on numerical precision of deep neural networks. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pp. 3007-3016, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL http://proceedings.mlr. press/v70/sakr17a.html .
  25. Sheng, T., Feng, C., Zhuo, S., Zhang, X., Shen, L., and Aleksic, M. A Quantization-Friendly Separable Convolution for MobileNets. mar 2018. doi: 10.1109/EMC2.2018.00011. URL http://arxiv.org/abs/1803.08607http: //dx.doi.org/10.1109/EMC2.2018.00011 . · DOI
  26. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S. E., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. CoRR , abs/1409.4842, 2014. URL http://arxiv.org/ abs/1409.4842 . · arXiv
  27. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vision. CoRR , abs/1512.00567, 2015. URL http:// arxiv.org/abs/1512.00567 . · arXiv
  28. Zhou, A., Yao, A., Guo, Y., Xu, L., and Chen, Y. Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights. feb 2017. URL https: //arxiv.org/abs/1702.03044 .
  29. Zhou, A., Yao, A., Wang, K., and Chen, Y. Explicit losserror-aware quantization for low-bit deep neural networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2018.

Cited by

Sorted by citations

Authors
Meller, E., Finkelstein, A., Almog, U., & Grobman, M.
Venue
arXiv
Publisher
arXiv
Published
2019
Pages
4486-4495
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1902.01917
arXiv
1902.01917
OpenAlex
W2911595986
Cited by
60
References
0
Citations by year
2025: 2 · 2024: 12 · 2023: 11 · 2022: 15 · 2021: 13 · 2020: 5 · 2019: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • International Conference on Machine Learning
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
Keywords
neural network quantization, channel scaling, MobileNet, embedded deep learning, network pruning, neural network regularization, interpretability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review