CoChat
Sign Up
PreprintCited by 17Advanced Neural Network ApplicationsStochastic Gradient Optimization TechniquesNeural Networks and Applications

Neural gradients are near-lognormal: improved quantized and sparse training

Open access
Chmiel, Brian, Ben-Uri, Liad, Shkolnik, Moran, Hoffer, Elad, Banner, Ron, Soudry, Daniel·2020·arXiv
DOI 10.48550/arxiv.2006.08173arXiv 2006.08173
Open this paper in CoChat — free accountOpen PDF

Abstract

While training can mostly be accelerated by reducing the time needed to propagate neural gradients back throughout the model, most previous works focus on the quantization/pruning of weights and activations. These methods are often not applicable to neural gradients, which have very different statistical properties. Distinguished from weights and activations, we find that the distribution of neural gradients is approximately lognormal. Considering this, we suggest two closed-form analytical methods to reduce the computational and memory burdens of neural gradients. The first method optimizes the floating-point format and scale of the gradients. The second method accurately sets sparsity thresholds for gradient pruning. Each method achieves state-of-the-art results on ImageNet. To the best of our knowledge, this paper is the first to (1) quantize the gradients to 6-bit floating-point formats, or (2) achieve up to 85% gradient sparsity -- in each case without accuracy degradation. Reference implementation accompanies the paper.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2006.08173

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LQNets, PACT, LSQ, and LSQ+ jointly optimize the clipping range and the neural network weights during training.

    “Notable work here are LQNets [276], PACT [36], LSQ [56], and LSQ+ [15] which jointly optimizes the clipping range and the weights in NN during training.”

    From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [36]

    The Related Work section (p. 2) mentions PACT among neural-network quantization methods, and the References (p. 9) identify its paper. Our copy does not say that PACT—or LQNets, LSQ, and LSQ+—jointly optimizes clipping ranges and network weights. Searches for that optimization claim and the four method names found no supporting passage. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 23

  1. Md Aamir Raihan and Tor M. Aamodt. Sparse weight activation training. arXiv:2001.01969 , 2020. · arXiv
  2. Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. In NeurIPS , 2018a.
  3. Ron Banner, Yury Nahshan, Elad Hoffer, and Daniel Soudry. Post-training 4-bit quantization of convolution networks for rapid-deployment. arXiv preprint arXiv:1810.05723 , 2018b. URL http://arxiv.org/abs/1810.05723 . · arXiv
  4. Chaim Baskin, Natan Liss, Yoav Chai, Evgenii Zheltonozhskii, Eli Schwartz, Raja Girayes, Avi Mendelson, and Alexander M Bronstein. Nice: Noise injection and clamping estimation for neural network quantization. arXiv preprint arXiv:1810.00162 , 2018. URL http://arxiv.org/ abs/1810.00162 . · arXiv
  5. Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Anima Anandkumar. signsgd: compressed optimisation for non-convex problems. In ICML , 2018.
  6. Gerard Biau and David M Mason. High-dimensional norms. In Mathematical Statistics and Limit Theorems , pp. 21-40. Springer, 2015.
  7. L´ eopold Cambier, Anahita Bhiwandiwalla, Ting Gong, Mehran Nekuii, Oguz H. Elibol, and Hanlin Tang. Shifted and squeezed 8-bit floating point format for low-precision training of deep neural networks. ArXiv , abs/2001.05674, 2020. · arXiv
  8. Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. ArXiv , abs/1805.06085, 2018b. · arXiv
  9. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pp. 248-255. Ieee, 2009.
  10. Jun Fang, Ali Shafiee, Hamzah Abdel-Aziz, David Thorsley, Georgios Georgiadis, and Joseph Hassoun. Post-training piecewise linear quantization for deep neural networks. arXiv: Computer Vision and Pattern Recognition , 2020.
  11. Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR , 2018.
  12. Greg Henry, Ping Tak Peter Tang, and Alexander Heinecke. Leveraging the bfloat16 artificial intelligence datatype for higher-precision computations. In 2019 IEEE 26th Symposium on Computer Arithmetic (ARITH) , pp. 69-76. IEEE, 2019.
  13. C. A. R. Hoare. Algorithm 65: Find. Commun. ACM , 4(7):321-322, July 1961. ISSN 0001-0782. doi: 10.1145/366622.366647. URL https://doi.org/10.1145/366622.366647 . · DOI
  14. Gao Huang, Zhuang Liu, and Kilian Q. Weinberger. Densely connected convolutional networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 2261-2269, 2017.
  15. Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through l0 regularization. ArXiv , abs/1712.01312, 2017. · arXiv
  16. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Frederick Diamos, Erich Elsen, David Garc´ ıa, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. ArXiv , abs/1710.03740, 2017. · arXiv
  17. Yury Nahshan, Brian Chmiel, Chaim Baskin, Evgenii Zheltonozhskii, Ron Banner, Alex M. Bronstein, and Avi Mendelson. Loss aware post-training quantization. ArXiv , abs/1911.07190, 2019. · arXiv
  18. Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR , abs/1409.1556, 2015. · arXiv
  19. N. Smirnov. Table for estimating the goodness of fit of empirical distributions. Ann. Math. Statist. 19 (1948), no. 2, 279-281. , 1948. URL https://projecteuclid.org/euclid.aoms/ 1177730256 .
  20. Xiao Sun, Jungwook Choi, Chia-Yu Chen, Naigang Wang, Swagath Venkataramani, Vijayalakshmi Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrishnan. Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks. In NeurIPS , 2019.
  21. Xu Sun, Xuancheng Ren, Shuming Ma, and Houfeng Wang. meprop: Sparsified back propagation for accelerated deep learning with reduced overfitting. arXiv:1706.06197 , 2017. · arXiv
  22. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. ArXiv , abs/1706.03762, 2017. · arXiv
  23. Xucheng Ye, Jianlei Yang, Pengcheng Dai, Yiran Chen, and Weisheng Zhao. Accelerating cnn training by sparsifying activation gradients. arXiv:1908.00173 , 2019. · arXiv

Cited by

Sorted by citations

Authors
Chmiel, B., Ben-Uri, L., Shkolnik, M., Hoffer, E., Banner, R., & Soudry, D.
Affiliations
  • Technion – Israel Institute of Technology (IL)
  • Intel (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2006.08173
arXiv
2006.08173
OpenAlex
W3092052590
Cited by
17
References
2
Citations by year
2025: 3 · 2024: 6 · 2023: 3 · 2022: 5
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
Keywords
gradient quantization, gradient sparsity, lognormal distribution, gradient scaling, memory-efficient training, computational complexity reduction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review