On Periodic Functions as Regularizers for Quantization of Neural Networks
Open accessAbstract
Deep learning models have been successfully used in computer vision and many other fields. We propose an unorthodox algorithm for performing quantization of the model parameters. In contrast with popular quantization schemes based on thresholds, we use a novel technique based on periodic functions, such as continuous trigonometric sine or cosine as well as non-continuous hat functions. We apply these functions component-wise and add the sum over the model parameters as a regularizer to the model loss during training. The frequency and amplitude hyper-parameters of these functions can be adjusted during training. The regularization pushes the weights into discrete points that can be encoded as integers. We show that using this technique the resulting quantized models exhibit the same accuracy as the original ones on CIFAR-10 and ImageNet datasets.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Fake quantization can help bandwidth-bound problems such as recommendation systems.
“In fact, fake quantization methods can be beneficial for problems that are bandwidth-bound rather than compute-bound, such as in recommendation systems [185].”
From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [185]
The Introduction (p. 1) says quantization can reduce model storage and compute requirements, and the Conclusion (p. 7) reports results for quantized ResNets. Searches for fake quantization, bandwidth-bound workloads, and recommendation systems found no passage addressing the claimed benefit in those settings. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Part of this paper’s reference list could not be split into single entries.
Li, F., Liu, B., Wang, X., Zhang, B., & Yan, J. (2016) · arXiv · doi:10.48550/arxiv.1605.04711 · Cited by 732 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [14]14. [NVIDIA. Turing architecture whitepaper. 2018. URL https://www.nvidia.com/ content/dam/en-zz/Solutions/design-\visualization/technologies/ turing-architecture/NVIDIA-Turing-Architecture-Whitepaper.pdf .](https://www.nvidia.com/content/dam/en-zz/Solutions/design-/visualization/technologies/turing-architecture/NVIDIA-Turing-Architecture-Whitepaper.pdf) - A. Parashar, M. Rhu, A. Mukkara, A. Puglielli, R. Venkatesan, B. Khailany, J. Emer, S. W. Keckler, and W. J. Dally. SCNN: An accelerator for compressed-sparse convolutional neural networks. ACM/IEEE 44th Annual International Symposium on Computer Architecture , 2017. - E. Park, S. Yoo, and P. Vajda. Value-aware quantization for training and inference of neural networks. CoRR , 2018. URL https://arxiv.org/abs/1804.07802 . - A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in PyTorch. Proc. Neural Information Processing Systems , 2017. - M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. XNOR-Net: ImageNet classification using binary convolutional neural networks. CoRR , 2016. URL https://arxiv.org/abs/ 1603.05279 . - B. Reagen, P. Whatmough, R. Adolf, S. Rama, H. Lee, S. K. Lee, J. M. Hernandez-Lobato, G.Y. Wei, and D. Brooks. Minerva: Enabling low-power, highly-accurate deep. ACM/IEEE 43rd Annual International Symposium on Computer Architecture , 2016. - A. L. Shimpi. ARM's Cortex A7: Bringing cheaper dual-core & more power efficient highend devices. AnandTech , 2011. URL https://www.anandtech.com/show/4991/ arms-cortex-a7-bringing-cheaper-dualcore-more-power-efficient/ /-highend-devices . - F. Stenger. Numerical methods based on sinc and analytic functions. Springer Series in Computational Mathematics , 20, 1993. - G. Strang and G. Fix. An Analysis of the Finite Element Method . Wellesley-Cambridge Press, 2nd Ed., 2008. - C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. CoRR , 2014. URL https://arxiv.org/abs/ 1409.4842 . - G. Venkatesh, E. Nurvitadhi, and D. Marr. Accelerating deep convolutional networks using lowprecision and sparsity. CoRR , 2016. URL https://arxiv.org/abs/1610.00324 . - P. Wang, Q. Hu, Y. Zhang, C. Zhang, Y. Liu, and J. Cheng. Two-step quantization for low-bit neural networks. Proc. Computer Vision and Pattern Recognition , 2018. - H. Wen, S. Zhou, Z. Liang, Y. Zhang, D. Feng, X. Zhou, and C. Yao. Training bit fully convolutional network for fast semantic segmentation. CoRR , 2016. URL https://arxiv.org/abs/ 1612.00212 . - Y. Zhou, S.M. Moosavi-Dezfooli, N.M. Cheung, and P. Frossard. Adaptive quantization for deep neural network. CoRR , 2017. URL https://arxiv.org/abs/1712.01048 . · arXiv
Cited by
Sorted by citations
A Survey of Quantization Methods for Efficient Neural Network Inference
Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., & Keutzer, K. (2022) · Low-Power Computer Vision · doi:10.1201/9781003162810-13 · Cited by 1,219 · see citing works
“Another class of mixed-precision methods uses periodic function regularization to train mixed-precision models by automatically distinguishing different layers and their varying importance with respect to accuracy while learning their respective bitwidths [179].”
- Authors
- Naumov, M., Diril, U., Park, J., Ray, B., Jablonski, J., & Tulloch, A.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1811.09862
- OpenAlex
- W2901554773
- Cited by
- 12
- References
- 1
- Citations by year
- 2024: 2 · 2023: 2 · 2022: 2 · 2020: 5 · 2019: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- neural network quantization, periodic functions, regularization, trigonometric functions, hat functions, model compression, loss function regularization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
