Hardware-oriented Approximation of Convolutional Neural Networks
Open accessAbstract
High computational complexity hinders the widespread usage of Convolutional Neural Networks (CNNs), especially in mobile devices. Hardware accelerators are arguably the most promising approach for reducing both execution time and power consumption. One of the most important steps in accelerator development is hardware-oriented model approximation. In this paper we present Ristretto, a model approximation framework that analyzes a given CNN with respect to numerical resolution used in representing weights and outputs of convolutional and fully connected layers. Ristretto can condense models by using fixed point arithmetic and representation instead of floating point. Moreover, Ristretto fine-tunes the resulting fixed point network. Given a maximum error tolerance of 1%, Ristretto can successfully condense CaffeNet and SqueezeNet to 8-bit. The code for Ristretto is available.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Floating-point backward passes are important because quantized gradient accumulation can produce zero or high-error gradients, especially at low precision.
“Performing the backward pass with floating point is important, as accumulating the gradients in quantized precision can result in zerogradient or gradients that have high error, especially in low-precision [42,80,81,107,159,186,204,231].”
From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [81]
The Fine-tuning section (p. 4) says Ristretto keeps layer outputs in floating point to compute parameter gradients analytically, and Fast fine-tuning (p. 4) says backward propagation can be performed in floating point. Neither passage says that accumulating gradients in quantized precision produces zero or high-error gradients, particularly at low precision. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 21 · Cited by 287
Research works printed but not linked: 16
- Bergstra, J. and Bengio, Y. Random Search for Hyper-Parameter Optimization. The Journal of Machine Learning Research , 13(1):281-305, 2012.
- Courbariaux, M., Bengio, Y., and David, J.-P. BinaryConnect: Training Deep Neural Networks with binary weights during propagations. In Advances in Neural Information Processing Systems , pp. 3105-3113, 2015.
- Courbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., and Bengio, Y. Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1. arXiv preprint arXiv:1602.02830 , 2016. · arXiv
- Gupta, S., Agrawal, A., Gopalakrishnan, K., and Narayanan, P. Deep Learning with Limited Numerical Precision. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15) , pp. 1737-1746, 2015.
- Hammerstrom, D. A VLSI Architecture for High-Performance, Low-Cost, On-chip Learning. In IJCNN International Joint Conference on Neural Networks, 1990 , pp. 537-544. IEEE, 1990.
- Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., and Dally, W. J. EIE: Efficient Inference Engine on Compressed Deep Neural Network. arXiv preprint arXiv:1602.01528 , 2016a. · arXiv
- Han, S., Mao, H., and Dally, W. J. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. In International Conference on Learning Representations , 2016b.
- Iandola, F. N., Moskewicz, M. W., Ashraf, K., Han, S., Dally, W. J., and Keutzer, K. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and < 0.5MB model size. arXiv:1602.07360 , 2016. · arXiv
- Jia, Y ., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. Caffe: Convolutional Architecture for Fast Feature Embedding. In Proceedings of the ACM International Conference on Multimedia , pp. 675-678. ACM, 2014.
- LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.
- Motamedi, M., Gysel, P., Akella, V., and Ghiasi, S. Design Space Exploration of FPGA-Based Deep Convolutional Neural Networks. In 2016 21st Asia and South Pacific Design Automation Conference (ASP-DAC) , pp. 575-580. IEEE, 2016.
- Qiu, J., Wang, J., Yao, S., Guo, K., Li, B., Zhou, E., Yu, J., Tang, T., Xu, N., Song, S., Wang, Y., and Yang, H. Going Deeper with Embedded FPGA Platform for Convolutional Neural Network. In Proceedings of the 2016 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays , pp. 26-35, 2016.
- Rastegari, M., Ordonez, V., Redmon, J., and Farhadi, A. XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks. arXiv preprint arXiv:1603.05279 , 2016. · arXiv
- Simonyan, K. and Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition. In International Conference on Learning Representations , 2015.
- Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going Deeper with Convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 1-9, 2015.
- Williamson, D. Dynamically scaled fixed point arithmetic. In IEEE Pacific Rim Conference on Communications, Computers and Signal Processing, 1991 , pp. 315-318. IEEE, 1991.
- Authors
- Gysel, P., Motamedi, M., & Ghiasi, S.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2016
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1604.03168
- OpenAlex
- W2337344472
- Cited by
- 287
- References
- 21
- Citations by year
- 2025: 4 · 2024: 4 · 2023: 12 · 2022: 19 · 2021: 29 · 2020: 48 · 2019: 56 · 2018: 68 · 2017: 40 · 2016: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Numerical Methods and Algorithms
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Low-power high-performance VLSI design
- Keywords
- convolutional neural networks, model compression, fixed-point arithmetic, hardware accelerators, numerical precision, weight quantization, fine-tuning, CaffeNet, SqueezeNet, mobile devices, power consumption
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
