Training Compact Neural Networks with Binary Weights and Low Precision Activations
Open accessAbstract
In this paper, we propose to train a network with binary weights and low-bitwidth activations, designed especially for mobile devices with limited power consumption. Most previous works on quantizing CNNs uncritically assume the same architecture, though with reduced precision. However, we take the view that for best performance it is possible (and even likely) that a different architecture may be better suited to dealing with low precision weights and activations. Specifically, we propose a "network expansion" strategy in which we aggregate a set of homogeneous low-precision branches to implicitly reconstruct the full-precision intermediate feature maps. Moreover, we also propose a group-wise feature approximation strategy which is very flexible and highly accurate. Experiments on ImageNet classification tasks demonstrate the superior performance of the proposed model, named Group-Net, over various popular architectures. In particular, with binary weights and activations, we outperform the previous best binary neural network in terms of accuracy as well as saving more than 5 times computational complexity on ImageNet with ResNet-18 and ResNet-50.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
CNNs under similar quantization constraints show near-lossless performance.
“This degradation starkly contrasts with the near-lossless performance observed in CNNs under similar quantization constraints [80], [81].”
From Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Survey (2024) · cites this paper as [80]
Cites
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015 · see citing works
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00716 · Cited by 9,982 · see citing works
SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., & Keutzer, K. (2016) · arXiv · doi:10.48550/arxiv.1602.07360 · Cited by 6,201 · see citing works
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and ActivationsPreprint
Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., & Bengio, Y. (2016) · Journal of machine learning research · doi:10.48550/arxiv.1609.07061 · Cited by 2,196 · see citing works
Progressive Neural Architecture Search
Liu, C., Zoph, B., Neumann, M., Shlens, J., Wei, H., Li, L.-J., Fei-Fei, L., Yuille, A., Huang, J., & Murphy, K. P. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_2 · Cited by 1,946 · see citing works
Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
Bengio, Y., Léonard, N., & Courville, A. (2013) · arXiv · doi:10.48550/arxiv.1308.3432 · Cited by 1,910 · see citing works
DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., & Zou, Y. (2016) · arXiv · doi:10.48550/arxiv.1606.06160 · Cited by 1,690 · see citing works
(2018) · International Conference on Machine Learning · Cited by 1,465 · see citing works
AMC: AutoML for Model Compression and Acceleration on Mobile Devices
He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J., & Han, S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_48 · Cited by 1,400 · see citing works
An Analysis of Deep Neural Network Models for Practical Applications
Canziani, A., Paszke, A., & Culurciello, E. (2016) · arXiv · doi:10.48550/arxiv.1605.07678 · Cited by 969 · see citing works
Li, F., Liu, B., Wang, X., Zhang, B., & Yan, J. (2016) · arXiv · doi:10.48550/arxiv.1605.04711 · Cited by 732 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey
Deng, B. L., Li, G., Han, S., Shi, L., & Xie, Y. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2020.2976475 · Cited by 996 · see citing works
Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
Wu, B., Wang, Y., Zhang, P., Tian, Y., Vajda, P., & Keutzer, K. (2018) · arXiv · doi:10.48550/arxiv.1812.00090 · Cited by 186 · see citing works
Ostwal, V., Zand, R., DeMara, R. F., & Appenzeller, J. (2019) · IEEE Journal on Exploratory Solid-State Computational Devices and Circuits · doi:10.1109/jxcdc.2019.2956468 · Cited by 45 · see citing works
A Review of State-of-the-art Mixed-Precision Neural Network Frameworks
Rakka, M., Fouda, M. E., Khargonekar, P. P., & Kurdahi, F. J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3394390 · Cited by 51 · see citing works
Efficient Visual Recognition: A Survey on Recent Advances and Brain-inspired Methodologies
Wu, Y., Wang, D., Lu, X., Yang, F., Yao, M., Dong, W., Shi, J., & Li, G. (2022) · Machine Intelligence Research · doi:10.1007/s11633-022-1340-5 · Cited by 21 · see citing works
Additive Noise Annealing and Approximation Properties of Quantized Neural NetworksPreprint
Spallanzani, M., Cavigelli, L., Leonardi, G. P., Bertogna, M., & Luca Benini, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.10452 · Cited by 14 · see citing works
Nazir, A., Mir, R. N., & Qureshi, S. (2020) · International Journal of Intelligent Computing and Cybernetics · doi:10.1108/ijicc-04-2020-0038 · Cited by 11 · see citing works
Efficient Deep Neural NetworksPreprint
Wu, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.08926 · Cited by 11 · see citing works
Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Survey
Du, D., Gong, G., & Chu, X. (2024) · arXiv · doi:10.48550/arxiv.2405.00314 · Cited by 4 · see citing works
“This degradation starkly contrasts with the near-lossless performance observed in CNNs under similar quantization constraints [80], [81].”
ResultFPGA Oriented Compression of DNN Using Layer-Targeted Weights and Activations Quantization
Hoang, M. Q., Nguyen, P. L., Tran, H. V., Nguyễn, H. Q., Nguyen, V. T., & Vo-Le, C. (2021) · International Conference on Communications and Electronics (ICCE) · doi:10.1109/icce48956.2021.9352106 · Cited by 4 · see citing works
Improving Bi-Real Net with block-wise quantization and multiple-steps binarization on activation
Le, D. H., & Pham, T. V. (2020) · RIVF International Conference on Computing and Communication Technologies (RIVF) · doi:10.1109/rivf48685.2020.9140782 · Cited by 2 · see citing works
- Authors
- Zhuang, B., Shen, C., & Reid, I.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1808.02631
- OpenAlex
- W2887204394
- Cited by
- 10
- References
- 11
- Citations by year
- 2024: 1 · 2022: 1 · 2021: 1 · 2020: 3 · 2019: 3 · 2018: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Advanced Memory and Neural Computing
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Low-power high-performance VLSI design
- Keywords
- binary neural networks, quantized convolutional neural networks, network expansion, compact neural network architecture, mobile devices, energy-efficient inference, ImageNet classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
