The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Extremely Low Bit Neural Network: Squeeze the Last Bit Out With ADMM
Open accessAbstract
Although deep learning models are highly effective for various learning tasks, their high computational costs prohibit the deployment to scenarios where either memory or computational resources are limited. In this paper, we focus on compressing and accelerating deep models with network weights represented by very small numbers of bits, referred to as extremely low bit neural network. We model this problem as a discretely constrained optimization problem. Borrowing the idea from Alternating Direction Method of Multipliers (ADMM), we decouple the continuous parameters from the discrete constraints of network, and cast the original hard problem into several subproblems. We propose to solve these subproblems using extragradient and iterative quantization algorithms that lead to considerably faster convergency compared to conventional optimization methods. Extensive experiments on image recognition and object detection verify that the proposed algorithm is more effective than state-of-the-art approaches when coming to extremely low bit neural network.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
(2016) · Cited by 31,590
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
SSD: Single Shot MultiBox Detector
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., & Berg, A. C. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_2 · Cited by 22,128
Boyd, S., Parikh, N., Chu, E., Peleato, B., & Eckstein, J. (2011) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000016 · Cited by 17,075
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 14,214
Boyd, S. (2010) · now publishers, Inc. eBooks · doi:10.1561/9781601984616 · Cited by 12,677
SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., & Keutzer, K. (2016) · arXiv · doi:10.48550/arxiv.1602.07360 · Cited by 6,201
Han, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11231 · Cited by 4,636
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_32 · Cited by 4,408
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1408.5093 · Cited by 3,752
BinaryConnect: Training Deep Neural Networks with binary weights during propagations
Courbariaux, M., Bengio, Y., & David, J.-P. (2015) · arXiv · doi:10.48550/arxiv.1511.00363 · Cited by 2,180
Courbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1602.02830 · Cited by 2,106
DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., & Zou, Y. (2016) · arXiv · doi:10.48550/arxiv.1606.06160 · Cited by 1,690
The extragradient method for finding saddle points and other problems
Korpelevich, G. M. (1976) · Medical Entomology and Zoology · Cited by 1,607
Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
Denton, R., Zaremba, W., Bruna, J., LeCun, Y., & Fergus, R. (2014) · arXiv · doi:10.48550/arxiv.1404.0736 · Cited by 1,414
Global Convergence of ADMM in Nonconvex Nonsmooth Optimization
Wang, Y., Yin, W., & Zeng, J. (2019) · Journal of Scientific Computing · doi:10.1007/s10915-018-0757-z · Cited by 1,268
Quantized Convolutional Neural Networks for Mobile Devices
Wu, J., Leng, C., Wang, Y., Hu, Q., & Cheng, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.521 · Cited by 1,212
Self-citationCompressing Deep Convolutional Networks using Vector Quantization
Gong, Y., Liu, L., Yang, M., & Bourdev, L. D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6115 · Cited by 925
Zhu, C., Han, S., Mao, H., & Dally, W. J. (2016) · arXiv · doi:10.48550/arxiv.1612.01064 · Cited by 865
Predicting Parameters in Deep Learning
Denil, M., Shakibi, B., Dinh, L., Ranzato, M., & de Freitas, N. (2013) · arXiv · doi:10.48550/arxiv.1306.0543 · Cited by 853
Accelerating Very Deep Convolutional Networks for Classification and Detection
Zhang, X., Zou, J., He, K., & Sun, J. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2502579 · Cited by 832
Li, F., Liu, B., Wang, X., Zhang, B., & Yan, J. (2016) · arXiv · doi:10.48550/arxiv.1605.04711 · Cited by 732
DeepTrust^RT: Confidential Deep Neural Inference Meets Real-Time!
Babar, M. F., & Hasan, M. (2024) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.ecrts.2024.13 · Cited by 708
Fixed Point Quantization of Deep Convolutional Networks
Lin, D. D., Talathi, S. S., & Annapureddy, V. S. (2015) · arXiv · doi:10.48550/arxiv.1511.06393 · Cited by 684
Deep Learning with Low Precision by Half-Wave Gaussian Quantization
Cai, Z., He, X., Sun, J., & Vasconcelos, N. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.574 · Cited by 498
Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
Zhou, A., Yao, A., Guo, Y., Xu, L., & Chen, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.03044 · Cited by 439
Neural Networks with Few MultiplicationsPreprint
Lin, Z., Courbariaux, M., Memisevic, R., & Bengio, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.03009 · Cited by 344
Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
Lebedev, V., Ganin, Y., Rakhuba, M., Oseledets, I. V., & Lempitsky, V. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6553 · Cited by 283
Liu, B., Li, F., Wang, X., Zhang, B., & Yan, J. (2023) · ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp49357.2023.10094626 · Cited by 191
8-Bit Approximations for Parallelism in Deep LearningPreprint
Dettmers, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.04561 · Cited by 165
Low precision storage for deep learningPreprint
Courbariaux, M., Bengio, Y., & David, J. (2014) · arXiv (Cornell University) · Cited by 142
Fast neural networks without multipliers
Marchesi, M., Orlandi, G., Piazza, F., & Uncini, A. (1993) · IEEE Transactions on Neural Networks · doi:10.1109/72.182695 · Cited by 125
Accelerating Deep Convolutional Networks using low-precision and sparsity
Venkatesh, G., Nurvitadhi, E., & Marr, D. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952679 · Cited by 123
Low precision arithmetic for deep learningPreprint
Courbariaux, M., Bengio, Y., & David, J. (2014) · arXiv (Cornell University) · Cited by 94
Weight discretization paradigm for optical neural networks
Fiesler, E., Choudry, A., & Caulfield, H. J. (1990) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.20700 · Cited by 91
A simple effective heuristic for embedded mixed-integer quadratic programming
Takapoui, R., Moehle, N., Boyd, S., & Bemporad, A. (2016) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.1109/acc.2016.7526551 · Cited by 76
Cheng, Z., Soudry, D., Mao, Z., & Lan, Z. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.03562 · Cited by 54
An Improved Three-Weight Message-Passing AlgorithmPreprint
Derbinsky, N., Bento, J., Elser, V., & Yedidia, J. S. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1305.1961 · Cited by 37
Cited by
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., & Kalenichenko, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00286 · Cited by 4,114
A Survey of Quantization Methods for Efficient Neural Network Inference
Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., & Keutzer, K. (2022) · Low-Power Computer Vision · doi:10.1201/9781003162810-13 · Cited by 1,219
“While STE is the mainstream approach [218, 280], other approaches have also been explored in the literature [2, 25, 31, 59, 142, 160].”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey
Deng, B. L., Li, G., Han, S., Shi, L., & Xie, Y. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2020.2976475 · Cited by 996
Pruning and quantization for deep neural network acceleration: A survey
Liang, T., Glossner, J., Wang, L., Shi, S., & Zhang, X. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.07.045 · Cited by 943
Binary Neural Networks: A Survey
Qin, H., Gong, R., Liu, X., Bai, X., Song, J., & Sebe, N. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107281 · Cited by 546
Metadata
- Authors
- Leng, C., Dou, Z., Li, H., Zhu, S., & Jin, R.
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2018-04-29
- Volume
- 32
- Issue
- 1
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W2739789140
- Cited by
- 275
- References
- 44
- FWCI
- 14.00
- Type
- conference paper
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.