The DOI matches its DataCite record.
Binarized Neural Networks
Open accessAbstract
We introduce a method to train Binarized Neural Networks (BNNs) - neural networks with binary weights and activations at run-time and when computing the parameters' gradient at train-time. We conduct two sets of experiments, each based on a different framework, namely Torch7 and Theano, where we train BNNs on MNIST, CIFAR-10 and SVHN, and achieve nearly state-of-the-art results. During the forward pass, BNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations, which might lead to a great increase in power-efficiency. Last but not least, we wrote a binary matrix multiplication GPU kernel with which it is possible to run our MNIST BNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The code for training and running our BNNs is available.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Such severe performance degradation of the compact model with quantization is also reported in other researches.
“Such severe performance degradation of the compact model with quantization is also reported in other researches [25, 12].”
From PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation (2021) · cites this paper as [25]
The Related Work section says low precision was once believed to be highly destructive to performance, but immediately says subsequent work proved the contrary. The Discussion and Future Work section reports that this paper's binarized networks performed no worse than comparable floating-point architectures and describes other quantized networks as having satisfactory or nearly identical performance. It does not report the claimed severe degradation. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 23
- Cheng, Zhiyong, Soudry, Daniel, Mao, Zexi, and Lan, Zhenzhong. Training Binary Multilayer Neural Networks for Image Classification using Expectation Backprop- gation. arXiv:1503.03562, (2012):8, 2015. · arXiv
- Courbariaux, Matthieu, Bengio, Yoshua, and David, Jean-Pierre. BinaryConnect: Training Deep Neural Networks with binary weights during propagations. Nips, pp. 1-9, 2015a. 1, 2, 2.1, 2.1, 3.3, 5.2, 3, 6
- DePasquale, Brian, Churchland, Mark M., and Abbott, L. F. Using Firing-Rate Dy- namics to Train Recurrent Networks of Spiking Model Neurons. pp. 1-17, 2016.
- Esser, Steve K and Arthur, John V. Backpropagation for Energy-Efficient Neuromor- phic Computing. Advances in Neural Information Processing Systems 28 (NIPS 2015), pp. 1-9, 2015.
- Ginkel, Robbert Van and Connor, Peter O. Discrete Parameter Autoencoders for Se- mantic Hashing. pp. 1-22, 2015.
- Gong, Yunchao, Liu, Liu, Yang, Ming, and Bourdev, Lubomir. Compressing Deep Convolutional Networks using Vector Quantization. pp. 1-10, 2014.
- Goodfellow, Ian J., Warde-Farley, David, Mirza, Mehdi, Courville, Aaron, and Bengio, Yoshua. Maxout Networks. arXiv preprint, pp. 1319-1327, 2013. 5.1.1, 3
- Graham, Benjamin. Spatially-sparse convolutional neural networks. pp. 1-13, 2014. 5.1.
- Hinton. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 15:1929-1958, 2014. ISSN 00905364. doi: 10. 1214/12-AOS1000. 1, 2, 3. · DOI
- Horowitz, Mark. Computing's Energy Problem (and what we can do about it). IEEE Interational Solid State Circuits Conference, pp. 10-14, 2014. ISSN 0018-9200. doi: 10.1109/JSSC.2014.2361354. 4, 1, 2, 4. · DOI
- Hwang, Kyuyeon and Sung, Wonyong. Fixed-point feedforward deep neural network design using weights +1, 0, and -1. 2014 IEEE Workshop on Signal Processing Systems (SiPS), pp. 1-6, 2014. doi: 10.1109/SiPS.2014.6986082. 2, 3, 6 · DOI
- Ioffe, Sergey and Szegedy, Christian. Batch Normalization: Accelerating Deep Net- work Training by Reducing Internal Covariate Shift. arXiv, 2015. 3.3
- Judd, Patrick, Albericio, Jorge, Hetherington, Tayler, Aamodt, Tor, Jerger, Natalie En- right, Urtasun, Raquel, and Moshovos, Andreas. Reduced-Precision Strategies for Bounded Memory in Deep Neural Nets. pp. 12, 2015.
- Kim, Minje and Paris, Smaragdis. Bitwise Neural Networks. ICML Workshop on Resource-Efficient Machine Learning, 37, 2015.
- Kingma, Diederik and Ba, Jimmy. Adam: A Method for Stochastic Optimization. arXiv:1412.6980 [cs], pp. 1-13, 2014. 3.4 · arXiv
- Krizhevsky, Alex, Sulskever, IIya, and Hinton, Geoffret E. ImageNet Classification with Deep Convolutional Neural Networks. Advances in Neural Information and Processing Systems (NIPS), pp. 1-9, 2012.
- Lin, Min, Chen, Qiang, and Yan, Shuicheng. Network In Network. arXiv preprint, pp. 10, 2013. 5.1.1, 3
- Lin, Zhouhan, Courbariaux, Matthieu, Memisevic, Roland, and Bengio, Yoshua. Neu- ral Networks with Few Multiplications. Iclr, pp. 1-8, 2015. 2, 3.3, 4.1, 5.2, 6
- Lomont, Chris. Fast Inverse Square Root. Indiana: Purdue University, 2003-02 [2006- 06-24]. http, pp. 12, 2003. 3.3
- Soudry, D, Hubara, I, and Meir, R. Expectation Backpropagation: parameter-free train- ing of multilayer neural networks with real and discrete weights. Neural Information Processing Systems 2014, 2(1):1-9, 2014. 1, 2, 6
- Soudry, Daniel, Di Castro, Dotan, Gal, Asaf, Kolodny, Avinoam, and Kvatinsky, Sha- har. Memristor-Based Multilayer Neural Networks With Online Gradient Descent Training. IEEE Transactions on Neural Networks and Learning Systems, 26(10): 2408-2421, 2015. ISSN 21622388. doi: 10.1109/TNNLS.2014.2383395. · DOI
- Spang, H.A. Reduction by Feedback*. IRE Transactions on Communications Systems, pp. 373-380, 1962.
- Wu, Zhirong, Lin, Dahua, and Tang, Xiaoou. Adjustable Bounded Rectifiers: Towards Deep Binary Representations. arXiv preprint, pp. 1-11, 2015.
Cited by
Sorted by citations
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., & Kalenichenko, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00286 · Cited by 4,114
Deep Learning for Generic Object Detection: A Survey
Liu, L., Ouyang, W., Wang, X., Fieguth, P., Chen, J., Liu, X., & Pietikäinen, M. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01247-4 · Cited by 2,965
A Survey of Quantization Methods for Efficient Neural Network Inference
Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., & Keutzer, K. (2022) · Low-Power Computer Vision · doi:10.1201/9781003162810-13 · Cited by 1,219
Dynamic Neural Networks: A Survey
Han, Y., Huang, G., Song, S., Yang, L., Wang, H., & Wang, Y. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3117837 · Cited by 848
A crossbar array of magnetoresistive memory devices for in-memory computing
Jung, S., Lee, H., Myung, S., Kim, H., Yoon, S. K., Kwon, S.-W., Ju, Y., Kim, M., Yi, W., Han, S., et al. (2022) · Nature · doi:10.1038/s41586-021-04196-6 · Cited by 644
The Computational Limits of Deep Learning
Thompson, N. C., Greenewald, K., Lee, K., & Manso, G. F. (2020) · arXiv · doi:10.48550/arxiv.2007.05558 · Cited by 538
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review
Shuvo, M. M. H., Islam, S. K., Cheng, J., & Morshed, B. I. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2022.3226481 · Cited by 436
XNOR-SRAM: In-Memory Computing SRAM Macro for Binary/Ternary Deep Neural Networks
Yin, S., Jiang, Z., Seo, J.-S., & Seok, M. (2020) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2019.2963616 · Cited by 429
Biswas, A., & Chandrakasan, A. P. (2019) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2018.2880918 · Cited by 414
Learning to Quantize Deep Networks by Optimizing Quantization Intervals With Task Loss
Jung, S., Son, C., Lee, S., Son, J., Han, J.-J., Kwak, Y., Hwang, S. J., & Choi, C. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00448 · Cited by 385
Bai, X., Wang, X., Liu, X., Liu, Q., Song, J., Sebe, N., & Kim, B. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108102 · Cited by 313
Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
Caldas, S., Konečny, J., McMahan, H. B., & Talwalkar, A. (2018) · arXiv · doi:10.48550/arxiv.1812.07210 · Cited by 312
ChamNet: Towards Efficient Network Design Through Platform-Aware Model Adaptation
Dai, X., Zhang, P., Wu, B., Yin, H., Sun, F., Wang, Y., Dukhan, M., Hu, Y., Wu, Y., Jia, Y., et al. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01166 · Cited by 279
Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
Wu, H., Judd, P., Zhang, X., Isaev, M., & Micikevicius, P. (2020) · arXiv · doi:10.48550/arxiv.2004.09602 · Cited by 246
Deep Learning for Generic Object Detection: A Survey
Liu, L., Ouyang, W., Wang, X., Fieguth, P., Chen, J., Liu, X., & Pietikäinen, M. (2018) · arXiv · doi:10.48550/arxiv.1809.02165 · Cited by 2,965
Amirsoleimani, A., Alibart, F., Yon, V., Xu, J., Pazhouhandeh, M. R., Ecoffey, S., Beilliard, Y., Genov, R., & Drouin, D. (2020) · Advanced Intelligent Systems · doi:10.1002/aisy.202000115 · Cited by 216
AdderNet: Do We Really Need Multiplications in Deep Learning?
Chen, H., Wang, Y., Xu, C., Shi, B., Xu, C., Tian, Q., & Xu, C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00154 · Cited by 199
Yin, S., Sun, X., Yu, S., & Seo, J.-S. (2020) · IEEE Transactions on Electron Devices · doi:10.1109/ted.2020.3015178 · Cited by 171
“Similar to how the original BNN algorithms [8], [9] still use multibit precision for the primary input, note that the 784 input neurons in this simple MLP also exhibit 8-bit precision.”
BackgroundResultInfluentialA Study of BFLOAT16 for Deep Learning Training
Kalamkar, D., Mudigere, D., Mellempudi, N., Das, D., Banerjee, K., Avancha, S., Vooturi, D. T., Jammalamadaka, N., Huang, J., Yuen, H., et al. (2019) · arXiv · doi:10.48550/arxiv.1905.12322 · Cited by 169
SwapAdvisor: Pushing Deep Learning Beyond the GPU Memory Limit via Smart Swapping
Huang, C.-C., Jin, G., & Li, J. (2020) · Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems · doi:10.1145/3373376.3378530 · Cited by 158
Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?
Zhu, S., Dong, X., & Su, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00506 · Cited by 153
“5) with the full-precision DNN [38, 50], DoReFa-Net (k-bit quantized weight and activation) [65], XNOR-Net (binary weight and activation) [50], BNN (binary weight and activation) [31]”
BackgroundMethodEfficient densely connected convolutional neural networks
Li, G., Zhang, M., Li, J., Lv, F., & Tong, G. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2020.107610 · Cited by 145
DeepShift: Towards Multiplication-Less Neural Networks
Elhoushi, M., Chen, Z., Shafiq, F., Tian, Y. H., & Li, J. Y. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00268 · Cited by 117
Rethinking Differentiable Search for Mixed-Precision Neural Networks
Cai, Z., & Vasconcelos, N. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00242 · Cited by 119
Online Fall Detection Using Recurrent Neural Networks on Smart Wearable Devices
Musci, M., De Martini, D., Blago, N., Facchinetti, T., & Piastra, M. (2021) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2020.3027454 · Cited by 128
LEARN Codes: Inventing Low-Latency Codes via Recurrent Neural Networks
Jiang, Y., Kim, H., Asnani, H., Kannan, S., Oh, S., & Viswanath, P. (2020) · IEEE Journal on Selected Areas in Information Theory · doi:10.1109/jsait.2020.2988577 · Cited by 87
PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation
Kim, J., Chang, S., & Kwak, N. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-248 · Cited by 85
A survey of model compression strategies for object detection
Lyu, Z., Yu, T., Pan, F., Zhang, Y., Luo, J., Zhang, D., Chen, Y., Zhang, B., & Li, G. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17192-x · Cited by 65
EasiEdge: A Novel Global Deep Neural Networks Pruning Method for Efficient Edge Computing
Yu, F., Cui, L., Wang, P., Han, C., Huang, R., & Huang, X. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3034925 · Cited by 58
Enhancing Reliability of Analog Neural Network Processors
Moon, S., Shin, K., & Jeon, D. (2019) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2019.2893256 · Cited by 45
Accelerating Mobile Applications at the Network Edge with Software-Programmable FPGAs
Jiang, S., He, D., Yang, C., Xu, C., Luo, G., Chen, Y., Liu, Y., & Jiang, J. (2018) · IEEE INFOCOM 2018 - IEEE Conference on Computer Communications · doi:10.1109/infocom.2018.8485850 · Cited by 40
Mask Technique for Fast and Efficient Training of Binary Resistive Crossbar Arrays
Fouda, M. E., Lee, S., Lee, J., Eltawil, A., & Kurdahi, F. (2019) · IEEE Transactions on Nanotechnology · doi:10.1109/tnano.2019.2927493 · Cited by 41
A Deep Look into Logarithmic Quantization of Model Parameters in Neural Networks
Cai, J., Takemoto, M., & Nakajo, H. (2018) · Proceedings of the 10th International Conference on Advances in Information Technology · doi:10.1145/3291280.3291800 · Cited by 31
Diagonalwise Refactorization: An Efficient Training Method for Depthwise Convolutions
Qin, Z., Zhang, Z., Li, D., Zhang, Y., & Peng, Y. (2018) · 2018 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2018.8489312 · Cited by 27
SpinDrop: Dropout-Based Bayesian Binary Neural Networks With Spintronic Implementation
Ahmed, S. T., Danouchi, K., Münch, C., Prenat, G., Anghel, L., & Tahoori, M. B. (2023) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2023.3242146 · Cited by 25
Quantformer: Learning Extremely Low-Precision Vision Transformers
Wang, Z., Wang, C., Xu, X., Zhou, J., & Lu, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3229313 · Cited by 22
FLightNNs: Lightweight Quantized Deep Neural Networks for Fast and Accurate Inference
Ding, R., Liu, Z., Chin, T.-W., Marculescu, D., & Blanton, R. D. (. (2019) · Proceedings of the 56th Annual Design Automation Conference 2019 · doi:10.1145/3316781.3317828 · Cited by 19
Yoo, J., & Ban, G. (2025) · Sensors · doi:10.3390/s25092918 · Cited by 10
Quantized Reservoir Computing on Edge Devices for Communication Applications
Liu, S., Liu, L., & Yi, Y. (2020) · 2020 IEEE/ACM Symposium on Edge Computing (SEC) · doi:10.1109/sec50012.2020.00068 · Cited by 5
LittleBit: Ultra Low-Bit Quantization via Latent Factorization
Lee, B., Kim, D., You, Y., & Kim, Y. (2025) · arXiv · doi:10.48550/arxiv.2506.13771 · Cited by 7
Metadata
- Authors
- Hubara, I., Soudry, D., & Yaniv, R. E.
- Affiliations
- Technion – Israel Institute of Technology (IL)
- Columbia University (US)
- ORCID
- Soudry, Daniel
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2016
- Volume
- 29
- Pages
- 4107-4115
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1602.02505
- OpenAlex
- W2267635276
- Cited by
- 1,667
- References
- 0
- Citations by year
- 2026: 2 · 2025: 65 · 2024: 117 · 2023: 184 · 2022: 233 · 2021: 321 · 2020: 310 · 2019: 222 · 2018: 162 · 2017: 48 · 2016: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- neural information processing systems
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Advanced Memory and Neural Computing
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- binary neural networks, binary weights, gradient computation, memory efficiency, power efficiency, GPU kernels, image classification, Theano
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
