Bitwise Neural Networks
Open accessAbstract
Based on the assumption that there exists a neural network that efficiently represents a set of Boolean functions between all binary inputs and outputs, we propose a process for developing and deploying neural networks whose weight parameters, bias terms, input, and intermediate hidden layer output signals, are all binary-valued, and require only basic bit logic for the feedforward pass. The proposed Bitwise Neural Network (BNN) is especially suitable for resource-constrained environments, since it replaces either floating or fixed-point arithmetic with significantly more efficient bitwise operations. Hence, the BNN requires for less spatial complexity, less memory bandwidth, and less power consumption in hardware. In order to design such networks, we propose to add a few training schemes, such as weight compression and noisy backpropagation, which result in a bitwise network that performs almost as well as its corresponding real-valued network. We test the proposed network on the MNIST dataset, represented using binary features, and show that BNNs result in competitive performance while offering dramatic computational savings.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Integer-only quantization uses low-precision integer arithmetic for all operations, allowing inference without floating-point dequantization.
“However, in integer-only quantization, all the operations are performed using low-precision integer arithmetic [113,132,154,193,267], as illustrated in Figure 6 (Right).This permits the entire inference to be carried out with efficient integer arithmetic, without any floating point dequantization of any parameters or activations.”
From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [132]
The introduction (p. 2) and Section 2 (p. 2) describe inference using binary inputs and parameters, with XNOR and bit counting rather than floating- or fixed-point arithmetic. This supports inference without floating-point dequantization, but the paper distinguishes its bitwise approach from quantized networks that perform low-precision integer arithmetic; it does not say that all operations use such arithmetic. (LLM, conf=0.86)
Experimental — accuracy not yet measuredINT8-only inference can be extended to BERT to enable up to 4.0× faster inference than FP32.
“Recently, [267] leverages mix-precision quantization to achieve 23% speedup for ResNet50, as compared to INT8 inference without accuracy degradation, and [132] extends INT8-only inference to BERT model to enable up to 4.0× faster inference than FP32.”
From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [132]
The Experiments section (p. 4) reports MNIST digit-recognition results, and the Conclusion (p. 4) describes a single-bit neural network. Searches for BERT, INT8-only inference, and a 4.0× speedup found no passage addressing the attributed claim. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260 · see citing works
Flagged first, then by citations
Research works printed but not linked: 17
- Baldauf, M., Dustdar, S., and Rosenberg, F. A survey on context-aware systems. International Journal of Ad Hoc and Ubiquitous Computing , 2(4):263-277, January 2007.
- Bengio, Y. Learning deep architectures for AI. Foundations and Trends in Machine Learning , 2(1):1-127, 2009.
- Courbariaux, M., Bengio, Y., and David, J.-P. Low precision arithmetic for deep learning. arXiv preprint arXiv:1412.7024 , 2014. · arXiv
- Cybenko, G. Approximations by superpositions of sigmoidal functions. Mathematics of Control, Signals, and Systems , 2(4):303-314, 1989.
- Fiesler, E., Choudry, A., and Caulfield, H. J. Weight discretization paradigm for optical neural networks. In The Hague'90, 12-16 April , pp. 164-173. International Society for Optics and Photonics, 1990.
- Golea, M., Marchand, M., and Hancock, T. R. On learning µ -perceptron networks with binary weights. In Advances in Neural Information Processing Systems (NIPS) , pp. 591-598, 1992.
- Goodfellow, I. J., Warde-Farley, D., Mirza, M., Courville, A., and Bengio, Y. Maxout networks. In Proceedings of the International Conference on Machine Learning (ICML) , 2013.
- Hinton, G. E., Osindero, S., and Teh, Y. A fast learning algorithm for deep belief nets. Neural Computation , 18 (7):1527-1554, 2006.
- Hornik, K. Approximation capabilities of multilayer feedforward networks. Neural Networks , 4(2):251-257, 1991.
- Hwang, K. and Sung, W. Fixed-point feedforward deep neural network design using weights +1, 0, and -1. In 2014 IEEE Workshop on Signal Processing Systems (SiPS) , Oct 2014.
- LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradientbased learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, November 1998.
- McCulloch, W. S. and Pitts, W. H. A logical calculus of the ideas immanent in nervous activity. The Bulletin of Mathematical Biophysics , 5(4):115-133, 1943.
- Pitt, L. and Valiant, L. G. Computational limitations on learning from examples. Journal of the Association for Computing Machinery , 35:965-984, 1988.
- Salakhutdinov, R. and Hinton, G. Semantic hashing. International Journal of Approximate Reasoning , 50(7):969 - 978, 2009.
- Soudry, D., Hubara, I., and Meir, R. Expectation backpropagation: Parameter-free training of multilayer neural networks with continuous or discrete weights. In Advances in Neural Information Processing Systems (NIPS) , 2014.
- Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research , 15(1):1929-1958, January 2014.
- Xu, Y., Du, J., Dai, L.-R., and Lee, C.-H. An experimental study on speech enhancement based on deep neural networks. IEEE Signal Processing Letters , 21(1):65-68, 2014.
Cited by
Sorted by citations
Recent advances in convolutional neural networks
Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G., Cai, J., et al. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.10.013 · Cited by 6,576 · see citing works
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review
Rawat, W., & Wang, Z. (2017) · Neural Computation · doi:10.1162/neco_a_00990 · Cited by 3,503 · see citing works
A Survey of Quantization Methods for Efficient Neural Network Inference
Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., & Keutzer, K. (2022) · Low-Power Computer Vision · doi:10.1201/9781003162810-13 · Cited by 1,219 · see citing works
“As such, there is a large body of work that has proposed different solutions to address this [18, 25, 47, 52, 77, 82, 91, 92, 118, 120, 122, 127, 129, 133, 139, 147, 152, 157, 190, 192, 210, 241, 243, 252, 254, 279, 281].”
FINN: A Framework for Fast, Scalable Binarized Neural Network Inference
Umuroglu, Y., Fraser, N. J., Gambardella, G., Blott, M., Leong, P., Jahre, M., & Vissers, K. (2017) · Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays · doi:10.1145/3020078.3021744 · Cited by 1,189 · see citing works
Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?
Zhu, S., Dong, X., & Su, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00506 · Cited by 153 · see citing works
- Authors
- Kim, M., & Smaragdis, P.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- ORCID
- Kim, Minje
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2016
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1601.06071
- OpenAlex
- W2266701264
- Cited by
- 166
- References
- 1
- Citations by year
- 2025: 2 · 2024: 9 · 2023: 4 · 2022: 12 · 2021: 13 · 2020: 22 · 2019: 33 · 2018: 32 · 2017: 24 · 2016: 13 · 2015: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Advanced Memory and Neural Computing
- Keywords
- Boolean functions, weight compression, resource-constrained devices, memory bandwidth reduction, computational efficiency, MNIST
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
