CoChat
Sign Up
PreprintCited by 33Advanced Neural Network ApplicationsParallel Computing and Optimization TechniquesLow-power high-performance VLSI design

BiQGEMM: Matrix Multiplication with Lookup Table For Binary-Coding-based Quantized DNNs

Open access
Jeon, Yongkweon, Park, Baeseong, Kwon, Se Jung, Kim, Byeongwook, Yun, Jeongin, Lee, Dongsoo·2020·arXiv
DOI 10.48550/arxiv.2005.09904arXiv 2005.09904
Open this paper in CoChat — free accountOpen PDF

Abstract

The number of parameters in deep neural networks (DNNs) is rapidly increasing to support complicated tasks and to improve model accuracy. Correspondingly, the amount of computations and required memory footprint increase as well. Quantization is an efficient method to address such concerns by compressing DNNs such that computations can be simplified while required storage footprint is significantly reduced. Unfortunately, commercial CPUs and GPUs do not fully support quantization because only fixed data transfers (such as 32 bits) are allowed. As a result, even if weights are quantized into a few bits, CPUs and GPUs cannot access multiple quantized weights without memory bandwidth waste. Success of quantization in practice, hence, relies on an efficient computation engine design, especially for matrix multiplication that is a basic computation engine in most DNNs. In this paper, we propose a novel matrix multiplication method, called BiQGEMM, dedicated to quantized DNNs. BiQGEMM can access multiple quantized weights simultaneously in one instruction. In addition, BiQGEMM pre-computes intermediate results that are highly redundant when quantization leads to limited available computation space. Since pre-computed values are stored in lookup tables and reused, BiQGEMM achieves lower amount of overall computations. Our extensive experimental results show that BiQGEMM presents higher performance than conventional schemes when DNNs are quantized.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2005.09904

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 52

  1. E. Li, L. Zeng, Z. Zhou, and X. Chen, "Edge AI: On-demand accelerating deep neural network inference via edge computing," IEEE Transactions on Wireless Communications, 2019.
  2. A. Reuther, P. Michaleas, M. Jones, V. Gadepally, S. Samsi, and J. Kepner, "Survey and benchmarking of machine learning accelerators," arXiv preprint arXiv:1908.11348, 2019. · arXiv
  3. T. Choudhary, V. Mishra, A. Goswami, and J. Sarangapani, "A comprehensive survey on model compression and acceleration," Artificial Intelligence Review, pp. 1-43, 2020.
  4. Y. Cheng, D. Wang, P. Zhou, and T. Zhang, "A survey of model compression and acceleration for deep neural networks," arXiv preprint arXiv:1710.09282, 2017. · arXiv
  5. Z. Zhou, X. Chen, E. Li, L. Zeng, K. Luo, and J. Zhang, "Edge intelligence: Paving the last mile of artificial intelligence with edge computing," Proceedings of the IEEE, vol. 107, no. 8, pp. 1738-1762, 2019.
  6. D. He, Z. Wang, and J. Liu, "A survey to predict the trend of AI-able server evolution in the cloud," IEEE Access, vol. 6, pp. 10591-10602, 2018.
  7. J. Von Neumann, "First draft of a report on the EDVAC," IEEE Annals of the History of Computing, vol. 15, no. 4, pp. 27-75, 1993.
  8. J. L. Hennessy and D. A. Patterson, Computer architecture: a quantitative approach. Elsevier, 2011.
  9. E. Eleftheriou, ""in-memory computing": Accelerating ai ap- plications," in 2018 48th European Solid-State Device Research Conference (ESSDERC), pp. 4-5, IEEE, 2018.
  10. S. Han, H. Mao, and W. J. Dally, "Deep compression: Compress- ing deep neural networks with pruning, trained quantization and huffman coding," arXiv preprint arXiv:1510.00149, 2015. · arXiv
  11. Z. Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell, "Rethinking the value of network pruning," arXiv preprint arXiv:1810.05270, 2018. · arXiv
  12. D. Lee, S. J. Kwon, B. Kim, P. Kapoor, and G.-Y. Wei, "Network pruning for low-rank binary indexing," arXiv preprint arXiv:1905.05686, 2019. · arXiv
  13. T. N. Sainath, B. Kingsbury, V. Sindhwani, E. Arisoy, and B. Ramabhadran, "Low-rank matrix factorization for deep neu- ral network training with high-dimensional output targets," in 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6655-6659, IEEE, 2013.
  14. Y. Li, Y. Liang, and A. Risteski, "Recovery guarantee of weighted low-rank approximation via alternating minimiza- tion," in International Conference on Machine Learning, pp. 2358-2367, 2016.
  15. C. Xu, J. Yao, Z. Lin, W. Ou, Y. Cao, Z. Wang, and H. Zha, "Al- ternating multi-bit quantization for recurrent neural networks," arXiv preprint arXiv:1802.00150, 2018. · arXiv
  16. A. Bhandare, V. Sripathi, D. Karkada, V. Menon, S. Choi, K. Datta, and V. Saletore, "Efficient 8-bit quantization of Transformer neural machine language translation model," arXiv preprint arXiv:1906.00532, 2019. · arXiv
  17. D. Zhang, J. Yang, D. Ye, and G. Hua, "LQ-Nets: Learned quan- tization for highly accurate and compact deep neural networks," in Proceedings of the European conference on computer vision (ECCV), pp. 365-382, 2018.
  18. P. Stock, A. Joulin, R. Gribonval, B. Graham, and H. Jégou, "And the bit goes down: Revisiting the quantization of neural networks," arXiv preprint arXiv:1907.05686, 2019. · arXiv
  19. M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi, "XNOR- Net: Imagenet classification using binary convolutional neural networks," in European conference on computer vision, pp. 525- 542, Springer, 2016.
  20. A. V. Aho and J. E. Hopcroft, The design and analysis of computer algorithms. Pearson Education India, 1974.
  21. Y. Guo, A. Yao, H. Zhao, and Y. Chen, "Network sketching: Exploiting binary structure in deep CNNs," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5955-5963, 2017.
  22. M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y. Ben- gio, "Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or -1," arXiv preprint arXiv:1602.02830, 2016. · arXiv
  23. Y. He and S. Han, "ADC: automated deep compres- sion and acceleration with reinforcement learning," CoRR, vol. abs/1802.03494, 2018. · arXiv
  24. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors," nature, vol. 323, no. 6088, pp. 533-536, 1986.
  25. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention is all you need," in Advances in neural information processing systems, pp. 5998- 6008, 2017.
  26. S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural computation, vol. 9, no. 8, pp. 1735-1780, 1997.
  27. D. Bahdanau, K. Cho, and Y. Bengio, "Neural machine trans- lation by jointly learning to align and translate," arXiv preprint arXiv:1409.0473, 2014. · arXiv
  28. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding," arXiv preprint arXiv:1810.04805, 2018. · arXiv
  29. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, "GLUE: A multi-task benchmark and analysis plat- form for natural language understanding," arXiv preprint arXiv:1804.07461, 2018. · arXiv
  30. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, "SQuAD: 100,000+ questions for machine comprehension of text," arXiv preprint arXiv:1606.05250, 2016. · arXiv
  31. K. Ahmed, N. S. Keskar, and R. Socher, "Weighted Trans- former network for machine translation," arXiv preprint arXiv:1711.02132, 2017. · arXiv
  32. P. Shaw, J. Uszkoreit, and A. Vaswani, "Self-attention with relative position representations," arXiv preprint arXiv:1803.02155, 2018. · arXiv
  33. M. Ott, S. Edunov, D. Grangier, and M. Auli, "Scaling neural machine translation," arXiv preprint arXiv:1806.00187, 2018. · arXiv
  34. S. Edunov, M. Ott, M. Auli, and D. Grangier, "Understanding back-translation at scale," arXiv preprint arXiv:1808.09381, 2018. · arXiv
  35. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, "Exploring the limits of transfer learning with a unified text-to-text Transformer," ArXiv, vol. abs/1910.10683, 2019. · arXiv
  36. Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le, "XLNet: Generalized autoregressive pretraining for language understanding," in Advances in neural information processing systems, pp. 5754-5764, 2019.
  37. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, "RoBERTa: A robustly optimized BERT pretraining approach," arXiv preprint arXiv:1907.11692, 2019. · arXiv
  38. Z. Zhang, X. Han, Z. Liu, X. Jiang, M. Sun, and Q. Liu, "ERNIE: Enhanced language representation with informative entities," arXiv preprint arXiv:1905.07129, 2019. · arXiv
  39. Y. Sun, S. Wang, Y. Li, S. Feng, H. Tian, H. Wu, and H. Wang, "ERNIE 2.0: A continual pre-training framework for language understanding," arXiv preprint arXiv:1907.12412, 2019. · arXiv
  40. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, "ALBERT: A lite BERT for self-supervised learning of language representations," arXiv preprint arXiv:1909.11942, 2019. · arXiv
  41. S. Karita, N. Chen, T. Hayashi, T. Hori, H. Inaguma, Z. Jiang, M. Someki, N. E. Y. Soplin, R. Yamamoto, X. Wang, et al., "A comparative study on Transformer vs RNN in speech applica- tions," arXiv preprint arXiv:1909.06317, 2019. · arXiv
  42. S. Karita, N. E. Y. Soplin, S. Watanabe, M. Delcroix, A. Ogawa, and T. Nakatani, "Improving Transformer-based end-to-end speech recognition with connectionist temporal classification and language model integration," Proc. Interspeech 2019, pp. 1408-1412, 2019.
  43. K. Irie, R. Prabhavalkar, A. Kannan, A. Bruguier, D. Rybach, and P. Nguyen, "On the choice of modeling unit for sequence-to- sequence speech recognition," Proc. Interspeech 2019, pp. 3800- 3804, 2019.
  44. C. Lüscher, E. Beck, K. Irie, M. Kitza, W. Michel, A. Zeyer, R. Schlüter, and H. Ney, "RWTH ASR systems for LibriSpeech: Hybrid vs attention-w/o data augmentation," arXiv preprint arXiv:1905.03072, 2019. · arXiv
  45. D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, "SpecAugment: A simple data augmenta- tion method for automatic speech recognition," arXiv preprint arXiv:1904.08779, 2019. · arXiv
  46. K. J. Han, R. Prieto, K. Wu, and T. Ma, "State-of-the-art speech recognition using multi-stream self-attention with dilated 1D convolutions," arXiv preprint arXiv:1910.00716, 2019. · arXiv
  47. G. Prato, E. Charlaix, and M. Rezagholizadeh, "Fully quan- tized Transformer for improved translation," arXiv preprint arXiv:1910.10485, 2019. · arXiv
  48. D. Lee, P. Kapoor, and B. Kim, "Deeptwist: Learning model compression via occasional weight distortion," arXiv preprint arXiv:1810.12823, 2018. · arXiv
  49. E. Wang, Q. Zhang, B. Shen, G. Zhang, X. Lu, Q. Wu, and Y. Wang, "Intel math kernel library," in High-Performance Computing on the Intel® Xeon Phi™, pp. 167-188, Springer, 2014.
  50. C. Nvidia, "cuBLAS library," NVIDIA Corporation, Santa Clara, California, vol. 15, no. 27, p. 31, 2008.
  51. V. Volkov and J. W. Demmel, "Benchmarking GPUs to tune dense linear algebra," in SC'08: Proceedings of the 2008 ACM/IEEE conference on Supercomputing, pp. 1-11, IEEE, 2008.
  52. S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer, "cuDNN: Efficient primitives for deep learning," arXiv preprint arXiv:1410.0759, 2014. · arXiv

Cited by

Sorted by citations

Authors
Jeon, Y., Park, B., Kwon, S. J., Kim, B., Yun, J., & Lee, D.
Affiliations
Samsung (South Korea) (KR)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Pages
1-14
ISSN
2167-4337 (ISSN-L)
DOI
10.48550/arxiv.2005.09904
arXiv
2005.09904
OpenAlex
W3131984434
Cited by
33
References
0
Citations by year
2026: 6 · 2025: 10 · 2024: 4 · 2023: 3 · 2022: 6 · 2021: 3 · 2020: 1
FWCI
1.00
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • International Conference for High Performance Computing, Networking, Storage and Analysis (SC) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
  • Physical Sciences › Engineering › Electrical and Electronic Engineering › Low-power high-performance VLSI design
Keywords
matrix multiplication, binary encoding, neural network quantization, look-up table, non-uniform quantization, weight quantization, memory bandwidth

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review