CoChat
Sign Up
PreprintCited by 42Advanced Neural Network ApplicationsTopic ModelingDomain Adaptation and Few-Shot Learning

BinaryBERT: Pushing the Limit of BERT Quantization

Open access
Bai, Haoli, Zhang, Wei, Hou, Lu, Shang, Lifeng, Jin, Jing, Jiang, Xin, Liu, Qun, Lyu, Michael +1 more·2020·arXiv
DOI 10.48550/arxiv.2012.15701arXiv 2012.15701
Open this paper in CoChat — free accountOpen PDF

Abstract

The rapid development of large pre-trained language models has greatly increased the demand for model compression techniques, among which quantization is a popular solution. In this paper, we propose BinaryBERT, which pushes BERT quantization to the limit by weight binarization. We find that a binary BERT is hard to be trained directly than a ternary counterpart due to its complex and irregular loss landscape. Therefore, we propose ternary weight splitting, which initializes BinaryBERT by equivalently splitting from a half-sized ternary network. The binary model thus inherits the good performance of the ternary one, and can be further enhanced by fine-tuning the new architecture after splitting. Empirical results show that our BinaryBERT has only a slight performance drop compared with the full-precision model while being 24x smaller, achieving the state-of-the-art compression results on the GLUE and SQuAD benchmarks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2012.15701

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 49

  1. H. Bai, J. Wu, I. King, and M. Lyu. 2020. Few shot network compression via cross distillation. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 3203-3210.
  2. Y. Bai, Y. Wang, and E. Liberty. 2018. Proxquant: Quantized neural networks via proximal operators. In International Conference on Machine Learning .
  3. A. Bulat, B. Martinez, and G. Tzimiropoulos. 2020. Bats: Binary architecture search. In European Conference on Computer Vision , pages 309-325.
  4. T. Chen, I. Goodfellow, and J. Shlens. 2016. Net2net: Accelerating learning via knowledge transfer. In International Conference on Learning Representations .
  5. M. Courbariaux, Y. Bengio, and J. David. 2015. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in neural information processing systems .
  6. M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and L. Kaiser. 2019. Universal transformers. In International Conference on Learning Representations .
  7. J. Devlin, M. Chang, K. Lee, and K. Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics .
  8. S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy, and D. S. Modha. 2019. Learned step size quantization. In International Conference on Learning Representations .
  9. A. Fan, E. Grave, and A. Joulin. 2019. Reducing transformer depth on demand with structured dropout. In International Conference on Learning Representations .
  10. A. Fan, P. Stock, B. Graham, E. Grave, R. Gribonval, H. Jegou, and A. Joulin. 2020. Training with quantization noise for extreme model compression. Preprint arXiv:2004.07320. · arXiv
  11. H. Habi, R. Jennings, and A. Netzer. 2020. Hmq: Hardware friendly mixed precision quantization block for cnns. In European Conference on Computer Vision , pages 448-463.
  12. Y. Hao, L. Dong, F. Wei, and K. Xu. 2019. Visualizing and understanding the effectiveness of BERT. In Conference on Empirical Methods in Natural Language Processing .
  13. L. Hou, Z. Huang, L. Shang, X. Jiang, X. Chen, and Q. Liu. 2020. Dynabert: Dynamic bert with adaptive width and depth. In Advances in Neural Information Processing Systems .
  14. L. Hou and J. T. Kwok. 2018. Loss-aware weight quantization of deep networks. In International Conference on Learning Representations .
  15. L. Hou, Yao Q., and J. T. Kwok. 2017. Loss-aware binarization of deep networks. In International Conference on Learning Representations .
  16. Z. Huang, L. Hou, L. Shang, X. Jiang, X. Chen, and Q. Liu. 2021. Ghostbert: Generate more features with cheap operations for bert. In Annual Meeting of the Association for Computational Linguistics .
  17. I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio. 2016. Binarized neural networks. In Advances in neural information processing systems .
  18. D. Kim, K Singh, and J. Choi. 2020. Learning architectures for binary networks. In European Conference on Computer Vision , pages 575-591.
  19. H. Kim, K. Kim, J. Kim, and J. Kim. 2019. Binaryduo: Reducing gradient mismatch in binary activation network by coupling binary activations. In International Conference on Learning Representations .
  20. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut. 2020. Albert: A lite bert for selfsupervised learning of language representations. In International Conference on Learning Representations .
  21. H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein. 2018. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems .
  22. Y. Li, X. Dong, and W. Wang. 2020a. Additive powersof-two quantization: a non-uniform discretization for neural networks. In International Conference on Learning Representations .
  23. Y. Li, W. Wang, H. Bai, R. Gong, X. Dong, and F. Yu. 2020b. Efficient bitwidth search for practical mixed precision neural network. Preprint arXiv:2003.07577. · arXiv
  24. H. Liu, K. Simonyan, and Y. Yang. 2019. Darts: Differentiable architecture search. In International Conference on Learning Representations .
  25. Z. Liu, Z. Shen, M. Savvides, and K. Cheng. 2020. Reactnet: Towards precise binary neural network with generalized activation functions. In European Conference on Computer Vision , pages 143-159.
  26. Z. Liu, B. Wu, W. Luo, X. Yang, W. Liu, and K. Cheng. 2018. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. In European Conference on Computer Vision .
  27. X. Ma, P. Zhang, S. Zhang, N. Duan, Y. Hou, D. Song, and M. Zhou. 2019. A tensorized transformer for language modeling. In Advances in Neural Information Processing Systems .
  28. B. Martinez, J. Yang, A. Bulat, and G. Tzimiropoulos. 2020. Training binary neural networks with real-tobinary convolutions. In International Conference on Learning Representations .
  29. P. Michel, O. Levy, and G. Neubig. 2019. Are sixteen heads really better than one? In Advances in Neural Information Processing Systems .
  30. Y. Nahshan, B. Chmiel, C. Baskin, E. Zheltonozhskii, R. Banner, A. M. Bronstein, and A. Mendelson. 2019. Loss aware post-training quantization. Preprint arXiv:1911.07190. · arXiv
  31. P. Rajpurkar, R. Jia, and P. Liang. 2018. Know what you don't know: Unanswerable questions for squad. Preprint arXiv:1806.03822. · arXiv
  32. M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. 2016. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision .
  33. S. Shen, Z. Dong, J. Ye, L. Ma, Z. Yao, A. Gholami, M. W. Mahoney, and K. Keutzer. 2020. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence .
  34. S. Sun, Y. Cheng, Z. Gan, and J. Liu. 2019. Patient knowledge distillation for bert model compression. In Conference on Empirical Methods in Natural Language Processing .
  35. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems .
  36. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. Preprint arXiv:1804.07461. · arXiv
  37. J. Wang, H. Bai, J. Wu, X. Shi, J. Huang, I. King, M. Lyu, and J. Cheng. 2020. Revisiting parameter sharing for automatic neural channel number search. In Advances in Neural Information Processing Systems , volume 33.
  38. K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han. 2019. Haq: Hardware-aware automated quantization with mixed precision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8612-8620.
  39. B. Wu, Y. Wang, P. Zhang, Y. Tian, P. Vajda, and K. Keutzer. 2018a. Mixed precision quantization of convnets via differentiable neural architecture search. Preprint arXiv:1812.00090. · arXiv
  40. J. Wu, Y. Zhang, H. Bai, H. Zhong, J. Hou, W. Liu, and J Huang. 2018b. Pocketflow: An automated framework for compressing and accelerating deep neural networks. In Advances in Neural Information Processing Systems, Workshop on Compact Deep Neural Networks with Industrial Applications .
  41. L. Wu, D. Wang, and Q. Liu. 2019. Splitting steepest descent for growing neural architectures. In Advances in Neural Information Processing Systems , volume 32.
  42. J. Xin, R. Tang, J. Lee, Y. Yu, and J. Lin. 2020. Deebert: Dynamic early exiting for accelerating bert inference. In Annual Meeting of the Association for Computational Linguistics .
  43. A. H. Zadeh and A. Moshovos. 2020. Gobo: Quantizing attention-based nlp models for low latency and energy efficient inference. Preprint arXiv:2005.03842. · arXiv
  44. O. Zafrir, G. Boudoukh, P. Izsak, and M. Wasserblat. 2019. Q8bert: Quantized 8bit bert. Preprint arXiv:1910.06188. · arXiv
  45. D. Zhang, J. Yang, D. Ye, and G. Hua. 2018. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In European conference on computer vision , pages 365-382.
  46. W. Zhang, L. Hou, Y. Yin, L. Shang, X. Chen, X. Jiang, and Q. Liu. 2020. Ternarybert: Distillation-aware ultra-low bit bert. In Conference on Empirical Methods in Natural Language Processing .
  47. R. Zhao, Y. Hu, J. Dotzel, C. De Sa, and Z. Zhang. 2019. Improving neural network quantization without retraining using outlier channel splitting. In International Conference on Machine Learning .
  48. S. Zhou, Y. Wu, Z. Ni, X. Zhou, H. Wen, and Y. Zou. 2016. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. Preprint arXiv:1606.06160. · arXiv
  49. W. Zhou, C. Xu, T. Ge, J. McAuley, K. Xu, and F. Wei. 2020. Bert loses patience: Fast and robust inference with early exit. In Advances in Neural Information Processing Systems .

Cited by

Sorted by citations

Authors
Bai, H., Zhang, W., Hou, L., Shang, L., Jin, J., Jiang, X., Liu, Q., Lyu, M., & King, I.
Affiliations
  • Chinese University of Hong Kong (HK)
  • Huawei Technologies (China) (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2012.15701
arXiv
2012.15701
OpenAlex
W3114304470
Cited by
42
References
4
Citations by year
2024: 3 · 2023: 16 · 2022: 8 · 2021: 15
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Chinese University of Hong Kong (14210717)
  • National Key Research and Development Program of China (2018AAA0100204)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
BERT, weight binarization, model compression, quantization, binary neural networks, loss landscape, fine-tuning, adhesives, SQuAD

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review