CoChat
Sign Up
PreprintCited by 186Advanced Neural Network ApplicationsNumerical Methods and AlgorithmsTopic Modeling

I-BERT: Integer-only BERT Quantization

Open access
Kim, Sehoon, Gholami, Amir, Yao, Zhewei, Mahoney, Michael W., Keutzer, Kurt·2021·arXiv
DOI 10.48550/arxiv.2101.01321arXiv 2101.01321
Open this paper in CoChat — free accountOpen PDF

Abstract

Transformer based models, like BERT and RoBERTa, have achieved state-of-the-art results in many Natural Language Processing tasks. However, their memory footprint, inference latency, and power consumption are prohibitive efficient inference at the edge, and even at the data center. While quantization can be a viable solution for this, previous work on quantizing Transformer based models use floating-point arithmetic during inference, which cannot efficiently utilize integer-only logical units such as the recent Turing Tensor Cores, or traditional integer-only ARM processors. In this work, we propose I-BERT, a novel quantization scheme for Transformer based models that quantizes the entire inference with integer-only arithmetic. Based on lightweight integer-only approximation methods for nonlinear operations, e.g., GELU, Softmax, and Layer Normalization, I-BERT performs an end-to-end integer-only BERT inference without any floating point calculation. We evaluate our approach on GLUE downstream tasks using RoBERTa-Base/Large. We show that for both cases, I-BERT achieves similar (and slightly higher) accuracy as compared to the full-precision baseline. Furthermore, our preliminary implementation of I-BERT shows a speedup of 2.4-4.0x for INT8 inference on a T4 GPU system as compared to FP32 inference. The framework has been developed in PyTorch and has been open-sourced.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2101.01321

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Layerwise quantization uses the same clipping range for all convolutional filters in a layer, is simple to implement, but often gives sub-optimal accuracy because filter ranges can vary greatly.

    “In this approach, the clipping range is determined by considering all of the weights in convolutional filters of a layer [133], as shown in the third column of Figure 3.”

    From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [133]

    Appendix A.2 (p. 14) says the scaling factor requires determining the range of parameters or activations, which supports the general link between weight ranges and quantization. It does not say that one clipping range is determined from all convolutional filters in a layer. Nor does it address the claimed simplicity or accuracy drawback of layerwise quantization. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • I-BERT quantizes all layers in a BERT model, including GELU, Softmax, and LayerNorm operations, to Int8 precision and achieves 2.4-4x speedup over FP32 BERT on Nvidia T4 GPU.

    “I-BERT [150] quantizes all the layers in a BERT model, GELU, Softmax, and LayerNorm operations to Int8 precision and achieves 2.4-4x speedup over FP32 BERT model on Nvidia T4 GPU.”

    From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [150]

    The Abstract (p. 1) reports a 2.4–4.0× speedup for INT8 inference over FP32 inference on a T4 GPU. But C.1 Implementation (p. 14) says GELU, Softmax, and LayerNorm are computed at INT32 precision, then requantized to INT8; the claim that those operations run at INT8 precision is incorrect. (LLM, conf=0.92)

    Experimental — accuracy not yet measured

  • I-ViT calculates the square root in LayerNorm using a lightweight, integer-based iterative method inspired by Newton's Method, refined with bit-shifting enhancements.

    “Additionally, I-ViT calculates the square root in LayerNorm using a lightweight, integer-based iterative method inspired by Newton's Method, as per [74], refined further with bit-shifting enhancements.”

    From Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Survey (2024) · cites this paper as [74]

  • FQ-ViT introduces Log-Int-Softmax, an integer-only variant of the softmax function that approximates the exponential component using a second-order polynomial.

    “FQ-ViT [43] introduces 'Log-Int-Softmax,' an integeronly variant of the softmax function that approximates the exponential component using a second-order polynomial [74].”

    From Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Survey (2024) · cites this paper as [74]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 75

  1. Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016. · arXiv
  2. Bai, H., Zhang, W., Hou, L., Shang, L., Jin, J., Jiang, X., Liu, Q., Lyu, M., and King, I. Binarybert: Pushing the limit of bert quantization. arXiv preprint arXiv:2012.15701 , 2020. · arXiv
  3. Bhandare, A., Sripathi, V., Karkada, D., Menon, V., Choi, S., Datta, K., and Saletore, V. Efficient 8-bit quantization of transformer neural machine language translation model. arXiv preprint arXiv:1906.00532 , 2019. · arXiv
  4. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 , 2020. · arXiv
  5. Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., and Specia, L. Semeval-2017 task 1: Semantic textual similaritymultilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055 , 2017. · arXiv
  6. Chen, T., Moreau, T., Jiang, Z., Zheng, L., Yan, E., Shen, H., Cowan, M., Wang, L., Hu, Y., Ceze, L., et al. TVM: An automated end-to-end optimizing compiler for deep learning. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18) , pp. 578-594, 2018.
  7. Choi, J., Wang, Z., Venkataramani, S., Chuang, P. I.-J., Srinivasan, V., and Gopalakrishnan, K. PACT: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv:1805.06085 , 2018. · arXiv
  8. Courbariaux, M., Bengio, Y., and David, J.-P. BinaryConnect: Training deep neural networks with binary weights during propagations. In Advances in neural information processing systems , pp. 3123-3131, 2015.
  9. Courbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., and Bengio, Y. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830 , 2016. · arXiv
  10. Crandall, R. and Pomerance, C. B. Prime numbers: a computational perspective , volume 182. Springer Science &Business Media, 2006.
  11. Dagan, I., Glickman, O., and Magnini, B. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop , pp. 177-190. Springer, 2005.
  12. Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., and Kaiser, Ł. Universal transformers. arXiv preprint arXiv:1807.03819 , 2018. · arXiv
  13. Detrey, J. and de Dinechin, F. A parameterized floatingpoint exponential function for fpgas. In Proceedings. 2005 IEEE International Conference on FieldProgrammable Technology, 2005. , pp. 27-34. IEEE, 2005.
  14. Dodge, J., Ilharco, G., Schwartz, R., Farhadi, A., Hajishirzi, H., and Smith, N. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305 , 2020. · arXiv
  15. Dolan, W. B. and Brockett, C. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005) , 2005.
  16. Esser, S. K., McKinstry, J. L., Bablani, D., Appuswamy, R., and Modha, D. S. Learned step size quantization. arXiv preprint arXiv:1902.08153 , 2019. · arXiv
  17. Fan, A., Grave, E., and Joulin, A. Reducing transformer depth on demand with structured dropout. arXiv preprint arXiv:1909.11556 , 2019. · arXiv
  18. Fan, A., Stock, P., Graham, B., Grave, E., Gribonval, R., Jegou, H., and Joulin, A. Training with quantization noise for extreme fixed-point compression. arXiv preprint arXiv:2004.07320 , 2020. · arXiv
  19. Gholami, A., Kwon, K., Wu, B., Tai, Z., Yue, X., Jin, P., Zhao, S., and Keutzer, K. SqueezeNext: Hardware-aware neural network design. Workshop paper in CVPR , 2018.
  20. Gordon, M. A., Duh, K., and Andrews, N. Compressing bert: Studying the effects of weight pruning on transfer learning. arXiv preprint arXiv:2002.08307 , 2020. · arXiv
  21. Han, S., Pool, J., Tran, J., and Dally, W. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems , pp. 1135-1143, 2015.
  22. Hauser, J. W. and Purdy, C. N. Approximating functions for embedded and asic applications. In Proceedings of the 44th IEEE 2001 Midwest Symposium on Circuits and Systems. MWSCAS 2001 (Cat. No. 01CH37257) , volume 1, pp. 478-481. IEEE, 2001.
  23. Hendrycks, D. and Gimpel, K. Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415 , 2016. · arXiv
  24. Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. Workshop paper in NIPS , 2014.
  25. Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., and Keutzer, K. SqueezeNet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv:1602.07360 , 2016. · arXiv
  26. Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 , 2015. · arXiv
  27. Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. Quantization and training of neural networks for efficient integerarithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 2704-2713, 2018.
  28. Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., and Liu, Q. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351 , 2019. · arXiv
  29. Jin, J., Liang, C., Wu, T., Zou, L., and Gan, Z. Kdlsqbert: A quantized bert combining knowledge distillation with learned step size quantization. arXiv preprint arXiv:2101.05938 , 2021. · arXiv
  30. Krishnamoorthi, R. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.08342 , 2018. · arXiv
  31. Kwon, K., Amid, A., Gholami, A., Wu, B., Asanovic, K., and Keutzer, K. Co-design of deep neural nets and neural net accelerators for embedded vision applications. In 2018 55th ACM/ESDA/IEEE Design Automation Conference (DAC) , pp. 1-6. IEEE, 2018.
  32. Lai, L., Suda, N., and Chandra, V. CMSIS-NN: Efficient neural network kernels for arm cortex-m cpus. arXiv preprint arXiv:1801.06601 , 2018. · arXiv
  33. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 , 2019. · arXiv
  34. LeCun, Y., Denker, J. S., and Solla, S. A. Optimal brain damage. In Advances in neural information processing systems , pp. 598-605, 1990.
  35. Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y ., Krikun, M., Shazeer, N., and Chen, Z. GShard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668 , 2020. · arXiv
  36. Levesque, H., Davis, E., and Morgenstern, L. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning . Citeseer, 2012.
  37. Li, H., Kadav, A., Durdanovic, I., Samet, H., and Graf, H. P. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 , 2016b. · arXiv
  38. Mao, H., Han, S., Pool, J., Li, W., Liu, X., Wang, Y., and Dally, W. J. Exploring the regularity of sparse structure in convolutional neural networks. Workshop paper in CVPR , 2017.
  39. Mao, Y., Wang, Y., Wu, C., Zhang, C., Wang, Y., Yang, Y., Zhang, Q., Tong, Y., and Bai, J. Ladabert: Lightweight adaptation of bert through hybrid model compression. arXiv preprint arXiv:2004.04124 , 2020. · arXiv
  40. Michel, P., Levy, O., and Neubig, G. Are sixteen heads really better than one? arXiv preprint arXiv:1905.10650 , 2019. · arXiv
  41. Mishra, A. and Marr, D. Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy. arXiv preprint arXiv:1711.05852 , 2017. · arXiv
  42. Molchanov, P., Tyree, S., Karras, T., Aila, T., and Kautz, J. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440 , 2016. · arXiv
  43. Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., and Auli, M. FairSeq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACLHLT 2019: Demonstrations , 2019.
  44. Park, E., Yoo, S., and Vajda, P. Value-aware quantization for training and inference of neural networks. In Proceedings of the European Conference on Computer Vision (ECCV) , pp. 580-595, 2018.
  45. Polino, A., Pascanu, R., and Alistarh, D. Model compression via distillation and quantization. arXiv preprint arXiv:1802.05668 , 2018. · arXiv
  46. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.
  47. Raganato, A., Scherrer, Y., and Tiedemann, J. Fixed encoder self-attention patterns in transformer-based machine translation. arXiv preprint arXiv:2002.10260 , 2020. · arXiv
  48. Rastegari, M., Ordonez, V., Redmon, J., and Farhadi, A. XNOR-Net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision , pp. 525-542. Springer, 2016.
  49. Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., and Bengio, Y. FitNets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550 , 2014. · arXiv
  50. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. MobilenetV2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 45104520, 2018.
  51. Schraudolph, N. N. A fast, compact approximation of the exponential function. Neural Computation , 11(4):853862, 1999.
  52. Shen, S., Dong, Z., Ye, J., Ma, L., Yao, Z., Gholami, A., Mahoney, M. W., and Keutzer, K. Q-BERT: Hessian based ultra low precision quantization of bert. In AAAI , pp. 8815-8821, 2020.
  53. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-LM: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053 , 2019. · arXiv
  54. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A. Y., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing , pp. 1631-1642, 2013.
  55. Sun, S., Cheng, Y., Gan, Z., and Liu, J. Patient knowledge distillation for bert model compression. arXiv preprint arXiv:1908.09355 , 2019. · arXiv
  56. Tang, R., Lu, Y., Liu, L., Mou, L., Vechtomova, O., and Lin, J. Distilling task-specific knowledge from bert into simple neural networks. arXiv preprint arXiv:1903.12136 , 2019. · arXiv
  57. Turc, I., Chang, M.-W., Lee, K., and Toutanova, K. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv:1908.08962 , 2019. · arXiv
  58. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. In Advances in neural information processing systems , pp. 5998-6008, 2017.
  59. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461 , 2018. · arXiv
  60. Wang, K., Liu, Z., Lin, Y., Lin, J., and Han, S. HAQ: Hardware-aware automated quantization. In Proceedings of the IEEE conference on computer vision and pattern recognition , 2019.
  61. Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., and Zhou, M. Minilm: Deep self-attention distillation for taskagnostic compression of pre-trained transformers. arXiv preprint arXiv:2002.10957 , 2020. · arXiv
  62. Waring, E. Vii. problems concerning interpolations. Philosophical transactions of the royal society of London , 1779.
  63. Warstadt, A., Singh, A., and Bowman, S. R. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics , 7:625-641, 2019.
  64. Williams, A., Nangia, N., and Bowman, S. R. A broadcoverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426 , 2017. · arXiv
  65. Wu, B., Wang, Y., Zhang, P., Tian, Y., Vajda, P., and Keutzer, K. Mixed precision quantization of convnets via differentiable neural architecture search. arXiv preprint arXiv:1812.00090 , 2018. · arXiv
  66. Wu, J., Leng, C., Wang, Y., Hu, Q., and Cheng, J. Quantized convolutional neural networks for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 4820-4828, 2016.
  67. Xu, C., Zhou, W., Ge, T., Wei, F., and Zhou, M. Bert-oftheseus: Compressing bert by progressive module replacing. arXiv preprint arXiv:2002.02925 , 2020. · arXiv
  68. Yang, T.-J., Chen, Y.-H., and Sze, V. Designing energyefficient convolutional neural networks using energyaware pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 56875695, 2017.
  69. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R. R., and Le, Q. V. XLNet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems , pp. 5753-5763, 2019.
  70. Yao, Z., Dong, Z., Zheng, Z., Gholami, A., Yu, J., Tan, E., Wang, L., Huang, Q., Wang, Y., Mahoney, M. W., and Keutzer, K. HAWQV3: Dyadic neural network quantization. arXiv preprint arXiv:2011.10680 , 2020. · arXiv
  71. Zadeh, A. H., Edo, I., Awad, O. M., and Moshovos, A. Gobo: Quantizing attention-based nlp models for low latency and energy efficient inference. In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) , pp. 811-824. IEEE, 2020.
  72. Zafrir, O., Boudoukh, G., Izsak, P., and Wasserblat, M. Q8BERT: Quantized 8bit bert. arXiv preprint arXiv:1910.06188 , 2019. · arXiv
  73. Zhang, D., Yang, J., Ye, D., and Hua, G. LQ-Nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European conference on computer vision (ECCV) , pp. 365-382, 2018.
  74. Zhang, W., Hou, L., Yin, Y., Shang, L., Chen, X., Jiang, X., and Liu, Q. Ternarybert: Distillation-aware ultra-low bit bert. arXiv preprint arXiv:2009.12812 , 2020. · arXiv
  75. Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., and Zou, Y. DoReFa-Net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160 , 2016. · arXiv

Cited by

Sorted by citations

Authors
Kim, S., Gholami, A., Yao, Z., Mahoney, M. W., & Keutzer, K.
Affiliations
University of California, Berkeley (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2101.01321
arXiv
2101.01321
OpenAlex
W3118568258
Cited by
186
References
12
Citations by year
2026: 1 · 2025: 28 · 2024: 55 · 2023: 57 · 2022: 35 · 2021: 10
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation
  • Intel Corporation
  • Nvidia
  • Samsung Advanced Institute of Technology
  • Samsung
  • Center for Long-Term Cybersecurity, University of California Berkeley
  • Office of Naval Research
  • Army Research Office
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Numerical Methods and Algorithms
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
BERT, RoBERTa, transformer models, softmax approximation, edge inference, inference acceleration, ARM processor

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review