Towards Fully 8-bit Integer Inference for the Transformer Model
Open accessAbstract
8-bit integer inference, as a promising direction in reducing both the latency and storage of deep neural networks, has made great progress recently. On the other hand, previous systems still rely on 32-bit floating point for certain functions in complex models (e.g., Softmax in Transformer), and make heavy use of quantization and de-quantization. In this work, we show that after a principled modification on the Transformer architecture, dubbed Integer Transformer, an (almost) fully 8-bit integer inference algorithm Scale Propagation could be derived. De-quantization is adopted when necessary, which makes the network more efficient. Our experiments on WMT16 En<->Ro, WMT14 En<->De and En->Fr translation tasks as well as the WikiText-103 language modelling task show that the fully 8-bit Transformer system achieves comparable performance with the floating point baseline but requires nearly 4x less memory footprint.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
L1 LayerNorm simplifies computation to achieve faster speed but its low approximation capability leads to non-trivial accuracy loss.
“For L1 LayerNorm, although it simplifies the computation to achieve faster speed, its low approximation capability leads to non-trivial accuracy loss.”
From I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference (2023) · cites this paper as [29]
Section 4.2 says L1 Layer Normalization simplifies the operation by replacing a square root with an absolute value. But Section 5.3 reports that it performs closely to standard layer normalization and calls it a good alternative, contrary to the claimed non-trivial accuracy loss. The passages do not establish a measured speed gain. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredInteger arithmetic units allow straightforward calculation of the mean and variance of data, but they fail to support the square root arithmetic for obtaining the standard deviation.
“The integer arithmetic units allow straightforward calculation of the mean and variance of the data, yet they fail to support the square root arithmetic for obtaining the standard deviation [29].”
From I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference (2023) · cites this paper as [29]
Section 4.2 says the square root used to compute standard deviation hinders INT8 inference because integer inputs do not guarantee integer outputs. Section 3.2 describes INT8 addition and multiplication, and Section 4.2 computes a mean, but the paper does not explicitly say that integer arithmetic units straightforwardly calculate variance or that the units themselves lack square-root support. (LLM, conf=0.82)
Experimental — accuracy not yet measuredFully-8bit employs L1 LayerNorm to replace the non-linear arithmetic of standard deviation.
“Fully-8bit [29] employs L1 LayerNorm to replace the non-linear arithmetic of standard deviation”
From I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference (2023) · cites this paper as [29]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Efficient Processing of Deep Neural Networks: A Tutorial and Survey
Sze, V., Chen, Y.-H., Yang, T.-J., & Emer, J. S. (2017) · Proceedings of the IEEE · doi:10.1109/jproc.2017.2761740 · Cited by 4,215 · cited 1× · see citing works
“Beyond this, INT8 is 10 × more energy efficient [ Johnson, 2018 ] and saves much less chip area than the commonly used 32-bit floating point (FP32) in hardware design [ Sze et al. , 2017 ] .”
Method
Flagged first, then by citations
Research works printed but not linked: 17
- Baevski and Auli, 2019] Alexei Baevski and Michael Auli. Adaptive input representations for neural language model- ing. In 7th International Conference on Learning Repre- sentations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
- Aishwarya Bhandare, Vamsi Sri- pathi, Deepthi Karkada, Vivek Menon, Sun Choi, Kushal Datta, and Vikram Saletore. Efficient 8-bit quantization of transformer neural machine language translation model. CoRR, abs/1906.00532, 2019. · arXiv
- Jiong Gong, Haihao Shen, Guoming Zhang, Xiaoli Liu, Shane Li, Ge Jin, Niharika Mahesh- wari, Evarist Fomenko, and Eden Segal. Highly effi- cient 8-bit low precision inference of convolutional neu- ral networks with intelcaffe. In Proceedings of the 1st on Reproducible Quality-Efficient Systems Tourna- ment on Co-designing Pareto-efficient Deep Learning, Re- QuEST@ASPLOS 2018, Williamsburg, VA, USA, March 24, 2018, page 2, 2018.
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. In 2015 IEEE International Conference on Computer Vi- sion, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 1026-1034, 2015.
- Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accurate normalization schemes in deep networks. In Advances in Neural Information Processing Systems 31: Annual Con- ference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada, pages 2164-2174, 2018.
- Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Bina- rized neural networks. In Advances in Neural Informa- tion Processing Systems 29: Annual Conference on Neu- ral Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 4107-4115, 2016.
- Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew G. Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In 2018 IEEE Confer- ence on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 2704-2713, 2018.
- Jeff Johnson. Rethinking floating point for deep learning. CoRR, abs/1811.01721, 2018. · arXiv
- Yanyang Li, Qiang Wang, Tong Xiao, Ton- gran Liu, and Jingbo Zhu. Neural machine translation with joint representation. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
- Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 -May 3, 2018, Conference Track Proceedings, 2018.
- Gabriele Prato, Ella Charlaix, and Mehdi Rezagholizadeh. Fully quantized transformer for im- proved translation. CoRR, abs/1910.10485, 2019. · arXiv
- Quinn and Ballesteros, 2018] Jerry Quinn and Miguel Ballesteros. Pieces of eight: 8-bit neural machine transla- tion. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL- HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 3 (Industry Papers), pages 114-120, 2018.
- Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Con- ference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Vol- ume 1: Long Papers, pages 3645-3650, 2019.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Sys- tems 30: Annual Conference on Neural Information Pro- cessing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 5998-6008, 2017.
- Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. Learning deep transformer models for machine translation. In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors, Proceedings of the 57th Conference of the Associ- ation for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 1810-1822. Association for Computational Linguis- tics, 2019.
- Ephrem Wu. Learning accurate integer transformer machine-translation models. CoRR, abs/2001.00926, 2020. · arXiv
- Tong Xiao, Yinqiao Li, Jingbo Zhu, Zhengtao Yu, and Tongran Liu. Sharing attention weights for fast transformer. In Sarit Kraus, editor, Proceedings of the Twenty-Eighth International Joint Conference on Arti- ficial Intelligence, IJCAI 2019, Macao, China, August 10- 16, 2019, pages 5292-5298. ijcai.org, 2019.
Cited by
Sorted by citations
Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers
Stevens, J. R., Venkatesan, R., Dai, S., Khailany, B., & Raghunathan, A. (2021) · 2021 58th ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac18074.2021.9586134 · Cited by 148 · see citing works
I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference
Li, Z., & Gu, Q. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01565 · Cited by 141 · see citing works
“• L1 LayerNorm in Fully-8bit [29]: using L1 norm to replace the calculation of standard deviation.”
MethodInfluential
- Authors
- Lin, Y., Li, Y., Liu, T., Xiao, T., Liu, T., & Zhu, J.
- Affiliations
- Northeastern University (CN)
- Institute of Psychology, Chinese Academy of Sciences (CN)
- Venue
- Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence
- Published
- 2020-07
- Pages
- 3759-3765
- OpenAlex
- W3035083896
- Cited by
- 30
- References
- 2
- Citations by year
- 2026: 3 · 2025: 5 · 2024: 2 · 2023: 8 · 2022: 4 · 2021: 6 · 2020: 2
- FWCI
- 1.54
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Joint Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (61876035, 61732005)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Numerical Methods and Algorithms
- Keywords
- transformer models, memory footprint reduction, low-bit neural networks, machine translation, language modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
