The DOI matches its DataCite record; 3 of the 3 references it cites were found in a registry or an index.
8-bit Numerical Formats for Deep Neural Networks
Open accessAbstract
Given the current trend of increasing size and complexity of machine learning architectures, it has become of critical importance to identify new approaches to improve the computational efficiency of model training. In this context, we address the advantages of floating-point over fixed-point representation, and present an in-depth study on the use of 8-bit floating-point number formats for activations, weights, and gradients for both training and inference. We explore the effect of different bit-widths for exponents and significands and different exponent biases. The experimental results demonstrate that a suitable choice of these low-precision formats enables faster training and reduced power consumption without any degradation in accuracy for a range of deep learning models for image classification and language processing.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 41
- Ankur Agrawal, Silvia M. Mueller, Bruce M. Fleischer, Jungwook Choi, Naigang Wang, Xiao Sun, and Kailash Gopalakrishnan. DLFloat: A 16-bit floating point format designed for deep learning training and inference. In Proceedings of 26th IEEE Symposium on Computer Arithmetic, ARITH 26 , 2019.
- Dzmitry Bahdanau, KyungHyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 [cs.CL] , 2014. · arXiv
- Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. arXiv preprint arXiv:1805.11046 [cs.LG] , 2018. · arXiv
- Workshop on Statistical Machine Translation. In Proceedings of 9th Workshop on Statistical Machine Translation , pages 12-58, 2014.
- Ondˇ rej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, et al. Findings of the 2016 Conference on Machine Translation. In Proceedings of 1st Conference on Machine Translation: Volume 2, Shared Task Papers , pages 131-198, 2016.
- Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Training deep neural networks with low precision multiplications. arXiv preprint arXiv:1412.7024 [cs.LG] , 2014. · arXiv
- Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. BinaryConnect: Training deep neural networks with binary weights during propagations. arXiv preprint arXiv:1511.00363 [cs.LG] , 2015. · arXiv
- Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or -1. arXiv preprint arXiv:1602.02830 [cs.LG] , 2016. · arXiv
- Dipankar Das, Naveen Mellempudi, Dheevatsa Mudigere, Dhiraj Kalamkar, Sasikanth Avancha, Kunal Banerjee, Srinivas Sridharan, Karthik Vaidyanathan, Bharat Kaul, Evangelos Georganas, Alexander Heinecke, Pradeep Dubey, Jesus Corbal, Nikita Shustrov, Roma Dubtsov, Evarist Fomenko, and Vadim Pirogov. Mixed precision training of convolutional neural networks using integer operations. arXiv preprint arXiv:1802.00930 [cs.NE] , 2018. · arXiv
- Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of 13th International Conference on Artificial Intelligence and Statistics, AISTAT 2010 , pages 249-256, 2010.
- Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning . MIT Press, Cambridge, MA, 2016.
- Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. arXiv preprint arXiv:1502.02551 [cs.LG] , 2015. · arXiv
- John L. Gustafson and Isaac Yonemoto. Beating floating point at its own game: Posit arithmetic. Supercomputing Frontiers and Innovations , 4(2):71-86, 2017.
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint arXiv:1905.11946 [cs.CV] , 2015. · arXiv
- Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. arXiv preprint arXiv:1609.07061 [cs.NE] , 2016. · arXiv
- Jeff Johnson. Rethinking floating point for deep learning. arXiv preprint arXiv:1811.01721 [cs.NA] , 2018. · arXiv
- Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 [cs.LG] , 2014. · arXiv
- Oleksii Kuchaiev, Boris Ginsburg, Igor Gitman, Vitaly Lavrukhin, Jason Li, Huyen Nguyen, Carl Case, and Paulius Micikevicius. Mixed-precision training for NLP and speech recognition with OpenSeq2Seq. arXiv preprint arXiv:1805.10387 [cs.LG] , 2018. · arXiv
- Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in Adam. arXiv preprint arXiv:1711.05101 [cs.LG] , 2017. · arXiv
- Henry B. Mann and Donald R. Whitney. On a test of whether one of two random variables is stochastically larger than the other. The Annals of Mathematical Statistics , pages 50-60, 1947.
- Dominic Masters, Antoine Labatie, Zach Eaton-Rosen, and Carlo Luschi. Making EfficientNet more efficient: Exploring batch-independent normalization, group convolutions and reduced resolution training. arXiv preprint arXiv:2106.03640 [cs.LG] , 2021. · arXiv
- Naveen Mellempudi, Sudarshan Srinivasan, Dipankar Das, and Bharat Kaul. Mixed precision training with 8-bit floating point. arXiv preprint arXiv:1905.12334 [cs.LG] , 2019. · arXiv
- Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843 [cs.CL] , 2016. · arXiv
- Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. arXiv preprint arXiv:1710.03740 [cs.AI] , 2017. · arXiv
- Daisuke Miyashita, Edward H. Lee, and Boris Murmann. Convolutional neural networks using logarithmic data representation. arXiv preprint arXiv:1603.01025 [cs.NE] , 2016. · arXiv
- Boris T. Poliak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics , 4(5):1-17, 1964.
- Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers , pages 186-191, 2018.
- Gabriele Prato, Ella Charlaix, and Mehdi Rezagholizadeh. Fully quantized transformer for improved translation. arXiv preprint arXiv:1910.10485 [cs.CL] , 2019. · arXiv
- Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) , 115(3):211-252, December 2015.
- Xiao Sun, Jungwook Choi, Chia-Yu Chen, Naigang Wang, Swagath Venkataramani, Vijayalakshmi Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrishnan. Hybrid 8-bit floating point (HFP8) training and inference for deep neural networks. In Advances in Neural Information Processing Systems 32, NeurIPS 2019 , 2019.
- Mingxing Tan and Quoc V. Le. EfficientNet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.11946 [cs.LG] , 2019. · arXiv
- Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop, Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning , 4(2), 2012.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv:1706.03762 [cs.CL] , 2017. · arXiv
- Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yu Chen, and Kailash Gopalakrishnan. Training deep neural networks with 8-bit floating point numbers. arXiv preprint arXiv:1812.08011 [cs.LG] , 2018. · arXiv
- Bernard Widrow and Istv´ an Koll´ ar. Quantization Noise: Roundoff Error in Digital Computation, Signal Processing, Control, and Communications . Cambridge University Press, Cambridge, 2008.
- Bernard Widrow, Istv´ an Koll´ ar, and Ming-Chang Liu. Statistical theory of quantization. IEEE Transactions on Instrumentation and Measurement , 45(2):353-361, 1996.
- Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. CutMix: Regularization strategy to train strong classifiers with localizable features. arXiv preprint arXiv:1905.04899 [cs.CV] , 2019. · arXiv
- Hongyi Zhang, Moustapha Ciss´ e, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 [cs.LG] , 2017. · arXiv
- Ruizhe Zhao, Brian Vogel, and Tanvir Ahmed. Adaptive loss scaling for mixed precision training. arXiv preprint arXiv:1910.12385 [cs.LG] , 2019. · arXiv
- Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. DoReFa-Net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160 [cs.NE] , 2016. · arXiv
- Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. arXiv preprint arXiv:1506.06724 [cs.CV] , 2015. · arXiv
Cited by
Sorted by citations
DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.19437 · Cited by 294
Efficient Methods for Natural Language Processing: A Survey
Treviso, M., Lee, J.-U., Ji, T., van Aken, B., Cao, Q., Ciosici, M. R., Hassid, M., Heafield, K., Hooker, S., Raffel, C., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00577 · Cited by 156
AI and ML Accelerator Survey and Trends
Reuther, A., Michaleas, P., Jones, M., Gadepally, V., Samsi, S., & Kepner, J. (2022) · 2022 IEEE High Performance Extreme Computing Conference (HPEC) · doi:10.1109/hpec55821.2022.9926331 · Cited by 109
Tortorella, Y., Bertaccini, L., Luca Benini, C., Rossi, D., & Conti, F. (2023) · Future Generation Computer Systems · doi:10.1016/j.future.2023.07.002 · Cited by 38
Boldo, S., Jeannerod, C.-P., Melquiond, G., & Muller, J. (2023) · Acta Numerica · doi:10.1017/s0962492922000101 · Cited by 30
Metadata
- Authors
- Noune, B., Jones, P., Justus, D., Masters, D., & Luschi, C.
- Affiliations
- Graphcore (United Kingdom) (GB)
- ORCID
- Masters, Dominic
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2206.02915
- OpenAlex
- W4281652308
- Cited by
- 19
- References
- 0
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 4 · 2023: 12 · 2022: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Numerical Methods and Algorithms
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Low-power high-performance VLSI design
- Keywords
- low-precision arithmetic, fixed-point representation, deep neural network, training efficiency, power consumption, bit-width optimization, image classification, language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
