The DOI matches its DataCite record; 34 of the 43 references it cites were found in a registry or an index.
A Study of BFLOAT16 for Deep Learning Training
Open accessAbstract
This paper presents the first comprehensive empirical study demonstrating the efficacy of the Brain Floating Point (BFLOAT16) half-precision format for Deep Learning training across image classification, speech recognition, language modeling, generative networks and industrial recommendation systems. BFLOAT16 is attractive for Deep Learning training for two reasons: the range of values it can represent is the same as that of IEEE 754 floating-point format (FP32) and conversion to/from FP32 is simple. Maintaining the same range as FP32 is important to ensure that no hyper-parameter tuning is required for convergence; e.g., IEEE 754 compliant half-precision floating point (FP16) requires hyper-parameter tuning. In this paper, we discuss the flow of tensors and various key operations in mixed precision training, and delve into details of operations, such as the rounding modes for converting FP32 tensors to BFLOAT16. We have implemented a method to emulate BFLOAT16 operations in Tensorflow, Caffe2, IntelCaffe, and Neon for our experiments. Our results show that deep learning training using BFLOAT16 tensors achieves the same state-of-the-art (SOTA) results across domains as FP32 tensors in the same number of iterations and with no changes to hyper-parameters.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This precision offers significant speedups for BERT models while maintaining comparable accuracy to FP32.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [164]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“All these formats discussed here offer several benefits over the traditional floating-point precision, enabling faster and more efficient inference with reduced memory bandwidth.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [164]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The emerging numerical formats such as Bfloat16 [164], Microsoft Floating Point (MSFP) [165], Tensorfloat32 [166], and FP8 [167, 168] provide better computational efficiency and accuracy for several transformer workloads.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [164]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 5×
“When training ResNet-50 on Imagenet using the AVX512BF16 instructions, we achieved a Top-1 accuracy of 75.62%, which matches the current state-of-the-art performance.”
MethodInfluentialVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“The network uses a “VGG loss” function using a pre-trained VGG19 network [33].”
MethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 5×
“Here we choose AlexNet (ILSVRC 2012) [22] and ResNet-50 (ILSVRC 2015) [18] as representative models for the BFLOAT16 evaluation.”
MethodInfluentialImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“However, all the aforementioned methods are shown to work for smaller benchmark model/data-sets only and invariably result in a non-trivial drop in accuracy with larger ImageNet data-set [9] and classification task [32].”
MethodImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 1×
“However, all the aforementioned methods are shown to work for smaller benchmark model/data-sets only and invariably result in a non-trivial drop in accuracy with larger ImageNet data-set [9] and classification task [32].”
MethodPhoto-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.19 · Cited by 12,696
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z.-H., et al. (2016) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1609.04802 · Cited by 12,624 · cited 5×
“Using multiple deep learning frameworks modified to insert appropriate Quantlib calls, we provide SOTA results for: AlexNet [22], ResNet-50 [18], DC-GAN [29], SR-GAN [24], DeepSpeech2 [4], GNMT [37], and two industrial workloads namely: a Deep and Cross Network, and a DNN Recommendation System.”
BackgroundMethodInfluentialUnsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235 · cited 7×
“We have implemented DC-GAN in Caffe and for our experiments all the input tensors (activations, weights) are converted to BFLOAT16 for convolution layers (in both the generator and the discriminator), while only the input activations are converted to BFLOAT16 for batch normalization layers; all other tensors are maint…”
BackgroundMethodInfluentialTensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408 · cited 2×
“There are at least three half-precision formats in the domain of mixed precision training of large neural networks: FP16 [28], 16-bit Integer based [9], and BFLOAT16 [10, 4, 3].”
MethodJulia: A Fresh Approach to Numerical ComputingPreprint
Bezanson, J., Edelman, A., Karpinski, S., & Shah, V. B. (2014) · SIAM Review · doi:10.48550/arxiv.1411.1607 · Cited by 7,077 · cited 1×
“The Julia language [5], which is designed to provide high-performance without sacrificing ease of programming, has also been shown to be benefited from BFLOAT16 [11].”
Background“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072
Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 5×
“Google’s Neural Machine Translation (GNMT) is the SOTA neural machine translation model using a recurrent network.”
MethodInfluentialHan, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662 · cited 1×
“Inference has witnessed a proliferation of mixed precision compute [19, 33, 27, 25] where different operations execute at different precision, all the way from binary/ternary operands to 16b floating point.”
BackgroundXNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_32 · Cited by 4,408 · cited 2×
“Inference has witnessed a proliferation of mixed precision compute [19, 33, 27, 25] where different operations execute at different precision, all the way from binary/ternary operands to 16b floating point.”
BackgroundMethodDeep Speech 2: End-to-End Speech Recognition in English and MandarinPreprint
Amodei, D., Anubhai, R., Battenberg, E., Case, C. T., Casper, J., Catanzaro, B., Chen, J., Chrzanowski, M., Coates, A., Diamos, G., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.02595 · Cited by 2,265
BinaryConnect: Training Deep Neural Networks with binary weights during propagations
Courbariaux, M., Bengio, Y., & David, J.-P. (2015) · arXiv · doi:10.48550/arxiv.1511.00363 · Cited by 2,180 · cited 1×
“This idea is further advanced in [7] where the authors showcase training with only binary weights while keeping all other tensors and operations in full precision.”
BackgroundHubara, I., Soudry, D., & Yaniv, R. E. (2016) · arXiv · doi:10.48550/arxiv.1602.02505 · Cited by 1,667
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and ActivationsPreprint
Hubara, I., Courbariaux, M., Soudry, D., El‐Yaniv, R., & Bengio, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.07061 · Cited by 1,663 · cited 2×
“Literature shows that various different reduced precision data representations have been investigated which can be broadly classified into two types: the more standard floating-point based formats [28, 15, 12] and custom fixed point based formats [37, 7, 18, 22, 23].”
BackgroundMethodDeep Learning with Limited Numerical Precision
Gupta, S., Agrawal, A., Gopalakrishnan, K., & Narayanan, P. (2015) · arXiv · doi:10.48550/arxiv.1502.02551 · Cited by 1,364 · cited 3×
“Numerous studies have shown that 16-bits of precision is sufficient for training deep neural networks[16],[26],[21],[8].”
BackgroundDeep & Cross Network for Ad Click Predictions
Wang, R., Fu, B., Fu, G., & Wang, M. (2017) · ADKDD@KDD · doi:10.1145/3124749.3124754 · Cited by 1,269 · cited 2×
“The accuracy of the recommendation system models is measured by the log loss [38, 43, 29], which predicts when the users will click on ads.”
MethodMicikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., et al. (2017) · arXiv · doi:10.48550/arxiv.1710.03740 · Cited by 1,179 · cited 16×
“The computations during forward pass and back propagation use FP16 datatype while results are accumulated into FP32.”
BackgroundMethodInfluentialDeep Learning Recommendation Model for Personalization and Recommendation Systems
Naumov, M., Mudigere, D., Shi, H.-J. M., Huang, J., Sundaraman, N., Park, J., Wang, X., Gupta, U., Wu, C.-J., Azzolini, A. G., et al. (2019) · arXiv · doi:10.48550/arxiv.1906.00091 · Cited by 411 · cited 2×
“The accuracy of the recommendation system models is measured by the log loss [38, 43, 29], which predicts when the users will click on ads.”
MethodSelf-citationTraining deep neural networks with low precision multiplications
Courbariaux, M., Bengio, Y., & David, J.-P. (2014) · arXiv · doi:10.48550/arxiv.1412.7024 · Cited by 407
Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural NetworksPreprint
Köster, U., Webb, T. J., Wang, X., Nassar, M., Bansal, A. K., Constable, W., Elibol, O. H., Gray, S., Hall, S., Hornof, L., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.02213 · Cited by 236 · cited 4×
“Literature shows that various different reduced precision data representations have been investigated which can be broadly classified into two types: the more standard floating-point based formats [28, 15, 12] and custom fixed point based formats [37, 7, 18, 22, 23].”
BackgroundInfluential8-Bit Approximations for Parallelism in Deep LearningPreprint
Dettmers, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.04561 · Cited by 164 · cited 1×
“Literature shows that various different reduced precision data representations have been investigated which can be broadly classified into two types: the more standard floating-point based formats [26, 13, 10] and custom fixed point based formats [34, 6, 16, 20, 21].”
BackgroundMixed Precision Training of Convolutional Neural Networks using Integer OperationsPreprint
Das, D., Mellempudi, N., Mudigere, D., Kalamkar, D. D., Avancha, S., Banerjee, K., Sridharan, S., Vaidyanathan, K., Kaul, B., Georganas, E., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.00930 · Cited by 163 · cited 3×
“There are at least three half-precision formats in the domain of mixed precision training of large neural networks: FP16 [26], 16-bit Integer based [8], and BFLOAT16.”
BackgroundSelf-citationImage Classification at Supercomputer ScalePreprint
Ying, C., Kumar, S., Chen, D., Wang, T., & Cheng, Y.-L. (2018) · arXiv.org · doi:10.48550/arxiv.1811.06992 · Cited by 133
Anatomy of High-Performance Deep Learning Convolutions on SIMD Architectures
Georganas, E., Avancha, S., Banerjee, K., Kalamkar, D. D., Henry, G., Pabst, H., & Heinecke, A. (2018) · International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.48550/arxiv.1808.05567 · Cited by 118
Self-citationTernary Neural Networks with Fine-Grained QuantizationPreprint
Mellempudi, N., Kundu, A., Mudigere, D., Das, D., Kaul, B., & Dubey, P. K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.01462 · Cited by 89 · cited 1×
“Inference has witnessed a proliferation of mixed precision compute [19, 33, 27, 25] where different operations execute at different precision, all the way from binary/ternary operands to 16b floating point.”
BackgroundSelf-citationHigh Performance Monte Carlo Simulation of Ising Model on TPU Clusters
Yang, K., Chen, Y.-F., Roumpos, G., Colby, C., & Anderson, J. R. (2019) · International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1145/3295500.3356149 · Cited by 57
Dynamically scaled fixed point arithmetic
Williamson, D. (1991) · IEEE Pacific Rim Conference on Communications, Computers and Signal Processing. Conference Proceedings · doi:10.1109/pacrim.1991.160742 · Cited by 47 · cited 1×
“The work reported in [34] leverages the dynamically scaled fixed point representation proposed in [36] to speed up convolution neural networks by 4× over an optimized floating point implementation on general purpose CPU hardware.”
BackgroundAutomatic Full Compilation of Julia Programs and ML Models to Cloud TPUs
Fischer, K., & Saba, E. (2018) · arXiv.org · doi:10.48550/arxiv.1810.09868 · Cited by 20
Ternary Residual NetworksPreprint
Kundu, A., Banerjee, K., Mellempudi, N., Mudigere, D., Das, D., Kaul, B., & Dubey, P. (2017) · arXiv.org · doi:10.48550/arxiv.1707.04679 · Cited by 9 · cited 1×
“Inference has witnessed a proliferation of mixed precision compute [19, 33, 27, 25] where different operations execute at different precision, all the way from binary/ternary operands to 16b floating point.”
BackgroundSelf-citation
Research works printed but not linked: 5
- [10][10] Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al. Large scale distributed deep networks. In Advances in neural information processing systems , pages 1223-1231, 2012. -
- [15][15] Boris Ginsburg, Sergei Nikolaev, and Paulius Micikevicius. Training of deep networks with half-precision float. NVidia GPU Technology Conference , 2017. -
- [34][34] Matthew Richardson, Ewa Dominowska, and Robert Ragno. Predicting clicks: Estimating the click-through rate for new ads. In Proceedings of the 16th International Conference on World Wide Web , WWW'07, pages 521-530, New York, NY, USA, 2007. ACM. -
- [37][37] Vincent Vanhoucke, Andrew Senior, and Mark Z Mao. Improving the speed of neural networks on CPUs. In Proc. Deep Learning and Unsupervised Feature Learning NIPS Workshop , volume 1, 2011. -
- [43][43] Jian Zhang, Jiyan Yang, and Hector Yuen. Training with low-precision embedding tables. In Systems for Machine Learning Workshop at NeurIPS 2018 . 2018.
Cited by
Sorted by citations
Large Language Models in Finance: A Survey
Li, Y., Wang, S., Ding, H., & Chen, H. (2023) · ACM International Conference on AI in Finance (ICAIF) · doi:10.1145/3604237.3626869 · Cited by 412
“In Section 5, we provide a more detailed discussion of which option is more favorable under different circumstances.”
BackgroundTen Lessons From Three Generations Shaped Google’s TPUv4i : Industrial Product
Jouppi, N. P., Yoon, D. H., Ashcraft, M. B., Gottscho, M., Jablin, T. B., Kurian, G. T., Laudon, J. P., Li, S., Ma, P., Ma, X., et al. (2021) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca52012.2021.00010 · Cited by 403
“Bfloat16 is a better match to DNNs than IEEE 754 fp16 [24], so the MXU was changed to become the first hardware to support it, with 16K MAC units (½ the side of the systolic array, so ¼ of the size).”
FP8 Formats for Deep LearningPreprint
Micikevicius, P., Stosic, D., Burgess, N., Cornea, M., Dubey, P., Grisenthwaite, R., Ha, S., Heinecke, A., Judd, P., Kamalu, J., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2209.05433 · Cited by 352
“Common floating point types for training include IEEE single precision, TF32 mode for single precision [19], IEEE half precision [14], and bfloat16 [9].”
Self-citationA domain-specific supercomputer for training deep neural networks
Jouppi, N., Yoon, D., Kurian, G., Li, S., Patil, N., Laudon, J., Young, C., & Patterson, D. A. (2020) · Communications of the ACM · doi:10.1145/3360307 · Cited by 312
“Beyond our experience that it works for training production applications, a recent Intel study corroborated its benefits.(21) However, fp16 requires adjustments to training software (loss scaling) to deliver convergence and efficiency.”
MethodResultEfficient Processing of Deep Neural Networks
Sze, V., Chen, Y., Yang, T.-J., & Emer, J. (2020) · Synthesis lectures on computer architecture · doi:10.2200/s01004ed1v01y202004cac050 · Cited by 125
Metadata
- Authors
- Kalamkar, D., Mudigere, D., Mellempudi, N., Das, D., Banerjee, K., Avancha, S., Vooturi, D. T., Jammalamadaka, N., Huang, J., Yuen, H., Yang, J., Park, J., Heinecke, A., Georganas, E., Srinivasan, S., Kundu, A., Smelyanskiy, M., Kaul, B., & Dubey, P.
- Affiliations
- Meta (United States) (US)
- Intel (United States) (US)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1905.12322
- OpenAlex
- W2947629474
- Cited by
- 169
- References
- 43
- Citations by year
- 2026: 4 · 2025: 14 · 2024: 15 · 2023: 33 · 2022: 32 · 2021: 37 · 2020: 32 · 2019: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Numerical Methods and Algorithms
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Keywords
- bfloat16, half-precision floating-point, mixed-precision training, deep learning training, rounding modes, image classification, speech recognition, language modeling, generative neural networks, recommender systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
